Boundary wall disputes with neighbour- fence wall touches the bounday

Our neighbour had constructed the boundary wall in the backside of our house. They also built a gate on the wall which opens into our property for their ease. Recently we built the fence wall which touches their boundary wall within our property. They want us to break the fence wall so that it doesnt touch the boundary wall and leave a gap to their boundary wall . They also have trees close to ther boundary wall and all the fruits fall on our roof sheet and cause damage to it. They brought in some party workers to ask us to get it rebuilt. However, we also asked them to cut the trees that spoil the roof sheet and also remove the gate that opens into our backyard as it is a concern of our safety. But they have refused to remove the gate and threaten to go legal to get the fence wall broken. Please suggest what is the legal way to proceed for this. Also in this time of Corona, they keep sending the party workers who come into our house to check the issue and tell us to break the fence.