Property dispute on common wall with neighbour

We have purchased a buildup property in 2005 with common wall 4.5 inch each with east side property(common wall is 9 inch) .On GPA paper its written 15 x 35 area, but in actual its width is more than 15.9 inch including 4.5 of common wall. Now when we are doing construction after demolish 4.5 inch of common wall and we have done the construction till ground floor, then our neighbours saying that he has 3 inch in my area. Actually our common wall is not straight, so in last portion of my property he is saying about 3 inch. As common wall is not straight thats why he is claiming for 3 or 2 inch at different positions. He is fighting with us and my labours and not allow to do more construction. He breaks my constructed wall as well on first floor. Kindly guid me what to do?