Dispute of property with illegal construction

Sir there was a mutual understanding between my father and uncle in our house. Our house is very old and divider is not straight so mutually we decided that some part of front area we give to my uncle and we will take same area from back of house. But we don't have anything in written.Now my uncle wants this to normal like before he wants to destroy everything which is coming in his area. And it's been more than 10 years that we have made construction in this area. He also constructed in our area. Sir my question is it possible that this property can be destroyed ? Though it was mutual but we don't have any proof also we made this property infront of them 10 years back then why they didn't stopped us from doing this at that time.