friend living on my property forcing transfer/sale of property

I am a government employee working in Delhi for the last 25 years. I have a parental property in Ballia, Uttar Pradesh. In 2013, a childhood friend of mine had to sell his property. He requested me to give some space for building two room set as he was unable to find suitable land to build a new house from the money he got from sale of his property. Since he was a childhood friend and working in Ballia, I agreed. We didn't enter into any kind of agreement. My friend also, touched by this gesture, gave me some money and asked me to keep it till he vacates the house in approx. 5 years. He built on my portion more than what was agreed upon and now he is putting pressure on me to either transfer the property in his name or sell it to returns the money he spent on construction. He is using other people to put pressure on me to act according to his will as he knows that my cousins who are the co-owners of the property would not support me in this case. Please advise what to do?