What can i do if house care taker is not ready to vacate house

I work in a different city with my family and my birth place is in a rural village but within same state West Bengal. About 8 years ago when I had been transferred from my native place to kolkata, I selected a family to take care of my house and the land. Since 8yrs I don't take any rent from them and an agreement was written in a A4 size paper to take care of my house without rent. Now I just retired from the job and want to live my native place, so for that I want to renovate my house in village. But the caretaker is not ready to vacant my place. With the help of village adult club member they are opposing to vacant the place and filed a case of harassment against me in nearby police station. They are now calling the place as their place. I didn't give any time any papers of land on which they can demand the place as their own. Now what can I do get back my native place? And can I get the place back? What is the procedure to get the land back? I am looking forward to hear from you.