House owner not returning security deposit

The house owner brotherinlaw is in airforce named Sudhakar. Sudhakar is the one who spoke to us while giving the house stating house owner is abroad and he is caretaker. He took 6 months of rent as security deposit which is 1.5 lakhs. Even after mutually agreeing date of vacation, he has not come on day of vacation. Instead he asking us to hand over key to third person. When asked about security deposit, he says he will come later and do inspection and return our advance. My questions: 1) As the key is still with me, do I have to pay him rent?? It's him who has not come till now... Last month he sent legal notice demanding that rent has not been paid till date and the notice is to pay rent till date and mental torture etc to give him more money. 2) what should I do with the house key?? Incase if I have to pay rent as long as I have key with me...actually the house in one state and the person lives in the other state of India. 3) the agreement is not registered as per new rule in ta