Refund of security for a vacant flat as it is being charged as rent.

I rented an apartment and the contract was getting over by 5th April,2020, and so I left the apartment before that date, and informed the broker as well. but due to the pandemic Covid_19, i wasn't able to return the keys of the apartment but, when I tried to return the keys or tried to contact, the broker didn't took any calls and went offline. Unfortunately, the broker gave us an answer a month later that he won't return the security deposit and will charge it as rent pay the vacant apartment. now, he is threathing us that I did a lot of parties at the apartment during the contract and there are a lot of complains against me and so I should not file a suit, and secondly he has made it clear that I should be the on responsible for paying the rent even after the agreement got over and I left it, instead of he trying to make a call and let me know about the situation and what should be done. the broker didn't even tried to contact me for a month and want to take the security money.