Is there any legal action if my brother do not vacate house completely

My younger brother with his wife and kid, grandmother and I with my wife were living in a rented house. We all were to shift to our parents house. We have given the house keys to the agent and vacated most of the house. Except my brother is not removing his cupboard giving lame excuses every single day. Its been more than 10 days now that we have shifted and he is just saying tomorrow I will remove it. The agreement is on my name. And the deposit has been paid by my grandmother and myself (majorly). Because of my brother creating issues, the owner isnt returning the deposit. Hes insisting on getting the house completely vacated first. I am stuck and worried that the owner might cut the amount from the deposit as days are passing by. I have to shift to a new city and need the money urgently. Already spoke to brother, he is taking things lightly. We are also not on talking terms. What action should I take to resolve this? Can the owner take some legal action against me ?