Client wants refund of advance payment

Hi, I am a freelancer and was supposed to work with a client for his requirement. I met the client twice at their location and a deal was stuck. I asked the client for a contract twice before starting the work but he kept delaying the same. He transferred a 30% advance to me to start my work and to onboard another freelancer. In good faith, I started my work and onboarded another freelancer. Then the client shared a contract with some ridiculous clauses and I refused to sign the same and the project was scrapped. Now the client is asking for a full refund of the advance payment. Since, I onboarded another freelancer on their request, did work for quite some time and spent time in meeting them as well, do I need to pay them back? What should I do?