My friend and his relative is threatening my family for borrowed money

I’m jobless from a very long time unfortunately. Because of some financial issues I asked for help from a close friend, he offered me help by lending money. I promised him to pay him partially through my ‘irregular’ part-time earning. I was somehow managing everything but in 2020 when pandemic was declared I lost the part-time source too and I could not pay him back as promised verbally on phone calls and sometimes on whatsapp messagesWe then agreed after discussing verbally to wait for some time as my father was about to retire and he will get retiral funds, and I told him that I will clear the amount in one goBecause of some legal issues with the department my father didn’t get the entire fund after the retirement and he also gets provisional pension which is very less for our family of four. But that friend has been threatening and insulting my family regularly. His relatives are also threatening us by reaching our home regularly. We didn’t sign any paperwork. Please help!