My lawyer emails me with blackmail emails ,criminal charges etc

Hi, I have a pending litigation of a false ipc 353 filed by some corrupt police men . I filed a complain against the police to magistrate in 2019 August through a lawyer . I noticed my lawyer is incompetent and greedy .Recently i decided to remove him and find a new lawyer.My lawyer blackmailed me saying he will file a complain to passport authority on his letterhead .he doesnot know my passport number ,but he has photo of my adhaar card,my date of birth and my address .can he cause any harm ? I have to renew my passport and fly abroad in 5 months .he also threatened me that he will file criminal charges of defamation and intimidation .we only exchanged a few unpleasant emails and I was angry on him on my emails only .i never defamed him in public nor did I cause any physical threat to him.I don't know what rubbish is he going to write .I am worried .should I be ?