Builder causing trouble and delaying possession. What to do?

We, 06 persons own a plot of 4 guntas at Manjhari Bk, Kesabnagar. In 2012, handed over to a local builder for construction of apartments. As the location is in gram panchayet & 11 km far from the pune rly stn, at that time other builder were not ready to take over the project and so, we gave Rs. 6 lakhs to the builder for start-up the project. In return we olny demanded 6 apartments from him. He had to return our apartments within 20 months wef 16/04/2012 to 16/12/2013. We had an Notary agreement with him.The builder only completed the plinth level const. & not ready to hand over to us.