Constructor made arbitrary agreement, now backing out and not paying.

We had entered into a contract with a house constructor, on the terms that if we back out from the agreement, then we'll pay him double of the amount he had paid to us at the time of signing the contract (ie: 10,00,000x2= 20 lakhs). If he backs out, then he won't be refunded the initial 10 lakhs. 10 days before the date of commencement of construction, he backed out saying that the costs are too high for him. We told him that we won't return the refund. Now, after 3-4 weeks, he has sent a legal notice stating that we refused from vacating the property and that we owe him 20 lakhs for fraud. What to do?