Irreversible contract. What legal implications if we break contract?

The legal implication of breaking the deal with the builder depends upon the terms of your contract with the builder. Generally any such contracts have termination clause and the amount which shall be paid back in case of termination. In case there is no such clause, then you can claim the amount back from him, however, the builder shall be entitled to deduct the amount he has already spent on the construction.

You have even stated that the contract is irrevocable, but even then if you can satisfy the court as to the reasons for termination of the contract, then you shall be entitled to the amount you have paid minus the amount already spent by the builder. 

With respect to your partner, it again depends on the terms of the partnership agreement. In a partnership, both the partners are jointly and severally liable for the liabilities.

You cannot stop the partner from terminating the contract, however you can claim the percentage share of his liability from him as per the split of 60:40.