Delay in payment to builder due to fault of builder charging penalty

I bought a flat from a builder in kolkata,on the basis of agreement executed between me and builder.Out of Flat price,10% was paid by me & 90% by bank loan.Builder delayed process of providing documents required for home loan to me & provided me documents in installments,resulting in delay in processing of same by my bank/lawyer and hence delay in payment to him. Builder had sent me mail to make payment within timelines to avoid penalty & I replied not to charge penalty as delay was from their end. That time builder didn't charge. But now after 3 months, they are demanding penalty.Pl. guide.