Paid 5.5 lakh advance to seller and bank loan not approved this proper

Hi Team, we have paid advance of Rs 5.5 lakh to purchase old house. But bank rejected the home loan, reason stating "property having one entrance i.e only 3 ft passage " . The owner had mentioned in the agreement "2 lakh will be deduct if we fails to purchase the property. My parents unknowingly signed the agreement. Now the seller taking advantage and deducting 2 lakh. And also the property dimension is not as per documents its passage acquired from neighbor around the property. Can we fight legally based on loan rejection from bank. Still we have around 10 days left to agreement expire.