As the house is in your father's name and he has died intestate (without a will), you need to file a suit for partition and permanent and mandatory injunction. 

Assuming you are Hindus, Section 8 of the Hindu Succession Act stipulates that the property of a male hindu dying intestate shall be divided equally among the Class I heirs and hence, the property shall be divided into 6 equal shares (1 widow + 3 brothers + 2 sisters). 

Thereafter the partition decree shall be passed by the Court and the proceeds after selling the house shall be divided equally.

In case the Court comes to a conclusion that it is impossible to partition the property in Six Equal parts, i.e. by metes and bounds, the Court will order sale of the property and divide the sale money equally among all.

You can get a protection order for your mother under Maintenance and Welfare of Parents and Senior Citizens Act 2007 whereby she shall be protected from ouster from her own house and if thrown out, your brother can be compelled to take your mother back to her house and maintain her as well.