Can my mother have a stake in grandfather's brother's property?

If the property is ancesteral, then your mother has a right on the property.
A pre-requisite condition required for a property to be Ancestral property is
that it should be owned by the great grand father followed by grand father, father & present generation all in this linear generation living & enjoying the property. The property should not have ever partitioned & share in it distributed to any of the Hindu Coparceners as in any linear generation as mentioned above, in other words, it should be intact single property which is being used jointly by all members of the Hindu Joint family.
However if the property is not an ancestral property, your mother will have no right on same.

If the property is ancestral then you can claim the property ,but if your grandfather brother had it by his own income you cannot claim .