I'm in agreement with a lot of the folks here. It is generally impolite, however, we have become accustomed to the more relaxed environment that some restaurants offer.
For example, McDonald's, I don't expect to see people take their hats off. For a casual or formal dining restaurant, absolutely.
Exceptions can exist however, such as at diners or truck stops. Or, as one person mentioned, in Texas, where hats have become a strong part of the culture.
On the other hand, if a woman is wearing what is considered a lady's hat (not a ball cap or something like that), then it is acceptable to keep it on. To me, the rule only applies to gentlemen.