I think young guys were the first to start wearing backward baseball caps at the dinner table. To me it looked like they were using them as a kind of hairband, to keep their too-long hair from getting into their faces while they ate. If this is your problem, try sitting up straighter, or slicking your hair back with gel. You might be even able to wear some kind of narrow headband. Girls with long hair have this problem all the time.
In any case, it is impolite to wear a hat or a baseball cap at the dinner table. It's just one of those cases where "let it all hang out" collides with custom and etiquette. Customs do change -- it used to be polite for a husband to wear a suit and tie to the family dinner table (except on Saturday perhaps). Then it became acceptable for him to leave off the suit coat and just wear his white shirt and tie. Nowadays he's just supposed to wear a shirt with sleeves of some kind.
Meanwhile, solve your hair problem some other way, and appear at the table without your cap.