The fact that you are a 56 year old nurse and don't know how to manage the most basic of social situations might be an indicator why the patients prefer other nurses to you.
Just pretend you didn't hear or laugh it off. I shouldn't have to explain to you that patients and their people are often incredibly stressed and it does not bring out the best in them. And some people are just jerks. Everyone has their biases whether it's ethnicity, gender, age, tattoos, the list goes on and on. You just deal with it. Your job is to care for the patient and to represent your employer's best interests.
The fact that you are 56, work with the public and have not yet learned this is astounding to me.
Why not just wink, smile and say, "I'm an oldie but a goodie."
Do you want to escalate the situation or relieve the pressure? Your choice.