Wrong, stupid, immature, idiotic, arrogant, inconsiderate, inappropriate and ignorant but not much of a criminal offense. I say not much of a criminal offense because it is a form of harassment. Unfortunately pressing charges for such a thing...the police will press the charges if you wish them to but it's really not going to do anything other than anger these idiots more and make their behavior worse toward you and other people. So the best thing to do is 'turn the other cheek' and leave it where it is, in the past, and just simply ignore people like that.
The reason people are racist and bully others is due to their own insecurities. Because a person around them is different an insecure-with-themself person feels uncomfortable because they aren't the same. It's a form of fear. To defend themselves (against nothing - an every day experience) they feel they have to encourage themselves to feel superior to the difference, so they gang up (out number) and bully, make inappropriate comments, etc, to make them selves feel better...about themselves.
They don't usually harass others if they're not in a group that out numbers the persons challenging them by being different. Different could be any difference; more attractive, better looking and/or cleaner clothes, smoother skin, eyes and teeth that aren't crossed as examples.
If these kind of mentally challenged people are alone they usually don't say or do anything. They just 'suck up' their own insecurity.
Your presence challenged them. You are the better person.
You were riding the bus just like they were.
No matter what a human looks like on the outside we're all the same inside.