I know you didn't ask for the drama but that's exactly what it is, drama! These people stepped over the line in saying that you are too rigid and introverted. Whether these statements are true or false, it's none of their business to treat you in this way or to say outrageous things to you.
In saying this, I think you need to stand up for yourself. I'd say to them, "I'm here to do a job, not to make friends or attempt to please in any way shape or form. It doesn't interest me what you think of my personally and i'm requesting from you to keep your thoughts to yourself and not get into the way of my work"
If they continue then I'd speak to the manager of the office or your boss of harassment.
Take down names, the times when these things happen, what was said, what you attempted to do to fix the situation on your own, and there responses. Try to see if there is anyone who will back you up and then make a formal complaint.
If you continue to let this go, it will go on. How do I know this? Well, thier comments were so outrageous they have proven that there is no limit to what they'll do.
Good luck with this. Passive is not the key if you want to continue on at your place of work. You should demand respect and put them in their place.
lets see how "rigid and introverted" they'll think you are after doing this.