"Everyone" is wrong. Height has absolutely nothing to do with when you show. Your uterus can grow up, down, to the back, sides, etc. Height doesn't impact that in any way, shape, or form. Neither does weight, build, being short or long waisted, etc. Personally, I wouldn't be hanging around people rude enough to tell me I must be having a premie since I "look too small."
I'm 4'11" and have had 6 full term pregnancies. I've showed everywhere from the end of my first trimester to not until I was seven months pregnant (and even then it was easily hidden until the final days).
Presumably you're having regular prenatal care. Your doctor would tell you if growth was a concern.