To me, a cat lady has to have at least 4 cats, doesn't keep her house very clean and lives alone.
You can't fill the stereotype with one cat. You can't fill the stereotype and have a beautifully clean house. You absolutely can't fit the stereotype with a lover. They kick you out of the club and your name is banned for life for the latter.
Why is it a bad thing?
I think that's obvious just because of the prerequesites:
Lack of hygiene is generally considered a bad thing. Such a person is far more likely to be the source of and/or spread disease.
If you are alone, for better or worse, particularly if you are a woman, people "wonder about you". In 2019, it's unlikely you'll be burned as a witch, but there is still the same distrust of isolated individuals, particularly women, as society generally believes they should have been able to pair up.
And as for having 4 cats - most people don't like large numbers of cats (or dogs). Maybe they're family members, but they're family members who periodically cough up hairballs or hork up their own feces or bring you the most excellent gift of half a dead mouse. I think this ultimately goes back to un-hygienic.
If you are relatively clean, can be social and have less than 4 cats, I really don't think you need to worry about this.