Personally, I think of APBT when I hear 'pit bull', but then again, I'm a groomer and I've been working with dogs for years, so I actually know my breeds. :-)
I think the mixup boils down to ignorance. People see any short, stocky, muscular dog, with short hair and an blocky head, and think, oh, that dog must be a pit bull. Someone posted a quiz up here once, it had photos of about 50 different dogs, only ONE of which was a true pit bull, and the rest of which were dog breeds commonly mistaken for pit bulls. It made a very good point, people just don't know. And with all the media hype, they are more than willing to jump to conclusions. My boyfriend has a 25lb Boston Terrier and people call HIM a pit bull! He looks closer to a pug than a pit. It's crazy.
Make more friends who are involved in the dog world- it keeps you from going insane when you deal with the ones that aren't. In my world, there are Pits, Staffies, Am-Staffs, etc. If anything, the proper blanket term for that group is the bull or bully breeds. Trust me, there are sane, rational, intelligent dog people out there. And the rest... well, the rest can just continue to cower when a 15lb mini bull terrier walks by, and we can laugh at them.