Grown men being interested in a cartoon whose primary audience is young girls comes off as weird to a lot of people, and suspicious to enough people that you hear of people making a "Bronie = pedo" connection.
Even though I find the Bronie phenomenon to be utterly bewildering, I don't have any particular beef with Bronies and I know it's patently unfair to make any accusations of pedophilia. There are some folks, however, whose rationality is insufficient to grasp the difference between "this seems strange to me" and "this is wrong". Those are the ones making the accusations.
It's not fair, and it's not pleasant to hear (particularly not if you're a Bronie), but that is the explanation.