I think part of it is the societal assumption that women touch to bond, comfort, communicate, while men touch as an overture to sex.
Is it always true? Of course not.
But men are FAR more reluctant to touch anyone than women are when we're not talking about romance, so, by extension, when they DO touch someone, it often is romantic/sexual in nature. Women touch each other all the time, and other women obviously know that. Men do NOT touch each other all the time. Why? Do you suppose that homophobia leaks into men simply not being able to touch unsexually?
I'm not claiming it's fair, but basic homophobia has repercussions in communication that we barely understand. I believe on effect is that men don't get to touch.