That's an interesting observation, and all I have is a guess. I dress very casually myself, not from the mens department, but usually simple pullover sweaters, jeans, and athletic shoes. In a setting where others get dressier, I'll switch out black pants and Payless boots for the jeans and athletic shoes. Why? Well, it's a matter of using my energy (including mental energy) on what is rewarding to me more than on what isn't rewarding to me. I haven't always managed to look impressive and fit in with the beautiful ladies, even when I tried my hardest, but I do always like the feel of wearing comfortable clothes. So one type of clothing choice didn't result in my life being better, and the other type of clothing choice did. Rinse and repeat across a lifetime, so habits get formed without me even thinking much about it.
I may not be on the autistic spectrum, but then again I have always thought maybe I could have been if they'd known in the 60's what they know now.
And for the record, one of the reasons I think that is that I do have sensory issues, and the softness of cotton clothing is MUCH preferable to me than most other fabrics.