Plenty of women are aroused by sight, and plenty of men (like my own husband) are aroused by touch. Rape is about using sex to wield power over women (or sometimes men), about being sexually aroused by the experience of wielding power and instilling fear, not about the way women dress.
But some people aren't very sophisticated or experienced. They think in simple black and white terms and find complexities, inconsistencies, and shades of gray too confusing to handle. Those things also wreak havoc with the desire for neat, simple rules that such people somehow (and mistakenly) believe will make everything clear and easy in terms of behavior.