Bad idea. I've done it, it sucks.
I have a pair of toms, I use to wear them a lot even though they were a pain, but hey, fashion hurts.
So every time I wore them with no socks, my feet started to ache, itch and sweat.
So I recommend wearing socks, especially when exercising!