You're right, that isn't smart generally. However, imagine a situation where there is a layer over the airport causing the weather reporting to call the ceiling 1000'. If you go a mile or so, that layer isn't there, and you can climb unrestricted. In this case, it would be legal and safe to take off, climb in the direction of the clear skies, and go on your way.
Of course, while a pilot wouldn't be violating the weather regulations by buzzing around at 500', he would almost certainly be operating in a careless or reckless manner, violating 91.13, and also probably violating the 500' minimum distance from people, structures, property, etc. throughout much of the flight.
The weather regulation is not meant to be the only limiting law in regards to whether you can or can not fly that day.