The Confederate Flag is often seen to represent racism for the reason you said - the south supported slavery. Slavery was the key issue in the Civil War and I think to discount it as THE cause of the Civil War would be incorrect (personal opinion, and I wasn't there). Because the Confederacy was so pro-slavery, civil rights proponents would like to remove all its symbols from common usage.
Throughout the years, the Confederacy itself has come to be known synonymously with pro-slavery/racism. Thus, so has the flag. The other causes of the south should not be forgotten, though. However, many people today flat out use the flag to proclaim their racism with pride. This small group of people has caused the flag to be seen in a negative light.
I agree that use of the Confederate Flag or allowance to fly the flag must be a careful decision. It can certainly have very innocent and even noble meaning, but it can easily be exploited, if you will, to promote racism and intolerance.
I remember all the fuss over flying the flag over a Confederate Cemetery (or something similar) and I thought that was a perfectly acceptable use of the flag. To ban it completely would be to forget an important part of our history and the strong beliefs of a lot of people.