Well, I am not really going to give you an opinion on this but I will try to explain the thinking behind it (if you really are interested -- I suspect you're trolling fer trouble):
The American Civil War was fought largely over slavery of Africans. Yes, states' rights were the ostensible issue but they would not have been an issue if it not for slavery. If you are truly ignorant of the causes and history behind the events that led up to the Civil War, you SHOULD ABSOLUTELY positively read up on them or at the very least watch Ken Burns' Civil War (one of the best films ever made).
So, you see, to make a long story short, a descendant of those slaves might not particularly like to see the representation of his ancestors' oppressors waved in his face and may counter that it's racist. If you were a descendant of an African slave, what would your reaction be to, what would be to you, a symbol of the bad ol' days for your people? This is an honest question, not a leading one, btw.
I'm from Richmond, Virginia -- former capitol of the Confederacy and KNOW ALL ABOUT IT!