Soldiers marching under the "Stars & Bars," the Confederate battle flag, killed and maimed so many Americans that al Qaeda look like a bunch of amateurs by comparison. For many people, that flag (like the Nazi and Viet Cong flags) represents not just hatred of America, but massive, violent anti-American bloodshed.
Although many Southerners today are sincere about their "heritage not hate" attitude when they display the Stars & Bars, the flag has also been adopted as a favorite symbol by the Klan and other racist hate groups. So people who perceive it as an emblem of racism are not imagining things. (The United States was not the only country where slavery was still legal in the 1860s, but it's certainly not true that "every other country in the world" was pro-slavery. In fact, it was largely because the Confederacy was so closely associated with slavery and racism that Confederate politicians and diplomats were not able to achieve international recognition for their new country.)
After the Civil War, many Southerners and Northerners devoted themselves to the cause of national unity. Grant and other Northern generals gave strict orders for respectful treatment of Confederate troops at Appomattox and other surrender ceremonies. Union and Confederate veterans shook hands at battlefield reunions. Former enemies paid their respects at each other's funerals. But there were some on both sides who tried to keep the old hatred alive. In the South, resistance to civil rights for black Americans was the main practical expression of that attitude. And displaying the Stars & Bars was the main symbolic expression. So for many Americans, that flag represents not only racism, but also the attitude that the U.S is still a nation divided, a nation at war, not a unified nation.