For the most part the Union army left freed slaves alone, in the sense that the Union army did not set out to terrorize them. And for the most part, as the Union Army began to advance into the deep south, slaves usually fled their masters and moved TOWARD the Union lines. Now, the Union Army was given the primary mission of destroying the Confederacy. While freeing the slaves became an ultimate goal for the Union, it was not something that every Union general directly concerned himself with while South was still fighting. This lead incidents in Sherman's March that the Union cut the pontoons to a bridge across a river leaving many freed slaves to try and swim the river to keep up with the march or surrender to Confederate guerrillas who were liable to shoot them if they didn't return them to the masters they had escaped from.
The greatest problem involving slaves and freed slaves came from the South's policy to blacks serving with the Union Army. After the Emancipation Proclamation the Confederate government issued a proclamation stating, "Any ***** taken in arms against the Confederacy will be returned to a state of slavery. Any ***** taken in Federal Uniform will be put to death. Any white officer in command of ***** troops will be deemed as inciting servile insurrection, and will be likewise, be put to death."
And the South made GOOD on this threat. As Lee's army marched into Pennsylvania before the Battle of Gettysburg, any african they came across (regardless of how long he had been free) was sent south IN to slavery. During the siege of Petersburg, after Burnside attempted to blow a hole in the Confederate lines, Confederates murdered African soldiers attempting to surrender at what became known as the "Crater". A similar incident occurred in the west. Nathan Bedford Forrest's cavalry overran a Federal post called Fort Pillow, predominantly manned by black soldiers. EVERY federal soldier in the post was killed or wounded, and many of them shot AFTER they surrendered. In response to such racist atrocities committed by the Confederacy, Ulysses S. Grant ended the system of prisoner exchanges which had been done up to that point in the war... unintentionally creating the horrors at Andersonville.