The Northern USA had water power and fabric mills in place to process the early fabrics, wool and flax. The cotton gin provided a cheap replacement of those more labor intensive fabrics and the Southern USA had the land, climate, wealth, political power, extensive use of water transportation and forced importation of labor to rapidly produce and transport the new raw material, cotton.
With the wealth of the new cotton the owners and laborers grew separated and distant. The labor became managed by a 'field boss' who was payed or could argue to be payed by a percentage of the profit he managed and produced.
Do you happen to be employed and managed by an owner, or a manager on a commission? Things can go very bad for an employee? Imagine adding an subordinate race, ownership factor, and absence of legal rights in that exchange?
Now the poor farmer-owner may have slept under the same roof, ate from the same pot, and by day labored side by side in the field. But still with an ownership in that situation … things could only be imagined as 'equal'!