Native Americans died from the deceases brought over by foreigners, which their immune systems couldn't handle.
With the foreigners came their slaves. A house of three white people sometimes would have (just a random number, but...) 10 black slaves. As time went on and they spread across America, they brought over more slaves.
Obviously, once the slaves were freed, they were able to spread out, have children, make large families... It's really not that hard.