Money drives society. And the 1% of the 1% drives the money. People whose incomes rival that of entire countries set up soul-less corporations that enjoy all the benefits of personhood with none of the responsibilities or liabilities.
Xenophobia may well be part of human nature but there's a lot more to racism than mere xenophobia. Before the rape of Africa, no one cared what color your skin was. The ruling class deliberately created racism to divide and conquer the rest of us. For many years, people of all colors were indoctrinated with the idea that white people were inherently superior to everyone else and that slavery was part of God's order. The churches and the media of the time had a heavy hand in this indoctrination.
The Virginia Slave Codes of 1705, the first law codifying racism in our society, were passed in the wake of Bacon's Rebellion. Jamestown folk of all races and classes joined forces to oust an unpopular Crown-appointed governor. The powers that be wanted to make sure that such an alliance would no longer be possible and so racism was born.