This question really belongs in the Politics section rather than Words and Wordplay.
First, I would argue that it's a false and leading assumption that 'racist' is used in 'every' disagreement about immigration. It is certainly used in 'some' discussions on the topic, so let's look at what the word actually means.
The dictionary defines 'racist' as:
1. (noun) a person who shows or feels discrimination or prejudice against people of other races, or who believes that a particular race is superior to another.
2. (adj) showing or feeling discrimination or prejudice against people of other races, or believing that a particular race is superior to another.
When a person refers to an entire class of people as rapists, criminals, and drug dealers, it certainly seems to come close to the above definition of racist.....even if it's followed by a half-hearted statement that a few of them may be good people.
Likewise, when all members of a particular country are assumed to be of a certain religion, and therefore terrorists, it also begs the above definition.
There may be some kernel of truth to statements like these. It can be debated what practical solutions are needed, but at least on the surface, statements like these do end up sounding racist.