If they're against it because they hate people of other races-- yes, it is racist.
You can be against illegal immigration for other reasons, however, in which case, it's not a racist idea.
However, many people (though not all) who are against illegal immigration
1) don't understand the issue or the economy completely, and
2) are xenophobes who emphasize the "illegal" part, but really just don't like non-white immigration in general