Unless you can present me with data that shows otherwise, generally speaking we assume that birth defects are distributed evenly and proportionally amongst all racial groups. So, no. There are other explanations for why you may think the way you do.
Chances are, as America is mostly white, the kids in your special ed class are mostly white because that's how demographics work. Unless you're in a mostly non-white area, it's expected that more kids in special ed will be white simply because there are more white people. Also, in the media, these issues are often displayed using white people afflicted with these conditions. Historically speaking, special ed classes where shown to be mostly white, so your confusion can stem from there. Lastly, are you sure that some of the Latino or Middle Eastern students don't pass as white? On that note, how do you count mixed race kids, particularly those brown-skinned individuals with one white parent and one black one? You could be familiar with a few of the special ed kids without realizing their true race.
One thing to note: Birth defect doesn't always translate to intellectual disability. Sure, Down's Syndrome is a birth defect...but so is a cleft lip, or a ventricular septal defect (basically a hole in the heart). Or spina bifida. There are a LOT of birth defects that don't put kids in special ed, or can be dealt with without involving special needs. Cleft lips can be fixed with plastic surgery, and I think they're the most common in Native Americans and not white people.