People may not agree but I think it's several things:
1. The beauty ideal has white women being the most desireable. In the minds of many, all black men want a white woman. You see it implied on this forum quite frequently. The question is always "why do so many black men date white women" and never "why do so many white women date black men" despite the fact that, of ncessity, the converse would also have to be true but to admit as much would be to somehow diminish the perceived higher desireability of whites.
2. Black women have, historically, been the least likely to date outside of their race and I think the American psyche is slow to catch up to the reality. Where I live, I actually see far more black women dating outside of their race than the other way around. It could just be the area that I live and the areas that I frequent but that is what I see.
3. Since black women are, typically, as far from white women as is physically possible comparatively, we are the furthest away from the beauty ideal so the idea of many is "why would you want a black woman when you could be with a white one?"
I know a number of white men who prefer to date black women and they have told me that they are constantly put in the position of having to explain themselves. Black men who prefer white women and white women who prefer black men are far less likely to have to defend their preferences. I do think that black women with white men, at the end of the day, are a more accepted pairing than the other way around. Women have, historically, absorbed the culture/status of the men in their lives. When a black woman is with a white man, her value is raised. The assumption is that she must be something pretty special. When you see a white woman with a black man, her value is diminished and she must be white trash (so say those who care anyway).