Well, assuming that you don't walk around with him like he's a trophy with your nose up at black women....
I would venture to guess that there's just as many white women that are disgusted with white men getting with black women, as there are black women that are upset that white women are getting with black men.
I would also take a wild guess that these women, whether white or black in each situation, simply have a low self-esteem, and playing the "race card" is the easiest way to boost it.
Since you married him and plan to be with him til death, you might as well sit back and get ready to deal with this til then, because I really don't think this problem will go away any time soon. That comes with the package of inter-racial relationships. Trust me, I know. My wife is asian.