Depends on if you mean infertile as in poor sperm count/quality, or you mean sterile as in has had a vasectomy or has a genetic condition that causes azoospermia. Because infertile and sterile are NOT the same thing.
And yeah, have to agree with the others, that if a guy says he "can't" get you pregnant because he's infertile and therefore shouldn't have to wear a condom, it's almost always a lie. Unless he has had a semen analysis proving he is shooting blanks, he has no way of knowing if he's infertile or not. A guy who is *actually* sterile due to medical causes knows the difference between sterility and infertility, as well, because it means he's seen a fertility specialist about it and knows exactly what his status is. Having had unprotected sex with other women without getting anyone pregnant is NOT proof he's infertile. That's just damn good luck.