You're not "justified" to cheat.
You're certainly justified to feel that he isn't living up to his responsibilities as your partner and husband. What you need to do is leave him, or kick him out. Tell him you want to separate, then divorce. Maybe that will get his attention. I'm sure that your life is difficult, with two babies to take care of, and he probably doesn't feel hot and bothered to have sex with you. You have a "newborn"...has it been long enough, post-partum, for you to have sex? Could he be worried that you're not fully healed yet?
Bottom line...if telling him that you're sexually frustrated enough to consider cheating didn't motivate him to act, then it really is probably time to separate. And if that doesn't make him realize how much he wants and needs you, then nothing will...file for divorce.