It would be wrong to give a definite opinion on what way you should move forward, even though you've asked. We don't know you or of your life so it's a complicated and heartbreaking thing for you.
But I will comment based on instinct, as someone on the outside looking in. What love does this man have for you? Children are a by-product of a union between two people. No-one can predict whether you will actually be able to produce them, on the day that you enter into the marriage contract. It's between the two of you, not between you and any future children you might hope to have.
Do you want to fight? If his mind's made up, it may be futile and I can't see any serious commitment to you as a person. No love. No consideration for 20 years of marriage. So, my instinct is to say let it go. He's not worth fighting for, in my opinion.