His violent actions toward you and your unborn child may be grounds for terminating his parental rights---maybe. He may even voluntarily relinquish his rights to a child he didn't even want. Or, alternatively, maybe he can appreciate his child now, and has turned his life around since the last time he was incarcerated.
Have you even formally acknowledged this man as the father of the child, as in, put his name on the birth certificate or registered him with a state registry? If not...it's an option to just leave things as they are.
If you have acknowledged him as the father, I know legally he has rights to the child at this point in time, as all fathers do. But if he did find out, he would have to take you to court and get an order of visitation, at which point, you could present your evidence of why he should not have visitation, or at the very least should have supervised visitation. It sounds like you know this man and know what is best for your child--you may hire a lawyer to see if you can get this man's rights terminated based on his violent history toward you and his threats toward the baby. If you cannot afford a lawyer, you may seek to contact a family violence organization or shelter in your town--they can often help with these sorts of things.