"I feel.guilt because they keep trying to say they miss me.
First, there's nothing strange in you remembering only the bad times. Don't forget, this isn't like an abusive relationship between 2 adults. You were a powerless, innocent child who had no control over any of it. An adult can leave another one at any time.
However, on the above, if they can produce guilt in you, they still have some power over you, and this isn't healthy for you. When people grow up like you did (assuming this was legit abuse), all experts recommend you cut off contact until you've gotten therapy and can interact with them as an equal. If you're an adult living on your own, don't assume they can no longer hurt you, because they can and they will.
If you haven't gotten therapy, do yourself a big favor and get some. Cut off contact with them or else keep any phone interaction very general and brief. Keep reminding yourself you do not owe them anything. You do, however, owe yourself a lot.