Unlike some who trash your viewpoint due to modern norms, let me be clear with you. Your concern makes sense, however, let me project for a moment. If I was the guy, sure, I have shared my care and body with that other women, but as for my heart? I don't attempt to state this with arrogance, but how many people can truly understand the heart of another? If I was that guy, I would've shared fun times but not my heart--after all, who can understand one's heart? In my eyes, only after being together for a lengthy period of time and going over a myriad of situations can another begin to sculpt out, in their mind, a shape of what your heart is like. However, even then, it wont be completely accurate.
Again, perhaps he did share his heart with her. Some people trust that others are capable of something so deep.
Regardless, as a guy, I can at least claim he will appreciate your body. As for the rest, that's left for your analysis. Presumably he chose you because what he views "your being" as one that aligns more with him than the other woman. Of course, this is where more info. would be needed for a clearer judgement call, i.e. why didn't he marry her.
Anyhow, relationships, especially on the level of marriage require trust. So you must figure out in your heart if you're willing to trust his love for you to be true. Only in taking that risk do you allow for a potentially long-lasting relationship. If not, you'll always be worried about his feelings being fake and, like a self-fulfilling prophecy, lead the relationship down the path of failure.
tl;dr concerns make sense. He chose you for a reason. Without trust, the relationship is doomed to failure.