He might have the benefits, but he really isn't a very good friend.
If I was casually nailing you on a semi-regular basis with no strings attached, I would make every effort to respect you as a person. I would even say nice things to you and offer to be there if you ever needed someone to talk to. Not because I want something from you, but because you deserve at least that much respect from other people.
I've had casual sex with more than a few friends, and most of them are still friends (or at least on friendly terms with me) to this day. That's because I never called any of them any insulting names, and because I respected them as fellow human beings.
Your 'friend' with benefits seems to think that having sex with him somehow cheapens or lessens you both as a friend and as a human being. And that's not very friendly of him, in fact it's very not-friendly to the extreme. He doesn't sound like a very good friend, in fact he doesn't sound friendly at all.
At least whores get paid. What do you get out of this relationship?