Caring about yourself is no bad thing, for we need to look after ourselves and value ourselves, and it's not wrong to enjoy legitimate pleasures either. After all, the beauty in the world, the pleasure of food and of having family etc are gifts we should all enjoy as they give immense satisfaction.
A problem only arises if you indulge in illegitimate pleasures (especially at great cost to others, such as raping a person), or care nothing for other people. The utter selfishness and self-centredness of caring only about yourself and your own pleasures means you are either immature and ignorant (like a two-year old) or so hard-hearted that you knowingly disregard others. The trouble is, to live selfishly is to deny yourself great pleasures in life, and the pleasures you pursue will be fleeting, with no lasting satisfaction. But just as two-year-olds have difficulty grasping mature concepts like sharing, duty and responsibility, so do adults who have never grown up in that regard. Even to ask such a question shows staggering lack of comprehension about basic human nature. Those who see nothing wrong with that will become very lonely, unsatisfied, bitter people, in time.