the one basic reason is because your right to do something ends as soon as it infringes on somebody else's right to do something more fundamentally important.
a second reason is that we have developed technological tools such that it is possible to more easily infringe upon one another.
a third reason is for social control. some people are more conservative than others, so we need to find a happy medium so that nudists, for example, don't exercise their rights to freedom of naked expression in front of the children of those with more puritan values. again, that's an example of the first.
a fourth (bonus) reason, is because humanity is not yet advanced enough to easily discern the grey areas between what is a right and what is a privilege, and when a denial of rights is okay, and when its oppression.