As of this reading, you have four answers, and while it is not unusual for me to disagree with other people's answers, I think this is a rare case in which either they or I completely misunderstand what's being asked. At any rate, I can only answer the questions as I understand them so if I misunderstand, please forgive me Carlton.
No one! There are forces and individuals who assume the right because they have the power, either through legislative, societal, psychological, or economic control, but this abrogation is given by the individuals controlled, often because of the threat of social castigation or physical harm. Ultimately, restriction of scope is an infringement of individual freedom, and is justified by those imposing limits using the rationale that it is for the benefit of the majority, or to further the goals of an individual or a group of individuals.
As to where the "right" originates, as I've implied in my previous paragraph, it stems from the power these entities have. It is assumed via the acquiescence of those being controlled. Often, it takes enormous courage, frequently at the risk of a person's life, to refuse to accept the restrictions imposed. History has given us examples of individuals who refused to accept these impositions and the consequences thereof. Just as there are living saints among us that will never be known or remembered, I'm sure there are persons who have sacrificed everything to defy such control. However, some famous persons who exemplify my assertion are Socrates, Gandhiji, Martin Luther King, and The Christ.