What is wrong with hate speech is obvious. It is offensive. There is a clear difference between a joke and saying harsh negative things about a particular sexual orientation or religion.
Almost anyhting anyone says can be offensive depending on the person who hears it and how they interpret it. But free speech is freedom to say anything and not EVERYTHING is offensive. If I walk down the street and say the word "the" to 100 people, not a single one will be offended. That doesn't make me saying the word the not free speech.
The United States federal government and state governments are broadly forbidden by the First Amendment of the Constitution from restricting speech. See, e.g., Gitlow v. New York (1925), incorporating the free speech clause. Generally speaking, the First Amendment prohibits governments from regulating the content of speech, subject to a few recognized exceptions such as defamation and incitement to riot. Even in cases where speech encourages illegal violence, instances of incitement qualify as criminal only if the threat of violence is imminent. This strict standard prevents prosecution of many cases of incitement, including prosecution of those advocating violent opposition to the government, and those exhorting violence against racial, ethnic, or gender minorities. See, e.g., Yates v. United States (1957), Brandenburg v. Ohio (1969).
Under Title VII of the Civil Rights Act of 1964, employers may sometimes be prosecuted for tolerating "hate speech" by their employees, if that speech contributes to a broader pattern of harassment resulting in a "hostile or offensive working environment" for other employees. See, e.g., Meritor Savings Bank v. Vinson (1986), Patterson v. McLean Credit Union (1989).
In the 1980s and 1990s, more than 350 public universities adopted "speech codes" regulating discriminatory speech by faculty and students. These codes have not fared well in the courts, where they are frequently overturned as violations of the First Amendment. See, e.g., Doe v. Michigan (1989), UWM Post v. Board of Regents of University of Wisconsin (1991), Dambrot v. Central Michigan University (1995), Corry v. Stanford (1995). Debate over restriction of "hate speech" in public universities has resurfaced with the adoption of anti-harassment codes covering discriminatory speech.