I would suggest looking at it from a different angle. As recenty as the 1950s and 1960s there were very, very few books intended for "Young Adults" in existence at all. (To Kill a Mockingbird, while read and loved by young adults, was intended as an adult book.) there was children's lit and adult fiction. Children's lit you had then (as now) vapid, popular series like Nancy Drew or the Hardy Boys (yes, they're fun but they are formulaic, just like Clique and Gossip Girl) that could be read by teens and then you had historical fiction and that was pretty much it.
Big, Big breakthroughs came in the 1960s and 1970s. Everything from Harriet the Spy to Are you there god, it's me, Margaret - knocked down the doors of what was considered appropriate fare for young adult and older juvenile readers. For example, the sequel to Harriet the Spy is the first book for juveniles to mention a girl getting her period (and the book came out in the mid 1960s.) it was banned from many libraries, but within 10 yrs of it's publication you had Judy Blume's books taking on the same topics (about physical development and social injustice) and they were (while still controversial!) able to be popularized to a wide audience. Another example of this is Anne Frank; the Diary of A Young Girl, which as many people know excised all references to her development and they weren't put back in until now.
So my point is that the entire category of Young Adult Literature didn't even exist until recently. As more and more "realistic" books (like The Outsiders and other SE Hinton books, Judy Blume books, or even Madeleine L'Engle's realistic books) became more and more popular and accepted, publishers started being more open to the idea that young adult books existed as it's own viable, strong category and the genre exploded. When I was a kid/teen in the 1980s, my library didn't even HAVE a teen section. I found books either downstairs in kids or scattered in Adults (usually in sci fi or fantasy.)
now, however, the genre is amazing. You have series novels like Clique or Gossip Girls but you have fascinating sci-fi/dystopian novels like Lois Lowry's The Giver Series or Scott Westerfields books, you have stunning fantasy like Golden Compass and the last 4 Harry Potter books, you have gothic romances appropriate for YA (like the Stephenie Meyer books) instead of just Anne Rice or awful books like VC Andrews series. And yes, you have tons of serious writers writing serious fiction. Think about Meg Rosoff (How I live Now), Edward Bloor (Tangerine), Walter Dean Myers' books, The Book Theif - you just have so much to choose from!
The main difference is what's popular and I think that it just takes time to see which books fall out of circulation and which books go on to become classics.
I love that "The Outsiders" is a classic... ah, the 80s - I am getting old!