Unless they made it clear in a written contract...or even a verbal contract, that there was a $50 broken appointment fee, then they do not have a leg to stand on. They only way someone can charge you ANY kind of fee is if you agree to it either in writing or verbally. And most verbal contracts are impossible to enforce.
So, unless you have a CONTRACT that you signed with your doctor, it just isn't going to happen. And, if they threaten to send you to collections, tell them you will contact the Federal Trade Commission and let them figure-out who is right or wrong. Them for charging you a fee that you didn't agree to in advance...or you for refusing to pay it. Lawyers cost about $400 an hour, and I don't see any doctor's office wishing to pay $2,000 to try to enforce and collect a $50 fee for a missed appointment.