Ghetto's were home to most of the black population in the day where segregation and other injustices were still happening. That is most likely why the population thinks as blacks as ghetto. The meaning probably changed from 'those African Americans from the ghetto' to 'ghetto African Americans'.
I think it's among the same lines of sick (illness -> awesome), awful (full of awe-> horrible), prude (polite, well mannered -> rude) we begin to associate words with each other, the human language is very flexible, it changes as generations go on. (We don't use thou or arts anymore, for example.) You can use ghetto now, to describe anything you'd like. 'Dude, my car is so ghetto' or something like that. It implies something that's run down or patched up horribly, it doesn't necessarily apply to African Americans or Jews.