Generally the reference is to a well suntanned face, implying an active outdoor life by a strong man. Possibly also to dark hair, though that is less important in the notion of strength and manliness carried in "tall, dark, and handsome".
Note also that in the days when that expression arose, respectable young ladies were expected to keep their skins very "white", and not to get suntans. Thus there was a contrast between their skin tones and those of tall, dark, handsome young men.
As I was growing up in Britain in the 1950s the notion of "black skin" would have been meaningless - where I was in a small seaside town we simply did not see black people, though, of course, I knew of the black races within the British Commonwealth as exotic people living in other parts of the world, maybe occasionally seen as visitors in London.
A similar word is "swarthy". It generally implies the skin colour of races outside the European field but not fully "black", and is sometimes derogatory. You would never see: "tall, swarthy, and handsome".