/h/ is not simply breathing in or out. It's a voiceless glottal fricative. It is produced by the same air motion with which you make an /s/, /f/, /v/, /z/ but its place place of articulation is in the glottis - like that catch in our throat when you say "uh-oh."
For that matter, we can all learn to make, or at least imitate sounds in other languages that we also have in English - like types of voicing, tones, etc. Their function in the english language is more pragmatic - meaning on the level of a whole sentence. Tone in Mandarin, for instance, is lexical. The pitch affects meaning on a word-by-word basis, not a sentence one like english.
Many languages have /h/ as a phoneme, meaning a segment of sound that has meaning when it is used in combination with other ones. Sure - a french person can make an h sound, but for historical reasons it is no longer meaningful for the language. It is still written out of convention.
All initial /h/ sounds underwent a weakening process in latin. They either vanished, became something else, or affected the vowel next to them while evolving into romance languages.
So "ha" in Spanish/Italian/French/Portuguese all sound like "a".