Yes the Marines did have cavalry units. Most notably in China, where the 4th Marines had a mounted (on horses) division from about 1907 until the Marines left China in 1941.
While the Peking Legation had the only organized 'Cavalry' unit, the Marine Corps used horses effectively in Mexico, South America, etc.
You can also call any mechanized (either by truck, tank, helicopter, etc) force cavalry. Cavalry is by definition fast moving forces.