"Yugoslavian" is a nationality, not an ethnicity. As with the Soviet Union, or Czechoslovakia, Yugoslavia was made up of a number of different ethnic groups, all speaking different languages.
In the early 90's, the nations of Slovenia, Croatia, Bosnia and Herzegovina, and Macedonia declared independence from Yugoslavia. What remained of the nation would split into Serbia, and Montenegro in the early 2000's.
Discussing the linguistic makeup of former Yugoslavia can be a little confusing, since there are many nationalist groups who are very keen one seeing, for example, "Montenegrin" classified as a distinct language, when it is, in reality, a dialect of Serbian.
Each of the nations of former Yugoslavia speak South Slavic languages. This means that they're relatively closely related to the West Slavic languages (e.g. Polish, Czech, Slovak) and the East Slavic languages (Russian, Belorussian, and Ukrainian), as well as the rest of the Indo-European family on a larger scale.
Slovenia speaks a distinct South Slavic language known as Slovene. Slovene is actually considered to be at risk of becoming endangered sometime in the next few decades, due to the comparatively low number of speakers, and the economic dependence of the tiny Alpine state on other EU member nations.
Serbia, Croatia, Bosnia, and Montenegro are perfect examples of political agendas taking priority over linguistic fact. As I have said, "Montenegrin" is indisputably a dialect of Serbian, while Croatia and Bosnia both speak languages that are extremely closely related to Serbian. Croatian and Serbian are often lumped together into one "Serbo-Croatian" macrolanguage, though, as the above answerer stated, the use of this classification is declining. These three languages [Serbian, Croatian, and Bosnian] are more or less mutually intelligible. One of the primary points of difference between Serbian and Croatian is that Serbian is written in the Cyrillic alphabet, while Catholic Croatia uses the Latin alphabet.
The tiny nation of FYROM Macedonia speaks Macedonian, which belongs to the East South Slavic subgroup of the South Slavic family, rather than the West South Slavic group to which Slovene, Serbian, Bosnian, and Croatian belong. Just as Croatian is often considered a dialect of Serbian, Macedonian is often connected with Bulgarian, which is extremely closely related to the language of this tiny republic.
If you're going to learn a South Slavic language, either Serbian or Croatian (Serbo-Croatian) would be the obvious choice. Not only do both languages have more speakers than Macedonian and Slovene combined, but they would be the most widely understood across all the former Yugoslav republics. Croatian also holds official status in Bosnia, where ethnic Croats make up a sizable minority. Once you know one South Slavic language, the rest will come extremely easily to you if you ever desire to learn another.