Each star we see is similar to our sun, in that it is a giant flaming ball of gas. Most of them are bigger than our sun. However, they do NOT all have their own 'universe'.
To put things in to scale: First comes Earth (containing all us people), then comes the Solar System (which the Earth and seven other planets orbit), then comes the Galaxy (which is made up of billions of other stars), and THEN comes the Universe, containing galaxies like the Milky Way.
Each star does not have it's own universe. When you talk about multiple universes, you are beginning to drift into the idea of parallel universes, alternate realities, and the fourth dimension, but I don't think that's what you're getting at.
What is true though, is that many of the stars you see in the night may have their very own Solar Systems, that is a number of planets orbiting it. But that is very different to a universe.
I am a year 12 physics student