There are approximately 250 billion stars in our Milky Way galaxy, alone. Even if we use the upper limit of your "millions of earth like planets in our milky way galaxy" (999,999,999), that still means that only four-tenths of 1 percent (999,999,999 ÷ 250.000,000,000 = 0.004) of the stars in the Milky Way galaxy have an "earth like planet" orbiting them. Even if every one of those planets (except Earth) had developed interstellar travel, that still leaves 99.6% (249 billion) of the stars in the Milky Way galaxy to explore. The chances of one of them having found Earth is pretty small, don't you think?
So, no, I don't think we can conclude that interstellar travel is impossible just because no extra-terrestrials have visited Earth.