The reason people are skeptical about visiting aliens is mainly because there is, in fact, no solid evidence that we've been visited. Why, after all these years, do we not have good photographic or video evidence of them, and why haven't we recovered artifacts that are unquestionably extraterrestrial in origin (made of unknown alloys, for instance)? What we have instead are grainy photos (many of them known forgeries), eyewitness accounts (which are notoriously unreliable), and the like.
The other reason for skepticism is that interstellar travel is much much more difficult than you probably realize. The distances between stars are enormous! That doesn't make it impossible, just difficult--and impractical, wouldn't you say, to send hundreds of valuable ships on what must be years-long journeys just to fly around and frighten people in the countryside?
Since interstellar travel is so difficult, it makes it improbable that we're being visited by aliens, especially on a continuous basis. Not impossible, but unlikely--now, couple that with a lack of evidence that they've been here, and the most reasonable conclusion is that we haven't been.
Don't get me wrong. I'm sure that life exists out there. Planets are apparently pretty common, and it's starting to look as though life takes root pretty easily when conditions are reasonably right for it. But, I expect that *intelligent* life is vanishingly rare. My guess is there might be 100 or so planets in the galaxy with it--which sounds like a lot, until you consider there are over 200 Billion stars in the Milky Way. I'd be surprised if there was an intelligent race closer than 100 light-years away, which is 20 times farther than the closest star. I would be super excited to learn we've made first contact with an alien race, but I just don't have any reason to think we have--and "I want it" is no reason to convince myself it's true.