Do you have to go to Lavender Town in Pokemon or can you just go past it without later game problems?

If I just walk past Lavender Town, and don't go into the tower (I fought Gary first), or do I have to do something in Lavender Town that will affect the game later? It's freaking me out.
2 answers 2