Baby/toddler stage was definitely the hardest. No sleep, no going to the bathroom by myself, constant supervision of everything, writing on walls, peeling off wallpaper, vomiting on upholstery, scream scream scream and tantrum... it was totally draining.
Now, they are so much fun and we have good times talking, doing schoolwork, listening to music, trying to recreate recipes from food network and stuff. Also, they take care of themselves and go to bed by 10 or 11 without drama. Of course the driver's ed makes me feel so nervous, but that is probably normal.
So anyway, when they are older you can do a lot of fun things with your kids and they can help you a lot.
So, I say teen years are the easiest, baby years the hardest.