It's so annoying! Watching someone play a game is like watching a really crappy improvised version of a TV show. I'm sort of a control freak, which may contribute to the fact that I'm so annoyed by it, but I don't like seeing someone play and having no control over it. This is why I quit watching Toby Turner's episodes of Terraria. He started messing up so much, and I couldn't stand it. The positive factor of a game is that you're able to control your character, and often times the environment around him/her. When someone else is playing, I may as well watch fish for a few hours.
Whenever my friends come over, or my brother plays, it's always co-op-compatible. At least then both of us get to enjoy playing. Plus, I usually end up completely pwning them in comparison, which further contributes to why it's so annoying watching people play.Don't worry, you're not alone.