It's probably not so much that the stove/chimney is letting cold air in as that it's letting warm air out. It will function as a chimney whether there's a fire in the stove or not. I found this out when I was up on my roof one summer day cleaning my chimney and found a tremendous draft of air coming up through it and the woodstove, strong enough to ruffle my hair, and get soot in my eyes.
Chimneys pretty much only get downdrafts in unusual circumstances and in strong wind unless the whole thing was very badly situated relative to the roof line. Maybe yours was.
All you have to do is block the air from getting up the chimney and out, either through the woodstove or through the chimney. Assuming this chimney is only there for the woodstove and doesn't have another appliance exhausting through it like a gas furnace, something as simple as a piece of plastic taped and/or wired over the top of the chimney will do it. You can try closing all the draft controls on the stove, or pull the whole thing out since it was illegally installed anyway, and close the chimney damper or stuff something in the chimney so air can't get up it.
Dampers have been known to warp and not close properly, so check for gaps with a strong light. I believe they even make something inflatable you can stick in a chimney to block the draft. It might take a little time and thought to fix this but it need not cost much at all.