"therefore" does not follow for the general condition as you interpret it. Want proof? Leave the heat off for 3 days. The inside temperature will be a lot closer to the (time average) outside temperature.
Leave the heat off forever and your inside temperature will start to mimic the average daily temperature, or perhaps more akin to the weekly average temperature, depending on how well insulated the house is and how much sunlight can affect inside heating.
I live in an area of very cold temperatures in the winter, so my house is fairly well insulated (could be better, especially if I replaced the windows). My house won't drop to 60F overnight when the temperature is only at freezing outside. I have lost power for a full day when it was well below freezing and the inside of the house was still only in the mid-50s F after that full day despite the deep cold outside. The basement actually helps a bit for buffering the temperature (the ground stays warmer than the air, so the basement, although usually cool, actually acts as a source of warmth in that situation; it is warmer in the basement than outside and tends to stay that way for a very long time (in the winter; it is the opposite in summer, of course).
My point, though, is that you need to plot a time versus temperature graph to get an idea of how well your house resists change to outside temperature. One point is not enough.