But thermal inertia works both ways. If it is large, then, as you rightly point out, it may take the building a couple of hours from when the heating is switched on to get up to comfortable temperature (and if your life is well regimented, this can be dealt with by using timers to switch on the heating a few hours before you get home). But it also means you can switch the heating off well before you leave, since it will also take the building some time to cool down before it becomes uncomfortable.
Mind you, and this is probably what you were hinting at, if the time lag is large, and "a couple of hours" turns out to be much longer than you're going to be away for, then it gets a little difficult.