Agreed. We only lock ours with the key when we're going away from home. There's a lockable porch door outside the door with the Yale lock, anyway - which is locked both at night when we're at home (with the key in the lock) and all the time when we're away (with the key removed). The Yale lock is never locked with the key when we're inside the house. When we go away, we lock it with the door open and then slam the door shut as we leave.
That suits us, but you obviously prefer a different setup.