Well, since you didn't mention that in your first post, why didn't you simply tell the caller "Sorry, I'm not able to come by the church the right now. Please try calling (insert names of the other key holders) to see if one of them is available."
If you are trying to not foster dependence on you and only you, you should be letting people know that they have other options.
Besides, how can you call my comments useless since I suggested that you set up a rotation amongst the key holders? If there's at least 6 people who have a key, then no one would be on call more than once every 6 weeks. My comment to set up a rotation still stands, but it's up to you to approach the others (or the board) to put the process in place.