There are multiple sub panels as the house has had wings added over time by previous owners.
The one problem with my "test" of hitting one of the breakers is that breaker could, I guess, have actually been going to a sub panel, in which case all circuits on that sub panel would have gone out.
I think someone said it doesn't really matter if it's on a sub panel anyway because the white wire is physically wired as a ground in the main panel and in the sub panels, not as a neutral.