I doubt I would personally own slaves. It sounds like a royal pain in the ass. Even free butlers, maids, gardeners and so forth hanging around would drive me crazy in short order.
otoh I don't think I would much care if other people had slaves. I think that was the case for most people in the southern states. The interesting question is whether I would put on the uniform. If I lived in the south, I think I would but not necessarily to preserve slavery. In the north, I think not. I have one ancestor who did fight in the Union army although I have no idea what his motivation was.