Yeah...the "helpful" homeowners are the very worst. If you are firm with them, they happily ignore you. This guy was perfectly aware that he was causing you an inconvenience. Frankly, the bullshit games were all part of his power trip.
Him - the wealthy, powerful homeowner YOU - the lowly service guy putz
So how to deal with him?
I have found that it helps to tell them how long the job SHOULD take. And politely let them know that if they can give me space to work, I should be able to accomplish the job in the specified amount of time. BUT, that if they are curious about what you're doing, you are more than happy to let them know what you're doing and why you're doing it. But that will take more time, and I'll have to charge you for it.
Because it really is all about money chief. If it takes you 7 hours to do a 2-hour job, I don't see where you should care one way or the next. As long as you're getting paid. So don't look at those customers as "inconvenience". Instead, look at them as the guys who are going to pay you 3X what the job is worth.
Also...STRONG SUGGESTION. Get a copy of a book titled How To Win Friends & Influence People by Dale Carnegie. It's a great book. It sites many many examples of difficult situations. Read this book and it will completely change the way you deal with people.
What you are selling on the job is your TIME and EXPERTISE. If they guy is costing you time, you're not making any money anyways. You won't earn dick when a 2-hour job drags into 4...5.......6 hours. And that shoudl be your only concern. So charge for you time and be the friendliest damn plumber your customers have ever met.