Yes, but you have an obligation to inspect (look at) the trees so you'll know if one is dead or dying. The level of obligation varies. It's one thing if it's 100 feet away from the house where you live, and another if it's 20 miles away and no one lives on your property, but iirc, you still have to check maybe once a year. I dont' think there is much difference between 20 miles and 1000 miles. You have to hire someone to inspect it, unless I guess the land borders nothing that a falling tree can damage. Even then, the odds are slim, but it could fall on a person.
AFAIK our president is misinformed about who owns the land behind these houses and afaik whoever does hasn't inspected anything in the 38 years I've been here. Or done any maintenance. I like that, except when it comes to this issue, then I'm not so sure.
(Once, the county sent a guy with a little tank, with treads like a tank, maybe 4' wide, that drove from the parking lot down into the stream bed, and over to the manhole that is 6 or 7 feet above ground level, and the guy used something from the tank to inspect the sewer line that runs along the stream. He said they do this every 3 years but I've never seen him before or seen tread-prints in the ground or the bushes broken off, since the path down the hill to the stream bed is only 2 feet wide and the tank is 4 feet wide. But the county is NOT the owner of the land, only of the sewer line.)
It sounds like the other owner is self-centered or, right or wrong, annoyed at you, only because while the guy was removing the part on his yard, he could have done both parts at a net savings. Well, probably just didn't think or it or didn't want to get involved trying to figure out how to split the cost with you, although the contractor ought to be able to apportion the cost. .