The fact is there are laws about this and if you violate the law, that neighbor can bring an action against you.
If you think that is not likely because you have nice neighbors, consider this. If the OP's neighbors were nice, why would he be posting the question, when a simple word to his nice thoughtful neighbor would be enough for that neighbor to take care of the problem.
If the OP were to violate the law, which varies depending on where you live, he could find himself in a far worse condition.
These kind of property rights issues are certainly not un-common.