All right, I was being over-critical. Yes, I know what "eyesore" means. It means something that does no actual harm but that one prefers not to look at. Part of the compromises we make in living around other people is realizing not all of them are exactly the same, and some are *bound* to do things that annoy others. Local codes generally prohibit conditions that are noxious or dangerous. I very much doubt that storing a canoe outside is one of them.
You asked if you were wrong in considering them "cranky". I think you were. If they'd wanted you to plan their landscaping, they would have asked. If you'd said when you planted it, "I want to plant this along the fence, and it may intrude on your property. Are you OK with this if I keep it trimmed?" I feel confident they would have been perfectly satisfied, even pleased. If they were "cranky," they would have yanked up the plants by the roots.
That is, their "eyesore" caused you nothing but aesthetic pain, while your plants have become a very real nuisance to them.