Has anyone ever thought of this ..............
In order for the man to build the pool and fill it up, the city/county had to sign off that the fence was of legal construction, that it was located properly, and complied with code. And since you weren't around, you have no real obligation or liability regarding the previous homeowner, structures built OFF of your property, or any agreements the previous owner entered into, either implied or expressed.
Since the pool was built and filled with the supervision of the local building department, the city/county has stated by their actions that the fence met ALL laws. If the fence is on the property line, or one inch on your property, it was an illegal structure that the city/county authorized as safe and legal.
Since the future fence will be located on his property, I would either tell him to pay for the thing himself, or offer to pay some small amount to keep the peace. I would insist that the fence be located at least one inch inside his property line so he can't come back at you later. Since he "notified" you that you would "have" to pay half, he may be incorrectly advised, not know the facts, or just be a garden variety putz.
If the fence IS on your property, even half of it, I would thank him for removing the fence, let the city/county know that THIS time you will make certain that the fence is legally located, that you DO NOT want it on your property, and encourage them to do the same. I would still offer to pay a SMALL amount just to keep the peace.
If your neighbor continues to be a putz, just make sure the fence is on his property by notifying the inspector and city/county of your concerns that the fence is PROPERLY AND LEGALLY situated, and let him pay for it all. It is ALL of his responsibility to fence in an "attractive nuisance" and NONE of yours. Is he going to want you to help pay for some pool furniture next?
This one requires the wisdom of Solomon. You have to live next to the guy, but if he is unreasonable, you gotta stand firm. Otherwise, he will be coming up with something new every time you turn around.
Steve