It can be harder to detect in some cases than others. And you know now it is something on the wood. All you need now if proper prep and you are on your way.
We run this very problem from time to time when we repainting door frames (and sometimes the doors themselves) when home owners give the hinges a little spritz of some lubricant. It gets on the doors and on the frame, and even when we prep it and prime it, after it sits for a while until we get to painting that stuff sneaks back out.
Several years ago the combined brain trust on the job noticed this type of paint problem and we were able to figure out what was going on because we could see the pattern of bad adhesion exactly lining up with the hinges. A quick word with the client confirmed this as the problem.
I clean with lacquer thinner as it is extremely/immediately effective on most lubricants and it flashes off really quick so you can continue working in minutes. But it is dangerous, and the fumes will knock down a horse. Imagine your significant other's nail polish fumes times 100 and you have it.
Here's something to consider, and I am just putting this out there since it works for me. Our local paint rep for my favorite paint spends a lot of time defending his product, and the problems are almost always improper application. I had the same adhesion problem on a door, under a knob and it was a streak that went from knob to slab (exterior door). He said that the solvent based primers often picked up enough oil or silicone to contaminate the primer, making hard for the primer to seal completely but also for paint to stick to the contaminated primer. (I immediately saw in my mind's eye one of the guys brushing the primer on, effectively spreading around the contaminates with his brush). Our primer was hard, but the paint still pulled away in small streaks while it was drying.
So... since we had nothing to lose, we sanded off the paint/primer back to good areas. I cleaned it with lacquer thinner, and then we put on WATER based Kilz 2 as he suggested. Yup, water base. I was really reluctant as I had never used water based stain killer/primer on anything. And thankfully it worked fine. I guess the water didn't have the solvents to activate dissolve whatever it was on the door. Finshed out the door as usual, and something new learned.
This is our modified method when we have a few hundred feet of moldings to prime/install/paint: put it all out on the saw horses and shoot it all with water based Kilz with the airless. If you use the airless, you will not spread around contaminates (barbecue grease, taco grease drippings, pizza grease, fried chicken grease, from the delivery/yard guys, or oil from a careless spraying, etc.) that seem to wind up on your moldings and material from time to time. I don't like the water based stuff for actual stain blocking as much as the old solvent based, but it makes a good primer.
Do what Mike said; sand all of it off the affected areas. Clean with some kind of solvent, then prime (think about the water based stuff if you sand back and it still doesn't work) and paint.
Since you have 3 spots to experiment on and rework, just do one, and if your method works then take the time needed to do all three areas. Let us know how you are doing.
Robert