Dealt with the exact same thing on my own gravel-ish RAP drive. The key thing I learned is that a pothole that keeps coming back after regrading is almost always a drainage or base problem, not a surface problem. Water is getting under the RAP at that spot, softening things up, and every time you grade it you're just putting a fresh bandaid over wet ground. So before you touch it again, wait for a good rain and go look at those spots. See where the water is sitting or running from. Nine times out of ten it's a low spot that's lost its crown or a spot where water is funneling across from a ditch or the yard.
Once you know where the water's coming from, fix that first. Sometimes it's as simple as pulling a shallow swale with the box blade a few feet to the side so water has somewhere to go besides straight through your bad spot. That alone killed half my problem.
For the holes themselves, regrading won't cut it. Dig the soft material out til you hit firm ground, which on mine was ussually only a few inches but it varies. Toss in some larger rock or crush and run, compact it as best you can, then put RAP on top and roll or drive-pack it. A cheap plate compactor makes a huge difference here, or you can just park your truck on it for a week like I did before I bought one. If you skip the digging step and just dump RAP in the hole, it'll pump right back out by fall, guaranteed. Also worth knowing, RAP settles and locks up best after it's been driven on wet a few times. Don't judge your repair until it's been through a couple rain cycles.