Remove as much of the pea gravel as possible. Stones and gravel in garden soil are a waste of valuable space, assuming that they are not part of a well designed drainage system which is well below the cultivation zone of the garden. Ideally, the inorganic material in your garden should all be in the 0.1 to 2.0 mm range. Pea gravel (a loosely defined term!) is in the range of 4 to 18 mm (1/8" to 3/4"). Every piece that you remove improves your garden quality.
I need many tons of pea gravel for some projects and I am certain that there is somebody in your area who is like me: Willing to come out with a truck, some helpers, wheel barrels, screens and shovels. I'd gladly screen and haul away that much free pea gravel. Every summer. Forever.
You should find somebody who wants the free gravel. Then stand back and watch the other guy and his crew remove the problem gravel for free. If you are nice, offer them some of the lemonade that you are sipping.
After you gets rid of that 3" layer of pea gravel, you should import 3" or more of good organic matter and possibly some gypsum to augment the existing soil. I know that you said that you have "good soil", but it is extremely rare to find any soil that isn't worthy of improvement. Is there such as thing as soil which is "too good"?
Personally, I believe that one will almost always reap the benefit of any soil improvement in the very first year of gardening. If it were my garden plot, I'd add about 38 cf of Sphagnum, 100 cf (or more) of compost, and some gypsum if the soil has many fines (clay).
If money is an issue, then just add compost. Municipal compost is very adequate. Yes, I'll hear opposing views on that statement. :)
If money is a very big issue, then add a smaller quantity of compost and start producing your own today.
Of course, I'm a fanatic at times. I'd pull some random 1 foot deep plugs of soil and perform my own Emerson Test on each strata of existing soil to determine exactly what the soil composition is at each strata. And pH testing is equally important and easy to perform. Why guess?
Remember: Gardening is fun with good soil. Gardening is work with average soil. Gardening is a pain in the ass with poor soil.
Good luck, Gideon