Can I stuff this space with rock wool insulation ?

Don't just cram batts in there loose, it won't do much. The problem isn't the insulation, it's the air movement. That daylight you're seeing near the sill plate is the whole issue. Wind washing straight through a loose wad of rock wool strips most of its insulating value, so you'd be spending an afternoon shoving fluffy stuff into what is basially a hallway for cold air.

What worked for me on a similar gap behind an old duct chase was to close off the back side first. I cut a piece of rigid foam board a little bigger than the opening, pushed it against the wall at the back or sides (wherever it borders that exterior cavity), and shot some low-expansion foam around all the edges to lock it and seal it in. You can do the same thing with a scrap of OSB or even stuffed and foamed if access is wierd. The goal is just to kill the airflow from the outside. Once it's air sealed, then yes, cut a rock wool piece to fit snug and press it in deliberately. Snug means friction fit, no gaps around the edges, but not squashed dense. The compression warning you heard is about real crushing, like forcing a 3-1/2 inch batt into a 2 inch slot. Rock wool still works slightly compressed, it just loses some R-value per inch, so don't worry over a small pinch but don't ball it up either.

Spray foam at the edges plus rock wool in the middle isn't overkill, it's actually the textbook combo. Foam handles air sealing, rock wool handles the bulk insulation. Foam alone in a 4 foot by 6 inch gap would waste a fortune in cans anyway. One thing to check while you're in there, keep an eye out for what the rest of the house is doing at the same junction. If the whole rim joist line is that leaky, this gap is just the spot where you noticed it. A few more hours with a foam can on the rim might be time well spent.

Did basically the same thing behind my old chimney chase last year. One thing I'd add is check what's in that cavity before you foam anything. Mine had an old knob-and-tube leftover in there, and I only spotted it when I went to shove the foam board back. Killed the plan pretty fast till I got it sorted. Also, do the sealing on a cold or windy day if you can. You can acutally feel where the drafts are coming from with the back of your hand, way easier than guessing. Then you know the foam board's actually sitting where the leak is.

Following this one because I've got a similar gap but mine is around a plumbing vent pipe where it goes through the rim, so it's more of an L shape than a rectangle. I stuffed some fiberglass in there last fall and the draft just pushed right through it, so the seal the back first advice makes sense to me now. Question for anyone who's used a rigid backer though: how thick does the board actually need to be? I've only got thin quarter inch scrap sheets and I can't tell if that's enough to block the cavity or if I need the thicker stuff. Also mine kept sliding down into the hole when I test fitted it, so curious how you hold it in place while the foam sets.

Agree with the seal-first approach, made a huge difference here too. I had a gap like this over my slab ledge where cold air was rolling thru a batt I'd shoved in years ago. Did the rigid backer and edge foam deal, then went back a week later during a cold snap and the batt was bone dry and warm to the touch instead of icy like before. Also cut my foam board about an inch oversized all around so it wedged in on its own while the foam cured, saved me trying to hold it in place with one hand while foaming with the other.

One detail I'd flag before you start is that the concrete at the top of a basement wall is rarely flat or plumb, so your rigid backer probaly won't sit flush against it without some coaxing. I ran a bead of backer rod into the low spots first, then foamed over that, and it sealed better than trying to fill a half inch void with foam alone. Also worth checking whether the daylight is coming between the sill plate and the foundation or through the cavity itself, since a shimmed and caulked sill gap is a slightly diffrent job than a hole in the wall.

Update: went up there with a flashlight before doing anything, since the chimney chase guy mentioned checking what's in the cavity first. Good call, beacuse there's an old cloth-wrapped wire running right through that gap. No foam anywhere near that thing then. Question now: can I still do the rock wool around it if I leave a couple inches clear, or does old wiring like that need to stay completly exposed?