What would cause a disconnect box to burn up like this?

Your instinct about a loose connection is probably right. I've pulled apart a few disconnects that looked exactly like that and the story is almost always the same. A lug or splice works loose or corrodes over the years, then you get arcing at that spot. Here's the part that confuses most people: the breaker doesn't care about heat. It only trips when current exceeds its rating for long enough. A failing connection draws the same current it always did, it just burns it off as heat right there at the bad spot. So the box cooks itself for months or years while the rest of the house never notices a thing. Breakers protect the wire, not the connections.

That old taped-up cable is a red flag too. Somebody spliced that at some point, and a splice in a disconnect area done with tape is exactly the kind of thing that turns into a slow burn. Add 25 years of AC season vibrations and thermal cycling and it doesn't take much. As for reuse, plan on the disconnect being a total loss. They're cheap enough that any electrician will just hang a new one rather than gamble on a box full of carbonized guts. The whip and wire are a case by case call. If the insulation is brittle, discolored, or smells burnt when they pull it back, they'll run new. If it tests out clean they can often keep it, but honestly on a unit that age most guys will just re-pull fresh and sleep better. Budget for the whole thing being new and be happy if it isn't. One more thing worth doing: have them check the lugs where the feeder lands in the main panel. Same age, same history, worth a look while they're there.

Not sure I buy the taped splice being the culprit though. Re-read the OP, the taped-up cable is coming in on one side but the heat damage is on the whip side, left side. If that old splice were the burn point I'd expect charring concentrated around it, not over by the flexible conduit. More likely the trouble started right at the disconnect lugs themselves or in the whip connection. The loose connection idea still holds, just maybe not at the spot people keep pointing at. When the electrician opens it up it'll be obvious which end cooked, the worst scarring marks the origin every time.

One detail that trips people up with these is that shutting the unit down for the season doesn't actually de-energize the box. The line side lugs are hot straight from the panel wheather the condenser is running or not, so a bad connection in there can cook away with zero load on the circuit, which lines up with what you described. I pulled apart a neighbor's disconnect a couple summers back that had a wasp nest built in behind the cover, and the moisture and debris in there had corroded one lug badly enough to pit the metal. Worth having whoever looks at yours check for water staining or bug intrusion, since a weep-hole situation plus 25 years outdoors does the same slow damage a loose lug does.

First time posting here so hope this is ok. We had a melted mess at our garage sub panel last year and the thing never tripped either. The electrician who fixed it did a sweep with one of those infrared cameras on the whole panel while he was there, so maybe worth asking yours if they can do the same on your main panel while they're out, not just look at the AC stuff separately.

Also curious if anyone knows, does soot spreading onto a joist like that make it worth calling your insurance company, or is that overkill? I never bothered with mine and now that I read how these things burn slowly for years I sort of wonder if I shoud have.

When mine went a few years back the culprit turned out to be water. The whip had a low spot that filled up every time it rained and the lugs just rotted away over time. Electrician pulled the cover and the bottom half of the box was rusted and green inside. Might be worth checking if yours sits under a gutter overflow or the AC line set drips anywhere near it, since that changes how you hang the new one.

Also took a bunch of photos before the electrician showed up, turned out to be useful for the insurance claim since there was some smoke staining on the wall too.