How many times...

I believe race hazards in hardware and software have been understood for many decades. Was this programmed in COBOL?

Well yes.

I once wrote a BIOS for an 8086 board with lots of interrupts and stuff. Then I handed to the boss and he came back a day later. It crashes sometimes' 'When? ' 'I dunno' just sometimes.

Back on the 8086 emulators and yes, after an hour or two it crashed.

It wasn't actually DOING anything though. Just running a loop waiting for keyboard interrupts.

But then I realised it WAS doing something, It was keeping tine via a timer interrupt every millisecond.

I patched the code to do it every microsecond and it crashed pretty much instantly.

I looked at where and realised there were a couple of adjacent assembler instructions that would go bad if any interrupt happened between them. I surrounded them with a Disable interrupt/Enable interrupt sequence and that was that. Never crashed again.

I can absolutely understand a bug like that going for years unnoticed

Its very hard to test against and you need to be a bit genius level to think of it as a possibility, and geniuses don't work for large software companies, they own smaller ones.

What is more pertinent is why there wasn't some sort of failover in place.

Duplicated data and another system on hot standby...

Well that's one way of putting it. It depends what operating system it was running on. My guess is its some real time system where everything is an interrupt to keep latency down, an d this one an instance of a particularly rare nested interrupt with unforeseen shared data between them, so no 2 interrupt f***ed up what no 1 was supposed to be doing. Designing that sort of system is no walk in the park.

Testing it is even worse.

BUT now they know what happened, its simple to look at the code, fix it and test that its fixed

I think so. Or rather to make provision for that data to be saved and restored during the interrupt.

That bit might well have been assembler, but was more likely C or C++

Understanding things does not stop them happening.