BA -----"Power Surge" was to blame???

You don't need a reset button, proper servers can by powered up or down over a separate, independent network connection. This is independent of the rest of the server, so it not affected by the system being locked up by a crash. They can even have their operating systems installed from scratch remotely. A person only needs to be there if a physical item such as a hard-disk or fan needs replacing.

The network itself should be designed so that it automatically restarts after power loss or consists of devices that can be remotely restarted in sequence 'til the full network is up and running.

SteveW

Some, well designed, linear ones did as well. I remember using a

SteveW

Which shouldn't have mattered. The guy from Ryanair was on the radio yesterday saying that they have a primary and two backup datacentres, with automatic failover. All at completely different locations.

Surely BA should have similar? And without the primary coming back up properly, one of the backup datacentres should have kept everything running as normal.

SteveW

Yes If you do have deep and very deep pockets! A backup diseasel is cheaper im most all cases.

Well had this debate with a friend who works in data centres and he reckons that not many have that luxury - one went down near that London Telehouse a while ago only had the one mains feed as there wasn't another available for quite some distance.

Decent UPS batteries and a well tested and fuelled diesel and the hope of course that BT's fibre plant is on some reserve power too;!

What to hear that one where so BT term box wasn't on a backed up supply to a data centre ,just got overlooked..

Yes mainly to do with lighting for TV and yes if you do have two separate mains supplies nice, and the diesel genny not a lot of money for a Footie club a weeks wages for a player perhaps;!.

Where does this "independant network" connection derive it's power? Both in the server and the required ancillary equipment?

Or PSU fuses, fried PSU(s).

If they still have functional PSU(s)...

At a minimum it consists of a POTS line and a modem. The modem you connect to a terminal server which has serial lines going to console ports on each of your servers, routers.

We also had ISDN lines to the terminal servers.

Any important device should have at least two power supplies.

Minimum of dual PSUs, connected to the independent supplies. Everything capable of running on just one of its PSUs. PSUs, fans, disks, etc. continuously monitored to report any failure or loss of supply.

In the case of ethernet networks (modern servers can use ethernet rather than serial for remote reboot, etc.), two parallel networks can be used as well.

If say both PSUs in a server failed, then another server can take its place.

Final resort, if the entire datacentre was (at least temporarily) unavailable, backup datacentre takes over automatically.

That's what you would expect as a minimum for the critical systems of a company like BA.

SteveW

Wrong sort of UPS. Get a Smart-UPS or equivalent.

My PPoE had two separate HV feeds. They entered the site from opposite directions and different 'HV substations', whatever they are called.

Well at least you didn't say 'what steel works?'.

Sounds more like it was a problem of their own making by making all the experienced old hands redundant in that there was noone left on site who understood how the legacy code worked and how to bring it back up again seamlessly. Once they got it out of sync all bets are off.

How do you know it was a "power surge"? Did your filament lights blow too?

PC PSU's do fail from time to time and with fan assisted combustion of the burning varnish can be quite magic smokey and sparks before the fuse goes. Exploding dodgy capacitors don't smell too good either.

'twas daylight.

Ah, now I thought the phrase "power surge" was what the engineers use to fob off management when they don't understand what happened.

Actually, interpolating a lot of guesses into the statements now being made, my guess is that what happened to them is a combination of what happened to me some years ago.

They lost power. The UPS kicked in and they shut down gracefully.

Power was restored and they attempted to reboot *everything* at once.

The switch on surge tripped something, and they lost the lot again, but this time they were in the middle of boot sequences and there was no graceful shutdown and many disks got corrupted, leaving them with a machine room full of kit that needed to be restored from backup or rebuilt.

Maybe they thought air traffic control was the critical part and so didn't worry too much about passenger convenience? No-one seems to have been injured as a result of the surge.

That's more like it, and a convenient line to feed the press when they ask questions.

And of course management quickly learn how to smell engineering bullshit a mile off.

It would be a real worry if the planes were flown by some cut price Indian IT centre.

They don't run air traffic control.

Indeed. However, I mentioned the example to point out the sorry state of the 'infrastructure' at the time. I fear that it may well not be much better in India now from some stories I've heard. In the UK, in fact in Europe generally, we take a 'stable' electric supply for granted but in some of the 'less developed' countries it simply isn't quite as stable as we enjoy- even in the cities. Even in some remote areas of the US, it is quite common for people to have standby generators 'just in case'.