BA -----"Power Surge" was to blame???

OK a power surge 'could' take out one data centre, but as they have class 3 independent power feeds - never been known. They have huge battery backup which keeps all running while own generators run up.

However how would it take out other data centres as the newtork would be dual redundant in Geographically remote locations.

Be interesting to see what comes out of this.

I expect the sun suddenly came out from behind a cloud, thousands of solar panels suddenly started pumping out squillions of watts and the grid didn't react quickly enough. Expect lots more in the future.

;-)

PS: what exactly is a 'power surge' on the grid, and how's it caused?

crap design by incompetent yes men in middle management

It appears that for some reason it took out everything including their phone system and that it took almost forever to get it going again. Way longer than you would expect for any PLC recovery plan. Something went badly wrong - last remaining sysyop wizard away for the Bank Holiday?

Possibly, or more likely I think that they made redundant all the people who actually knew how things worked and how to maintain it.

The system might well have been OK if it had been maintained and rebooted by people who actually knew what they were doing.

Indeed but since when has senior "management" ever known anything about IT?.

Its been suggested over at uk.railway that:

"BA appear to have both their main data centres on the same industrial site near Heathrow. Presumably both on the same mains distribution feed!"

That would make sense to an airline, wouldn't it? After all, you wouldn't have the co-pilot flying in a different plane to the pilot, would you?

Sigh. Its seems that 'power surge' = We powercyled the kit, like what you do with Windows and Netgear innit? and it didn't all come up and play nice.

I suffered a "power surge" at home 2 weeks ago. One of my computers started emitting magic smoke.

On 30-May-17 8:54 AM, Martin Brown wrote: ...

From what was being said last night, it might have been OK if somebody had thought to check whether the backup system actually worked when needed.

They apparently know enough about it to decide it will be better out sourced to India. Hope the penalties etc they have to pay after this fiasco shows them a headline cost may not always be the true one.

Wouldn't be the first time (or the last) that the "regulary tested" backup generator fails to start (or the auto start system doesn't) or if it does start falls over within minutes.

Reasons: It takes a brave man to truely simulate a mains failure by opening the breaker on the incoming supply(s) without warning or any preparation of down stream systems(*). It's "safer" to leave the supply(s) connected, press "start" on the generator and when warmed and up to speed, manually operate the changeover, run on generator for a cuple of mins manually drop back to the supplies (that have never been isolated) and shutdown generator. How many things

(*) One assumes BA would have doubley rendundant, hot spare, core systems, so "just in case" you make sure that the breaker(s) you are about to pull are only feeding the "non-live" core system.

What ever it was I've seen one report that there had to be some "hardware changes", followed by databases being out of sync with each other. A few hours ought to sort the hardware, what really screws you up is bad/missing/mismatched data.

Some years ago, in the early 80s, I was in Pakistan on Business. I was doing some equipment trials. I decided to test the equipment in the hotel room. I was in a major city. I had a simple linear power supply which 'dropped' the mains voltage to that required for the equipment. The equipment itself would normally work over a wide range of input voltages but it wouldn't power up correctly.

Out came the multimeter and at first it looked like the power supply had failed. However, it turned out the mains voltage was way below the supposed 240V, as I recall something under 200V. I spoke to our local rep and it seems this was far from unusual- as were power cuts etc. This was in a major city. Sure enough, that night when I needed to answer a 'call of nature', there was no power. The next morning, the mains measured 265V.

Across the road, they were building another hotel. The scaffolding was all made of bamboo. The hotel was already up to about 10 floors. The workman were climbing it in bare feet, no hard hats or any safety kit to be seen.

So why haven't you got a UPS? It's not as if a small APC costs very much. Something like this:

Good question, and not one that seems to have been answered so far.

The ones I have seen were direct lightning strikes to the metal roof or nearby electric pylons. In both cases damage was done to kit that was notionally protected by heavy duty industrial grade surge arresters.

The worst one vaporised the telephone lines in reception leaving a 4" wide charred mess down the wall and a dazed unconsolable receptionist. Took days to get the line mended and the phone switchboard was toast. That was after a strike to the metal roof of the building.

It was notable how getting one mainframe back online was made more difficult by multiple systems failures in different zones. Normal CPU failures usually are single point and neighbouring bits whereas a big lightning strike seems to find perverse and unusual destructive paths. Sacrificial protection bits somehow manages to save themselves at the expense of parts that are much harder to access and replace.

However, BA don't seem to be claiming that it was from a lightning strike but a "Power surge". Presumably they can back up their claim of power surge by peak voltage/current spike detection and logging...

No lighting conductors?

Must admit I'd thought the actual hardware would still be in the UK? But under the control of their newly out sourced experts? Who might just have a problem popping down the corridor to press the reset button in event of a failure?

That's why I have three UPS units.

The power surge quote was courtesy of Alex Cruz, BA CEO.

The same Alex Cruz who, after a total breakdown of BA's IT systems caused a suspension of all the airlines services worldwide for around 24 hours, was quoted as saying

"We would never compromise the integrity and security of our IT systems"

Which does make you wonder how much worse things could have been, if that had been his intention all along

michael adams

...