Saturday's Airport and flight chaos.

[Michael Chare]

We are being told that the current airport/flight chaos is caused by a problem with the air traffic controllers phone system.

Is this anything other than a failure to have proper standby arrangements?

[Nightjar]

AIUI, the problem is that they are having to work with what is, effectively, the standby system and that doesn't give them enough positions to handle all the traffic of the full system.

Colin Bignell

[The Natural Philosopher]

AIUI its stuck in 'overnight mode' and not programmed to handle full weight

[Nightjar]

It is not programming, which they did have a problem with last year, but the number of controller positions that are available on the night system as compared to the day system.

Colin Bignell

[Bill Wright]

They want to go and have a look behind all the consoles. Some bloody cleaner will have unplugged something to plug her her vac in.

Bill

[Brian Gaff]

From my dropping of the eaves on Saturday, I understood the following. At night, they have fewer controllers on the system, and in the morning ready for the main rush of the overnight long haul etc, they ramp up the numbers and presumably the load on the system. They said that when people started to log in in the morning many could not do so, and the system remained in night mode. This to me begs some questions. Why should there be a night mode in the first place? IE if its just a case of fewer controllers, then why change the system? The second question is, why is there no complete back up of a known good system to switch to for safety, as from what I could hear on air yesterday, the stress of the controllers and other staff was showing in their voices, and one mistake if busier than they could cope with could have been very dangerous, given than to flights were already in the air when the problem surfaced.

It makes me wonder if there was more going on, maybe a software change that did not go well or a wrong module left on the system for some reason, or maybe hardware replacement was done and it was not fully populated in time. However it seems to last until early evening to my knowledge and a recovery plan surely should have got results well before that, assuming they had one. Brian

[Brian Gaff]

Yes, but why have a night system in the first place? Does not seem right to me see my other comment. Brian

[Brian Gaff]

Or its mould or a spider or rat infestation... Brian

[Nightjar]

The reason they can have fewer controllers on is that night mode the country is divided into larger control areas. Each controller needs to be able to speak to the controllers in the adjacent areas, but generally has no need to speak to those two areas away. Thus the structure of the communications system has to change completely between night and day.

To an extent, the night system is the backup. It works, but not as well as the full day system. Completely duplicating everything that might possibly go wrong would be prohibitively expensive.

Colin Bignell

[Andrew Gabriel]

Interview with head of NATS - when asked why it would take until 7pm to fix, she said something that sounded to me like that's how long it takes to rebuild the software.

So my guess is a software upgrade screwed up, and they had no rapid back-out or fall-back procedure. It could be that synchronised changes across many software systems meant that backing one out wasn't viable. That's not an excuse of course.

[Nightjar]

Yesterday they specifically denied it had anything to do with software.

Colin Bignell

[Michael Chare]

The head of NATS did say they could not immediately restore the service because of the time taken to reload (AIUI) the (phone system?) software.

Absolutely no excuse for not having adequate fallback procedures though.

The head said that they wanted to learn from their mistakes. Getting rid of a head who did not ensure that there were proper fallback procedures is one step that they should take.

[Clive Page]

That would be a start. One thing that used to surprise me was that the airlines didn't immediately sue them for huge amounts of damages, given that they have all had their schedules completely messed up and will have had to compensate vast numbers of delayed passengers serious amounts of money. My guess is that they don't do that because NATS is semi-privatised, and the principal private owner is a consortium of airlines. So they would effectively be suing themselves if they did that.

It doesn't explain why they don't use their influence, as major shareholders, to get NATS to have appropriate backup arrangements. These major outages seem to happen every year or so.

[Tim Watts]

On Monday 09 December 2013 10:35 Clive Page wrote in uk.d-i-y:

How come the GPO managed to develop and run nearly bomb proof telephone systems[1] for decades (not withstanding their dire end-customer service).

[1] I'm including the 2 I know of - the public network and the Government Telephone Network. British Rail of course ran a 3rd system.

And yet, this critical system fell over...

[charles]

perfectly simply. Underfunding.