Power loss at Verizon data centre disrupts JetBlue systems and flights

What happened

On Thursday 14 January 2016, at 11.37 a.m. Eastern Time, power was lost at a Verizon data centre that hosted JetBlue Airways systems. The outage took down check-in, gate systems, the JetBlue website, the mobile app and the toll-free customer service number, and around 200 flights were delayed. JetBlue said power was disrupted during a maintenance operation at the Verizon site. Power was restored to the data centre at 1.50 p.m.

Eastern Time, and JetBlue's systems were fully restored by 8 p.m. that day. Flights were operating normally by 6 a.m. on Friday 15 January. JetBlue said the event was not a cyber security issue. The location of the data centre was not given in the joint statement issued by the two companies.

Contributing factors

JetBlue said power was disrupted during a maintenance operation at the Verizon site. The joint statement from the two companies gave few details. No further cause was reported.

Lessons learned

1. Maintenance work on power feeds to a shared data centre should be planned with a verified fallback supply and a tested rollback procedure before work begins.
2. Operators that rely on a third-party facility should know where their systems are hosted and identify single points of failure in that hosting.
3. Customer-facing channels such as check-in and mobile apps need a degraded mode or alternative access that still works when the primary hosting site loses power.
4. Status updates to customers and staff should be issued promptly and kept consistent across channels.

More lessons on worker safety and safe systems of work, outages, backup supply and restoration, from every incident on the register.

Source

rcrwireless.com