Small fire at Delta data centre in Atlanta follows electrical failure

What happened

In August 2016, a piece of electrical equipment at Delta Air Lines Delta data centre in Atlanta failed and shut down the transformer supplying its data centre. The system moved to backup power, but not all servers were connected, and Delta Air Lines reported a cascading computer outage. The equipment failure also started a small fire, which Delta Air Lines said was put out immediately without calling the fire service. Delta Air Lines reported more than 2,100 flight cancellations that week while systems were disrupted; it did not say how far the fire itself contributed.

Contributing factors

An electrical component at the Delta data centre in Atlanta failed and took down the transformer feeding the data centre. Not all servers were connected to the backup supply. Delta Air Lines did not describe the failed part or how the fire was extinguished.

Lessons learned

1. Check that every critical server is actually fed from the backup supply, not only that a backup path exists.
2. Treat the supply into a data centre as a single point of failure and test what happens if the upstream transformer trips.
3. Record what failed, how any fire was put out, and what else was damaged, so later analysis is not left to press statements.

Source

mprnews.org