Power outage brings down British Airways data centre near Heathrow

What happened

On 27 May 2017, a power outage apparently brought down a British Airways data centre near Heathrow that was operated by CBRE. British Airways had two data centres close to Heathrow, Boadicea House and Comet House. Backup systems evidently failed to cope, and the disruption lasted three days. British Airways cancelled 672 flights over a Bank Holiday weekend, at an estimated cost of £58 million. Early reports suggested that a UPS issue shut Boadicea House, but no published explanation has said how that happened or why Comet House also failed. Eighteen months later there was still no agreement on the cause.

Contributing factors

The cause has not been agreed. Early reports suggested a power-supply or UPS issue at one site, and that backup systems did not keep operations going. Those reports have not been confirmed, and it is not known why the second site also failed.

Lessons learned

1. Operators of paired data centres should test whether a supply or UPS fault at one site can take that site offline and whether the other site can carry the load.
2. Early reports of cause should be treated as unconfirmed until a root-cause finding is published.
3. Failover between sites should be exercised so that a fault at one facility does not become a multi-day loss of service.

Source

datacenterdynamics.com