Power loss and apparent failed generator changeover at London Docklands data centre

What happened

On the afternoon of 17 November 2015, the Sovereign House data centre in London Docklands, operated by TelecityGroup, lost utility power at around 14:00. The facility appears to have failed to switch over to its backup generators. Power was restored but then failed again. The London Internet Exchange (LINX), which had infrastructure in the facility, and AWS Direct Connect customers linked to Amazon's eu-west-1 region in Ireland were affected, and intermittent packet loss was reported.

Amazon said power had been restored about two hours later. TelecityGroup told a customer, EX Networks, that it would need to shut down power to two suites housing LINX to repair the electrical infrastructure, and that it planned to install six new power feeds from the unaffected supply. TelecityGroup had not stated the root cause at the time of reporting.

Contributing factors

The source reports that the facility appears to have failed to transfer to backup generators after the loss of utility power. The reason for the failed transfer and for the second power loss had not been stated at the time of reporting.

Lessons learned

1. Test automatic transfer from the utility supply to standby generators under load at regular intervals, because a generator that does not take the load is of no use in a real outage.
2. Verify switchgear and UPS condition before restoring supply after a failure, since a second outage can follow restoration if the underlying fault has not been found.
3. Plan repairs to shared electrical infrastructure so that customers are not left on unprotected mains supply, and give customers clear notice of any planned shutdown.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

datacenterknowledge.com