Faulty UPS causes brief power outage at Equinix London data centre

What happened

On the morning of Wednesday 20 July 2016, a brief power outage occurred at the former Telecity LD8 data centre at 8/9 Harbour Exchange, in London Docklands. Equinix, which owns the site, said the outage began at 07:55 BST. DatacenterDynamics reported that power was lost until about 08:17 BST and that a London Internet Exchange peering node housed in the facility was affected until it was fixed at 09:15 BST.

Engineers diagnosed the root cause as a faulty uninterruptible power supply (UPS) system. BT and Plusnet customers across the United Kingdom lost connectivity, and BT said around 10 per cent of customers' internet usage was affected. Equinix said a limited number of customers were impacted and that service was restored within minutes, although end users reported connection problems through the morning. A separate circuit breaker trip occurred at Telehouse North the following day.

Contributing factors

Engineers identified the faulty UPS system as the root cause. The source does not report any further contributing factors, such as maintenance history or the specific fault mode within the UPS.

Lessons learned

1. UPS systems are a single point of failure for the power chain of a data centre and their failure can take down upstream network exchange nodes, so resilience testing of UPS units and their bypass and failover paths should be verified regularly.
2. Operators hosting internet exchange or peering equipment should map the downstream effects of a power loss, because a short outage can disrupt many customers across a wide area.
3. Restoration to service in minutes does not mean the root cause has been resolved, so faulty power equipment should be isolated, diagnosed and repaired promptly to avoid repeat events.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

computing.co.uk