Telecity reports breaker fault behind total power loss at London data centre

What happened

On the morning of Wednesday 27 July 2011, Telecity's Meridian Gate data centre in London lost all mains power. Tenant C4L reported that the UPS and generator did not take over, taking all customers offline and breaking its 10G network ring. Telecity told C4L it had identified a fault on a breaker in the power distribution system. Telecity said its engineers restored power within around 20 minutes and that it had apologised to affected customers.

C4L first lost both of its 10G fibre network legs to the site at about 9:15 am BST, and the facility was reported to be running on mains power, not generators, by about 9:40 am. Half of C4L's Infrastructure-as-a-Service platform was affected, and several customer websites went down. It was Telecity's second London data centre outage that month, following a mains power loss on the seventh floor of its Harbour Exchange site in early July.

Contributing factors

Telecity attributed the outage to a fault on a breaker in the power distribution system. The source does not say why the UPS and generator did not take the load, or what caused the breaker fault.

Lessons learned

1. Operators should test UPS and generator changeover under real load so that a failure in the distribution path does not leave the whole facility without backup.
2. Tenants relying on a single site should have diverse network paths, so that losing one facility does not take down their whole platform.
3. Prompt, specific updates to tenants during a power incident reduce uncertainty and help customers plan their own response.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

datacenterdynamics.com