Automatic transfer switch failure causes partial outage at Colo4 Dallas data centre

What happened

On 10 August 2011 an automatic transfer switch (ATS) failed at one of the six service entrances, where utility power enters the Colo4 data centre in Dallas, Texas. The first alert of a power issue was posted at 12:40 p.m. The partial outage lasted several hours and kept numerous customers' equipment down. Technicians installed a spare ATS kept on site as a temporary replacement until a new primary unit could be installed.

Power to distribution gear from the temporary ATS was restored around 5:40 p.m., and HVAC units were brought back online. Power distribution units were restored in bypass mode so UPS batteries could charge, and the facility ran on generator power during charging before returning to utility power. Network connectivity was affected for some customers.

Contributing factors

The source does not state the cause of the ATS failure. Colo4 said bypassing an ATS is not a simple fix and that the safety of its people and contractors had to take priority, which lengthened restoration. One network switch was connected to only one power feed, although most routing gear had redundant power, and Colo4 said this was a mistake that it corrected on the day.

Lessons learned

1. Check that every network and IT device is connected to both power feeds, because a single-corded device can lose connectivity when one feed fails.
2. Keep a spare transfer switch on site, but recognise that installing it safely and restoring downstream systems can still take many hours.
3. Plan the restoration sequence for HVAC, UPS, distribution units and customer load in advance, and give customers regular status updates during the event.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

datacenterdynamics.com