Underground vault short circuit shuts down Washington state data centre

What happened

A short circuit in an equipment vault on the Washington State government campus in Olympia caused a power surge that shut off power to the state government building housing the state's data centre in August 2011. The data centre lost utility power, and its backup generators took over the load as designed. A second surge followed, which forced the generators to shut down, as they are designed to do to protect electrical equipment from damage.

The outage lasted several hours and was the first shutdown of the data centre in 20 years. The state put the cost at US$500,000, including a new electrical vault switch costing US$130,000, generator fuel, staff overtime and consultant fees for damage assessment. The state had recently completed a new data centre and had begun moving staff into the building.

Contributing factors

The source describes the initiating fault only as an equipment short circuit in the campus underground vault. The shutdown of the generators on the second surge was a designed protective function rather than a fault in the generators themselves.

Lessons learned

1. Test how backup generators respond to repeated or sequential electrical faults, since protective shutdowns can remove backup power during an outage.
2. Maintain and inspect underground vault switchgear that feeds critical buildings, because a fault there can cut supply to the whole site.
3. Count the full cost of an outage, including replacement switchgear, fuel, overtime and assessment fees, when setting budgets for resilience.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

datacenterdynamics.com