Mistaken fire trip shuts UPS group at Sydney data centre

What happened

On Thursday 14 July 2011, a power interruption at Global Switch's Sydney data centre in Australia brought down a number of customers' servers. Global Switch said a contractor mistakenly activated the fire-trip system, which shuts down UPS units in case of a fire. A group of UPS systems was shut down as a result, cutting power to a very small number of customers. The affected equipment was connected to only one of the facility's two power feeds.

The feed was shut down at around 10:20 am, and the feed and the affected UPS array were restored within one hour. Companies named in reporting as affected included PacNet, HarbourMSP, Exigent and Servers Australia. PacNet said its core network was not affected but some edge services were. Global Switch said no service level agreements were breached.

Contributing factors

A contractor mistakenly activated the fire-trip function, as attributed by Global Switch. Affected customers were connected to only one of the two power feeds. The source does not say what work the contractor was carrying out.

Lessons learned

1. Emergency shutdown controls on UPS systems need physical guarding, clear labelling and access control so they cannot be activated by mistake during site work.
2. Contractors working near critical power controls need briefing on what each control does and supervision while they work.
3. Customers with critical loads should use dual, independently fed power supplies, because single-feed equipment is exposed to a single point of failure.

More lessons on outages, backup supply and restoration, worker safety and safe systems of work, from every incident on the register.

Source

datacenterdynamics.com