UPS inverter damage and control board faults cut power at Newark data centre

What happened

At about 19:35 UTC on 18 July 2018 utility power to the colocation data centre in Newark, New Jersey, housing Linode's US-East deployment was interrupted, and the critical load transferred to generator. The UPS system alerted the colocation provider to a possible failure, and technicians found that one unit in the N+1 UPS set had been damaged and taken offline for repair. The system continued in an N configuration. At 23:04 UTC, when power was switched back from generator to utility, the two remaining UPS units failed to take the load and transferred to bypass, causing a power loss of about three minutes. Power was fully restored at about 23:07 UTC. Most infrastructure and customer instances were back online by about 02:34 UTC on 19 July 2018.

Contributing factors

The colocation provider's investigation found that a power surge during the utility failure damaged the inverter on one UPS unit. The provider stated that the remaining UPS units were unable to handle the load during the retransfer to utility power because of a control board malfunction, and went into bypass mode.

Lessons learned

1. Test UPS retransfer behaviour under load, since a system that runs on generator or in a reduced configuration may fail when switching back to utility power.
2. Surge protection and inspection after a utility failure can reveal damaged power electronics before they cause a second outage.
3. Replacing a faulty component on one unit is not enough if sibling units share the same control board fault; audit and replace common components across the set.
4. Review power circuit redundancy for critical hardware after a power event.

More lessons on outages, backup supply and restoration, electrical protection and isolation, maintenance, inspection and ageing assets, from every incident on the register.

Source

status.linode.com