Three UPS failures cause outages at Internap data centres in New York

What happened

Internap experienced three outages in six days at two of its Manhattan data centres in May 2014, all caused by component failures in uninterruptible power supply (UPS) systems. On 16 May an output breaker on a UPS at 111 8th Avenue failed and registered a major fault. The UPS was taken out of the lineup and the utility feed was shut off, disrupting about 20 customers, including Livestream and StackExchange.

At 75 Broad Street, electrical capacitors failed on one UPS at about 10:30am on 20 May. Single-corded customers lost power for about three seconds and remained on non-UPS utility power until the capacitors were replaced the next morning. On 22 May, at about 10:40pm, an output breaker on a UPS at 75 Broad Street failed, leaving single-corded customers without power for about four hours until engineers replaced it. Customers on redundant feeds were not affected at 75 Broad Street.

Contributing factors

The capacitors that failed at 75 Broad Street had an average lifespan of about seven years and had been replaced about three years earlier. The UPS vendors serviced the systems annually. 75 Broad Street is described as a legacy site without the most recent support infrastructure, and single-corded customers, fewer than half of tenants, did not have redundant power. The faulty output breaker on 22 May blocked all power distribution downstream of the UPS, so affected equipment could not transfer to utility power. The operator described the 111 8th Avenue failure as very odd, as the site had never previously had an outage. Root causes were still under investigation at the time of reporting.

Lessons learned

1. Redundant power feeds kept customers running during both 75 Broad Street events, so operators should make the reliability difference between single-corded and redundant options clear to customers.
2. Components with a known service life, such as capacitors, should be tracked and replaced on schedule, because annual vendor servicing alone did not prevent failure.
3. A breaker fault that blocks downstream power can stop an automatic transfer to utility supply, so failure modes should be tested to confirm a path to backup remains available.
4. Keeping spare critical breakers on site shortened repair time to a few hours.

More lessons on outages, backup supply and restoration, maintenance, inspection and ageing assets, electrical protection and isolation, from every incident on the register.

Source

datacenterknowledge.com