UPS stops after wrong cable connected during data centre power switchover

What happened

On Monday 25 November 2019, a power failure occurred at 13:58 at a data centre operated by NTT Communications in Gifu Prefecture, Japan. The failure happened during switchover work on the data centre's power equipment. An excessive current was generated, and an uninterruptible power supply (UPS) forming part of the power equipment stopped. As a result, power could not be supplied to some customer equipment housed in the data centre.

The power equipment was restored first, and customer equipment was then brought back online in sequence. Service was restored at 19:35 the same day, an outage of 5 hours 37 minutes. The company attributed the cause to human error, in which an incorrect cable was connected during the switchover work. NTT Communications said it would review work processes and manuals to prevent recurrence.

Contributing factors

The source attributes the incident to a human error during switchover work, in which the wrong cable was connected. The excessive current generated during the switchover caused the UPS to stop. The source does not say whether the switchover was checked or independently verified before work began.

Lessons learned

1. Verify every cable connection during power switchover work against a checked drawing, and use an independent second person to confirm the connections before energising.
2. Include the UPS bypass and isolation steps in written method statements and rehearse them before live switchover work.
3. Document the restoration sequence for customer equipment downstream of a UPS so that recovery after a trip is fast and orderly.

More lessons on electrical protection and isolation, outages, backup supply and restoration, from every incident on the register.

Source

ntt.com