UPS battery failure during power cut drill halts CTBC Bank services

What happened

On 5 February 2010, at about 6.20pm, the machine room of CTBC Bank (Chinatrust Commercial Bank) in Taiwan suffered a major outage. Credit card, online banking, ATM, customer service and Taiwan Lottery computer ticket systems were interrupted. The machine room was running a power cut drill, in which the Taipower supply was cut to test whether the generator would take over. The uninterruptible power supply (UPS) was expected to carry the load for about 30 seconds until the generator started.

Its batteries did not have enough charge, so the equipment lost power and shut down. Emergency repairs restored services at about 7.20pm. The bank said in a press release that the cause was a UPS fault.

Contributing factors

The UPS batteries had been in use for seven years and their capacity had fallen with age. According to iThome, the UPS used a parallel architecture, but this did not help once the batteries were depleted.

Lessons learned

1. Measure the capacity of UPS batteries under load on a regular schedule, and replace them on measured condition as well as on age.
2. Confirm battery state of charge before a power cut drill that relies on UPS ride-through, and have a plan to abort the test if the batteries are not ready.
3. Treat UPS battery health as a common-mode risk, because parallel or redundant UPS units provide no protection if their battery strings are all depleted.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

ithome.com.tw