Power outage at IBM Japan data centre stops bank core system

What happened

On Sunday 23 February 2020, at about 09:25, a power outage at a data centre operated by IBM Japan brought down the core banking system of Sumishin SBI Net Bank. Power was restored at about 09:29, but equipment needed to restart the core banking system had been damaged by the outage, so the system could not be restarted. Recovery work followed. ATM transactions for most ATMs resumed at 15:58 and Seven Bank ATMs at 16:43.

Online account opening and transactions, transfers, loan services, balance enquiries, the API link to external partners, the SBI Securities link and debit card services were restored in stages between 16:09 and 17:00. The failed equipment was replaced and other equipment was checked. The bank apologised to customers and announced a temporary system maintenance.

Contributing factors

The outage at the data centre damaged equipment required to restart the core banking system, so restoring power did not bring services back. The bank's notice does not state the cause of the power outage itself. IT Leaders reported that IBM Japan attributed the blackout to a power equipment failure at its data centre and said it would analyse the cause; this is not confirmed in the bank's notice.

Lessons learned

1. Restart equipment for critical core systems should be tested for resilience to power disturbances, not only to loss of the main supply, because damage at restoration can prevent recovery.
2. Operators relying on third-party data centres should confirm the facility's power failure analysis, backup arrangements and restart procedures, and the recovery time commitments in their contracts.
3. Recovery plans should set out staged restoration of channels such as ATMs, transfers and external interfaces, with prompt customer communication during the outage.

More lessons on outages, backup supply and restoration, worker safety and safe systems of work, from every incident on the register.

Source

netbk.co.jp