UPS malfunction at Singapore Exchange data centre halts trading

What happened

On 5 November 2014 the trading systems of the Singapore Exchange (SGX) went down. SGX said the cause was a malfunction in the backup power systems at its primary data centre. The primary data centre is fed by two separate substations, each connected to its own uninterruptible power supply (UPS) system. A momentary fluctuation in the power supply, caused by lightning, should have led the UPS systems to keep the equipment running.

Instead the UPS systems switched to their internal power source, which malfunctioned. SGX said the reason was still unknown. The UPS systems had last been tested in August. Because the disruption was brief and could have caused a mismatch between primary and secondary order books, SGX did not fail over to its secondary data centre. It restarted both primary and secondary systems instead. Recovery was completed at 4.45pm for the securities market and 6pm for the derivatives market.

Contributing factors

Lightning caused the initial fluctuation in the power supply. The UPS systems' internal power source malfunctioned for reasons SGX had not determined at the time of reporting. SGX last tested the UPS systems in August 2014.

Lessons learned

1. Test UPS transfer and internal power source performance under realistic fault conditions at regular intervals, and record the results.
2. Define in advance the criteria for failing over to a secondary site after a transient power disturbance, so that the decision is not taken under pressure during an outage.
3. Check that UPS transfer logic responds as designed to brief supply fluctuations caused by lightning, and that protection at the incoming supply limits such disturbances.

More lessons on outages, backup supply and restoration, weather and external hazards, from every incident on the register.

Source

techgoondu.com