Storm-related power outage and control failure hit Fujitsu data centre in Perth

What happened

A Fujitsu data centre in Perth, Western Australia, lost power on a Sunday in early February 2015 during severe thunderstorms. The first outage lasted about 1.5 hours. According to Fujitsu, backup systems responded as designed, but a second failure in a control system then caused a longer interruption to service. Fujitsu said the facility's protection mechanisms activated to protect computer equipment and site safety, which interrupted service. Customers affected reportedly included a state government health department and a bank that is an anchor tenant of the facility, whose ATM and phone banking systems were reported down for 12 hours.

The bank did not confirm it was affected. Lightning strikes may have caused the initial failure, but this was not confirmed. Fujitsu said it was investigating the cause.

Contributing factors

Severe thunderstorms in the Perth area. Possible lightning strikes, not confirmed. A second failure in an undisclosed control system that prolonged the outage after backup systems had responded.

Lessons learned

1. Test that backup power and transfer systems hold through a full storm-related grid disturbance, not only a short interruption.
2. Review control system logic and failure modes so that a fault in one control layer cannot extend an outage after backup systems have already taken over.
3. Map which customers depend on the facility, and set out how they will be told and supported during an extended outage.

More lessons on weather and external hazards, outages, backup supply and restoration, from every incident on the register.

Source

datacenterknowledge.com