Transformer explosion and fire disrupts Calgary city services and IBM data centre

What happened

A transformer exploded at Shaw Communications' downtown headquarters, Shaw Court, in Calgary, Alberta, on the afternoon of Wednesday 11 July 2012. The explosion caused a fire. The building's sprinkler system activated, and the water damaged the backup systems housed on site, so both primary and backup systems were knocked out. An IBM data centre in the building, which maintained Alberta's property and vehicle information databases, was disrupted. Calgary's 311 service went down, Alberta's motor vehicle and land title services were unavailable, and Alberta Health Services' computer network failed, which led to the postponement of hundreds of elective surgical procedures.

Some Shaw landline customers downtown could not reach 911. Health network service was restored late on the Friday, and Shaw said service was restored to all customers on the Saturday morning. IBM Canada reportedly flew backup tapes of vehicle and property registration data to a backup facility in Markham, Ontario.

Contributing factors

The sources do not establish why the transformer exploded. Backup systems were housed on the same site as the primary systems, so one event disabled both. The sprinkler discharge damaged equipment as well as the fire.

Lessons learned

1. Locate backup systems and recovery sites away from the primary site so that a single transformer failure or fire cannot disable both.
2. Assess whether fire suppression discharge, such as sprinkler water, can damage IT and backup equipment, and plan protection accordingly.
3. Keep tested off-site copies of critical data and a documented restoration plan, and confirm that essential public services such as emergency call routing have failover arrangements.

More lessons on outages, backup supply and restoration, fire suppression and firefighting, emergency response and public communication, from every incident on the register.

Source

datacenterknowledge.com