Transformer failure cuts power to Amazon and Microsoft data centres in Dublin

What happened

At 18:16 local time on Sunday 7 August 2011, a 110kV transformer operated by ESB Networks failed at Citywest, Dublin, Ireland. The failure caused an explosion and fire, which ESB said were localised to the transformer's bushings or insulators, and serious internal damage to the transformer. About 100 customers in the Citywest area lost supply, including Amazon and a number of other data centres. Supply was disrupted for about one hour, and a partial outage continued in the area until 23:00.

Amazon said the electrical disturbance from the explosion reached the phase control system that synchronises its backup generators, disabling some of them, so one EC2 Availability Zone lost power completely. Bringing the generators online required manual synchronisation. The source reports that Microsoft's BPOS online services were also affected. Amazon said recovery of some storage volumes could take 24 to 48 hours. ESB said the fire did not require emergency services. Lightning was first given as the cause, but ESB later ruled it out.

Contributing factors

Amazon and Microsoft said fluctuations in the incoming utility power interfered with the automated systems that start backup generators once power failed. ESB said the fault was unexpected and gave no advance warning, and that a voltage dip was experienced by customers in parts of Dublin for the duration of the fault. The cause of the transformer failure was still under investigation at the time of reporting.

Lessons learned

1. Test backup generator start and synchronisation logic against utility voltage disturbances and transients, not only complete loss of supply.
2. Maintain and rehearse manual synchronisation procedures for backup generators, because automatic paralleling can fail during a grid fault.
3. Where several critical sites depend on one utility substation or transformer, map shared dependencies and seek diverse supply routes.
4. Verify the cause of an outage before publishing it, since early attributions such as lightning can later be ruled out.

More lessons on outages, backup supply and restoration, worker safety and safe systems of work, weather and external hazards, from every incident on the register.

Source

datacenterknowledge.com