Lightning strike causes cooling shutdown at Azure data centre in Texas

What happened

On 4 September 2018, storms over Southern Texas caused voltage sags and swells at several Microsoft Azure data centres in the South Central US region, according to Microsoft's preliminary findings as reported by The Register. At 08:42 UTC a lightning strike caused one data centre to switch to generator power. The strike also overloaded surge suppressors on the mechanical cooling system, which shut down. The facility ran on thermal buffers until these were depleted.

Temperatures then rose until a shutdown began, and storage units and network devices were damaged by heat. Customers in the region experienced near-total outages from 09:29 UTC on 4 September. Most services were restored by 11:00 UTC on 5 September, and full mitigation was completed at 08:40 UTC on 7 September. Engineers chose not to fail over to another site in order to protect customer data, which lengthened recovery. Services in other regions, including Azure Active Directory and Visual Studio Team Services, were also affected because of dependencies on the site.

Contributing factors

Cooling equipment was protected by surge suppressors that were overloaded by the electrical disturbance from the lightning strike. Thermal buffers were exhausted before temperatures could be controlled. The decision not to fail over to another data centre, to protect customer data, extended recovery time. Cross-region dependencies on the affected site, including the legacy Azure Service Manager, which uses South Central US as its primary metadata site and does not support automatic failover, and Azure Active Directory North America sites, widened the impact. These points are taken from Microsoft's preliminary findings as reported by The Register.

Lessons learned

1. Surge protection on cooling and other critical plant should be assessed for lightning and voltage swell, since an overloaded suppressor can take cooling offline during an electrical disturbance.
2. Thermal buffers should be sized against the realistic time needed to restore cooling and to fail over, with alarms that trigger before buffers run out.
3. Operators should identify where a single site holds primary metadata, directory or authentication functions for other regions, and provide automatic failover for them.

More lessons on weather and external hazards, detection and monitoring, from every incident on the register.

Source

theregister.com