Utility power surge trips cooling at Microsoft Singapore data centre

What happened

On 8 February 2023 a utility power surge in the Southeast Asia region took a subset of cooling units offline at a Microsoft Azure data centre in Singapore, one of three availability zones in the region. Temperatures in the facility rose, and Microsoft proactively powered down a number of compute and storage units to avoid damage to hardware and reduce cooling system load. Azure services were disrupted, and organisations including the Central Provident Fund Board, EZ-Link, the Esplanade and Nanyang Technological University reported disruption to their web services.

Microsoft restored cooling, brought temperatures back within normal operating limits and carried out a structured power-up of the affected compute and storage resources. Services were reported restored by 9 February. Microsoft described it as the second such outage in a few weeks.

Contributing factors

A utility power surge in the Southeast Asia region is the stated trigger. The source does not identify the cause of the surge. Industry commentators suggested that the Tier 3 design and power quality, rather than power quantity, may have contributed, but this is opinion and was not confirmed by Microsoft.

Lessons learned

1. Treat utility supply disturbances as a credible trigger for sudden loss of cooling, and test how the facility responds to it.
2. Set temperature thresholds and staged shutdown sequences so that compute and storage can be powered down in an orderly way before hardware is damaged.
3. Assess power quality as well as capacity when judging supply resilience, and track repeat events at the same facility.
4. Plan a controlled, staged power-up once cooling has been restored, so that resources return without a further fault.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

w.media