Electrical distribution failure during generator transfer causes Microsoft data centre outage in Netherlands

What happened

On 20 October 2023 Microsoft detected instability on the utility grid, in the form of voltage sags and swells, at one of its data centres in the AZ-01 Availability Zone of its West Europe region in the Netherlands. The company decided to transfer the load from the grid to backup generators. During this transfer a critical failure occurred in a section of the electrical distribution system, which prevented 10 percent of the generators from taking load.

The main distribution system went offline and the redundant system could not be accessed, and about 1 percent of server racks in the zone lost power. Services including App Service, Cosmos DB, SQL DB, Storage and Virtual Machines were affected between 07:31 and 09:15 UTC. Five storage scale units were hit. One needed hardware diagnostics and part replacements on about 5 percent of its storage nodes, and full restoration was complete by 17:10 UTC. The nature and cause of the distribution failure were not detailed.

Contributing factors

Voltage sags and swells on the utility grid prompted the transfer to generators. The source states that an issue with generator startup also led to an outage for some racks. The nature and cause of the distribution failure were not detailed.

Lessons learned

1. Test generator transfer sequences under load so that faults in the distribution path are found before a grid disturbance forces a transfer.
2. Verify that redundant distribution paths are independent and remain accessible during a fault, not only on paper.
3. Plan storage recovery so that a single slow scale unit does not extend restoration for dependent services.

Source

datacenterdynamics.com