Remote power panel breaker trip causes Azure East US storage outage
Mar 2017Published 12 Oct 2026Responsible editor: Chief Editor
Get the next one by email
Free alerts on data centre incidents in the United States, and an email when EIR-2290 is updated. No account needed; unsubscribe in one click.
Get a free sample brief Three complete verified records and the incidents-per-GW table, by email within a minute.
By signing up you agree to the privacy policy.
Companies named in this entry: Microsoft
What happened
Beginning at 22:19 UTC on 15 March 2017, a main breaker tripped in one of the redundant upstream remote power panels serving a storage scale unit in Microsoft Azure's East US region. As load transferred to the remaining power sources, the transfer caused a cascading power loss across the scale unit, including all of its server racks and its network rack. Customers using Storage in the affected scale unit saw errors and timeouts, and virtual machines with VHDs in that unit shut down.
Microsoft reported knock-on effects for Azure SQL Database, Redis Cache, Event Hub, Service Bus Premium, Azure Search, Site Recovery and Backup. Power was restored one rack at a time using safe power recovery procedures to protect data integrity. Storage and dependent services were fully recovered by 08:32 UTC on 16 March 2017, roughly ten hours after onset. Failed rack power distribution units were sent for analysis.
Contributing factors
Microsoft's initial investigation identified the main breaker trip in one redundant upstream remote power panel as the trigger, with the load shift to remaining sources producing the cascade. The source does not establish why the remaining sources could not carry the transferred load. An analyst commentary questioned whether the redundant design accounted for the remaining capacity, but this is opinion and is not confirmed by the operator's account.
Lessons learned
- Test how redundant power paths behave under full transfer load, not only when a single component is isolated, and confirm that remaining sources can carry the transferred load.
- Plan power restoration in controlled stages, such as rack by rack, so that equipment and data are protected, and factor the restoration time into recovery targets.
- Customers of cloud and hosted services should check which dependent services share a single storage or power domain and set up cross-region failover where downtime is unacceptable.
More lessons on electrical protection and isolation, outages, backup supply and restoration, from every incident on the register.
Source
Other entries that name Azure
Published 2026-10-12. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.
