Data centreUnited StatesEIR-2323

Generator overheating and breaker misconfiguration cause AWS data centre power loss

June 2012Published 13 Oct 2026Responsible editor: Chief Editor

SignificantElectrical

Estimated damageModelled by the register's adjusterHow? Know the real figure?

Low confidence

out of service for about •• daysShown in the record →Full estimate

From the recordRoot cause Failed part

By signing up you agree to the privacy policy.

Companies named in this entry: Amazon Web Services

What happened

On 14 June 2012 an Amazon Web Services data centre in the US-East-1 availability zone in Ashburn, Virginia, lost utility power at 8:44 p.m. Pacific time. The facility transferred to backup generators as designed. About nine minutes later a defective cooling fan caused one backup generator to overheat and shut down. The affected instances and volumes moved to a secondary backup power circuit, but a breaker on that circuit had been configured to open at too low a power threshold.

It opened when the load transferred at 8:57 p.m., leaving those systems with no primary, backup or secondary power. Power was restored at 10:19 p.m. after the generator fan was repaired and the generator restarted. A primary Elastic Block Storage datastore lost power and did not fail cleanly, causing further disruption. Customers including Heroku, Pinterest, Quora and HootSuite were affected, along with smaller sites. Afterwards AWS audited its backup power circuits and found one more breaker needing correction.

Contributing factors

A breaker on the secondary backup power circuit was configured to open at too low a power threshold and tripped when the load transferred. A defective cooling fan caused a backup generator to overheat and shut down. A primary EBS datastore lost power and did not fail cleanly. Customers needed time after power returned to fully restore databases and applications.

Lessons learned

Check breaker trip settings on backup power circuits against the real load expected during a transfer, since a configuration that looks correct can still trip at changeover.
Monitor generator cooling systems and alarm on faults early, because a single failed fan can take a generator out of service during an outage.
Audit every backup power breaker after an incident, not only the one that failed, and test transfers under load.
Ensure storage systems shut down or fail cleanly when power is lost.

More lessons on electrical protection and isolation, detection and monitoring, from every incident on the register.

Source

datacenterknowledge.com

Published 2026-10-13. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.

See the register's stories first in Google: add it as a preferred source.

Chart to reuse with this story: Data centre incidents in the United States, by year, free with a credit (PNG).

The record behind this entry

The record behind EIR-2323 holds the site, the operator, the root cause, what failed, the type of facility, how long the outage lasted and the services that went down as fields you can filter, count and export, with its source and their verification status. The cause is not hidden: the contributing factors above give it as the sources report it. The record adds it as a field.

Site: ••••••••••••
Operator: •••••••••
Failed part: •••••••
Type of facility: ••••••••
Outage: •••••
Services affected: ••••••••••••
Sources: ••••••••

Opens with this page:

Or , both for reading; or £39 a month, which also lets you propose edits to entries. See a complete record first, if you like.

Open the record in Lookup

Or get this one record free by email

Data centre incidents in the United States: 54 entries, with a free alert for the next one.

Browse the full register or submit an incident.