Utility power loss and backup generator failure at AWS data centre, Virginia
Aug 2019Published 12 Oct 2026Responsible editor: Chief Editor
Get the next one by email
Free alerts on data centre incidents in the United States, and an email when EIR-2284 is updated. No account needed; unsubscribe in one click.
Get a free sample brief Three complete verified records and the incidents-per-GW table, by email within a minute.
By signing up you agree to the privacy policy.
What happened
On Saturday 31 August 2019, at about 04:33 PDT, one of the ten data centres in one of the six Availability Zones of the AWS US-East-1 region in Northern Virginia lost utility power. AWS said its backup generators came online immediately but began quickly failing at around 06:00 PDT, for reasons it was still investigating. By 06:10 PDT about 7.5 per cent of EC2 instances in that Availability Zone had failed, with similar impact on EBS storage volumes.
Most instances were recovered over the following hours. AWS told some customers that a small number of volumes were hosted on hardware damaged by the power event, that the underlying EBS servers had not recovered, and that the data was unrecoverable. Customers without backups lost data. AWS did not comment to The Register.
Contributing factors
The utility supply to the data centre failed. The backup generators then failed for reasons AWS had not explained at the time of reporting. Damage from the power event affected hardware hosting some EBS volumes. No root cause had been published in the source.
Lessons learned
- Test backup generators under realistic load and monitor their fault states, so that a failure is found before a utility outage exposes it.
- Keep customer-critical data backed up independently of the primary storage volumes and the availability zone that hosts them.
- Design for the loss of a single availability zone by spreading workloads, and check regularly that snapshots and backups can actually be restored.
More lessons on detection and monitoring, spacing, layout and fire spread, from every incident on the register.
Source
Published 2026-10-12. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.
