Sydney data centre loses power after diesel rotary UPS fails
June 2016Published 13 Oct 2026Responsible editor: Chief Editor
Get the next one by email
Free alerts on data centre incidents in Australia, and an email when EIR-2310 is updated. No account needed; unsubscribe in one click.
Get a free sample brief Three complete verified records and the incidents-per-GW table, by email within a minute.
By signing up you agree to the privacy policy.
Companies named in this entry: Amazon Web Services
What happened
On 4 June 2016 at 10:25 PM PDT (5 June Sydney time), a utility provider suffered a loss of power at a regional substation during severe weather, cutting utility power to multiple Amazon Web Services facilities in the Sydney Region. The event was an unusually long voltage sag rather than a full outage. In one facility, several power delivery line-ups failed to transfer load to their generators. Breakers meant to isolate the diesel rotary UPS (DRUPS) units from the degraded grid opened too slowly, so the DRUPS energy reserve drained into the grid and the units shut down before their generators could take over.
Instances lost both primary and secondary power, affecting EC2 instances and EBS volumes in a single Availability Zone. Power was restored at 11:46 PM PDT after on-site technicians confirmed it was safe to re-engage the line-ups. Recovery was slowed by DNS resolution failures, a latent software bug and API delays, and nearly all instances were recovered by 8 AM PDT. Less than 0.01% of EBS volumes could not be recovered because of failed hard drives.
Contributing factors
Severe weather caused the substation failure. The voltage sag was unusually long, and the isolating breakers did not open quickly enough to protect the DRUPS energy reserve. Several power line-ups failed together, so the redundancy design did not hold. A latent bug in instance management software slowed recovery of the remaining instances. Some storage servers suffered failed hard drives during the event.
Lessons learned
- Test that protective breakers open fast enough for long, partial voltage sags as well as full outages.
- Verify that generators can start and take load before UPS energy reserves are depleted, under degraded-grid conditions.
- Regularly test recovery workflows on long-running production hosts to find latent software faults before a real event.
- Design the power and redundancy architecture so that correlated line-up failures cannot take down both primary and secondary supply.
More lessons on electrical protection and isolation, outages, backup supply and restoration, from every incident on the register.
Source
Published 2026-10-13. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.
