Conflicting procedures delay failover after power outage at Google data centre
Feb 2010Published 12 Oct 2026Responsible editor: Chief Editor
Get the next one by email
Free alerts on data centre incidents in the United States, and an email when EIR-2302 is updated. No account needed; unsubscribe in one click.
Get a free sample brief Three complete verified records and the incidents-per-GW table, by email within a minute.
By signing up you agree to the privacy policy.
Companies named in this entry: Google
What happened
On 24 February 2010 a power failure occurred at Google's primary data centre used by App Engine, the company's cloud application platform. Google did not disclose the cause of the failure. When power was lost, about 25 percent of the servers in the facility did not receive backup generator power and crashed, taking App Engine down. After engineers decided to fail over to an alternate data centre, they found two conflicting sets of procedures, a result of a recent Datastore migration.
While staff worked out which set was correct, one engineer moved all traffic to the alternate facility, where a configuration problem stopped it working properly. The team later wrongly believed the primary facility had recovered and moved traffic back, so all traffic had to be moved again. Issues were first identified at 7:48 am and App Engine was restored at about 10:10 am, although lingering problems continued for several days.
Contributing factors
Conflicting operating procedures, left over from a recent Datastore migration, delayed the failover. A configuration problem at the alternate facility and a premature belief that the primary site had recovered added to the disruption. Google did not disclose the cause of the power failure, and the report does not say why roughly a quarter of servers did not receive generator power.
Lessons learned
- Audit operations documents regularly and retire superseded procedures after a migration, so on-call staff do not face conflicting instructions during an outage.
- Run regular drills for on-call staff and give them a clear policy for making failover decisions under pressure.
- Confirm that a primary facility has fully recovered through verified status before moving traffic back.
- Test that backup generator transfer reaches every critical server, since a quarter of servers missing it caused the crash.
More lessons on worker safety and safe systems of work, outages, backup supply and restoration, from every incident on the register.
Source
Other entries that name Google
Published 2026-10-12. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.
