UPS failure caused Google Cloud outage in Columbus

What happened

On 29 March 2025 a loss of utility power occurred in the us-east5-c zone of Google's US East cloud region in Columbus, Ohio. As reported from Google's incident report, the outage lasted six hours and ten minutes. The power loss caused a cascading failure in the UPS system, which is meant to bridge the gap until generators start and which experienced a critical battery failure, leaving it unable to keep supply.

Virtual machine instances in the zone lost power. Secondary effects included packet loss in the zone and some storage disks being brought down. More than 20 services had degraded performance or were unavailable. Engineers bypassed the failed UPS and restored power from a generator just under two hours after notification. Most services recovered then, though some needed manual actions.

Contributing factors

Utility power loss exposed a battery failure inside the UPS that prevented it from maintaining supply until generators engaged.

Lessons learned

1. UPS battery systems need monitoring and maintenance so they can bridge a utility outage until generators start.
2. Power-failure recovery paths should be hardened against cascading effects.
3. Operators should work with UPS vendors to understand and remediate battery backup faults.

Source

datacenterdynamics.com