Google Cloud data centre power failure causes cooling outage

What happened

Three Google Cloud services experienced a 15-hour outage after a power failure in the datacentre serving the europe-west4-a zone. An electrical fault on the utility grid upstream of the datacentre disrupted electrical distribution gear and cooling equipment. The DRUPS back-up power system for one side failed to take over due to electrical component failures. A chiller controller dropped offline, the redundant chilled water source was unavailable due to construction, and temperatures reached 44°C, leading to machine shutdowns. Workloads were turned down to protect data.

Contributing factors

Upstream utility grid fault, failure of DRUPS transfer due to component failures and load deployment discrepancy, chiller controller failure during voltage transient, redundant cooling source unavailable due to ongoing construction.

Lessons learned

1. Verify that managed services advertised as zone-resilient do not have single-datacentre dependencies.
2. Ensure backup power transfer systems are tested under realistic transient conditions.
3. Maintain redundant cooling sources even during construction work at facilities.

Source

theregister.com