Arc flash in power distribution unit disrupts Google Cloud zone in Frankfurt
Oct 2024Published 12 Oct 2026Responsible editor: Chief Editor
Get the next one by email
Free alerts on data centre incidents in Germany, and an email when EIR-2261 is updated. No account needed; unsubscribe in one click.
Get a free sample brief Three complete verified records and the incidents-per-GW table, by email within a minute.
By signing up you agree to the privacy policy.
Companies named in this entry: Google Cloud
What happened
On 23 October 2024 at 18:22 US/Pacific time, an electrical arc flash occurred in one of the power distribution units serving the europe-west3-c zone of Google Cloud's Frankfurt region in Germany. The fault caused a partial power outage in a single data centre. The loss of power degraded the building's cooling infrastructure and ambient temperature rose. To prevent thermal damage, some IT equipment was shut down, taking virtual machines offline and affecting several cloud services in the zone, including Persistent Disk, Compute Engine, Pub/Sub, Dataflow, Dataproc, Cloud Build, Kubernetes Engine and Vertex AI Batch Prediction.
Google engineers were alerted at 18:39 and at 20:43 shut down equipment and diverted traffic away from the affected infrastructure. Power was manually restored at 21:44 by transferring load away from the failed components. Full restoration of services in the zone was completed at 02:09 on 24 October. Google gave the duration of the event as seven hours and 39 minutes, although its stated end time in the report header is 02:01.
Contributing factors
Google said the root cause investigation was ongoing and that it would repair the affected power distribution unit and check that the arc flash causes were not present in other data centres. Google also reported that a misconfiguration in access control services affected multiple europe-west3 zones, causing increased latency and errors when creating instances and attaching disks from 19:12 to 23:34 US/Pacific. Google stated that it had hardened that component against zonal outages.
Lessons learned
- Assess arc flash risk in power distribution units and verify that protection and containment are effective, and check other sites for the same causes after a fault.
- Treat loss of cooling as a time-critical failure, because thermal shutdown of IT equipment can extend an outage beyond the electrical fault itself.
- Make control-plane and access services resilient to the loss of a single zone, and test that a zonal failure does not spread to other zones.
- Customers should distribute critical workloads across zones or regions so that a single facility failure does not take them offline.
More lessons on worker safety and safe systems of work, outages, backup supply and restoration, spacing, layout and fire spread, from every incident on the register.
Source
Published 2026-10-12. Written from public reporting. Descriptive, not investigative. See the data accuracy disclaimer.
