Transformer failure and generator start failure cut power at Frankfurt data centre

What happened

On 14 May 2024 a transformer supplying power to the Interxion/Digital Realty FRA2 data centre in Frankfurt failed, causing loss of critical IT load to equipment in the space. The site's generators also failed to start. Around 11:56 UTC a generator was running and the A feed was restored, but racks remained without power because the B feed and other infrastructure had problems. Power returned to one feed briefly and was then lost again.

The site then ran on generator while engineers investigated the low-voltage switchgear on the transformer. The power loss also affected cooling, which was later restored. By 14:22 UTC the site was back on grid power with UPS protection. Customer notices put the outage at roughly four hours. The operator said the root cause was still under investigation.

Contributing factors

The operator reported that the transformer supplying power had an issue and that the generators failed to start. The low-voltage switchgear on the transformer was under investigation. The root cause had not been confirmed in the source. Access restrictions to the site limited the ability of customer technicians to check equipment.

Lessons learned

1. Test generator start-up and automatic transfer under load on a regular schedule, because a generator that fails to start turns a single transformer fault into a full outage.
2. Re-energise a site in stages after a fault, checking each feed before restoring load, since a feed that came back was lost again.
3. Customers relying on a single site should plan for loss of cooling and for limited on-site access during a power event.

More lessons on outages, backup supply and restoration, design, standards and commissioning, from every incident on the register.

Source

status.core-backbone.com