Reported transformer fault and generator shutdown cause total power loss at Hillsboro data centre

What happened

On 2 November 2023 Portland General Electric (PGE) had an unplanned maintenance event that shut down one of the utility feeds into the Flexential PDX04 data centre near Hillsboro, Oregon. Flexential started its generators to supplement the lost feed and kept the remaining utility feed running. At about 11:40 UTC a ground fault was reported on a utility transformer at the site, which Cloudflare believes stepped down power for the second feed.

The protective measures also shut down all of the site's ten generators, leaving the facility on UPS batteries meant to last about ten minutes. Cloudflare reports the batteries began to fail after about four minutes. Between 11:44 and 12:01 UTC the batteries ran out and all customers of the data centre lost power. Generators were restarted at 12:48 UTC, but faulty circuit breakers delayed the return of power to customer circuits. Flexential confirmed clean power to both utility feeds at 22:48 UTC. Cloudflare's control plane and analytics services were disrupted until 4 November.

Contributing factors

Cloudflare reports, without official confirmation from Flexential or PGE, that the generators had to be restarted manually because the ground fault had tripped circuits, that the site access control system was not on battery backup, and that the overnight staff included no experienced operations or electrical expert. Flexential did not inform Cloudflare that it had switched to generator power. Cloudflare says that when power was restored, several circuit breakers were found to be faulty, and it does not know whether the ground fault or an earlier condition caused this. Cloudflare also reports that some of its services depended on systems hosted only at this facility, and that its failover testing had not covered a full loss of the facility.

Lessons learned

1. Test the total loss of a facility, not only partial failures, and map cross-site dependencies before relying on high availability designs.
2. Verify UPS runtime under realistic load, since batteries may fail well before their rated duration.
3. Ensure generators can restart after a protective trip and that site access control remains on backup power.
4. Require tenants to be notified promptly when a facility changes its power source, and keep overnight staffing sufficient to handle electrical faults.

More lessons on design, standards and commissioning, outages, backup supply and restoration, emergency response and public communication, from every incident on the register.

Source

blog.cloudflare.com