Data centre power outage affects Cloudflare facility in Portland

What happened

On 26 March 2024 at 14:58 UTC the PDX01 data centre in the Portland, Oregon region lost all power to Cloudflare's equipment. Cloudflare reported that four Flexential-owned and operated switchboards failed at the same time, so both primary and redundant power paths were lost. After the switchboards were reset and settings adjusted, the data centre cold start took roughly ten hours. The control plane recovered automatically within about seven minutes using capacity in other Portland facilities. Analytics services were not fully restored until later that day. The data plane handling customer traffic was unaffected.

Contributing factors

An initial assessment pointed to breaker coordination settings on the four circuit switch boards being set too low relative to downstream loads, which can cause nuisance tripping and, in this case, was reported to have led to a cascading failure. Cloudflare said it did not know when those settings were established.

Lessons learned

1. Pre-provision redundant capacity and test automatic failover of control-plane services before a major facility loss occurs.
2. Move configuration databases to a highly available topology across multiple facilities with weekly cut tests.
3. Review breaker coordination settings and operational procedures at all critical data centre facilities on a recurring basis.

Source

blog.cloudflare.com