UPS maintenance trip causes brief power outage at Equinix Manchester data centre

What happened

Equinix's MA1 data centre in Manchester Science Park, Manchester, United Kingdom, lost power on the morning of 24 May 2022. A customer status page reported that both A and B power feeds failed at about 10:03, while Equinix said load was transferred to generators. Internet Exchange, Metro Connect, Equinix Fabric and Equinix Connect services in Manchester were affected. One observer posting on Reddit reported customers on different providers in other parts of the UK going down for about ten minutes.

Equinix said the trip followed planned maintenance on the site's uninterruptible power supply (UPS) to replace one communications board. It said power was restored after a few milliseconds, with some services down for up to ten minutes while they rebooted.

Contributing factors

Equinix said the trip followed planned UPS maintenance to replace a communications board. The cause of the trip was still under investigation with vendors when the reports were published. The site was described as having N+1 power redundancy.

Lessons learned

1. Treat planned work on UPS control boards as high-risk and confirm the transfer path to backup power before the work starts.
2. Verify that a single control fault cannot take out both A and B power feeds, and test that isolation under maintenance conditions.
3. Customers should test their failover arrangements and avoid depending on a single facility for network-critical services.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

thestack.technology