Power loss in one AWS data centre affects EC2 in US-EAST-1

What happened

On 22 December 2021 Amazon Web Services (AWS) reported a loss of power within a single data centre in one Availability Zone, USE1-AZ4, of its US-EAST-1 region in the United States. Issues began around 7:30 a.m. ET and were still affecting services at 1 p.m. ET. Availability and connectivity were affected for EC2 instances in the affected data centre, and AWS reported elevated error rates for instance launches in that Availability Zone.

The single sign-on service in the region also saw increased error rates. AWS said that connectivity and power to other data centres in the same zone, and to other zones in the region, were not affected. Customers were advised to move workloads away from the affected zone where possible. AWS later said the issue was resolved and the service was operating normally. The article reported this as the third outage in the US-EAST-1 region that month. The earlier events it lists were a networking issue on 7 December and a connectivity outage between two West Coast regions on 17 December.

Contributing factors

The source gives no root cause beyond the loss of power within the single data centre, and does not say which power equipment failed. The article notes earlier outages in US-EAST-1 that month, including a networking issue on 7 December, but does not link them to this event.

Lessons learned

1. Design cloud workloads to fail over across Availability Zones so that a power loss in one data centre does not take down the service.
2. Give customers early, specific guidance on which zone is affected and how to move workloads away from it.
3. Track repeated failures in one region, since several events in a short period can indicate a shared weakness in resilience.

More lessons on design, standards and commissioning, outages, backup supply and restoration, from every incident on the register.

Source

w.media