Fallen trees and failed backup switch cause University of Virginia data centre outage

What happened

On the late evening of Sunday 30 April 2023, fallen trees affected the grid connection supplying the on-site data centre at the University of Virginia in Charlottesville. The facility was designed to move automatically to a backup generator, but the backup switch failed and the data centre went down. Power was restored from 11:30am on Monday 1 May 2023 after university staff manually transferred power to the data centre. The outage affected Wi-Fi services and disabled access to applications that require NetBadge authentication, including Workday, Collab, the Student Information System and UVABox. The account is based on a report by Data Centre Dynamics, which cites the university publication UVAToday.

Contributing factors

Fallen trees affected the grid connection to the on-site data centre. The automatic transfer to backup generator power did not operate as intended because the backup switch failed. Restoration depended on manual transfer of power by university staff.

Lessons learned

1. Test automatic transfer switches under load at regular intervals, since a switch that has not been exercised may fail when a real grid loss occurs.
2. Confirm that manual transfer procedures to backup generation are documented, rehearsed and can be carried out quickly by on-call staff.
3. Map which IT services depend on the on-site facility and plan communications and alternative access for users during a prolonged outage.
4. Assess the risk from vegetation near overhead supply lines to critical power connections and include it in maintenance plans.

More lessons on worker safety and safe systems of work, outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

datacenterdynamics.com