Grid power interruption damages university computing centre IT despite generator start

What happened

On Thursday 20 August 2015, at about 10:30, the power supply to the Neuenheimer Feld campus of Heidelberg University was interrupted for about one and a half minutes. The diesel emergency generators started within seconds, as planned. However, fluctuations in the electrical grid during the event caused considerable damage, in part, to the IT infrastructure of the university computing centre. As a result, email, parts of the telephone system and access to the university websites were disrupted or failed completely.

The university administration, the rectorate and several institutes and facilities were affected. The university reported on 24 August 2015 that the affected IT services had been stabilised and were largely back in use, with the computing centre working to restore full normal operation.

Contributing factors

Power fluctuations on the grid following the interruption were reported as the cause of the damage to IT equipment, even though the backup generators started as designed. The source does not describe the specific equipment damaged or why the fluctuations were not contained before reaching the IT infrastructure.

Lessons learned

1. A backup generator starting within seconds does not protect IT equipment if the interruption and the supply disturbances that follow reach the load, so ride-through of uninterruptible power supplies and power conditioning should be verified for both outages and voltage fluctuations.
2. Test changeover and recovery sequences under realistic grid disturbance conditions, not only with a clean loss of supply.
3. Plan a staged restart and integrity check of IT systems after a power event, because services such as email, telephony and websites can fail in different ways and may take time to stabilise.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

uni-heidelberg.de