UPS reportedly fails to take over during thunderstorm at Hetzner data centres in Falkenstein

What happened

On 24 May 2018 a thunderstorm reportedly caused voltage fluctuations in the local grid at Falkenstein in Saxony, Germany, affecting three data centres operated by Hetzner Online. The uninterruptible power supply (UPS) apparently did not take over as intended, and problems with the emergency power supply were reported. When mains power returned, some servers reportedly did not restart automatically and had to be started by hand, which kept the technical team busy.

Many customer websites were reported to be affected, and Hetzner's own status page was offline at first. WinFuture reported on 25 May that parts of the service were still offline. A company spokesperson said it was not yet known why the UPS had not started, and technicians from the UPS manufacturer were on site analysing the cause. Hetzner apologised to affected customers and said it would publish a final statement once the exact cause was confirmed.

Contributing factors

Voltage fluctuations in the local grid, reportedly caused by a thunderstorm. The UPS apparently did not take over as intended. Some servers reportedly did not restart automatically when power returned. The cause of the UPS failure had not been established at the time of reporting.

Lessons learned

1. Test UPS transfer under grid disturbances such as voltage fluctuations and short dips, not only full outages.
2. Configure servers to restart automatically and in a controlled sequence after power restoration, so that manual recovery does not become the bottleneck.
3. Keep status and customer communication systems on independent power and infrastructure so that customers can be informed during an outage.
4. Agree in advance with UPS manufacturers how root cause analysis will be carried out and findings shared with the operator.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

winfuture.de