Utility cable fault and failed generator start shut down OVH Strasbourg data centre

What happened

On 9 November 2017 at about 07:04, a fault on the external electricity network cut both 20kV supply cables to the OVH Strasbourg (SBG) site in France. The two cables shared a single circuit breaker, so both feeds were lost. The site's high-voltage backup generators did not start. The post-mortem attributes this to a control PLC in the normal-emergency inverter system, which was locked in a default 'locked automatics' mode and never sent the start command.

One of the two low-voltage generator units was in maintenance mode. The two high-voltage units then could not synchronise, and when they were run separately, a single 2MVA unit could not carry the load and shut down. UPS batteries ran out between 07:12 and 07:21, and servers in SBG1, SBG2 and SBG4 shut down. The operator reports that 40,386 servers were shut down. Grid power returned at 10:39. By 22:00 that day, 97 percent of servers were running, and a minority of services remained affected until 12 November.

Contributing factors

Both supply cables were protected by one circuit breaker, so a single cable fault removed both feeds. The control PLC for the automatic generator start was in a locked mode that the operator says it has not yet explained. One low-voltage generator unit was out of service for maintenance, which reduced backup capacity. The high-voltage generators could not synchronise and could not carry the site load on their own. The operator states the site did not meet its internal standard of two independent 20kV feeds, and that the last full backup recovery test was in May 2017. The cause of the cable damage was under investigation by the network operator.

Lessons learned

1. Supply cables feeding critical loads should be protected by independent circuit breakers, so that one fault cannot remove both feeds.
2. Automatic generator start and transfer logic should be tested under real fault conditions and at intervals that catch controller lock-ups, not only through periodic simulations.
3. Generator maintenance should be planned so that the remaining backup units still meet the full load, including in the reduced configurations that maintenance creates.
4. Generator synchronisation and load-sharing should be verified in every configuration the site may run in, and UPS ride-through time should be checked against the actual generator start time.

More lessons on electrical protection and isolation, outages, backup supply and restoration, from every incident on the register.

Source

network.status-ovhcloud.com