Generator fire at WebNX data centre causes outage in Ogden

What happened

An emergency generator at a WebNX data centre in Ogden, Utah, experienced a catastrophic failure and caught fire on a Sunday afternoon over the Easter weekend in 2021, after city power was disrupted and backup generators switched on. Fire officials cut power to the building. The company said no fire damage was inflicted on customer servers, but servers in one main bay were exposed to water and a small number may have been damaged while crews fought the fire. By the following Wednesday many servers were back online, while some were expected to remain offline for several weeks pending inspection.

Contributing factors

The generator failed during automatic switchover after city power was disrupted. The company said the unit had been recently tested and benchmarked for that situation. The precise mechanical cause of the failure is not reported.

Lessons learned

1. Recent testing does not remove the need to check how backup generators behave during a real utility-to-generator transition.
2. Plan for water reaching nearby equipment when a generator fire is fought, even if the fire itself stays in the plant.
3. Keep customers informed and stage inspection and restart when suppression or firefighting may have wet servers.

Source

datacenterfrontier.com