Grid surge disables generator start at San Jose data centre

What happened

Utility power was lost at NaviSite's data centre in San Jose, California, during severe storms that had caused a power outage across a wide area of the Bay Area. NaviSite said an electrical surge from the grid had damaged relays in the facility's transfer switches, which prevented automatic switching to generator power. The generators did not receive a signal to start. The facility ran on UPS power until the batteries drained, and the whole data centre was without power for about 40 minutes, until an electrician was dispatched to the site and manually started the generators and transferred the load.

More than 100 customers were affected, and some customer environments took up to 24 hours to restore. The company said it would compensate affected customers under their service level agreements.

Contributing factors

Severe storms caused a wide area outage in San Jose, and the utility PG&E attributed the outage to the storms. The grid surge damaged relays in the transfer switches. The generators therefore did not start automatically. No qualified electrician was on site to transfer the load manually, so recovery depended on dispatching one to the facility.

Lessons learned

1. Test automatic transfer and generator start sequences against grid disturbances such as voltage surges, not only against clean loss of supply.
2. Keep a qualified person on site around the clock who can start generators and transfer load manually when automatic transfer fails.
3. Compare UPS runtime with the realistic time needed to restore generator power, and treat any gap as a risk to be closed with staffing or equipment changes.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

datacenterdynamics.com