Backup generator fails to start during Amsterdam grid outage

What happened

On 17 January 2017 a major grid outage affected Amsterdam from about 04:15. The Datacenter Group (TDCG) data centre in Amsterdam switched to backup power. All backup generators received automatic start commands, but one did not start after three attempts, so the A power feed was not covered. The B feed was backed up successfully. The A feed was without power for about 1 hour 20 minutes, until the generator was started manually at about 05:50.

Mains power was restored at about 07:00, and the generators returned to normal operation automatically. TDCG said that customers with redundant supplies were not affected, but some customers, including hosting providers, reported problems. Afterwards the operator found a mechanical fault: a defective actuator coil on a mechanical safety device that is needed before the generator can start. The coil was replaced and tested.

Contributing factors

TDCG said the generators had passed a test on 22 December 2016. The source does not say when the coil fault developed. The cause of the grid outage was not confirmed in the source, which said it began at a high-voltage distribution station on the Hemweg in Amsterdam.

Lessons learned

1. Test backup generators through the full automatic start sequence, including safety interlocks and their actuators, and not only by running the engine.
2. Treat a failed automatic start on one unit of a redundant pair as an immediate alarm, so that manual intervention happens within minutes rather than over an hour.
3. Check that each feed has independent backup capacity, so that a single generator fault does not leave a feed unprotected.

More lessons on outages, backup supply and restoration, detection and monitoring, from every incident on the register.

Source

computable.nl