UPS module failure causes outage at Rackspace London data centre

What happened

On 18 January 2010 a module on an uninterruptible power supply (UPS) failed at the LON03 data centre operated by Rackspace in London, United Kingdom. The power interruption began at 09:19 local time, and the unit failed to transfer the load properly. Power was restored for most customers by 11:30, but about 220 servers, which Rackspace described as Whiteboxes, took several hours to return to service. Engineers had to intervene manually, replacing power supplies in servers and firewalls, reconfiguring switches and logging on to servers to get them to boot properly.

Rackspace said the UPS vendor had been notified and that it would investigate the cause of the UPS issues. DatacenterDynamics reported that more than 200 servers were down for several hours.

Contributing factors

The source reports that the UPS module failed and the unit did not transfer the load as expected. Restart problems on about 220 servers required manual intervention. The cause of the UPS failure had not been confirmed at the time of reporting.

Lessons learned

1. Test UPS transfer and failover under realistic load so that a failed module is detected and contained before it affects the load.
2. Maintain documented restart sequences and boot procedures for servers and network equipment so that recovery after a power event is not dependent on manual troubleshooting.
3. Keep spare power supplies, firewalls and switches on site to shorten recovery time.
4. Require the UPS vendor to provide a root cause report and to act on any defect it identifies.

More lessons on detection and monitoring, worker safety and safe systems of work, spacing, layout and fire spread, outages, backup supply and restoration, from every incident on the register.

Source

datacenterknowledge.com