Smoking power distribution unit evacuates Telehouse North data centre in London

What happened

At about 6am on Monday 14 November 2016, internet connectivity was affected for a number of broadband ISPs after a fault developed in one of the Power Distribution Units (PDUs) at Telehouse North, a major data centre in London. The faulty unit produced smoke, which set off the building's fire alarm, and the building was evacuated. Power was largely restored by about 7:30am. A number of ISPs across the United Kingdom reported some disruption to their networks, although the source suggests most end users would not have noticed.

The article notes that a major power failure at the same site had occurred in July 2016 during a similar early-morning period, and describes the November disruption as much less significant. Full service was expected to return shortly after publication.

Contributing factors

The source attributes the disruption to a fault in one Power Distribution Unit. No further cause was reported.

Lessons learned

1. Power distribution units can fail with smoke production, so monitoring for overheating and early fault signs in PDUs and similar equipment is important.
2. A single power fault in a shared facility can disrupt many dependent networks at once, so customers should plan for upstream failure and have alternative connectivity routes.
3. Repeated power incidents at one site within a few months call for a root-cause review and remediation that addresses the recurring pattern, not only each event in isolation.

More lessons on detection and monitoring, outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

ispreview.co.uk