Scorched bus bars cause power failure at Level 3 London data centre

What happened

A fault in the electrical system took the Level 3 Braham Street data centre in London offline on 10 July 2012. The outage began at about 03:30 GMT and power was restored by 09:00. Technicians found four scorched bus bars, according to an incident report obtained by The Register. Initial reports blamed the uninterruptible power supply (UPS), but the investigation found the fault lay outside the UPS, in electrical bus-bar components.

According to the UPS supplier, the UPS units remained stable and carried the electrical load for the duration of the battery autonomy. Hosting and connectivity customers were affected, including ADSL customers of a hosting provider that connects to the BT network at the site. The facility has five floors, about 87,000 sq ft of technical space and 4MVA of site power.

Contributing factors

The source identifies scorched bus bars in the electrical distribution system, outside the UPS, as the point of failure. It does not state what caused the bus bars to scorch. The initial attribution to the UPS was corrected after investigation.

Lessons learned

1. Confirm the fault location before attributing an outage to a specific system, as initial reports named the UPS when the fault lay in bus-bar components.
2. Inspect bus-bar joints and connections and thermally scan distribution equipment, as scorching may point to a high-resistance connection or overload.
3. Record how long the UPS carried load on battery during fault isolation, to confirm autonomy and transfer performance.
4. Notify dependent customers and downstream providers early when a shared site loses power.

More lessons on outages, backup supply and restoration, electrical protection and isolation, from every incident on the register.

Source

datacenterdynamics.com