Relay setting error during works cuts power at Qic Data Center, Japan

What happened

On 23 November 2019, at about 06:40, systems at the Qic Data Center in Fukuoka, Japan, went down. The data centre is run by QTnet, a Kyushu Electric Power Group company. The site was undergoing renewal of its power equipment. An overcurrent relay in the indoor receiving and distribution panel had a setting defect, which caused the normal power feed to cut out automatically. The seamless changeover to backup supply had been disabled for a planned 10-minute step of the work, and the fault occurred during that step.

Power was interrupted for about 7 seconds, which stopped servers belonging to around 260 customers. Power equipment was restored the same day. On 26 November, 16 customers were still recovering their systems.

Contributing factors

The overcurrent relay had a setting defect. The automatic changeover to backup power, which would normally avoid an interruption, had been switched off for a planned step of the works, so the fault was not covered by the backup feed during that step.

Lessons learned

1. Verify protection relay settings against the design before energising equipment after renewal work.
2. Do not disable seamless backup transfer during live works unless the step is short, pre-checked and has a fallback plan.
3. Schedule and record any temporary disabling of protection or transfer functions, and confirm their restoration before the work ends.

More lessons on electrical protection and isolation, outages, backup supply and restoration, from every incident on the register.

Source

it.impress.co.jp