Power cut during fire alarm test takes down Canadian prison data centre

What happened

On 3 October 2017, at about 6:45 p.m., a fire alarm test at a data centre used only for Correctional Service Canada systems went wrong and power to the facility was cut for three hours. The cut was caused by human error by a third party. Shared Services Canada runs the site, and neither agency would disclose its location for security reasons. The outage lasted almost 20 hours and affected email, BlackBerry service and 82 systems, including those used to manage offenders and share security intelligence with police.

Neither notices nor alarms were triggered by the shutdown, which made it harder for staff to identify the fault and plan recovery. Systems were fully restored by 3 p.m. the following day. A letter from the former commissioner of Correctional Service Canada described the failed hardware as very old and said it had never been updated. All departmental work was delayed or done manually, and landlines were used in place of BlackBerry devices and email.

Contributing factors

According to a letter from the former commissioner of Correctional Service Canada, the hardware that failed was very old and had never been updated. The shutdown raised no alarms or notices, so staff struggled to find the cause. A federal minister said earlier problems with server storage space and uptime monitoring could have been remediated more quickly, or avoided, with better monitoring by Shared Services Canada. The source does not say how the fire alarm test was carried out or why power was cut for that long.

Lessons learned

1. Fire alarm and other safety tests at data centres should be planned with a written procedure that stops any test which would remove power to critical loads, and third-party contractors should be briefed on those limits.
2. Power loss to critical systems should trigger automatic alarms and notifications, so operators do not have to discover the outage manually.
3. Legacy power and IT hardware should be on a documented refresh and monitoring programme, since ageing equipment slows recovery and raises the risk of extended outages.

More lessons on worker safety and safe systems of work, detection and monitoring, from every incident on the register.

Source

cbc.ca