Unsuccessful UPS maintenance cuts power at Vocus data centre in Sydney

What happened

On 13 February 2018, shortly before 7am AEST, power was lost at Vocus's Alexandria data centre in Sydney, described in the report as a core point of presence. The report describes the outage as following "unsuccessful UPS maintenance". Servers Australia, which hosted equipment at the site, said that both the A and B power feeds to its equipment were lost, which removed redundancy, and reported substantial damage to server power supply units.

Vocus customers reported major disruption. Services were restored gradually through the morning, but a router failure within Vocus's Ipera network was still affecting business DSL services at the time of reporting. Servers Australia said it had restored the majority of its services and was working through the remaining affected list. It also said it would move its customers out of the site within weeks.

Contributing factors

The report links the outage to unsuccessful UPS maintenance. The loss of both A and B feeds meant the equipment had no redundancy at the time. The source does not say what specifically failed during the maintenance, or why the power transfer did not hold.

Lessons learned

1. Plan UPS maintenance with a documented method of procedure, a rollback plan and a clear go or no-go point before work starts.
2. Confirm that A and B power feeds stay independent during maintenance, because losing both removes redundancy.
3. Check that server power supply units and downstream equipment can tolerate a power interruption, and record damage for replacement planning.
4. Track restoration of every service against a written list, since some equipment may not recover when power returns.

More lessons on worker safety and safe systems of work, maintenance, inspection and ageing assets, outages, backup supply and restoration, from every incident on the register.

Source

itnews.com.au