Load shedding outage causes cooling failure at MTN Gallo Manor data centre, Johannesburg

What happened

During Stage 4 load shedding by Eskom, South Africa's grid operator, a cooling failure was reported at MTN's Gallo Manor data centre in Johannesburg. MTN said a power outage on the Wednesday afternoon affected one of its coolers at the site. MTN said the cooling system is redundant, did not fully shut down, and was fully restored within 45 minutes. Hosting firm Afrihost, which hosts some web servers in the facility, said the cooling failure damaged certain of its hosting servers, and that its disruption began at 09:07 local time on the day before the article was published.

Afrihost said email services and websites hosted through it were offline for upwards of thirty hours. At the time of reporting, Afrihost said 12.6 percent of affected servers had recovered.

Contributing factors

Extended Stage 4 load shedding, with outages scheduled in blocks of two to four hours. MTN said its battery back-up at sites could not recharge sufficiently between outages, and that the outage schedule did not allow enough time for batteries to charge.

Lessons learned

1. Cooling plant that depends on grid power needs backup sized for repeated outages across a load-shedding schedule, not only for a single outage.
2. Redundant cooling units can limit the impact of a single unit failure, but operators should confirm recovery times and whether hosted equipment tolerates a loss of cooling.
3. Tenants in shared facilities should understand the power and cooling dependencies of the site they occupy and maintain their own continuity and customer communication plans.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

datacenterdynamics.com