Power outage and UPS failure at SmileServ data centre in South Korea

What happened

In the early morning of 8 September 2021, a power outage hit the cloud data centre operated by SmileServ, a South Korean cloud service provider, at its IDC-Smile 1 Center, second server room. The company said harmonics stopped the static transfer switch (STS) and the uninterruptible power supply (UPS). Power to the block storage equipment was cut at the same time, and the original and replica data map information was lost.

Posts on the company's bug and improvement board said servers could not be reached and that the error had lasted more than a week. A customer told the newspaper that the company had said recovery of some data was impossible. On 14 September the company apologised to customers in a notice, said it would assess damage and pay compensation, and set out plans to improve the UPS and STS, distribute power to block storage and build an external backup for storage map data.

Contributing factors

The company attributed the failure to harmonics. An industry source quoted in the article said that a UPS failure should not cut power to the whole data centre, which points to the power design, and that dual backup of block storage would have prevented the large losses. The article also reported criticism that routine data backups had not been properly carried out.

Lessons learned

1. Test how UPS and static transfer switches behave under harmonic distortion before a fault occurs, not only under standard load conditions.
2. Feed critical storage and other key loads through redundant power paths so that a single UPS or STS fault does not cut power to them.
3. Keep an external backup of storage mapping metadata as well as customer data, so that a loss of the map does not make data unrecoverable.

More lessons on outages, backup supply and restoration, from every incident on the register.

Source

inews24.com