Smoke from power room equipment prompts evacuation of Oracle NetSuite data centre

What happened

On 14 February 2023 smoke was reported at a data centre in Boston, United States, used by Oracle NetSuite. According to a report in The Register, cited by Network World, the smoke came from electrical equipment in a power room at the site. The same report says firefighters turned off power to the site and evacuated it. Services were disrupted from around 12:15 p.m. ET on 14 February until they were restored around 11:46 a.m.

ET on 15 February, a little over 23 hours. Customers reported on Reddit that data recorded in the roughly 30 minutes before the outage could not be recovered. A customer quoted a NetSuite statement saying the restoration point was about 30 minutes before the outage. Oracle did not detail the cause of the incident.

Contributing factors

Oracle did not detail the cause. The source does not say what failed in the power room or how the smoke started.

Lessons learned

1. Electrical equipment in power rooms needs condition monitoring and early alarms so that overheating is detected before smoke appears.
2. Operators should have emergency power isolation procedures agreed with fire services, so that a site can be made safe quickly and evacuated without delay.
3. Cloud and hosted service customers should confirm recovery point objectives in their contracts and keep independent copies of transactional data, so that a data gap of the kind reported here can be recovered.

More lessons on detection and monitoring, worker safety and safe systems of work, outages, backup supply and restoration, from every incident on the register.

Source

networkworld.com