Battery cabinet fire takes Aruba data centre offline in Arezzo

What happened

At about 04:30 on 29 April 2011 a short circuit inside the battery cabinets serving the UPS systems at Aruba's server farm in Arezzo, Italy, started a fire, according to a statement from Aruba reproduced by Punto Informatico. The fire detection system switched off the air conditioning and activated the extinguishing system. Smoke from burning battery plastic filled the building. The system read the persisting smoke as a continuing fire and automatically cut electrical power to the facility.

Firefighters worked at the site until about 08:00, but access to the rooms was possible only about two hours after the fire was extinguished because of smoke. Many websites and services hosted by Aruba, which it said held about 1.5 million domains, were unreachable for much of the day. Aruba said no servers, storage or people were harmed and no data was lost. Power was restored to the server rooms in stages during the late morning and afternoon, with the whole farm back by mid-afternoon according to Aruba. The UPS services remained out of service at the time of the reports.

Contributing factors

The batteries were installed inside the data centre building. Smoke from burning battery plastic spread through the whole facility and triggered an automatic total power cut. Smoke delayed access for responders and staff. Aruba said the UPS units, batteries and maintenance were supplied by Eaton, whose technicians were investigating the exact cause of the fault. Aruba stated that the batteries at Arezzo and its other data centres would be moved to separate rooms outside the main building.

Lessons learned

1. Keep battery cabinets for UPS systems in separate, fire-separated rooms rather than in server halls, so that a battery fire does not affect IT areas.
2. Review automatic shutdown logic so that residual smoke after fire suppression does not trigger a total power cut without operator confirmation.
3. Plan for smoke clearance and safe re-entry times, and for staged restoration of server rooms, in emergency procedures.
4. After a short-circuit fire in a battery system, inspect and replace affected strings and UPS units, and confirm the root cause with the equipment supplier before restarting.

More lessons on outages, backup supply and restoration, fire suppression and firefighting, worker safety and safe systems of work, from every incident on the register.

Source

punto-informatico.it