Smoke detection causes data centre power shutdown at Wells Fargo

What happened

Wells Fargo said smoke detected during routine maintenance at one of its main data centre facilities triggered an automatic power shutdown. Applications were re-routed to back-up data centres. Online banking, mobile services and debit cards were affected, while some ATM services and branches remained operational. The company said the outage was not a cybersecurity event. A separate report said the fire department attended a Wells Fargo data centre in Minnesota after a suppression system was tripped and found no fire, but the source does not establish that this was the same event.

Contributing factors

The company said routine maintenance created a smoke condition that triggered the automatic shutdown. It did not describe the equipment or the maintenance task.

Lessons learned

1. Routine maintenance in data centres can produce conditions that activate safety shutdowns.
2. Automatic power shutdowns need tested failover if customer services are to continue.
3. Operators should say clearly which site was affected and what is still unknown.

Source

businessinsider.com