Incident Response – Suarez Style

irssi

Yesterday, after some other relaxing JKD training session in the w00dz, a #suarez #daemon style incident was awaiting one of my customers – after nearly a year w/o any trouble: A technician was literally hit by an electric arc or fault arc in one of global access’ datacenters in munich, forcing them to shut down the power completely for one hour, resulting in subsequent UPS failures, switch failures and much more trouble.

May it have been Murphy, the Daemon, or something completely different: Either way, this again proves, how vulnerable our technically equipped world has become, and how important it is to have good concepts, including failover, redundancy, and – most importantly – a regular testing of these (you see,  Murphy can become handy here as well).

One rule applies most of the time: If something bad happens, the probability of similar problems should be very low for quite some while.

Update: Another similar story from 291015 (via Slashdot):

UK contractors Balfour Beatty and Norland have been fined £380,000 ($580k) after an electrician was electrocuted while working on a data center owned by finance firm Morgan Stanley. The fine follows mounting concern that safety is being compromised because of the need for data centers to remain online non-stop. This leads to pressure for contractors to work on live power supplies.