Emergent Chaos

I forgot exactly where I saw the link to Ben Neumann's Views from the Trenches, but the opening lines of his post "Network Outage" are great, doubly for what he's just gone through:

Today was a NIGHTMARE-DAY! Globat.com just emerged from a major outage - the worst in company history and everybody - customers and staff alike - still feel extremely beaten up. Here's what happened:

At approximately 5:00am Pacific Time on Thursday, February 21, 2008 we suffered a major network outage, which effected nearly all Globat.com customers, our own Web sites and service infrastructure as well as our phone systems.

He goes on to explain what went wrong, and what he's doing to prevent it from happening again.

This sort of thing is fairly common in computer operations. People talk about what's gone wrong. And their customers, while annoyed, prefer this to the bravado and bull they get about security incidents.

In fact, it's common in a lot of industries to have failures discussed. And while it leads to some Monday-morning quarterbacking, it also leads to operational improvement.