The IT Manager is heading to bed on Thanksgiving evening. It’s been a good day and freezing rain is starting to come down. Perhaps he will have a good excuse to take the day off tomorrow and catch up on some things around the house. Two hours of happy dreaming later he awakes to overhear his aging but still beautiful trophy wife talking to someone on the phone:
(Wife to phone): He said to just power it off and then wait 30 seconds and power it back on.
(IT Manager): What are you doing?! Is that the computer room?
(Wife): (turning to husband aghast and pouting) You told me that’s what I should tell them if I couldn’t wake you up!
(IT Manager): That was only once after a party. Gimme the phone. Hello. Who is this?
(Night shift operator): Is that you sir? We have a problem. I can’t reach the shift supervisor and you told me not to call the Data Center Manager until she got back from maternity leave and… … … the main console just went dead, sir… it just went black!!!
(IT Manager): (thinking to self… whoa… haven’t had to do this in a while… lets try to calm down and get through this) Okay, the console server is clearly marked. It’s always been the first one in the rack on the bottom. Check to see if it still has power and if not, check the plug in, it’s probably right under the raised floor.
(Night shift operator): (Five minutes later, speaking into the phone in an extremely scared voice.) Oh my gosh! Oh my gosh!!… I raised the tile behind the server like you said and there is like six inches of water down there!! Should I do something!!
(IT Manager): (pictures of electrocuted computer operators flashing through mind) DONT TOUCH A THING! I’LL BE RIGHT THERE!
The IT Manager urgently gets dressed and heads to the Data Center. As he drives the slippery roads his mind is racing with thoughts. This is bad!! We might be down for hours or days, on the busiest shopping day of the year!! First and foremost in his thoughts is his backup employment plan of working as a sales clerk in the sporting goods department at the local Wal-Mart. But as the reality of the situation hits, he realizes this isn’t going to pay adequately for the lifestyle he (and his trophy wife) have become accustomed to. He catalogs the list of disaster recovery choices he made and things that aren’t quite ready to go yet.
So all in all, with a real situation facing him, the IT Manager decides he really doesn’t have an actionable disaster recovery plan that is going to save his job if he has to activate it. He is going to need to figure out another way.He soon arrives at the Data Center and assesses the situation. Since the Data Center is in the basement of the home office, there is a pump on the air handling system to get rid of the water from the air handler. The pump has failed. Probably weeks ago. Now the water had accumulated until it started tripping breakers and getting in data cables. If only he had put in a water detector under the floor!
He knows he ought to hit the emergency power off, but some of the equipment is still running and hasn’t been powered off in 13 years. He is particularly worried about the OS/2 voice response unit with the special hardware in it that can’t be bought anymore… [Crap… that’s another thing I don’t have at the hot site, he laments.]
By this time, the facility people have arrived and want him to power everything off. He tells them that they are going to help him vacuum the water and leave everything on! Citing various code and health reasons, they not so politely decline and retire to wherever facility people go when you can’t find them. Still not wanting to fry an operator (ironically, he was less worried about the facility people), the IT Manager sends the operator out for buckets, mops, wet vacs, rags and anything else that can help them get rid of six inches of water, meanwhile starting to dip five gallon buckets of water out and carrying them to a working drain.
As he goes through the process of notifying executive management, they “advise” him that he should be ready to activate the DR plan they have heard so much about at board meetings. He “advises” them that they better be prepared to be down for days and have ATMs offline for even longer if they go to another site and invites them to help him carry water.
Twenty hours later, with the last server dried out and the last circuit breaker finally back in service, batch is running again and the ATMs are back online. Customers are furious, of course, since all of the branches were offline. Fortunately the bank had its ATM switch provider stand in and most customers had access to ATMs.
The IT Manager decides that should he still have a job at the first of the week he is going to turn over a new leaf:
Cornerstone Advisors can help you assess the risk in your current computing environment and develop business continuity strategies to reduce that risk.
Ask us how.