Panic in the Night

The IT Manager is heading to bed on Thanksgiving evening. It’s been a good day and freezing rain is starting to come down. Perhaps he will have a good excuse to take the day off tomorrow and catch up on some things around the house. Two hours of happy dreaming later he awakes to overhear his aging but still beautiful trophy wife talking to someone on the phone:

(Wife to phone): He said to just power it off and then wait 30 seconds and power it back on.

(IT Manager): What are you doing?! Is that the computer room?

(Wife): (turning to husband aghast and pouting) You told me that’s what I should tell them if I couldn’t wake you up!

(IT Manager): That was only once after a party. Gimme the phone. Hello. Who is this?

(Night shift operator): Is that you sir? We have a problem. I can’t reach the shift supervisor and you told me not to call the Data Center Manager until she got back from maternity leave and… … … the main console just went dead, sir… it just went black!!!

(IT Manager): (thinking to self… whoa… haven’t had to do this in a while… lets try to calm down and get through this) Okay, the console server is clearly marked. It’s always been the first one in the rack on the bottom. Check to see if it still has power and if not, check the plug in, it’s probably right under the raised floor.

(Night shift operator): (Five minutes later, speaking into the phone in an extremely scared voice.) Oh my gosh! Oh my gosh!!… I raised the tile behind the server like you said and there is like six inches of water down there!! Should I do something!!

(IT Manager): (pictures of electrocuted computer operators flashing through mind) DONT TOUCH A THING! I’LL BE RIGHT THERE!

The IT Manager urgently gets dressed and heads to the Data Center. As he drives the slippery roads his mind is racing with thoughts. This is bad!! We might be down for hours or days, on the busiest shopping day of the year!! First and foremost in his thoughts is his backup employment plan of working as a sales clerk in the sporting goods department at the local Wal-Mart. But as the reality of the situation hits, he realizes this isn’t going to pay adequately for the lifestyle he (and his trophy wife) have become accustomed to. He catalogs the list of disaster recovery choices he made and things that aren’t quite ready to go yet.

1. The contract with his hot site provider provides for the mainframe backup, but it doesn’t have any servers defined in the contract. So there are no provisions for acquiring the 50 or so servers it would take to restore production at another site.
2. Even if he had the servers, he’s pretty sure that it would take days for his server guys to get them all restored. In some cases, he would bet his staff doesn’t even know how to re-install the applications because he contracted with the vendors to set them up originally.
3. The bandwidth between the branches and the hot site provider was never sized for all the upgrades they have done lately on the voice network and those new fat client mortgage loan origination apps they put in the branches.
4. He recalls the conversation he had with the ATM switch provider and the Internet banking provider about putting a backup line into the hot site provider’s site, but he had about fainted when he saw the prices, so he never set the redundant lines up. Knowing that it will take days if not weeks to get something set up now, he just shakes his head and keeps driving.

So all in all, with a real situation facing him, the IT Manager decides he really doesn’t have an actionable disaster recovery plan that is going to save his job if he has to activate it. He is going to need to figure out another way.

He soon arrives at the Data Center and assesses the situation. Since the Data Center is in the basement of the home office, there is a pump on the air handling system to get rid of the water from the air handler. The pump has failed. Probably weeks ago. Now the water had accumulated until it started tripping breakers and getting in data cables. If only he had put in a water detector under the floor!

He knows he ought to hit the emergency power off, but some of the equipment is still running and hasn’t been powered off in 13 years. He is particularly worried about the OS/2 voice response unit with the special hardware in it that can’t be bought anymore… [Crap… that’s another thing I don’t have at the hot site, he laments.]

By this time, the facility people have arrived and want him to power everything off. He tells them that they are going to help him vacuum the water and leave everything on! Citing various code and health reasons, they not so politely decline and retire to wherever facility people go when you can’t find them. Still not wanting to fry an operator (ironically, he was less worried about the facility people), the IT Manager sends the operator out for buckets, mops, wet vacs, rags and anything else that can help them get rid of six inches of water, meanwhile starting to dip five gallon buckets of water out and carrying them to a working drain.

As he goes through the process of notifying executive management, they “advise” him that he should be ready to activate the DR plan they have heard so much about at board meetings. He “advises” them that they better be prepared to be down for days and have ATMs offline for even longer if they go to another site and invites them to help him carry water.

Twenty hours later, with the last server dried out and the last circuit breaker finally back in service, batch is running again and the ATMs are back online. Customers are furious, of course, since all of the branches were offline. Fortunately the bank had its ATM switch provider stand in and most customers had access to ATMs.

The IT Manager decides that should he still have a job at the first of the week he is going to turn over a new leaf:

1. He is going to start thinking about planning for a disaster as if one is actually going to happen. (not something he would have thought of 24 hours ago)
2. He is going to be fanatical about getting and keeping his telecom infrastructure up to date so that if something like this happens again, the bandwidth and circuits will be preconfigured to run from another site.
3. He is going to get some servers pre-acquired (hey… maybe it’s time to look at those virtual server gizmo-doohickies) and he is going to make sure his server guys know how to restore every server in production.
4. He is going to go through all of his apps and make sure that they are all on current technology that he can buy and prestage hardware for, including that old IVR.
5. He is going to start power cycling the Data Center equipment on regular intervals so he can solve power supply and other start-up issues before a disaster causes him to power cycle everything.
6. He is going to work closer with the facility people to understand the age and service intervals on all of the infrastructure parts in his Data Center.

It’s now midnight of the next day as the IT Manager heads for home with aching muscles from bending over, vacuuming and carrying water. At least the roads have thawed out and he can drive home half asleep without killing himself. You would think everybody would have recognized what a hero he was since he had single handedly got the computers running again. But it seemed like everyone was mad at him. He arrives at home and climbs into bed, thinking maybe that the Wal-Mart sporting goods job might just work out after all if he could get a second job as a night clerk at a liquor store! … … oh, and better remember to order those below floor water detectors on Monday.
-br

Cornerstone Advisors can help you assess the risk in your current computing environment and develop business continuity strategies to reduce that risk.

Ask us how.