The ability to carry on business, at least at some level, in the face of a debilitating event is critical to a bank’s operations. A question I frequently hear from bank executives around budget time (when the new information technology budget in under review) is, “Why is business resumption so expensive?”
The whole issue of business resumption (sometimes called business continuity) generates for executives a mixture of discomfort, exposure, and a little fear.
To understand the expense associated with business resumption, it is first necessary to answer these three questions:
Typically, executives and business unit management (with a little board involvement) can be counted on to answer these questions. However, once they do, it is up to the IT department to handle the cost, planning and execution.
To elevate you GonzoBankers to a higher plane, here’s a bit of elaboration to help you effectively answer the three questions:
Which applications or services must be restored?
Whew, this is the easy one; let’s just make everything work. Making everything work is nice, but it’s not realistic.
Think through this scenario: A small fire that broke out in your operations facility took out the building’s electrical systems, including the ability to supply backup power to your data center. Best guess is five to 10 working days to restore reliable power to the operations center.
OK, so your bank operations center is crippled for up to two weeks. Do you believe every system must be made available in the interim? Or do you think you could “get by” with some of the applications and services and do “workarounds” for the remainder? (Gonzo Hint: “Yes” is the right answer to the second question.)
In most cases, it is possible to continue serving customers and employees without restoring every application and service. (Of course, if the outage for some reason persisted, it is unlikely that you would be willing to operate for months without most or even all of the systems available.)
A crucial step in the BR plan is to categorize systems and services. Three categories are typical:
Discrete applications for mid-size banks can easily exceed 100. Make sure there is a strong business case for every application in the critical category because cost will rise with every added application.
How fast must applications or services be restored?
On the surface, this appears the easiest question to answer. If you believe your critical applications must be available within minutes of an outage, it is likely to be the single largest contributor to high business resumption costs. If your bank wants to have one or more fault-tolerant applications, your IT group is likely discussing such topics as “hot backup site,” “real time replication” and “synchronization delay time.”
A site that provides near real time backup of critical applications is expensive to build and operate – think millions. It is important to consider whether the potential loss, financial or non-financial, of an important application exceeds the cost to build and operate this type of facility. Extending the time between failure and recovery is the single best method to reduce cost of business continuity. Would a workaround be acceptable for two to three hours or until the next morning?
Let me digress for a minute. GonzoBanker’s alter ego, Cornerstone Advisors, will soon release The Cornerstone Report: Benchmarks and Best Practices for Mid-Size Banks. This is our third mid-size bank study, and this year we are introducing a new technical statistic to reflect the rapid growth of server-based applications. Mid-size banks reported having a server for every 10 employees. That means 100 servers will likely be required to support 1,000 employees.
What does this mean for business resumption? Setting up a server on the network, loading the required software and making it function properly is a complex task. For planning purposes, it is reasonable to assume a full staff day to set up each server. If you want to bring up 30 applications at the hot site, plan to provide 30 staff days to get them all working.
How fast do you need those 30 applications? “Hmmmm, 30 applications requiring one day each and I have three server technicians. That equals 10 days!” This should make it easier to see why both the number of applications and time to recover is critical in determining cost.
What performance level is required?
Last but not least, do you expect backup to deliver the same response time? If the response time is usually five seconds or less, could your users “get by” for two weeks if it is 10 to 15 seconds? I suspect they can easily accept this degradation knowing it will be for a short period.
Once again, the requirement placed on IT is to make the backup perform at the same level as before. Make sure you really need it, though, because it is very expensive to build and maintain the required infrastructure that will deliver comparable performance.
I hope that this little primer clarifies what drives business resumption costs. Business units drive applications, timing and performance needs. IT builds the necessary architecture, staffing and plans to meet those requirements. IT’s budget holds most of the dollars and gets the brunt of the pressure to reduce BR costs.
Maybe you will never be required to implement your BR plan. However, you can get rid of that queasy feeling in your stomach if you have the right amount of protection.
-caf