DISASTER RECOVERY THE HANDBOOK SECOND EDITION This page intentionally left blank DISASTER RECOVERY THE HANDBOOK SECOND EDITION A Step-by-Step Plan to Ensure Business Continuity and Protect Vital Operations, Facilities, and Assets MICHAEL WALLACE LAWRENCE WEBBER American Management Association New York • Atlanta • Brussels • Chicago • Mexico City • San Francisco Shanghai • Tokyo • Toronto • Washington, D.C. Bulk discounts available. For details visit: www.amacombooks.org/go/specialsales Or contact special sales: Phone: 800-250-5308 Email: specialsls@amanet.org View all the AMACOM titles at: www.amacombooks.org This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert assistance is required, the services of a competent professional person should be sought. Library of Congress Cataloging-in-Publication Data Wallace, Michael The disaster recovery handbook : a step-by-step plan to ensure business continuity and protect vital operations, facilities, and assets / Michael Wallace, Lawrence Webber. — 2nd ed. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-8144-1613-6 ISBN-10: 0-8144-1613-6 1. Emergency management—Handbooks, manuals, etc. 2. Crisis management—Handbooks, manuals, etc. 3. Computer security—Handbooks, manuals, etc. 4. Data protection—Handbooks, manuals, etc. 5. Data recovery (Computer science)—Planning—Handbooks, manuals, etc. 6. Business planning—Handbooks, manuals, etc. I. Webber, Larry. II. Title. HD49.W36 2010 658.4'77—dc22 2010010633 © 2011 Michael Wallace and Lawrence Webber. All rights reserved. Printed in the United States of America. This publication may not be reproduced, stored in a retrieval system, or transmitted in whole or in part, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of AMACOM, a division of American Management Association, 1601 Broadway, New York, NY 10019 About AMA American Management Association (www.amanet.org) is a world leader in talent development, advancing the skills of individuals to drive business success. Our mission is to support the goals of individuals and organizations through a complete range of products and services, including classroom and virtual seminars, webcasts, webinars, podcasts, conferences, corporate and government solutions, business books and research. AMA’s approach to improving performance combines experiential learning—learning through doing—with opportunities for ongoing professional growth at every step of one’s career journey. Printing number 10 9 8 7 6 5 4 3 2 1 CONTENTS Getting Started: Overview of the Project 1 Building the Business Case: Measuring the Impact on the Business 23 Evaluating Risk: Understanding What Can Go Wrong 35 Selecting a Strategy: Setting the Direction 71 Build an Interim Plan: Don’t Just Sit There, Do Something 85 CHAPTER 6 Writing the Plan: Getting It Down on Paper 101 CHAPTER 7 Administrative Plan: Orchestrating the Recovery 115 CHAPTER 8 Crisis Management Plan: Minimizing the Damage 133 Technical Recovery Plan: Putting Humpty Dumpty Back Together Again 149 Work Area Recovery Plan: Getting the Office Up and Running 165 Pandemic Plan 179 Emergency Operations Center: Take Control of the Situation 199 CHAPTER 13 Testing Your Plans: Test, Test, Test 221 CHAPTER 14 Electrical Service: Keeping the Juice Flowing 249 Telecommunications and Networking: Your Connection to the World 267 CHAPTER 1 CHAPTER 2 CHAPTER 3 CHAPTER 4 CHAPTER 5 CHAPTER 9 CHAPTER 10 CHAPTER 11 CHAPTER 12 CHAPTER 15 CONTENTS v vi CONTENTS CHAPTER 16 Vital Records Recovery: Covering Your Assets 291 CHAPTER 17 Data: Your Most Irreplaceable Asset 315 CHAPTER 18 Workstations: The Weakest Link 335 CHAPTER 19 Customers: Other People to Worry About 349 CHAPTER 20 Suppliers: Collateral Damage 355 CHAPTER 21 Fire: Burning Down the House 365 CHAPTER 22 Human Resources: Your Most Valuable Asset 381 CHAPTER 23 Health and Safety: Keeping Everyone Healthy 399 CHAPTER 24 Terrorism: The Wrath of Man 413 Index 423 About the Authors 439 CHAPTER 1 GETTING STARTED Overview of the Project Nothing is impossible for the man who doesn’t have to do it himself. —A.H. Weiler INTRODUCTION The job of a business executive requires coordination of the many activities necessary to create a successful business. Markets must be analyzed, potential customers identified, strategies for creating and delivering products and services must be developed, financial goals established and reported, legislative mandates followed, and many different stakeholders satisfied. To ensure that all of these objectives are met, businesses eventually develop a series of processes designed to produce the desired result. But the world is a dangerous place. Earthquakes, floods, tornadoes, pandemics, snow storms, fire, and other natural disasters can strike at any time and interrupt these important processes. Terrorism, riots, arson, sabotage, and other human-created disasters can also damage your business. Accidents and equipment failures are guaranteed to happen. As an executive responsible for the well-being of your organization, it is critical that you have a plan in place to ensure that your business can continue its operations after such a disaster and to protect vital operations, facilities, and assets. You do this just like you do any other important task; you analyze the situation and create a plan. A disaster recovery plan keeps you in business after a disaster by helping to minimize the damage and allowing your organization to recover as quickly as possible. While you can’t prevent every disaster, you can with proper planning mitigate the damage and get back to work quickly and efficiently. The key is having a well thought out and up-to-date disaster recovery plan. This chapter will lead you through the creation and implementation of a project plan for creating an effective disaster recovery plan. GETTING STARTED 1 2 THE DISASTER RECOVERY HANDBOOK THE DISASTER RECOVERY PLAN PROJECT Building a disaster recovery or business continuity plan is much like any other business project. A formal project management process is necessary to coordinate the various players and company disciplines required to successfully deliver the desired results of the project. This chapter will give you a high-level roadmap of what you should expect as you prepare to lead or manage a disaster recovery project. A sample project plan is included on the CD-ROM accompanying this book. Adapt this chapter and the project plan to fit your business goals, company timeline, and scope of project. Most projects tend to run in a well-defined sequence. For example, to build a new house, first you clear the land, then build the foundation, then build a floor, and so on. Many things cannot begin until the previous step is completed. A business continuity plan (BCP) project is a bit different. In its early stages, most actions logically follow each other. However, once the basic elements are in place, the project bursts out on to parallel tracks, as each department documents its own area. How you proceed in your company is, of course, determined by your corporate culture, the resources you have to work with to complete the process, and the level of visible support from the project’s sponsor. Most business continuity projects follow these steps: 1. An executive within the organization decides that a business continuity plan is needed. This might be due to an auditor’s report or the result of a business disruption that was more painful than it would have been if a plan had been in place. Or it could be that an alert employee realized that a good plan did not exist and brought this to the executive’s attention. This executive normally becomes the sponsor for the project. 2. The first (and most important) step that the sponsor takes is to select someone to lead the project. This person is most often called the Business Continuity Manager and is responsible for the successful completion of the project. 3. The project sponsor and the Business Continuity Manager meet to clearly define the scope of the project, the project timeline, and expectations. The Business Continuity Manager must be comfortable that the resources available are adequate to meet all the objectives of the project. 4. The Business Continuity Manager selects the team that will work together to complete the project. Both technical and political considerations are important in selecting a team that can successfully develop a workable business continuity plan. 5. The Business Continuity Manager together with the team now develops the project plan to be used in managing the project. Tasks are identified and assigned, task durations calculated, and activities are sequenced as the project plans are developed. 6. The project plans are executed. The Business Continuity Manager oversees the project as the plan unfolds, keeping everyone focused on completing their GETTING STARTED 3 tasks, and ensuring that milestones are met and that important stakeholders are kept informed as to the project’s progress. It is here where the actual continuity plans for the organization are created. 7. Once the business continuity plans have been developed and tested, the Business Continuity Manager closes the project by making sure that everything was documented properly and handing the project results over to the individual(s) responsible for keeping the plan up to date. Each affected department will normally have someone responsible for keeping their portion of the plan current. A report is also generated for the sponsor recapping the project and documenting lessons learned. In many organizations, the job of Business Continuity Manager is not taken as seriously as it should be. Management in these organizations only wants you to write something, anything to make the auditors go away. That’s OK because as you build the plan, and as they begin to see the benefits, their interest and support will grow. A project plan organizes the team so members focus their skills on specific actions to get the job done. This respects their time and brings the project to a prompt, but successful, solution. INITIATING THE PROJECT Every project starts with a sponsor. A sponsor should be a person with enough organizational influence to give the project credibility, financing, and strategic direction. The sponsor should also be in a position to ensure the willing cooperation of other departments and to ensure that the project is adequately funded. Building a business continuity plan in many cases involves changing people’s attitudes and some of their tried-and-true business processes. Business continuity planning is a logical step toward mistake-proofing a business. So, to suppress the reluctance to change or even participate in the project, it is important for the sponsor to be of sufficient stature as to overcome objections before they are raised. Ideally, the sponsor is the company’s CEO, or the Vice President in charge of the local facility. However, sometimes it is a department manager who realizes that something must be done. Whoever assumes this role must remain involved with the project throughout its lifetime. As the sponsor’s interest fades, so will the interest of your team. Find out why they want to sponsor the project. It will tell you how much support to expect. In some cases, the sponsor honestly believes the project is a good idea and is personally interested in seeing it is completed. In other cases, the sponsor may have been required to start this project due to an auditor’s citation of a poor business practice. In this situation, the sponsor may only want the minimum 4 THE DISASTER RECOVERY HANDBOOK recovery plan to satisfy the audit citation. Spend some time early in the project digging out what is motivating support for this project. By understanding what motivates the sponsor, you can gauge how much time and money will be available to you. It is also possible for you to educate the sponsor on the many advantages in having a well-written company-wide plan. The sponsor’s first task is the selection of the Business Continuity Manager, who will act as the project manager. In most companies, the cynics say that if you raised the issue, then the job is yours! This isn’t a bad way to assign projects because only the people who believe in something would raise the issues. Still, the selection of the right Business Continuity Manager will help make this project a success and the wrong one will make success much more difficult to attain. The sponsor has the additional duties of approving the plan’s objectives, scope, and assumptions. The sponsor must also obtain approval for funding. THE BUSINESS CONTINUITY MANAGER The selection of the person to spearhead this project is the single most important part of building a plan. The Business Continuity Manager should be someone who can gain the willing cooperation of team members and their supervisors. To help ensure the support of everyone in the organization, the Business Continuity Manager should be publicly assigned to this task with the sponsor’s unqualified support. This is essential to overcome internal politics and to let everyone know that their assistance is important and required. As the project moves forward, regular public displays of support are required if the project is to result in a complete and usable plan. Form 1-1 on the CD-ROM is an example of a letter appointing the Business Continuity Manager. Some sponsors begin a business continuity project by hiring an outside consultant to build the plan. This can be a good way to get the project started and to mentor someone in the organization to assume the Business Continuity Manager position. Generally speaking, it takes more effort and expertise to organize and develop the plan than it does to administer it. As the plan is built, the consultant can teach the Business Continuity Manager the ropes. Understand that even though the consultant is guiding the project, the consultant should not assume the role of Business Continuity Manager. Every company, every facility, every computer site is unique. The actions necessary to promptly restore service are the result of the key people at each site writing down what to do and how to do it. Outside consultants can provide considerable insight into the basic services (electrical, telephone, water, data processing), but lack indepth experience at your company. They don’t know your business processes. They don’t understand the pulse of your business and what its key elements are. Building a solid plan will take a lot of time. An experienced consultant working with an internal Business Continuity Manager can help move the project along quicker. The Business Continuity Manager is also the logical candidate to become the plan’s ongoing administrator once the initial project is completed. This person GETTING STARTED 5 will be responsible for keeping the plan relevant and current. Writing a plan and then filing it away is a waste of money. Whoever builds the plan will be intimately familiar with it. That person can easily continue responsibility for maintaining it and teaching others how to keep their portion of it current. Using an outside consultant as a Business Continuity Manager raises the possibility that no one has internal ownership to ensure it is updated and tested periodically. The plan must be kept up to date if it is to be useful when it is needed most. As the plan administrator, the Business Continuity Manager will ensure that as new equipment enters the building, as new products are rolled out, and as new business processes are implemented, they are reflected in the business continuity plan. The Business Continuity Manager also schedules and evaluates the ongoing testing of the plan by department, or by a specific threat, such as the loss of electrical power, to ensure it works. Once the plan is written, the Business Continuity Manager’s role will evolve into ensuring the plan is an integral part of the company’s ongoing operations. No new company process or piece of equipment should begin operation until the mitigation and recovery plans have been tested and approved. SCOPE OF THE PROJECT One of the first tasks the Business Continuity Manager must perform is to come to an agreement with the project sponsor as to the scope of the project. The scope of the project defines its boundaries. It identifies what is included in the project and what is not. If the project is too vast, it will probably fail. If it is too small, then it would be best assigned to a single person like any other office detail. The scope of the project must be given a lot of thought. If in doubt, start with a narrow focus on a specific department or function to demonstrate the plan’s value and build up from there. One guideline commonly used is any event that would cost (in lost wages, sales, etc.) more than 5% of your quarterly revenues merits its own plan. So if a temporary outage of a critical machine stops the entire factory, then it needs a plan. If the same machine stoppage means that three extra workers must drill holes with hand tools until the machine is repaired, then it probably does not need a plan. A good way to approach the plan is to address areas that everyone uses, such as security, data processing, electrical, etc. Don’t try to tackle too much, too fast. Start with building services, then security and safety, then data processing, etc. In this way, if the project is killed, you still have some useful documents. If your recovery plans will encompass many sites, or a large complex, then start with a pilot project for a single building, a business function, or even for your Data Processing department. This will build your team’s expertise and confidence, 6 THE DISASTER RECOVERY HANDBOOK resulting in a very useful document, and demonstrate real value to top management. The scope of the project will drive the resource requirements for the project in terms of how many people it will involve, how long it will take, and the budget required to complete it. The project scope must be a written statement. Here are three examples with gradually narrowing requirements. As you read these scope statements, imagine what sort of implied tasks these statements carry (or as they say, “The devil is in the details!”). Follow up on the scope statement by clarifying the timelines, criteria for success, and overall expectations for this project. Otherwise, you would be digging up information and writing forever. Example #1 If you were in a factory’s Data Processing department, your scope statement might be: “Develop, implement, and provide ongoing testing for a business continuity plan for the factory’s automated systems to include the computer rooms, the internal and external telephone system, the shop floor control systems, and data connections to both internal and external sites. This plan will provide specific action steps to be taken up to and including emergency replacement of the entire computer and telecommunications rooms.” Note that this statement does not include the factory machines (drill presses, mills, conveyors, etc.) or the front offices. It is focused on the telephone system and the internal data processing processes. Example #2 If you were the Director for Building Security, your scope might be: “Write an emergency contingency plan to address the possibility of fire, personal injury, toxic material spill, and structural collapse. Include escalation procedures, emergency telephone numbers, employee education, and specific emergency actions. Make recommendations concerning potential mitigation actions to take before a disaster strikes. Ensure the plan conforms to all legal, regulatory, and insurance requirements.” The project scope described in this statement does not include flood controls, security actions, etc. Although some security tasks may be implied, very little is called for. GETTING STARTED 7 Example #3 An even narrower approach might be: “Document all the payroll procedures and recovery processes to ensure that paychecks are always on time and that the automated vacation balance tracking system is available even during an electrical outage.” Note that this scope statement does not include time clocks, exception reporting, or interfaces with your accounting system. Most people do not have any idea of what a disaster plan would look like. They imagine some large book just sitting on the shelf. In this situation, you could demonstrate the usefulness of the plan by building it a piece at a time. You might build the part that covers the core utilities for a facility (electricity, gas, telecommunications, water, and heating and air conditioning). As you review with the sponsor how these essential services will be recovered after a disaster, the sponsor will begin to see the usefulness of your work. If your company has multiple sites, it might work better for you to build the plan one site at a time. Timelines, Major Milestones, and Expectations The output of a scope statement is to build a list of goals for the project. These are specific results against which the success of the project will be judged. Detail any expectations as to a completion date or major milestone dates. If this project is in response to an internal audit item, then the due date might be when the auditor is scheduled to return. If the Board of Directors required this to be done, then progress reports might be due at every directors meeting. Ensure all key dates are identified and explain why they were selected. The term “expectations” can also be described as the criteria for success. Be clear in what you are asking for. A business continuity plan should only include critical processes. A critical process is usually defined as a process whose interruption would cause a material financial and operational impact over some period of time that you define (5% or greater of quarterly revenues is standard). You can’t plan for what to do down to the front door being stuck open. That level of detail would be too difficult to maintain. Focus on the critical business functions and the processes that support them. Your long-run goal is that the business continuity planning process will become an integral part of how business will be conducted in the future. Some example criteria for success include: ➤ Every department’s continuity plan must provide for employee and visitor safety by detailing to them any dangers associated with this device or type of technology. ➤ Each department’s continuity plan must be understandable to anyone familiar with that type of equipment or technology. 8 THE DISASTER RECOVERY HANDBOOK ➤ A business continuity plan will be submitted for every critical piece of equipment or critical process in the facility. ➤ At the end of the project, the Business Continuity Manager will submit a list of known weaknesses in the processes or equipment along with long-term recommendations to address them. ➤ All continuity plans will be tested by someone other than the plan’s author and certified by the department manager as suitable for the purpose. ➤ This project shall commence on June 1 and be completed by December 31. By that time, all plans must be complete, tested and approved by the department managers. In terms of a timeline, the length of your project will depend on how supportive the team members are of this effort, how complex your operations are, and how detailed your plan must be. Generally, these projects have an initiation phase and then the various departments break off and work in parallel to write their respective plans. During this phase, they also perform initial testing of the plan. At the end, all the plans are compared and modified so as to avoid duplicate mitigation actions and to ensure one person’s mitigation step doesn’t cause problems for someone else. The capstone event is the system-wide disaster test. As a general guideline, most plans can be completed in about 6 months, depending on the project’s scope, the degree of management support, the number of locations to be included in the plan, and the amount of resources available. One month is spent on the start-up administration and training. About 3 months are needed to draft and test the departmental plans. Be sure to stay on top of these people so they don’t forget about their plans! The final synchronization and testing should take an additional 2 months. However, as your team members are probably assigned to this project part time, their level of participation will vary according to their availability. The Business Continuity Manager must be flexible but, in the end, is responsible for driving the project to its completion. ADEQUATE FUNDING One of the indicators of the seriousness of a project is the presence of a separate budget item to support its activities. It is the Business Continuity Manager’s responsibility to track the funds spent on the project and to demonstrate the benefit they provided. If a separate budget is not available, then clear guidelines on a spending ceiling for the project must be set. Some of the items to include in the project budget are: ➤ The Business Continuity Manager and key team members should attend formal business continuity planning training to obtain a thorough grounding in its principles. This speeds the project along and removes some of the guesswork of building a plan. ➤ You may need to pay a consultant to advise the project and mentor the Business Continuity Manager as the plan is being developed. GETTING STARTED 9 ➤ Sometimes the folks with the most knowledge about your processes are not available during normal working hours. For these people, you may need to schedule meetings on weekends or offsite to gain their full attention. This may incur overtime expense or the cost of a consultant to backfill the person while they work on the plan. ➤ Temporary help might be needed for administrative assistance, such as documenting the wiring of your data networks, transcribing notes for those without the time or inclination to type, conducting an asset inventory, etc. ➤ It is amazing what a few pastries brought into a meeting can do for attendance. ➤ It is a good practice to build team spirit for the project to carry you over the rough times. This might be shirts, hats, special dinners, performance bonuses, and many other things to build team cohesion. Visible recognition helps to maintain the team’s enthusiasm. Visible Ongoing Support If the goal of this project was to determine which employees deserved to have their pay doubled, you would be inundated with folks clamoring to join your team. Unfortunately, an assignment to a business continuity planning team may not be considered a high-profile assignment. This could discourage the enthusiastic support of the very people you need to make this project a success. To minimize this possibility, the visible, vocal, and ongoing support of the sponsor is very important. Once the sponsor and the Business Continuity Manager have agreed on the scope, the sponsor should issue a formal memo appointing the Business Continuity Manager in a letter to the entire organization. This letter should inform all departments of the initiation of the project and who has been appointed to lead it. It should also describe the project’s scope, its budget or budget guidelines, and major milestones and timelines, as well as alert the other departments that they may be called upon to join the project and build their own recovery plans. This memo will detail who, what, where, when, why, and how the project will unfold. The closing paragraph should include a call for their assistance in ensuring the project will be a success. The sponsor should provide periodic updates to senior management on the progress of this project, which should include milestones met and problems that need to be overcome. Regular visibility to senior management can go a long way toward the continued support of each department with which you’ll be working. SELECTING A TEAM Once the sponsor and the coordinator have defined the scope of the project, the next step is to create a team. As you begin the project and start selecting your team, be ready for a chorus of resistance. Some departments will be indignant about being forced to join this project since they already have a plan (it’s just no 10 THE DISASTER RECOVERY HANDBOOK one can find it). Even if they have a plan, it does not mean that it is a good plan, or it may have interdependences with other areas and needs to be linked to other plans. Some will already have a plan being developed, but under scrutiny you see it has been under development for the last 10 years. So, with the naysayers in tow, prepare to select your team. In the case of existing, workable plans, ask that a liaison be appointed. For the plans under development, ask that you be able to enfranchise these hard-working people. As for any parsimonious financial people trying to kill your project’s training request, ask the sponsor to override objections and allow the team to attend training on the latest business continuity best practices. Identify the Stakeholders As you form your team, take time to identify the project’s stakeholders. A stakeholder is anyone who has a direct or indirect interest in the project. Most stakeholders just want to know what is going on with the project. Stakeholders need to be kept regularly informed about the project’s progress or problems with which they need to assist. For all stakeholders, identify their goals and motivation for this project. Based on this list, you will determine what to communicate to them, how often, and by which medium. Some stakeholders’ interests are satisfied by a monthly recap report. Some will want to hear about every minor detail. Form 1-2 (see CD) is a Stakeholder Assessment Map. Use it to keep track of what the key stakeholders are after in this project so you do not lose sight of their goals. The strategy is an acknowledgment that you may need to apply some sort of specific attention to a particular person to keep them supporting this important project. Form the Team The size and makeup of your team depends on how you will roll out the project. In the very beginning, it is best to start with a small team. Always respect people’s time. Don’t bring anyone into the project before they are needed. The initial team lays the groundwork for the project by arranging for instructors, coordinating training on building disaster plans, helping to sharpen the focus of what each plan should contain, etc. The core team should consist of the sponsor, the Business Continuity Manager, an Assistant Business Continuity Manager, and an administrative assistant. This group will prepare standards, training, and processes to make the project flow smoother. Several other key people will eventually need to join the team. You may want to bring them in early or as they are needed. This may include people such as: ➤ Building maintenance or facilities manager. They can answer what mitigation steps are already in place for the structure, fire suppression, electrical service, environmental controls, and other essential services. GETTING STARTED 11 ➤ Facility safety and security. They should already have parts of a disaster plan in terms of fire, safety, limited building and room access, theft prevention, and a host of other issues. If these plans are adequate, this may save you from writing this part of the plan. Be sure to verify that these plans are up to date and of an acceptable quality. ➤ Labor union representative. In union shops, the support of the union makes everyone’s job easier. Show leadership how a carefully created plan will help keep their members working and they will be very helpful. ➤ Human resources. The HR people have ready access to up-to-date information about the individuals who are important to the plan. ➤ Line management. These individuals tend to know the most about what is critical for getting the work done in their areas of responsibility. ➤ Community relations. A disaster may affect more than just your operations. You may need help from the surrounding community while recovering from a disaster. ➤ Public information officer. This is your voice to the outside world. The role is critical in getting accurate information out to customers and vendors when dealing with a disaster. ➤ Sales and marketing. These people know your customers the best and can provide insight on what level of service is required before customers begin to fade away. ➤ Finance and purchasing. These people know your vendors the best and can provide insight on what kind of support you can expect from vendors while recovering from a disaster. ➤ Legal. You need more than just common sense when taking action during an emergency. Your legal team can provide important insight on the legal ramifications of activities performed in response to an emergency. The next step is to make a few tool standardization decisions. The company’s technical support staff usually makes these for you. Announce to the group the standard word processing program, spreadsheet, and, most importantly, the project management software everyone will need on their workstations. Most people have the first two, but few will have the project management software already loaded. Be sure that as people join the team, copies of the software are loaded onto their workstations and training is made available on how to use this tool. You will get the best results by investing some time training team members on how to write their portion of the plan and providing administrative help if they have a lot of paperwork to write up (such as network wiring plans). Every person reacts differently to a new situation and being assigned to this team is no exception. If you will take the time to assemble a standard format for the plan and a process to follow to write it, then people will be a lot more comfortable being on the team. 12 THE DISASTER RECOVERY HANDBOOK A project of this type will generate a lot of paper. If possible, the accumulation of the various plans, wiring diagrams, manuals, etc. should be shifted from the Business Continuity Manager to an administrative assistant. An administrative assistant will also free the Business Continuity Manager from coordinating team meetings, tracking the project costs, etc. Although these tasks are clerical in nature, this person may also be the Assistant Business Continuity Manager. Another value of appointing an Assistant Business Continuity Manager is that it provides a contingency back-up person in case something happens to the Business Continuity Manager, as they will quickly learn about all aspects of the plan. Once you are ready to roll out the project plan to the world, you will need to pull in representatives from the various departments involved. When tasking the department managers to assign someone, ensure they understand that they are still responsible for having a good plan so that they send the proper person to work on the team. This person need not know every aspect of their department, but they should understand its organization, its critical hardware and software tools, and its major workflows. Depending on the project’s scope, you might end up with someone from every department in the company. This would result in too many people to motivate and keep focused at one time. Break the project down into manageable units. Start with an area you are most familiar with or that needs the most work. Involving too many people in the beginning will result in chaos. Plan on inviting in departments as you begin to review their area. An example is fire safety. Although it touches all departments, it is primarily a Safety/Security department function. Given all this, just what skills make someone a good team member? An essential skill is knowledge of the department’s processes. This allows the team member to write from personal knowledge and experience instead of spending a lot of time researching every point in the plan. Members should also know where to find the details about their departments that they don’t personally know. Another useful skill is experience with previous disasters. Even the normal problems that arise in business are useful in pointing out problem areas or documenting what has fixed a problem in the past. And of course, if they are to write a plan, they need good communications skills. Department managers should appoint a representative to the business continuity planning project team by way of a formal announcement. However, the Business Continuity Manager must approve all team members. If someone with unsuitable qualifications is sent to represent a department, they should be sent back to that manager with a request to appoint someone who is more knowledgeable about that department’s processes. When rejecting someone from the team, be sure to inform your sponsor and the originating manager as to why that person is unsuitable. The people on the initial project team are the logical ones to spread the good word of business continuity planning back to their departments. Time spent educating them on the continuity planning principles and benefits will pay off for GETTING STARTED 13 the company in the long run. They can also learn more about the company by proofreading the plans submitted by the other departments. This has an additional benefit of broadening the company perspective of a number of employees. Use Form 1-3 (see CD) to map out the responsibilities of each member of the team. Rolling Out the Project to the Team Team meetings are an opportunity to bring everyone together so they all hear the same thing at the same time. This is when you make announcements of general interest to everyone. It is also a good time to hear the problems that the team has been encountering and, if time permits, to solicit advice from the other team members on how to approach the issue. A properly managed meeting will keep the team members focused on the project and the project moving forward. In the beginning, conduct a project rollout meeting with an overview of why this project is important and an explanation of what you are looking for. This is your most critical team-building meeting (you never get a second chance to make a good first impression). In most meetings, you will work to bring out from the people their thoughts and impressions on the project. But at the first meeting, be prepared to do most of the talking. Lay out the roles of each player and set their expectations about participation in the project. Information makes the situation less uncertain and the people can begin to relax. This is your first big chance to teach, cheerlead, and inspire your team! Sell your project to them! The team members should leave the meeting with a clear idea that this project is of manageable size—not a never-ending spiral of work. Use this meeting and every meeting to informally teach them a bit about business continuity planning. As the project progresses, you will be surprised how hard it is to get business continuity information out of people. Some people are worried that others will use it to dabble with their systems. Some folks just don’t know what they would do in a disaster and intend to ad lib when something happens, just like they always have. Have patience, ask leading questions, and get them to talk. When they have declared their plan complete (and you know it is only a partial plan), conduct a meeting with the team member, their manager, and the sponsor to review the plan. Step through it item by item. By the time that meeting is over, team members will realize that they will be accountable for the quality of their plans. PLANNING THE PROJECT Refer to the sample plans included on the CD-ROM for ideas to include in your plan. Any plan that you use must be tailored to your site and management climate. Always keep your plan in a software tool like Microsoft Project. Such programs will recalculate the project’s estimated completion date as you note which tasks are complete. It can also be used to identify overallocated resources. 14 THE DISASTER RECOVERY HANDBOOK OK, now it is time to build the project plan. This is best done with input from your team. There are four basic processes to building your plan: identifying the activities, estimating how long each task will take, deciding who should do what (or what skills this person should have), and then sequencing the tasks into a logical flow of work. The general term for this is a work breakdown schedule, which describes it quite nicely. Identifying the Activities What must be done? Your core project team members can be a great help here by identifying the steps they see as necessary to complete this project. Although some tasks will logically seem to follow others, the focus here is to identify what needs to be done. How deeply you “slice and dice” each task is up to you. Unless it is a critical activity, you should rarely list any task that requires less than 8 hours of work (1 day). The times in the sample plan are calendar time, not how long the task will actually take. This is because your team members may only work on this project part time. Write a brief paragraph describing each task. This will be very useful in estimating the time required to complete it. It also keeps the task’s scope from spiraling out of control. You may understand what you mean for a task, but remember, someone else will probably execute the task, so an explanation will be very useful. Always document your planning assumptions. When discussing the plan with others later, this explanation of what you were thinking at the time the plan was drafted will be very useful. By listing your assumptions, you can discuss them point by point with the team and your sponsor to avoid areas that the plan should not address and to identify why a specific course of action was followed. Along with the assumptions, list all the known constraints for the project. This might be a specific due date to meet a business or legal obligation; it might be project funding issues or even a limit on the number of people available to be on the team. A major benefit of listing your project constraints is that upon examination they may be less than you think or can be used to prevent the scope of the project from expanding. Determining Activity Durations Once the tasks are laid out, estimate how much time should be set aside for each task to be completed. Creating reasonable time estimates for someone else is tough. You may think you know what needs to be done, but you could underestimate the true work required. Also, not everyone has your strengths—or weaknesses. Therefore, the estimates you assign at this stage are a starting point. When a task is assigned to a team member, take the time to discuss with them what each task involves and see how long they think it will require. Be sure that they understand what each task entails so they can estimate accordingly. Update the plan GETTING STARTED 15 with their estimated task durations and start dates. It is unfair to the team members to drop a task on them and demand a date without any further explanation. Once you negotiate the duration of a task with someone, encourage them to stick with it. Other people further along in the project may be depending on this task to be completed before they can start. Who Should Do It? Some tasks are easy to assign. If the task is to validate the key locker security, it will go to the security manager. If that person chooses to delegate it to someone else, then it is still his or her responsibility to ensure the task is properly completed on time. Some tasks will be more general in nature and need to be spread around the team fairly. If a task is not needed, don’t hesitate to delete it. If it is necessary, don’t hesitate to assign it! This is a good time to identify any gaps in your available labor. If you see a large time commitment for the Data Network Manager and little likelihood that team members will be available to do the assigned work, you might generate a task to bring in some temporary help to assist them. Other time issues may be on the horizon. For example, if you need to involve the Accounting Controller, and the project will run over the calendar time for closing the fiscal year accounts, then you would schedule their project participation to avoid this time period. Sequencing the Activities Now, put all the tasks in some sort of order. In this type of project, the beginning of the project is somewhat sequential. Later, many tasks will run in parallel when the various groups break off to write their respective plans. Select an estimated start date, and place some dates on your plan. With the plan held up against a calendar, check to see if any tasks need to be resequenced or if they conflict with some other critical company activity. If your task contingencies are in place, the project management software will fill in the plan dates for you. If when you save the plan you select the option to save without a baseline, you can easily change the start date later. Next, you should level your resources so one person isn’t asked to complete more than 8 hours of work in 1 day. This occurs when people are assigned too many tasks that are running simultaneously. Plan Risk Assessment So now that you have a rough plan, with time estimates and in some sort of a logical flow, it is time to scrutinize the plan for problems. Are there any labor resources overobligated? Look at each task area. What is the risk that an item won’t be completed on time? Yes, there is always a risk that a key person won’t be available. List any other underlying issues. 16 THE DISASTER RECOVERY HANDBOOK Most projects share the same basic risks to their success. In addition, each project has its own risks unique to what you are trying to accomplish and to your environment. Common project plan risks include: ➤ The amount of experience the Business Continuity Manager has in leading this type of project. Less experience adds risk to the project. Extensive experience makes for lower risk. ➤ The level of management support for the project. If you have low management support, you will have high project risk, and vice versa. ➤ Adequate funding to complete the project with a top-quality result. Don’t let needed training, support activities, or mitigation actions be cut from the budget. ➤ How many locations will this project involve at one time? The more locations that are involved, the greater the project’s risk of failure. If possible, run a separate project for each site and do not attempt to do them all at the same time. ➤ The number of departments involved with the project at one time. Like trying to work across too many sites, trying to handle too many departments will fragment the Business Continuity Manager’s time and increases the likelihood of failure. Consider tackling fewer departments at one time. ➤ The frequency and length of business interruptions to the project. This could be an upcoming ISO audit, it could be a quarterly wall-to-wall inventory, it might even be the end of the fiscal year, etc. The more interruptions to the project’s flow you can foresee, the higher the risk of failure. ➤ The time required to complete your business continuity plans will depend on the knowledge and quality of the people assigned by the various departments. Typically, the Data Processing department has the most to write and will take the longest. ➤ A mandated completion date may not be realistic. EXECUTING AND CONTROLLING Now you have your sponsor, your budget, your plan, and a core team assigned. It is time to get your project underway! A Business Continuity Manager must be the inspiring force behind the project. At those times when everyone is piling work on your team members’ desks, you must be the driving force in keeping this job as a priority project until it is finished. As the project progresses, you will make decisions as to what is included in your project charter and what is not. This “scope verification” may mean that as the project progresses, you discover that it must involve specific actions that were not foreseen when the project was started. It may also involve the “nice-to-have” things that pop up as a project moves on. In either case, recognize these things as they occur and make a conscious decision to accept or reject them. Do not let anyone else add tasks to the plan without your approval or your tightly planned project will turn into an untamed monster! GETTING STARTED 17 Communications Plan Every person within your organization has different information needs and preferred channels for receiving that information. The sponsor shouldn’t be burdened with minute details; the department managers should be responsible for tracking what their people are doing. To provide the right level of information to the right person at the appropriate time, you need to build a communications plan. The more people involved with your project, the greater your need for communication. A communications plan details who needs to report about what, and when. For example, who should receive project status reports? Who needs copies of the team meeting minutes? Who needs to know about minor project delays, etc.? To manage this, build a matrix that accounts for the information needs of all stakeholders. Your communications plan will address a wide range of audiences. Be sure to identify the person responsible for generating the communication and its major focus. Evaluate every report and every meeting in your communications plan as to whether it will be worth the effort to prepare for it. Some reports may require more effort than they are worth. Some meetings are just a waste of time. Effective communication is important for focusing a team to a goal, but you must strike a balance between enough communication and the time wasted generating too much. Use Form 1-4 (see CD) to plan who is responsible for what communications. The communications plan will encompass more than memos floating around the office. It should include meetings with your team, meetings with your sponsor, and presentations to the various departments. Another important communications task is to raise the awareness of the employees of your project and how it impacts them. Posters, newsletter articles, and open meetings all serve to answer their questions and are useful for instilling a business continuity culture in your company. The information that you need to communicate falls into three main categories: 1. Mandatory communications are things that must be done, such as status reports to the sponsor, meeting minutes to the team members, etc. Skipping a mandatory communication may affect your project’s support or credibility. 2. Informational communications include reports to the interested and curious. Many people will see the plan under development and believe that it directly or indirectly will involve them. Your informational communications will pass on project accomplishments, testing schedules, and things that may not directly affect them, but they would want to know about. Informational communications can help to shape expectations, so interested people can better understand what is next instead of being surprised or disappointed. 3. Similar to informational communications is marketing communications. Here you are out to build a positive image of your project to the rest of the company. Your marketing communications will help to educate the company as a whole on the business continuity planning principles (risk analysis, mitigation, documentation, etc.) and how they can relate to their own work processes. One effective method is to give a presentation on business recovery 18 THE DISASTER RECOVERY HANDBOOK planning to each of the various department staffs. The more they understand it, the greater your support is across the company. Form 1-5 (see CD) is a sample stakeholder reporting matrix. Modify it to reflect your project team and business requirements. In this matrix, you will identify which persons might only want to see monthly status reports with summary comments, such as the sponsor. Who might need a weekly status report with specific accomplishments, such as the department managers? Who might want short stories on accomplishments, such as the facility’s employee newsletter? The stakeholder reporting matrix also indicates the best way to deliver these reports. Do some of your executives ignore their e-mail? Do some require face-toface reports? Indicate the method of delivery to which they would be most receptive. Reporting Using the Communications Plan As the project progresses, you should occasionally revisit the project’s risk assessment. Things change; people come and go on a project; and what was once a looming challenge may at closer glance appear to be nothing at all. In addition, business conditions are in constant flux and that must also be figured into the update of your risk analysis. Controlling is the process used to identify variation from the plan in the areas of: ➤ Change control. ➤ Scope control. ➤ Cost control. ➤ Quality control. ➤ Performance reporting. ➤ Risk response. Your best tool for focusing the team on its goals will be a weekly team meeting. There are many fine books dealing with the proper way to conduct a meeting, but a few basics follow: ➤ First, always publish an agenda before the meeting. It acts as an anchor to keep people from drifting too far off the subject. ➤ Second, keep the meeting pertinent. Focus on recent achievements over the past 2 weeks and upcoming events of the next 2 weeks. ➤ Third, keep it under an hour. People lose focus the longer a meeting drones on. Side conversations should be stopped and taken outside the meeting. If you are finished in a half hour, cut it off! People will respect the meeting time limit as much as you do, so set a good example. ➤ Have your meeting at the same place and time every week, even if not much is happening. Try to make it a habit for them. GETTING STARTED 19 ➤ When planning your team meetings, involve a bit of showmanship to keep people involved. If they sit there passively, ask specific people questions, but never to embarrass them if they are late. If the discussions seem tedious, jump in once in a while to keep them focused and interesting. ➤ Use slack time in the agendas to fill in with short training topics and visits by the sponsor or department managers. ➤ Publish a meeting recap as soon after the meeting as possible. Detailed meeting minutes may become too burdensome but a recap of the high points gives you a document to talk from at the beginning of the next meeting. ➤ Always include a copy of the updated project plan. Test “Completed” Plans The quickest way to snap people out of lethargy is to publicly test the first plans submitted. You don’t need to pull the plug on a computer to do this. An easy test is to verbally walk through it. If the plan authors know that it is really going to be read and see how you test it, they will be more thorough. Do the first desktop walk-through with the plan’s author. You will uncover glossed-over steps where they clearly knew what to do but where, based on the plan, you had no clue as to what was next. After updating that version, do the same walk-through with the author’s manager (who may very well be called on to execute this plan) and look for gaps. Reward those contributors who complete their plans on time. This is where your sponsor comes in. Everyone likes to be appreciated, and some liberal rewards for the first few completed plans will go a long way toward motivating the rest of the team. You’d be surprised how fast this kind of word spreads throughout a company. Set Up and Enforce a Testing Schedule As the departmental plans roll in, update the project plan’s testing schedule. Testing will uncover gaps and inconsistencies in the current draft. Normally, this is a multiple step process: ➤ The team member and the manager initially check completed plans by using a desktop walk-through. ➤ The next level is to walk through the plan with someone familiar with the area, but not involved with the plan development. ➤ Run a departmental test. ➤ Once enough plans are ready, it is time to schedule a simulated major disaster. This might be over a holiday period or whenever the systems are lightly used. Testing will teach people some of what to expect in a disaster. It will also make them more familiar with the procedures of other functions. 20 THE DISASTER RECOVERY HANDBOOK Always follow testing or a disaster event with an “after-action” meeting and report detailing the lessons learned and updates made to the plan. Be sure to praise its high points and to privately express what it is lacking. Depending on how well your group members know one another, you can use team members for a peer evaluation. People must feel free to speak at these meetings without fear of retaliation or their full value will not be realized. After-action reviews are a very powerful learning tool. They require a moderator to keep them focused and moving through the following five questions. An after-action discussion follows a simple format: ➤ What happened? ➤ What should have happened? ➤ What went well? ➤ What went poorly? ➤ What will we do differently in the future? Appoint someone to take notes on these lessons learned. Send a copy to each participant, and the Business Continuity Manager should maintain a file of these reports. Refer to this file when updating the plan. CLOSING THE PROJECT Once you have your plan written and the initial tests are completed, it is time to close the project. All good things come to an end, as when the plan is transformed from a project to an ongoing business process. The transition involves reporting the project results to management, closing out the project’s budget, identifying known exposures for future action, and thanking your team members for their efforts. Closing the project involves the following steps: ➤ Turn all files over to the Plan Administrator. What was once your project may become someone else’s regular responsibility. If the Business Continuity Manager is not to be the Plan Administrator, accumulate all files pertaining to this project and hand them over to the Plan Administrator. It is now the administrator’s job to ensure the ongoing test plan is enforced, that plan updates are issued in a timely fashion, etc. Make a final update to the project plan. It may be useful if sister companies want to use it for building their own business continuity plans. You can also refer to it when estimating task duration for future projects. ➤ Report results to management. To wrap up your project, draft a recap of the progression of the project to management. In this, point out any major successes that occurred during the project, such as low-cost solutions found to important problems, materials found stashed away in closets that could be put to good use, and so on. In the report, be sure to point out the benefit of the cross-functional training received by the project team as they worked with each other during plan development and testing. GETTING STARTED 21 You should provide a final account of the funds spent on the project, broken down as to what part of the project they supported. This will assist in estimating the funds required for similar projects in the future. ➤ Identify known exposures. A business reality is that not every worthwhile activity can be funded. During your risk analysis and mitigation efforts, you very likely uncovered a number of areas where there were single points of failure that called for redundant solutions, unmasked obsolete equipment that must be replaced, or other mitigation actions that would make your business processes more stable. Roll up these exposures into a report to management. List each item separately along with a narrative explanation of why it is important. Detail the advantages and disadvantages of this course of action along with estimated (or known) costs. These narratives may not be reviewed again for many months, so the clearer the business reasons behind funding this action, the better. When your capital budgeting cycle rolls around, use this list as input to the budget. ➤ Thank the team. Hopefully, careful notes were kept during the course of the project so that team members could be recognized for their contributions to the project. In particular, those team members who overcame major obstacles to complete their plan and thoroughly test them are due special recognition. Acknowledgment of a job well done should be made as soon as possible after the fact. At the end of the project, it is time to again acknowledge these welldone jobs to remind everyone and management of the individual accomplishments during the project. CONCLUSION After reading this chapter, you should now have a good idea as to the overall strategy for developing a useful business continuity plan. Your odds for a successful project increase dramatically when you have a well-thought-out plan. The major steps for getting your project off to a good start are these: 1. Make sure the scope of the project is clearly defined. You need adequate time, funding, and support to be successful. 2. Carefully select the right team members. They must have a good understanding of the important processes within their departments and be able to clearly communicate the importance of the project back to their coworkers. 3. Identify the activities required, their durations, and who should do the work. 4. Communicate not only within the team but with the entire organization, as what you are doing is important for everyone’s survival. 5. Test, test, test. If a plan isn’t tested, you won’t know whether it will work until it’s too late. This page intentionally left blank CHAPTER 2 BUILDING THE BUSINESS CASE Measuring the Impact on the Business If you don’t know where you are going, any road will get you there. —Lewis Carroll INTRODUCTION Once your team is in place and the scope of your disaster recovery planning is determined, the next step is to determine exactly what vital functions need to be included in the plan. Can you easily identify the most vital functions? What happens to the business if one or more functions are suddenly unavailable due to a system failure or other disaster? What is the cost if a function is unavailable? Intuitively, some functions must be more valuable than others, but what is that value? How can this value be measured? In a time of scarce resources, which functions need to be heavily protected and which if any can be safely ignored? In a major disaster affecting many functions, which functions are essential for the company’s survival? All of these questions are pertinent. Often, decisions are based on the perceived value of a particular function when comparing two functions and the resources for only one of them is available. Capital spending, major improvement projects, and, of course, support staff training often are decided by the perceived value that a function provides the company. But what is this value based on? Where are the data that support this value? How old are the data? Has the value provided by a function changed over time? The problem with the business-as-usual approach is that it is based on a limited understanding or personal whim—not on the facts. A long-time manager might be acting on “rules-of-thumb” or assumptions that were valid at one time, but may not be any longer. A new manager lacks the “institutional knowledge” BUILDING THE BUSINESS CASE 23 24 THE DISASTER RECOVERY HANDBOOK about which previous failures have caused the greatest damage. Another caveat is that the business impact of a function changes over time. Companies compete in an ever-shifting business environment. Yesterday’s cash cow may be today’s cash drain. Yesterday’s cash drain may be today’s regulatory compliance requirement and must be working smoothly to keep the government at arm’s length! Unfortunately, few executives fully appreciate which of their functions are truly critical. They draw on personal experience, but that is limited to the areas with which they are familiar. They can ask their peers, but each person sees the world through the narrow view of his or her own situation. The accounting department will identify all of its functions as critical since it handles the money. The materials management team will identify its functions as critical since the company’s assets are reflected in a fragile collection of materials. The engineering department will think it is the most critical since its technology holds the company’s valuable intellectual property. To some extent, all of these are right! To determine where the true benefits lie, conduct a detailed Business Impact Analysis that breaks the business down by its major functions, and assigns value to each function in terms of cash flow and regulatory obligations. Then the systems that support these functions are identified and the functions rolled up. Based on this data—based on these facts—an executive can more efficiently assign resources for the greater benefit of the organization. BUSINESS IMPACT ANALYSIS A Business Impact Analysis (BIA) is an exploratory review of the important functions that are essential for the operation of the business. This review is used to quantify the value of each function to the business and to identify any risks to the most valuable functions. It also suggests mitigation actions to reduce the likelihood or impact of these risks. In the event of a disaster, the BIA indicates how much is lost per hour or per day for the length of the outage. Many of these functions are linked to an IT system that supports them (lose the IT system, and that function can no longer continue). A BIA is a snapshot of vital business functions at a given point in time. Any major changes in the operation of the business will require an update to the BIA. An organization’s critical functions depend on its primary mission. For a call center, a BIA would focus on the key telecommunication services required to service the callers. For a manufacturing firm, this might be the functions required to make the end product. A bank might identify the various financial services offered to its customers. An online store would value availability of its Web page, speed of processing, and security of customer data. And of course each department within the organization will have its own list of critical functions. BUILDING THE BUSINESS CASE 25 A BIA provides many benefits to an organization, many of which are valuable beyond the scope of a business continuity project. These include: ➤ Quantifying the tangible and qualifying the intangible costs of the loss of a critical function. ➤ Identifying the most critical functions to protect. ➤ Pinpointing the critical resources necessary for each function to operate, such as people, equipment, software, etc. ➤ Determining the recovery time objective (RTO) of critical functions. The RTO is the length of time that the organization can operate with a function disabled before the effect of the loss of the function affects other functions. ➤ Identifying vital records and the impact of their loss. ➤ Prioritizing the use of scarce resources if multiple functions are affected at the same time. There are numerous ways that the loss of a function can have a negative financial impact on the organization. The tangible financial costs of a disaster can include: ➤ Direct loss of revenue because products cannot be shipped or services not delivered. ➤ Increased waste from the spoilage of materials or finished goods. ➤ Penalties levied by customers for late shipments or lost services. ➤ Legal penalties for not conforming to government regulations or reporting requirements. Intangible costs due to the loss of a vital business function can be harder to quantify, but are no less damaging. Intangible losses can include: ➤ Loss of customer goodwill. ➤ Reduced confidence in the marketplace that your organization is a reliable supplier. ➤ Employee turnover caused by concern for the viability of the organization. ➤ Damaged image in the community if your disaster harms the local community. ➤ Loss of confidence in the organization’s executive management by key stakeholders. A well-executed BIA can provide much valuable information to executive management about the organization’s vulnerabilities. This includes: ➤ The maximum acceptable outage (MAO) that the organization can suffer before the organization will have difficulty meeting its objectives. ➤ The recovery time objective (RTO)—the amount of time that a function can be unavailable before the organization is negatively impacted—for each 26 THE DISASTER RECOVERY HANDBOOK vital function. The cost of the recovery or mitigation solution selected will typically rise as the RTO decreases. This is a major driver of your disaster recovery plan. ➤ The recovery point objective (RPO) for each function that relies on data. The RPO is the amount of data that can be lost without causing serious damage to a function. The cost of the recovery or mitigation solution selected will typically rise as the RPO decreases. Managing a BIA Project To be successful, a BIA must be run as its own project within your overall disaster recovery project. The project must be supported financially and politically from the highest levels of the organization. Every part of the organization will be touched by a BIA; it is therefore important to appoint a senior executive as the sponsor of the project. Many department heads may be reluctant to share sensitive information about their department due to legitimate concerns about the use of the information or because they are concerned that the information could be used for political purposes. The sponsor’s role is to: ➤ Work with the Business Continuity Manager to select the project manager (who could be the Business Continuity Manager). ➤ Approve the project budget. ➤ Communicate to every department the importance of its participation in the BIA. ➤ Address any objections or questions raised about the BIA. ➤ Approve the BIA report for submission to the executive team. A well-run BIA will build credibility for the overall disaster recovery planning project; a poorly run BIA will make a disaster of your disaster recovery project. The key to a successful BIA (as with any other project) is the selection of the right project manager. For a BIA it is especially important, as the BIA will expose every part of the organization to the light of day. The BIA project manager must be able to moderate discussions among department heads about the true value of internal functions. In many cases, there has been no formal examination of the functions performed within each department, which may cause heated discussions about the value of each department. In choosing a project manager, the executive sponsor has two options: 1. Internal—An employee of the organization is appointed as the project manager. The advantages of this approach are that this person already understands the corporate structure, is familiar with the personalities involved, knows where to find people, etc. This approach also builds internal expertise. A possible disadvantage is that the project manager could be caught in the middle of any political battles over the BIA, which could negatively impact the manager’s career at the organization. BUILDING THE BUSINESS CASE 27 2. External—A person from outside the organization is brought in to lead the project. The possible advantages are that this person does not have any internal ties and loyalty is to the executive paying the bill. A potential problem is that the organization’s business functions, finances, and problems will be exposed to this third party. The BIA project manager is responsible for developing a formal project plan, which is critical for the success of the project. In a large organization, many people have to be interviewed, many meetings need to be held, interim reports must be prepared, and deliverables have to be created. A formal project plan is vital for managing this process. The project plan will be used to manage the activities of the BIA team, which typically consists of several business analysts. BIA Data Collection Once the BIA team is created, the next step is to begin the data collection process. The goal of the BIA is to identify the most vital functions in the organization; just what is vital will vary depending on whom you ask. An effective data collection process will help quantify the value of each function in terms of its financial and legal impacts. The level of success of the BIA is directly related to the quality of the information collected. You cannot have a high-quality disaster recovery plan without a foundation of accurate data about your vital business functions. Your data collection plan must address what data to collect and from whom it is to be collected. It may also be important to consider when to collect the data. As this process takes people away from the important business of their departments, it is critical that the data be collected only once. Time spent in careful development of the questionnaire will save time later by only having to collect the data one time. A data collection plan consists of the following steps: 1. Identify who will receive the questionnaire using an up-to-date organization chart. 2. Develop the questionnaire to be used to collect the data from each department. Many organizations will begin with a standard form which is then modified for use. 3. Provide training to small groups (usually a department at a time) on how to respond to the questionnaire. 4. Follow up with each department to ensure timely completion of the questionnaire. 5. Review responses with respondents if the responses are not clear or are incomplete. 6. Conduct review meetings with each department to discuss responses. 7. Compile and summarize the BIA data for review by the various levels of the organization. 28 THE DISASTER RECOVERY HANDBOOK IDENTIFY RESPONDENTS The first step in identifying who should receive the BIA questionnaire is to obtain a current organizational chart. The organizational chart should identify the different departments or business units within the organization and who their leaders are. These leaders are made responsible for the completion of the questionnaire(s) for their areas. Your executive sponsor must provide you with support in ensuring their cooperation. Each department first needs to identify the vital functions performed in its area. A form such as Form 2-1, Department Function Identification Form (see the CD-ROM), can be used to develop this list. A separate function is typically identified if it has different resource requirements (e.g., IT systems or machines), staffing roles, or service providers who perform other functions in the department. Each department can have many business functions to report. Therefore, each department numbers its forms according to how many functions it is reporting. This reduces the chance of missing a questionnaire. Consider including suppliers where their activities are critical to your business. DEVELOP THE QUESTIONNAIRE At this time, you should select a single department or business unit as a test case for your questionnaire. This might be a department under the sponsor’s direct control or one where the department head has voiced support for the project. This test department can provide valuable feedback on the questionnaire, including its instructions, the clarity of the questions, or if something is missing. Often what is clear to the BIA team is obscure or has a different meaning to someone who is not familiar with the subject. Next, develop the questionnaire. Because the end result of the data collection process is the creation of an aggregated report, it is important that everyone responding to the questionnaire use important terms consistently. To ensure consistency, create a glossary of terms as part of the questionnaire. A glossary not only improves reporting consistency, but also speeds up responses and makes it obvious when something new or unexpected is encountered. The use of consistent terminology can also be enforced by using an electronic form for the questionnaire (such as an Excel spreadsheet) with checklists or dropdown lists that confine the answers to a predefined set of answers or range of numbers. If you choose this approach, have an “Other” option available for unexpected situations. Otherwise, the respondent may stop filling out the questionnaire if such a question is encountered. By allowing the choice of “Other,” you can go back later for clarification rather than have the respondent hold the questionnaire until informed about how to respond to a particular question. BUILDING THE BUSINESS CASE 29 A question can be answered in two ways: qualitatively and quantitatively. Qualitative data represent attributes for which you cannot assign a numerical value, such as color or gender. Quantitative data are represented by a numerical value, such as length of time or dollars. Quantitative data can be aggregated, averaged, etc., which makes it easier to analyze a series of responses. As much as possible, make the answers to the BIA questions quantitative; some questions are naturally quantitative, but others may need to be framed in such a way as to require a quantitative response. The BIA questionnaire begins with an identification block that indicates the department and function to which the questionnaire applies (see Form 2-2, Business Impact Analysis Questionnaire, as an example). The business function name must be the one that it is most commonly known by within the organization. When the final report is reviewed, executives will question high values for functions that no one can recognize, so be sure to use the function’s common name. The name in the function manager field will be used by the BIA team as the contact person if there are any questions. The form should also include the name of the person who completed the form and the date the form was completed. The next series of questions on the example questionnaire are designed to get a sense of the time sensitive nature of the function: Does the function have to be performed at a certain time? Can it operate at a reduced level for some period of time? How long can it be unavailable before other functions are affected? It is also important to know if this function depends on things outside the control of this department, including a dependency on any particular technology. If yes, this helps the IT department in developing its specific plans and for financial justification to purchase redundant equipment to reduce the likelihood or duration of an outage. To ensure consistency among the answers, the IT department provides a list of all applications on all platforms (desktop, server, mainframe, online). The list is included in the instructions accompanying the form. Be sure to include both the official name and the commonly used name (if one is better known). Respondents can select from this list to minimize variation of system names. This section also documents whether the function depends on outside suppliers. The next section in the example questionnaire is a matrix that is used to quantify important categories of impact (across the top) with a time scale (along the vertical axis). It is the heart of the analysis and must be tuned to the local requirements. Categories used in the example questionnaire are: 1. Cumulative Financial Loss (revenue lost plus costs incurred)—measured in dollars. This might include: a. lost revenues. b. lost sales. c. financial penalties. d. wages paid for no work. e. overtime wages paid to catch up. 30 THE DISASTER RECOVERY HANDBOOK f. spoiled materials and finished goods. 2. Legal Compliance Impact—Yes or No. For this and the following items, space is provided later for an explanation. 3. Impact on Customer Confidence—Answers can be Low, Medium, or High. 4. Loss of Supplier Confidence—Answers can be Low, Medium, or High. 5. Damaged Public Image—Answers can be Low, Medium, or High. Rate each of the impact categories according to its impact over time. For example, what is the Cumulative Financial Loss for one hour of outage? Some examples include: Example #1 If the function is a busy online catalog, then a one-hour outage might have a significant financial impact because buyers may look elsewhere for goods. Loss of customer confidence and a damaged public image would also come into play. Example #2 If the function is the shipping department for a factory, then a one-hour outage would mean that shipments would leave the dock late that day. A four-hour outage might involve shipments arriving late to the customer. Beyond four hours, late shipments would be widespread and, depending on the purchasing stipulations, may be refused by the customer. There may even be penalties for late deliveries. Also, at some point, the rest of the factory is shut down since finished goods are piled up with nowhere to go. Example #3 If the payroll department was down for an hour, then the clerks can tidy up around the office or even leave early for lunch, and the cost is minimal. However, if the same payroll department was inoperable for a week, the company may not have lost revenue but the employees definitely would be angry. If the employees belonged to a union, they might walk off the job. Other categories to consider adding to the questionnaire include: ➤ Shareholder Confidence. ➤ Loss of Financial Control. BUILDING THE BUSINESS CASE 31 ➤ Employee Morale. ➤ Customer Service. ➤ Employee Resignations. ➤ Vendor Relations. ➤ Potential Liability. ➤ Competitive Advantage. ➤ Health Hazard. ➤ Additional Cost of Credit. ➤ Additional Cost of Advertising to Rebuild Company Image and Reliability. ➤ Cost to Acquire New Software and to Re-Create Databases. ➤ Damage to Brand Image. ➤ Potential Reduction in Value of Company Stock Shares. The next section on the sample questionnaire is used to document any documents or other vital records that are critical for the success of the function. Departments that originate, use, or store vital business records must be identified. This information can be used to develop protection plans for this data. It can also identify documents that should be properly destroyed instead of stored on-site. Next on the sample questionnaire is a section in which to document critical non-IT devices that may be difficult or impossible to replace. This can spawn a project to modify the function to eliminate these unique devices (and thereby reduce the chance of a business function outage due to the failure of a special machine). The last question on the sample questionnaire offers the department an opportunity to give a subjective rating of the importance of a specific function to the overall functioning of the department. This information will be used in conjunction with the financial impact data to help prioritize the functions to be restored in the event of a disaster. Once the questions have all been determined, develop a set of written instructions to be distributed with the questionnaire. The instructions should explain how every field on the form will be used and what the respondent should fill in for each field. Ideally, include a telephone number for someone on the BIA project team to quickly answer questions; the quicker you can resolve questions the more likely the questionnaire will be completed. COLLECT THE DATA Once the questionnaire has been developed, you need to distribute it to the various departments. An important first step is to meet with each of the department leaders and help them to draft the list of vital business functions within their domains. Use this list to provide a numbered stack of questionnaires. Assign a 32 THE DISASTER RECOVERY HANDBOOK number to each person the department leaders indicate should receive one. An important management tool is a log of which form number went to which person. This is used to verify that all of the forms are returned. Next, coordinate a series of meetings with the various departments to review the questionnaire and give people a chance to ask questions. While this will be time consuming, it will speed up the process by helping to prevent the completion of the questionnaire from getting sidetracked. Try to keep the groups smaller than 20 people. This provides opportunities to ask questions. During these meetings: ➤ Explain the purpose of the BIA and how it will help the company and their department—sell the concept to them! ➤ Provide copies of the letter from the executive sponsor that supports this project; this serves to reinforce the importance of this project. ➤ If possible, ask the executive sponsor to drop by the meetings for a brief word of “encouragement.” ➤ Provide copies of the questionnaires, along with a printed explanation of what each item means. ➤ Walk through every item in the questionnaire and provide examples of how they might be filled in. ➤ Set a deadline (typically one week) for the questionnaire to be completed and returned. Check vacation and travel schedules to ensure that all respondents will be available to complete the questionnaire. If not, make sure that an appropriate substitute is identified. For collecting data from departments with a limited number of functions and highly paid employees (such as the legal department), it may be more time and cost effective to have the BIA team interview critical members of the department and fill out the questionnaires for them. As questionnaires are returned to the BIA team, carefully track which teams have returned their questionnaires. Visit any department you think might be less than diligent in filling out the questionnaires. Make the visit a friendly reminder of the deadline and use it as an opportunity to answer any questions or respond to any problems with the questionnaire. As the deadline for each department passes, visit each department that has not returned the questionnaires to see if help is needed and to encourage them to complete the form. As the forms are returned, be sure to check them for: ➤ Clarity. Ensure that you understand the answers. BUILDING THE BUSINESS CASE 33 ➤ Completeness. Return any incomplete forms and ask if department members need help in completing the questionnaire. If only a few items are missing, it is likely that they simply did not understand them. ➤ Other. Review any items answered “Other” to see if one of the existing categories may have been a fit or if a new category is needed. Reporting the Results Once all of the questionnaires have been returned, it is time to compile the reports. The reports are organized into a hierarchy of reports, starting with each business function. Depending on the size of the organization, you might have several layers between each function and the overall organization. A typical organization will use the following levels for the BIA report: 1. Function 2. Workgroup 3. Department 4. Business Unit 5. Overall Organization The example below shows a workgroup report for the A/R function within the Accounting department. Each business function is listed along the left side, with the time ranges used in the questionnaire across the top. Each column then shows the impact if that function is unavailable for that amount of time. Workgroup Report Workgroup: Accounts Receivable Cumulative Impact Business Function 1 hour 4 hours 1 day 2 days 1 week 2 weeks Generate invoices $0 $5,000 $10,000 $20,000 $100,000 $250,000 Daily cash balance $0 $0 $5,000 $15,000 $75,000 $200,000 Process checks $0 $0 $0 $0 $10,000 $30,000 Once the workgroup report is completed, you should meet with everyone who responded to the questionnaire and their next level manager. A copy of the report is provided to all participants, which is then reviewed with the group one line at a time. The entire group then must reach a consensus about each line item. The BIA analyst’s job is to remain nonjudgmental and to only guide the discussion. During this process, the collective knowledge of the group is used to correct any errors, point out any missing functions, and discuss options that may be available to reduce potential losses. 34 THE DISASTER RECOVERY HANDBOOK The amount of time a vital business function can tolerate downtime and at what cost determines the disaster recovery strategy. The less tolerant a business function is to an outage, the more expensive the disaster recovery strategy must be and the more urgent it becomes that business continuity mitigation is implemented. Every line in the report should either be validated or updated. In this way, the BIA report is the product of both the team and that workgroup’s management. The entire discussion is important, because the workgroup’s management must defend the workgroup’s consensus at the next level of data validation. This process is then repeated at the next level. If the next level is a department, then the impact of the loss of each workgroup that makes up the department is reviewed by each workgroup manager along with the manager of the department. As each team reviews its report, expect vigorous discussion about what is important and the impact on the organization. For many managers this process is very educational. Many are often surprised at the impact some functions really have and how vulnerable they are to a loss of that function. An important consequence of performing a BIA is to get the different departments at least thinking about how their functions fit within the mission of the organization, which makes improvements easier to identify. CONCLUSION After reading this chapter, you should now be able to determine which functions are vital to the success of your organization, as well as the priority in which these functions should be restored. Performing a BIA can be a tricky process politically, as each department within an organization will naturally believe that its functions are the most critical and may be hesitant to share details with someone outside of the department. A successful BIA requires the following: ➤ Strong and vocal support from senior management. ➤ A capable project leader. ➤ A well-crafted questionnaire. ➤ Complete and honest answers from each department. With a complete and accurate BIA in hand, you are now ready to begin evaluating the actual risks to your organization’s vital functions and develop a strategy for dealing with them. CHAPTER 3 EVALUATING RISK Understanding What Can Go Wrong Luck: 1a, a force that brings good fortune or adversity; 1b, the events or circumstances that operate for or against an individual; 2, favoring chance. INTRODUCTION The heart of building a business continuity plan is a thorough analysis of events from which you may need to recover. This is variously known as a threat analysis or risk assessment. The result is a list of events that could slow your company down or even shut it down. We will use this list to identify those risks your business continuity plan must address. First, let’s define the terminology we’ll use when discussing risk: ➤ The potential of a disaster occurring is called its risk. Risk is measured by how likely this is to happen and how badly it will hurt. ➤ A disaster is any event that disrupts a critical business function. This can be just about anything. ➤ A business interruption is something that disrupts the normal flow of business operations. Whether an event is a business interruption or a disaster sometimes depends on your point of view. An interruption could seem like a disaster to the people to whom it happens, but the company keeps rolling along. An example might be a purchasing department that has lost all telephone communication with its suppliers. It is a disaster to the employees because they use telephones and fax machines to issue purchase orders. The facility keeps running because their mitigation plan is to generate POs on paper and use cell phones to issue verbal material orders to suppliers. EVALUATING RISK 35 36 THE DISASTER RECOVERY HANDBOOK Risk is defined as the potential for something to occur. It could involve the possibility of personal injury or death. For example, insurance actuaries work to quantify the likelihood of an event occurring in order to set insurance rates. A risk could be an unexpected failing in the performance of duties by someone you had judged as reliable. It could be a machine failure or a spilled container of toxic material. Not all risks become realities. There is much potential in our world that does not occur. Driving to work today, I saw clouds that indicate the potential of rain. Dark clouds don’t indicate a certainty of precipitation, but they do indicate a greater potential than a clear sky. I perceive an increased risk that I will get wet on the long walk across the company parking lot, so I carry an umbrella with me. The odds are that it will not rain. The weatherman says the clouds will pass. I can even see patches of blue sky between the massive dark clouds. Still, to reduce my risk of being drenched, I carry an umbrella. Some risks can be reduced almost to the point of elimination. A hospital can install a backup generator system with the goal of ensuring 100% electrical availability. This will protect patients and staff against the risk of electrical blackout and brownouts. However, it also introduces new risks, such as the generator failing to start automatically when the electricity fails. It also does not protect the hospital against a massive electrical failure internal to the building. Some risks are unavoidable and steps can only be taken to reduce their impact. If your facility is located on the ocean with a lovely view of the sea, defenses can be built up against a tidal surge or hurricane, but you cannot prevent them. You can only minimize their damage. Some risks are localized, such as a failure of a key office PC. This event directly affects at most a few people. This is a more common risk that should not be directly addressed in the facility-wide business continuity plan. Rather, localized plans should be developed and maintained at the department level, with a copy in the company-wide master plan. These will be used mainly within a department, whose members address these challenges as they arise. If a problem is more widespread, such as a fire that burns out just those offices, all the combined small reaction plans for that office can be used to more quickly return that department to normal. Other risks can affect your entire company. An example is a blizzard that blocks the roads and keeps employees and material from your door. We all appreciate how this can slow things down, but if you are a just-in-time supplier to a company in a sunnier climate, you still must meet your daily production schedule or close your customer down! In building the list, we try to be methodical. We will examine elements in your business environment that you take for granted. Roads on which you drive. Hallways through which you walk. Even the air you breathe. In building the plan, a touch of paranoia is useful. As we go along, we will assign a score to each threat and eventually build a plan that deals with the most likely or most damaging events (see Figure 3-1). EVALUATING RISK 37 Predictability Scope Advance Warning Time of Day Risk Location Day of Week Likelihood Impact FIGURE 3-1: Attributes of risk. BUILDING A RISK ANALYSIS At this point we can differentiate among several common terms. We will begin with a risk analysis. A risk analysis is a process that identifies the probable threats to your business. As we progress, this will be used as the basis for a risk assessment. A risk assessment compares the risk analysis to the controls you have in place today to identify areas of vulnerability. The recommended approach is to assemble your business continuity planning team and perform the layers 1, 2, and 3 risk analyses (see the section below on The Five Layers of Risk) together. Your collective knowledge will make these reviews move quickly. Such things as the frequency of power or telephone outages in the past, how quickly these were resolved, and types of severe weather and its impact are all locked in the memories of the team members. 38 THE DISASTER RECOVERY HANDBOOK What Is Important to You? A risk analysis begins with a written statement of the essential functions of your business that will be used to set priorities for addressing these risks. Essential functions could be business activities, such as the availability of telephone service. It could be the flow of information, such as up-to-the-second currency exchange rates. It is anything whose absence would significantly damage the operation of your business. Most functions of a business are nonessential. You may think of your company as being tightly staffed and the work tuned to drive out waste. But think about the functions whose short-term loss would not stop your essential business from running. One example is payroll. Losing your payroll function for a few days would be inconvenient, but should not shut your business down. Most people can’t delay paying their bills for long, so over a longer period of time, this rises to the level of critical. This illustrates how a short-term noncritical function can rise to be a critical function if it is not resolved in a timely manner. Another example is a manufacturing site that states its essential functions as building, shipping, and invoicing its products. Anything that disturbs those functions is a critical problem that must be promptly addressed. All other functions that support this are noncritical to the company, although the people involved may consider them critical. On a more local scale, there may be critical functions for a department or a particular person’s job. These are also important to resolve quickly. The difference is one of magnitude. Company-wide problems have company-wide impact and must be resolved immediately. Another aspect to consider is the loss of irreplaceable assets. Imagine the loss or severe damage to vital records that must be retained for legal, regulatory, or operational reasons. Safeguarding these records must be added to your list of critical functions. Included in this category are all records whose loss would materially damage your company’s ability to conduct business. All other records are those that can be reproduced (although possibly wit...
Purchase answer to see full attachment