IT GOVERNANCE COURSES FOR CIO

Data Processing Continuity Planning

The various means of processing backup services are all important elements to the disaster recovery plan. Here we look at the most common alternate processing types:

• Mutual aid agreements

• Subscription services

• Multiple centers

• Service bureaus

• Other data center backup alternatives

Subscription Services

Another type of alternate processing scenario is presented by subscription services. In this scenario, third-party commercial services provide alternate backup and processing facilities. Subscription services are probably the most common of the alternate processing site implementations. They have very specific advantages and disadvantages, as we will see.

There are three basic forms of subscription services with some variations:

• Hot site

• Warm site

• Cold site

Hot Site

This is the Cadillac of disaster recovery alternate backup sites. A hot site is a fully configured computer facility with electrical power, heating, ventilation, and air conditioning (HVAC) and functioning file/print servers and workstations. The applications that are needed to sustain remote transaction processing are installed on the servers and workstations and are kept up-to-date to mirror the production system. Theoretically, operators and other personnel should be able to walk in and, with a data restoration of modified files from the last backup, begin full operations in a very short time. If the site participates in remote journaling - that is, mirroring transaction processing with a high-speed data line to the hot site - even the backup time may be reduced or eliminated.

This type of site requires constant maintenance of the hardware, software, data, and applications to ensure that the site accurately mirrors the state of the production site. This adds administrative overhead and can be a strain on resources, especially if a dedicated disaster recovery maintenance team does not exist.

The advantages to a hot site are numerous. The primary advantage is that 24/7 availability and exclusivity of use are ensured. The site is available immediately (or within the allowable time tolerances) after the disruptive event occurs. The site can support an outage for a short time as well as a long-term outage.

Some of the drawbacks of a hot site are as follows:

• It is seriously the most expensive of any alternative. Full redundancy of all processing components (e.g., hardware, software, communications lines, and applications) is expensive, and the services provided to support this function will not be cheap.

• It is common for the service provider to oversell its processing capabilities, betting that not all its clients will need the facilities simultaneously. This situation could create serious contention for the site’s resources if a disaster is large enough to affect a major geographic region.

• There also exists a security issue at the hot site, because the applications may contain mirrored copies of live production data. Therefore, all the security controls and mechanisms that are required at the primary site must be duplicated at the hot site. Access must be controlled, and the organization must be aware of the security methodology implemented by the service organization.

• Also, a hot site may be administratively resource-intensive because controls must be implemented to keep the data up to date and the software patched.

Warm Site

A warm site could best be described as a cross between a hot site and cold site. Like a hot site, the warm site is a computer facility readily available with electrical power, HVAC, and computers, but the applications may not be installed or configured. It may have file/print servers, but not a full complement of workstations. External communication links and other data elements that commonly take a long time to order and install will be present, however.

To enable remote processing at this type of site, workstations will have to be delivered quickly, and applications and their data will need to be restored from backup media.

The advantages to this type of site, as opposed to the hot site, are primarily as follows:

• Cost. This type of configuration will be considerably less expensive than a hot site.

• Location. Because this type of site requires less extensive control and configuration, more flexibility exists in the choice of site.

• Resources. Administrative resource drain is lower than with the maintenance of a hot site.

The primary disadvantage of a warm site, compared to a hot site, is the difference in the amount of time and effort it will take to start production processing at the new site. If extremely urgent critical transaction processing is not needed, this may be an acceptable alternative.

Cold Site

A cold site is the least ready of any of the three choices, but it is probably the most common of the three. A cold site differs from the other two in that it is ready for equipment to be brought in during an emergency, but no computer hardware (servers or workstations) resides at the site. The cold site is a room with electrical power and HVAC, but computers must be brought on-site if needed, and communications links may be ready or not. File and print servers have to be brought in, as well as all workstations, and applications will need to be installed and current data restored from backups.

A cold site is not considered an adequate resource for disaster recovery, because of the length of time required to get it going and all the variables that will not be resolved before the disruptive event. In reality, using a cold site will most likely make effective recovery impossible. It will be next to impossible to perform an in-depth disaster recovery test or to do parallel transaction processing, making it very hard to predict the success of a disaster recovery effort.

There are some advantages to a cold site, however, the primary one being cost. If an organization has very little budget for an alternative backup-processing site, the cold site may be better than nothing. Also, resource contention with other organizations will not be a problem, and neither will geographic location likely be an issue.

The big problem with this type of site is that having the cold site could engender a false sense of security. But until a disaster strikes, there’s really no way to tell whether it works or not, and by then it will be too late.

TERTIARY SITES

A tertiary site is a secondary backup site which can be used in case the primary backup site (regardless of whether it’s hot, warm, or cold) is not able to handle the recovery process or is completely unavailable. If an organization requires an extremely low MTD, or is not totally comfortable with just one backup site, a tertiary site may be designed and built.

Disaster recovery plans often get out of date. A similarity common to all recovery plans is how quickly they become obsolete, for many different reasons. The company may reorganize, and the critical business units may be different from the ones existing when the plan was first created. Most commonly, changes in the network or computing infrastructure may change the location or configuration of hardware, software, and other components. The reasons may be administrative: Complex disaster recovery plans are not easily updated, personnel lose interest in the process, or employee turnover may affect involvement.

Whatever the reason, plan maintenance techniques must be employed from the outset to ensure that the plan remains fresh and usable. It’s important to build maintenance procedures into the organization by using job descriptions that centralize responsibility for updates. Also, create audit procedures that can report regularly on the state of the plan. It’s also important to ensure that multiple versions of the plan do not exist, because they could create confusion during an emergency. Always replace older versions of the text with updated versions throughout the enterprise when a plan is changed or replaced.

Emergency management plans, business continuity plans, and disaster recovery plans should be regularly reviewed, evaluated, modified, and updated. At a minimum, the plan should be reviewed at an annual audit. The plan should also be reevaluated:

• After tests or training exercises, to adjust any discrepancies between the test results and the plan

• After a disaster response or an emergency recovery, as this is an excellent time to amend the parts of the plan that were not effective

• When personnel, their responsibilities, their resources, or organizational structures change, to familiarize new or reorganized personnel with procedures

• When polices, procedures, or infrastructures change

Reasons for Testing

In addition to the general reasons for testing that we have previously mentioned, there are several specific reasons to test, primarily to inform management of the recovery capabilities of the enterprise. Other specific reasons are as follows:

• Testing verifies the accuracy of the recovery procedures and identifies deficiencies.

• Testing prepares and trains the personnel to execute their emergency duties.

• Testing verifies the processing capability of the alternate backup site.

Creating the Test Document

To get the maximum benefit and coordination from the test, a document outlining the test scenario must be produced, containing the reasons for the test, the objectives of the test, and the type of test to be conducted (see the five following types). Also, this document should include granular details of what will happen during the test, including the following:

• The testing schedule and timing

• The duration of the test

• The specific test steps

• Who will be the participants in the test

• The task assignments of the test personnel

• The resources and services required (supplies, hardware, software, documentation, and so forth)

Certain fundamental concepts will apply to the testing procedure. Primarily, the test must not disrupt normal business functions. Also, the test should start with the easy testing types (see the following section) and gradually work up to major simulations after the recovery team has acquired testing skills.

It’s important to remember that the reason for the test is to find weaknesses in the plan. If no weaknesses were found, it was probably not an accurate test. The test is not a graded contest on how well the recovery plan or personnel executing the plan performed. Mistakes will be made, and this is the time to make them. Document the problems encountered during the test and update the plan as needed, and then test again.

TEST YOUR BACKUP REGULARLY!

If you don’t know whether the data can be retrieved quickly and accurately, or if the process has not been tested to your level of comfort, it’s not a working backup. One of us had an experience with a small New York securities firm that was in the middle of merger negotiations. Their primary server crashed, and at that point they discovered that all their backup tapes were blank; although the backup was running, no data was ever written to them. They had never tested the restore procedure. The crash was so severe that external third-party disk data restorers weren’t able to restore much data. Although some paper records existed, the value of the company tanked, and the merger failed.

The same one of us also worked with a major university that had had its e-mail system sabotaged by the recently fired systems administrator, and its backups were rendered useless. It took many weeks to build a new e-mail system, using multiple platforms, and although legal action was successfully initiated against the sysadmin, the VP of IT was forced to resign.

The Five Disaster Recovery Plan Test Types

Disaster recovery/emergency management plan testing scenarios have several levels and can be called different things, but there are generally five types of disaster recovery plan tests. The listing here is prioritized, from the simplest to the most complete testing type. As the organization progresses through the tests, each test is progressively more involved and more accurately depicts the actual responsiveness of the company. Some of the testing types, such as the last two, require major investments of time, resources, and coordination to implement. The CISSP candidate should know all of these and what they entail.

The following are the testing types:

• Checklist review. During a checklist type of disaster recovery plan, copies of the plan are distributed to each business unit’s management. The plan is then reviewed to ensure that the plan addresses all procedures and critical areas of the organization. This is considered a preliminary step to a real test and is not a satisfactory test in itself.

• Table-top exercise or structured walk-through test. In this type of test, members of the emergency management group and business unit management representatives meet in a conference room setting to discuss their responsibilities and how they would react to emergency scenarios by stepping through the plan. The goal is to ensure that the plan accurately reflects the organization’s ability to recover successfully, at least on paper. Each step of the plan is walked through in the meeting and marked as performed. Major glaring faults with the plan should be apparent during the walk-through.

• Walk-through drill or simulation test. The emergency management group and response teams actually perform their emergency response functions by walking through the test, without actually initiating recovery procedures. During a simulation test, all the operational and support personnel expected to perform during an actual emergency meet in a practice session. The goal here is to test the ability of the personnel to respond to a simulated disaster. The simulation goes to the point of relocating to the alternate backup site or enacting recovery procedures, but it does not perform any actual recovery process or alternate processing.

• Functional drill or parallel test. This type tests specific functions such as medical response, emergency notifications, warning and communications procedures, and equipment, although not necessarily all at once. This type of test also includes evacuation drills, in which personnel walk the evacuation route to a designated area where procedures for accounting for the personnel are tested. A parallel test is a full test of the recovery plan, utilizing all personnel. The goal of this type of test is to ensure that critical systems will actually run at the alternate processing backup site. Systems are relocated to the alternate site, parallel processing is initiated, and the results of the transactions and other elements are compared.

• Full-interruption or full-scale exercise. A real-life emergency situation is simulated as closely as possible. This test involves all the participants who would be responding to the real emergency, including community and external organizations. The test may involve ceasing some real production processing. The plan is totally implemented as if it were a real disaster, to the point of involving emergency services (although for a major test, local authorities might be informed and help coordinate).

 lists the five disaster recovery plan testing types in priority.

PLAN VIABILITY

Remember: The functionality of the recovery plan will directly determine the survivability of the organization. The plan shouldn’t be a document gathering dust in the CIO’s bookcase. It has to reflect the actual capability of the organization to recover from a disaster, and therefore needs to be tested regularly.

The Recovery Team

A recovery team will be clearly defined with the mandate to implement the recovery procedures at the declaration of the disaster. The recovery team’s primary task is to get the predefined critical business functions operating at the alternate backup-processing site.

Among the many tasks the recovery team will have will be the retrieval of needed materials from off-site storage - that is, backup tapes, media, workstations, and so on. When this material has been retrieved, the recovery team will install the necessary equipment and communications. The team will also install the critical systems, applications, and data required for the critical business units to resume working.

The Salvage Team

A salvage team, separate from the recovery team, will be dispatched to return the primary site to normal processing environmental conditions. It’s advisable to have a different team, because this team will have a different mandate from the recovery team. They are not involved with the same issues the recovery team is concerned with, such as creating production processing and determining the criticality of data. The salvage team has the mandate to quickly and, more importantly, safely clean, repair, salvage, and determine the viability of the primary processing infrastructure after the immediate disaster has ended.

Clearly, this cannot begin until all possibility of personal danger has ended. Firefighters or police might control the return to the site. The salvage team must identify sources of expertise, equipment, and supplies that can make the return to the site possible. The salvage team supervises and expedites the cleaning of equipment or storage media that may have suffered from smoke damage, the removal of standing water, and the drying of water-damaged media and papers.

This team is often also given the authority to declare when the site is up and running again - that is, when the resumption of normal duties can begin at the primary site. This responsibility is large, because many elements of production must be examined before the green light is given to the recovery team that operations can return.

Normal Operations Resume

This job is normally the task of the recovery team, or another, separate resumption team may be created. The plan must have full procedures on how the company will return production processing from the alternate site to the primary site with the minimum of disruption and risk. It’s interesting to note that the steps to resume normal processing operations will be different from the steps in the recovery plan; that is, the least critical work should be brought back first to the primary site.

WHEN IS A DISASTER OVER?

When is a disaster over? The answer is very important. The disaster is not over until all operations have been returned to their normal location and function. A very large window of vulnerability exists when transaction processing returns from the alternate backup site to the original production site. The disaster can be officially called over only when all areas of the enterprise are back to normal in their original home, and all data has been certified as accurate.

It’s important to note that the emergency is not over until all operations are back in full production mode at the primary site. Reoccupying the site of a disaster or emergency should not be undertaken until a full safety inspection has been done. Ideally the investigation into the cause of the emergency has been completed and all damaged property has been salvaged and restored before returning. During and after an emergency, the safety of personnel must be monitored, any remaining hazards must be assessed, and security must be maintained at the scene. After all safety precautions have been taken, an inventory of damaged and undamaged property must be done to begin salvage and restoration tasks. Also, the site must not be reoccupied until all on-site investigative processes have been completed. Detailed records must be kept of all disaster-related costs, and valuations must be made of the effect of the business interruption.^[*]

All elements discussed here involve well-coordinated logistical plans and resources. To manage and dispatch a recovery team, a salvage team, and perhaps a resumption team is a major effort, and the short descriptions we have here should not give the impression that it is not a very serious task.

Gathering Assessment Materials

The initial step of the BIA is identifying which business units are critical to continuing an acceptable level of operations. Often, the starting point is a simple organizational chart that shows the business units’ relationships to each other. Other documents may also be collected at this stage in an effort to define the functional interrelationships of the organization.

As the materials are collected and the functional operations of the business are identified, the BIA will examine these business function interdependencies with an eye toward several factors, such as determining the business success factors involved, establishing a set of priorities between the units, and deciding what alternate processing procedures can be utilized.

The Vulnerability Assessment

The vulnerability assessment is often part of a BIA. It is similar to a Risk Assessment in that there is a quantitative (financial) section and a qualitative (operational) section. It differs in that the vulnerability assessment is smaller than a full risk assessment and is focused on providing information that is used solely for the business continuity plan or disaster recovery plan.

A function of a vulnerability assessment is to conduct a loss impact analysis. Because there will be two parts to the assessment (a financial assessment and an operational assessment), it will be necessary to define loss criteria both quantitatively and qualitatively.

Quantitative loss criteria can be defined as follows:

• Incurring financial losses from loss of revenue, capital expenditure, or personal liability resolution

• The additional operational expenses incurred because of the disruptive event

• Incurring financial loss from resolution of violation of contract agreements

• Incurring financial loss from resolution of violation of regulatory or compliance requirements

Qualitative loss criteria can consist of the following:

• The loss of competitive advantage or market share

• The loss of public confidence or credibility, or incurring public embarrassment

During the vulnerability assessment, critical support areas must be defined in order to assess the impact of a disruptive event. A critical support area is defined as a business unit or function that must be present to sustain continuity of the business processes, maintain life safety, or avoid public relations embarrassment.

Critical support areas could include the following:

• Telecommunications, data communications, or information technology areas

• Physical infrastructure or plant facilities, transportation services

• Accounting, payroll, transaction processing, customer service, purchasing

The granular elements of these critical support areas will also need to be identified. By granular elements we mean the personnel, resources, and services that the critical support areas need to maintain business continuity.

Common steps to performing a vulnerability assessment could be^[*]:

1. List potential emergencies, both internally to your facility and externally to the community. Natural, man-made, technological, and human errors are all categories of potential emergencies and errors.

2. Estimate the likelihood that each emergency could occur, in a subjective analysis.

3. Assess the potential impact of the emergency on the organization in the areas of human impact (death or injury), property impact (loss or damage), and business impact (market share or credibility).

4. Assess external and internal resources required to deal with the emergency, and determine whether they are located internally or whether external capabilities or procedures are required.

Analyzing the Information

During the analysis phase of the BIA, several activities take place, such as documenting required processes, identifying interdependencies, and determining what an acceptable interruption period would be.

The goal of this section is to clearly describe what support the defined critical areas will require to preserve the revenue stream and maintain predefined processes, such as transaction processing levels and customer service levels. Therefore, elements of the analysis will have to come from many areas of the enterprise.

Documentation and Recommendation

The last step of the BIA entails a full documentation of all the processes, procedures, analyses, and results and the presentation of recommendations to the appropriate senior management.

The report will contain the previously gathered material, list the identified critical support areas, summarize the quantitative and qualitative impact statements, and provide the recommended recovery priorities generated from the analysis.

Defining the Continuity Strategy

To define the BCP strategy, the information collected from the BIA is used to create a continuity strategy for the enterprise. This task is large, and many elements of the enterprise must be included in defining the continuity strategy, such as:

• Computing. A strategy needs to be defined to preserve the elements of hardware, software, communication lines, applications, and data.

• Facilities. The strategy needs to address the use of the main buildings or campus and any remote facilities.

• People. Operators, management, and technical support personnel will have defined roles in implementing the continuity strategy.

• Supplies and equipment. Paper, forms, HVAC, or specialized security equipment must be defined as they apply to the continuity plan.

In developing plans, consideration should be given to both short-term and long-term goals and objectives. Short-term goals can include:

• Vital personnel, systems, operations, and equipment

• Priorities for restoration and mitigation

• Acceptable downtime before restoration to a minimum level of operations

• Minimum resources needed to accomplish the restoration

Long-term goals and objectives can include^[*]:

• The organization’s strategic plan

• Management and coordination of activities

• Funding and fiscal management

• Management of volunteer, contractual, and entity resources

THE INFORMATION TECHNOLOGY DEPARTMENT

The IT department plays a very important role in identifying and protecting the company’s internal and external information dependencies. Also, the information technology elements of the BCP should address several vital issues, including:

• Ensuring that the organization employs an adequate data backup and restoration process, including off-site media storage

• Ensuring that the company employs sufficient physical security mechanisms to preserve vital network and hardware components, including file and print servers

• Ensuring that the organization uses sufficient logical security methodologies (authentication, authorization, etc.) for sensitive data

• Ensuring that the department implements adequate system administration, including up-to-date inventories of hardware, software, and media storage

Documenting the Continuity Strategy

Documenting the continuity strategy simply refers to the creation of documentation of the results of the continuity strategy definition phase. You will see the word documentation a lot in this chapter. Documentation is required in almost all sections, and it is the nature of BCP/DRP to require a lot of paper.