5. Security Incident Handling

 < Day Day Up > 



5. Security Incident Handling

This section will supply guidance to be used before, during, and after a computer security incident occurs on a host, network, site, or multi-site environment. The operative philosophy in the event of a breach of computer security is to react according to a plan. This is true whether the breach is the result of an external intruder attack, unintentional damage, a student testing some new program to exploit software vulnerability, or a disgruntled employee. Each of the possible types of events, such as those just listed, should be addressed in advance by adequate contingency plans.

Traditional computer security, while quite important in the overall site security plan, usually pays little attention to how to actually handle an attack once one occurs. The result is that when an attack is in progress, many decisions are made in haste and can be damaging to tracking down the source of the incident, collecting evidence to be used in prosecution efforts, preparing for the recovery of the system, and protecting the valuable data contained on the system.

One of the most important but often overlooked benefits for efficient incident handling is an economic one. Having both technical and managerial personnel respond to an incident requires considerable resources. If trained to handle incidents efficiently, less staff time is required when one occurs.

Due to the worldwide network, most incidents are not restricted to a single site. Operating system vulnerabilities apply (in some cases) to several millions of systems, and many vulnerabilities are exploited within the network itself. Therefore, it is vital that all sites with involved parties are informed as soon as possible.

Another benefit is related to public relations. News about computer security incidents tends to be damaging to an organization's stature among current or potential clients. Efficient incident handling minimizes the potential for negative exposure.

A final benefit of efficient incident handling is related to legal issues. It is possible that in the near future, organizations may be held responsible if one of their nodes is used to launch a network attack. In a similar vein, people who develop patches or work-arounds may be sued if the patches or work-arounds are ineffective, resulting in compromise of the systems, or if the patches or work-arounds themselves damage systems. Knowing about operating system vulnerabilities and patterns of attacks and then taking appropriate measures to counter these potential threats is critical to circumventing possible legal problems.

The sections in this appendix provide an outline and starting point for creating your site's policy for handling security incidents:

  1. Preparing and planning (what are the goals and objectives in handling an incident?)

  2. Notification (who should be contacted in the case of an incident?)

    • Local managers and personnel

    • Law enforcement and investigative agencies

    • Computer security incidents handling teams

    • Affected and involved sites

    • Internal communications

    • Public relations and press releases

  3. Identifying an incident (is it an incident and how serious is it?)

  4. Handling (what should be done when an incident occurs?)

    • Notification (who should be notified about the incident?)

    • Protecting evidence and activity logs (what records should be kept before, during, and after the incident?)

    • Containment (how can the damage be limited?)

    • Eradication (how to eliminate the reasons for the incident?)

    • Recovery (how to reestablish service and systems?)

    • Follow up (what actions should be taken after the incident?)

  5. Aftermath (what are the implications of past incidents?)

  6. Administrative response to incidents

The remainder of this appendix will detail the issues involved in each of the important topics listed and provide some guidance as to what should be included in a site policy for handling incidents.

5.1 Preparing and Planning for Incident Handling

Part of handling an incident is being prepared to respond before the incident occurs. This includes establishing a suitable level of protections, as explained in the preceding chapters. Doing this should help your site prevent incidents as well as limit potential damage resulting from them when they do occur. Protection also includes preparing incident handling guidelines as part of a contingency plan for your organization or site. Having written plans eliminates much of the ambiguity that occurs during an incident, and will lead to a more appropriate and thorough set of responses. It is vitally important to test the proposed plan before an incident occurs through "dry runs." A team might even consider hiring a tiger team to act in parallel with the dry run. Learning to respond efficiently to an incident is important for a number of reasons:

  1. Protecting assets that could be compromised

  2. Protecting resources that could be utilized more profitably if an incident did not require their services

  3. Complying with government or other regulations

  4. Preventing the use of your systems in attacks against other systems (which could cause you to incur legal liability)

  5. Minimizing the potential for negative exposure

As in any set of preplanned procedures, attention must be paid to a set of goals for handling an incident. These goals will be prioritized differently, depending on the site. A specific set of objectives can be identified for dealing with incidents:

  1. Figure out how it happened

  2. Find out how to avoid further exploitation of the same vulnerability

  3. Avoid escalation and further incidents

  4. Assess the impact and damage of the incident

  5. Recover from the incident

  6. Update policies and procedures as needed

  7. Find out who did it (if appropriate and possible)

Due to the nature of the incident, there might be a conflict between analyzing the original source of a problem and restoring systems and services. Overall goals (such as assuring the integrity of critical systems) might be the reason for not analyzing an incident. Of course, this is an important management decision; but all involved parties must be aware that without analysis the same incident may happen again.

It is also important to prioritize the actions to be taken during an incident well in advance of the time an incident occurs. Sometimes an incident may be so complex that it is impossible to do everything at once to respond to it; priorities are essential. Although priorities will vary from institution to institution, the following suggested priorities may serve as a starting point for defining your organization's response:

  1. Protect human life and people's safety; human life always has precedence over all other considerations.

  2. Protect classified and sensitive data. Prevent exploitation of classified and sensitive systems, networks, or sites. Inform affected classified and sensitive systems, networks, or sites about penetrations that have already occurred. Be aware of regulations by your site or by government.

  3. Protect proprietary, scientific, managerial, and other data because loss of data is costly in terms of resources. Prevent exploitations of other systems, networks, or sites and inform already affected systems, networks, or sites about successful penetrations.

  4. Prevent damage to systems (e.g., loss or alteration of system files, damage to disk drives, etc.). Damage to systems can result in costly downtime and recovery.

  5. Minimize disruption of computing resources (including processes). It is better in many cases to shut a system down or disconnect from a network than to risk damage to data or systems. Sites will have to evaluate the trade-offs between shutting down and disconnecting, and staying up. There may be service agreements in place that may require keeping systems up even in light of further damage occurring. However, the damage and scope of an incident may be so extensive that service agreements may have to be overridden.

An important implication for defining priorities is that once human life and national security considerations have been addressed, it is generally more important to save data than system software and hardware. Although it is undesirable to have any damage or loss during an incident, systems can be replaced. However, the loss or compromise of data (especially classified or proprietary data) is usually not an acceptable outcome under any circumstances.

Another important concern is the effect on others beyond the systems and networks where the incident occurs. Within the limits imposed by government regulations, it is always important to inform affected parties as soon as possible. Due to the legal implications of this topic, it should be included in the planned procedures to avoid further delays and uncertainties for the administrators.

Any plan for responding to security incidents should be guided by local policies and regulations. Government and private sites that deal with classified material have specific rules that they must follow.

The policies chosen by your site that determine how it reacts to incidents will shape your response. For example, it may make little sense to create mechanisms to monitor and trace intruders if your site does not plan to take action against the intruders if they are caught. Other organizations may have policies that affect your plans. Telephone companies often release information about telephone traces to law enforcement agencies only.

Handling incidents can be tedious and requires any number of routine tasks that could be handled by support personnel. To free the technical staff, it may be helpful to identify support staff that will help with tasks such as photocopying, faxing, etc.

5.2 Notification and Points of Contact

It is important to establish contacts with various personnel before a real incident occurs. Many times, incidents are not real emergencies. Indeed, often you will be able to handle the activities internally. However, there will also be many times when others outside your immediate department will need to be included in the incident handling. These additional contacts include local managers and system administrators, administrative contacts for other sites on the Internet, and various investigative organizations. Getting to know these contacts before incidents occur will help to make your incident handling process more efficient.

For each type of communication contact, specific points of contact (POC) should be defined. These may be technical or administrative in nature and may include legal or investigative agencies as well as service providers and vendors. When establishing these contacts, it is important to decide how much information will be shared with each class of contact. It is especially important to define ahead of time what information will be shared with the users at a site, with the public (including the press), and with other sites.

Settling these issues is especially important for the local person responsible for handling the incident, because that is the person responsible for the actual notification of others. A list of contacts in each of these categories is an important time-saver for this person during an incident. It can be quite difficult to find an appropriate person during an incident when many urgent events are ongoing. It is strongly recommended that all relevant telephone numbers (also electronic mail addresses and fax numbers) are included in the site security policy. The names and contact information of all individuals who will be directly involved in the handling of an incident should be placed at the top of this list.

5.2.1 Local Managers and Personnel

When an incident is underway, a major issue is deciding who is in charge of coordinating the activity of the multitude of players. A major mistake that can be made is to have a number of people who are each working independently but are not working together. This will only add to the confusion of the event and will probably lead to wasted or ineffective effort.

The single POC may or may not be the person responsible for handling the incident. There are two distinct roles to fill when deciding who will be the POC and who will be the person in charge of the incident. The person in charge of the incident will make decisions as to the interpretation of policy applied to the event. In contrast, the POC must coordinate the effort of all the parties involved with handling the event.

The POC must be a person with the technical expertise to successfully coordinate the efforts of the system managers and users involved in monitoring and reacting to the attack. Care should be taken when identifying who this person will be. It should not necessarily be the same person who has administrative responsibility for the compromised systems because such administrators often have knowledge sufficient for the day-to-day use of the computers but lack in-depth technical expertise.

Another important function of the POC is to maintain contact with law enforcement and other external agencies to assure that multi-agency involvement occurs. The level of involvement will be determined by management decisions as well as legal constraints.

A single POC should also be the single person in charge of collecting evidence because, as a rule of thumb, the more people that touch a potential piece of evidence, the greater the possibility that it will be inadmissible in court. To ensure that evidence will be acceptable to the legal community, collecting evidence should be done following predefined procedures in accordance with local laws and legal regulations.

One of the most critical tasks for the POC is the coordination of all relevant processes. Responsibilities may be distributed over the whole site, involving multiple independent departments or groups. This will require a well-coordinated effort in order to achieve overall success. The situation becomes even more complex if multiple sites are involved. When this happens, rarely will a single POC at one site be able to adequately coordinate the handling of the entire incident. Instead, appropriate incident response teams should be involved.

The incident handling process should provide some escalation mechanisms. In order to define such a mechanism, sites will need to create an internal classification scheme for incidents. Associated with each level of incident will be the appropriate POC and procedures. As an incident is escalated, there may be a change in the POC that will need to be communicated to all others involved in handling the incident. When a change in the POC occurs, the old POC should brief the new POC in all background information.

Finally, users must know how to report suspected incidents. Sites should establish reporting procedures that will work both during and outside normal working hours. Help desks are often used to receive these reports during normal working hours, while beepers and telephones can be used for out-of-hours reporting.

5.2.2 Law Enforcement and Investigative Agencies

In the event of an incident that has legal consequences, it is important to establish contact with investigative agencies (e.g., the FBI and Secret Service in the U.S.) as soon as possible. Local law enforcement, local security offices, and campus police departments should also be informed as appropriate. This section describes many of the issues that will be confronted, but it is acknowledged that each organization will have its own local and governmental laws and regulations that will impact how they interact with law enforcement and investigative agencies. The most important point to make is that each site needs to work through these issues.

A primary reason for determining these points of contact well in advance of an incident is that once a major attack is in progress, there is little time to call these agencies to determine exactly who the correct point of contact is. Another reason is that it is important to cooperate with these agencies in a manner that will foster a good working relationship, and that will be in accordance with the working procedures of these agencies. Knowing the working procedures in advance, and the expectations of your point of contact is a big step in this direction. For example, it is important to gather evidence that will be admissible in any subsequent legal proceedings, and this will require prior knowledge of how to gather such evidence. A final reason for establishing contacts as soon as possible is that it is impossible to know the particular agency that will assume jurisdiction in any given incident. Making contacts and finding the proper channels early on will make responding to an incident go considerably more smoothly.

If your organization or site has a legal counsel, you need to notify this office soon after you learn that an incident is in progress. At a minimum, your legal counsel needs to be involved to protect the legal and financial interests of your site or organization. There are many legal and practical issues, a few of which are:

  1. Whether your site or organization is willing to risk negative publicity or exposure to cooperate with legal prosecution efforts.

  2. Downstream liability - if you leave a compromised system as is so it can be monitored and another computer is damaged because the attack originated from your system, your site or organization may be liable for damages incurred.

  3. Distribution of information - if your site or organization distributes information about an attack in which another site or organization may be involved or the vulnerability in a product that may affect ability to market that product, your site or organization may again be liable for any damages (including damage of reputation).

  4. Liabilities due to monitoring - your site or organization may be sued if users at your site or elsewhere discover that your site is monitoring account activity without informing users.

Unfortunately, there are no clear precedents yet on the liabilities or responsibilities of organizations involved in a security incident or who might be involved in supporting an investigative effort. Investigators will often encourage organizations to help trace and monitor intruders. Indeed, most investigators cannot pursue computer intrusions without extensive support from the organizations involved. However, investigators cannot provide protection from liability claims, and these kinds of efforts may drag out for months and may take a lot of effort.

On the other hand, an organization's legal council may advise extreme caution and suggest that tracing activities be halted and an intruder shut out of the system. This, in itself, may not provide protection from liability, and may prevent investigators from identifying the perpetrator.

The balance between supporting investigative activity and limiting liability is tricky. You will need to consider the advice of your legal counsel and the damage the intruder is causing (if any) when making your decision about what to do during any particular incident.

Your legal counsel should also be involved in any decision to contact investigative agencies when an incident occurs at your site. The decision to coordinate efforts with investigative agencies is most properly that of your site or organization. Involving your legal counsel will also foster the multi-level coordination between your site and the particular investigative agency involved, which in turn results in an efficient division of labor. Another result is that you are likely to obtain guidance that will help you avoid future legal mistakes.

Finally, your legal counsel should evaluate your site's written procedures for responding to incidents. It is essential to obtain a "clean bill of health" from a legal perspective before you actually carry out these procedures.

It is vital, when dealing with investigative agencies, to verify that the person who calls asking for information is a legitimate representative from the agency in question. Unfortunately, many well intentioned people have unknowingly leaked sensitive details about incidents, allowed unauthorized people into their systems, etc., because a caller has masqueraded as a representative of a government agency. [4]

A similar consideration is using a secure means of communication. Because many network attackers can easily reroute electronic mail, avoid using electronic mail to communicate with other agencies (as well as others dealing with the incident at hand). Nonsecured phone lines (the phones normally used in the business world) are also frequent targets for tapping by network intruders, so be careful!

There is no one established set of rules for responding to an incident when the local government becomes involved. Normally (in the U.S.), except by legal order, no agency can force you to monitor, to disconnect from the network, to avoid telephone contact with the suspected attackers, etc. Each organization will have a set of local and national laws and regulations that must be adhered to when handling incidents. It is recommended that each site be familiar with those laws and regulations, and identify and get to know the contacts for agencies with jurisdiction well in advance of handling an incident.

5.2.3 Computer Security Incident Handling Teams

There are currently a number of Computer Security Incident Response Teams (CSIRTs) such as the CERT Coordination Center, the German DFN-CERT, and other teams around the globe. Teams exist for many major government agencies and large corporations. If such a team is available, notifying it should be of primary consideration during the early stages of an incident. These teams are responsible for coordinating computer security incidents over a range of sites and larger entities. Even if the incident is believed to be contained within a single site, it is possible that the information available through a response team could help in fully resolving the incident.

If it is determined that the breach occurred due to a flaw in the system's hardware or software, the vendor (or supplier) and a Computer Security Incident Handling team should be notified as soon as possible. This is especially important because many other systems are vulnerable, and these vendor and response team organizations can help disseminate help to other affected sites.

In setting up a site policy for incident handling, it may be desirable to create a subgroup, much like those teams that already exist, that will be responsible for handling computer security incidents for the site (or organization). If such a team is created, it is essential that communication lines be opened between this team and other teams. Once an incident is underway, it is difficult to open a trusted dialogue between other teams if none had existed before.

5.2.4 Affected and Involved Sites

If an incident has an impact on other sites, it is good practice to inform them. It may be obvious from the beginning that the incident is not limited to the local site, or it may emerge only after further analysis.

Each site may choose to contact other sites directly or it can pass the information to an appropriate incident response team. It is often very difficult to find the responsible POC at remote sites and the incident response team will be able to facilitate contact by making use of already established channels.

The legal and liability issues arising from a security incident will differ from site to site. It is important to define a policy for the sharing and logging of information about other sites before an incident occurs.

Information about specific people is especially sensitive, and may be subject to privacy laws. To avoid problems in this area, irrelevant information should be deleted and a statement of how to handle the remaining information should be included. A clear statement of how this information is to be used is essential. No one who informs a site of a security incident wants to read about it in the public press. Incident response teams are valuable in this respect. When they pass information to responsible POCs, they are able to protect the anonymity of the original source. But, be aware that, in many cases, the analysis of logs and information at other sites will reveal addresses of your site.

All the problems discussed above should not be taken as reasons not to involve other sites. In fact, the experiences of existing teams reveal that most sites informed about security problems are not even aware that their site had been compromised. Without timely information, other sites are often unable to take action against intruders.

5.2.5 Internal Communications

It is crucial during a major incident to communicate why certain actions are being taken, and how the users (or departments) are expected to behave. In particular, it should be made very clear to users what they are allowed to say (and not say) to the outside world (including other departments). For example, it would not be good for an organization if users replied to customers with something like, "I'm sorry the systems are down, we've had an intruder and we are trying to clean things up." It would be much better if they were instructed to respond with a prepared statement like, "I'm sorry our systems are unavailable, they are being maintained for better service in the future."

Communications with customers and contract partners should be handled in a sensible but sensitive way. One can prepare for the main issues by preparing a checklist. When an incident occurs, the checklist can be used with the addition of a sentence or two for the specific circumstances of the incident.

Public relations departments can be very helpful during incidents. They should be involved in all planning and can provide well-constructed responses for use when contact is necessary with outside departments and organizations.

5.2.6 Public Relations: Press Releases

There has been a tremendous growth in the amount of media coverage dedicated to computer security incidents in the United States. Such press coverage is bound to extend to other countries as the Internet continues to grow and expand internationally. Readers from countries where such media attention has not yet occurred, can learn from the experiences in the United States and should be forewarned and prepared.

One of the most important issues to consider is when, who, and how much to release to the general public through the press. There are many issues to consider when deciding this particular issue. First and foremost, if a public relations office exists for the site, it is important to use this office as liaison to the press. The public relations office is trained in the type and wording of information released, and will help to assure that the image of the site is protected during and after the incident (if possible). A public relations office has the advantage that you can communicate candidly with them, and provide a buffer between the constant press attention and the need of the POC to maintain control over the incident.

If a public relations office is not available, the information released to the press must be carefully considered. If the information is sensitive, it may be advantageous to provide only minimal or overview information to the press. It is quite possible that any information provided to the press will be quickly reviewed by the perpetrator of the incident. Also note that misleading the press can often backfire and cause more damage than releasing sensitive information.

While it is difficult to determine in advance what level of detail to provide to the press, some guidelines to keep in mind are:

  1. Keep the technical level of detail low. Detailed information about the incident may provide enough information for others to launch similar attacks on other sites, or even damage the site's ability to prosecute the guilty party once the event is over.

  2. Keep the speculation out of press statements. Speculation of who is causing the incident or the motives is very likely to be in error and may cause an inflamed view of the incident.

  3. Work with law enforcement professionals to assure that evidence is protected. If prosecution is involved, assure that the evidence collected is not divulged to the press.

  4. Try not to be forced into a press interview before you are prepared. The popular press is famous for the "2 a.m." interview, where the hope is to catch the interviewee off guard and obtain information otherwise not available.

  5. Do not allow the press attention to detract from the handling of the event. Always remember that the successful closure of an incident is of primary importance.

5.3 Identifying an Incident

5.3.1 Is It Real?

This stage involves determining if a problem really exists. Of course many if not most signs often associated with virus infection, system intrusions, malicious users, etc., are simply anomalies such as hardware failures or suspicious system/user behavior. To assist in identifying whether there really is an incident, it is usually helpful to obtain and use any detection software that may be available. Audit information is also extremely useful, especially in determining whether there is a network attack. It is extremely important to obtain a system snapshot as soon as one suspects that something is wrong. Many incidents cause a dynamic chain of events to occur, and an initial system snapshot may be the most valuable tool for identifying the problem and any source of attack. Finally, it is important to start a log book. Recording system events, telephone conversations, time stamps, etc., can lead to a more rapid and systematic identification of the problem, and is the basis for subsequent stages of incident handling.

There are certain indications or "symptoms" of an incident that deserve special attention:

  1. System crashes

  2. New user accounts (the account RUMPELSTILTSKIN has been unexpectedly created) or high activity on a previously low usage account

  3. New files (usually with novel or strange file names, such as data.xx or k.xx)

  4. Accounting discrepancies (in a UNIX system you might notice the shrinking of an accounting file called/usr/admin/lastlog, something that should make you very suspicious that there may be an intruder)

  5. Changes in file lengths or dates (a user should be suspicious if .EXE files in an MS DOS computer have inexplicably grown by over 1800 bytes)

  6. Attempts to write to system (a system manager notices that a privileged user in a VMS system is attempting to alter rightslist.dat)

  7. Data modification or deletion (files start to disappear)

  8. Denial of service (a system manager and all other users become locked out of a UNIX system, now in single user mode)

  9. Unexplained, poor system performance

  10. Anomalies ("GOTCHA" is displayed on the console or there are frequent unexplained "beeps")

  11. Suspicious probes (there are numerous unsuccessful login attempts from another node)

  12. Suspicious browsing (someone becomes a root user on a UNIX system and accesses file after file on many user accounts

  13. Inability of a user to log in due to modifications of his account

By no means is this list comprehensive; we have just listed a number of common indicators. It is best to collaborate with other technical and computer security personnel to make a decision as a group about whether an incident is occurring.

5.3.2 Types and Scope of Incidents

Along with the identification of the incident is the evaluation of the scope and impact of the problem. It is important to correctly identify the boundaries of the incident in order to effectively deal with it and prioritize responses.

In order to identify the scope and impact, a set of criteria should be defined that is appropriate to the site and to the type of connections available. Some of the issues include:

  1. Is this a multi-site incident?

  2. Are many computers at your site affected by this incident?

  3. Is sensitive information involved?

  4. What is the entry point of the incident (network, phone line, local terminal, etc.)?

  5. Is the press involved?

  6. What is the potential damage of the incident?

  7. What is the estimated time to close out the incident?

  8. What resources could be required to handle the incident?

  9. Is law enforcement involved?

5.3.3 Assessing the Damage and Extent

The analysis of the damage and extent of the incident can be quite time consuming but should lead to some insight into the nature of the incident, and aid investigation and prosecution. As soon as the breach has occurred, the entire system and all of its components should be considered suspect. System software is the most probable target. Preparation is crucial to be able to detect all changes for a possibly tainted system. This includes checksumming all media from the vendor using an algorithm that is resistant to tampering.

Assuming original vendor distribution media is available, an analysis of all system files should commence, and any irregularities should be noted and referred to all parties involved in handling the incident. It can be very difficult, in some cases, to decide which backup media are showing a correct system status. Consider, for example, that the incident may have continued for months or years before discovery, and the suspect may be an employee of the site, or otherwise have intimate knowledge or access to the systems. In all cases, the pre-incident preparation will determine what recovery is possible.

If the system supports centralized logging (most do), go back over the logs and look for abnormalities. If process accounting and connect time accounting are enabled, look for patterns of system usage. To a lesser extent, disk usage may shed light on the incident. Accounting can provide much helpful information in an analysis of an incident and subsequent prosecution. Your ability to address all aspects of a specific incident strongly depends on the success of this analysis.

5.4 Handling an Incident

Certain steps are necessary to take during the handling of an incident. In all security related activities, the most important point to be made is that all sites should have policies in place. Without defined policies and goals, activities undertaken will remain without focus. The goals should be defined by management and legal counsel in advance.

One of the most fundamental objectives is to restore control of the affected systems and to limit the impact and damage. In the worst-case scenario, shutting down the system, or disconnecting the system from the network, may be the only practical solution.

As the activities involved are complex, try to get as much help as necessary. While trying to solve the problem alone, real damage might occur due to delays or missing information. Most administrators take the discovery of an intruder as a personal challenge. By proceeding this way, other objectives as outlined in the local policies may not always be considered. Trying to catch intruders may be a very low priority, compared to system integrity, for example. Monitoring a hacker's activity is useful but it might not be considered worth the risk to allow the continued access.

5.4.1 Types of Notification and Exchange of Information

When you have confirmed that an incident is occurring, the appropriate personnel must be notified. How this notification is achieved is very important to keeping the event under control both from a technical and emotional standpoint. The circumstances should be described in as much detail as possible, in order to aid prompt acknowledgment and understanding of the problem. Great care should be taken when determining to which groups detailed technical information is given during the notification. For example, it is helpful to pass this kind of information to an incident handling team, as they can assist you by providing helpful hints for eradicating the vulnerabilities involved in an incident. On the other hand, putting the critical knowledge into the public domain (e.g., via USENET newsgroups or mailing lists) may potentially put a large number of systems at risk of intrusion. It is invalid to assume that all administrators reading a particular newsgroup have access to operating system source code, or can even understand an advisory well enough to take adequate steps.

First of all, any notification to either local or offsite personnel must be explicit. This requires that any statement (be it an electronic mail message, phone call, fax, beeper, or semaphore) providing information about the incident be clear, concise, and fully qualified. When you are notifying others that will help you handle an event, a "smoke screen" will only divide the effort and create confusion. If a division of labor is suggested, it is helpful to provide information to each participant about what is being accomplished in other efforts. This will not only reduce duplication of effort but allow people working on parts of the problem to know where to obtain information relevant to their part of the incident.

Another important consideration when communicating about the incident is to be factual. Attempting to hide aspects of the incident by providing false or incomplete information may not only prevent a successful resolution to the incident but may even worsen the situation.

The choice of language used when notifying people about the incident can have a profound effect on the way that information is received. When you use emotional or inflammatory terms, you raise the potential for damage and negative outcomes of the incident. It is important to remain calm both in written and spoken communications.

Another consideration is that not all people speak the same language. Due to this fact, misunderstandings and delay may arise, especially if it is a multi-national incident. Other international concerns include differing legal implications of a security incident and cultural differences. However, cultural differences do not only exist between countries. They even exist within countries, between different social or user groups. For example, an administrator of a university system might be very relaxed about attempts to connect to the system via telnet, but the administrator of a military system is likely to consider the same action as a possible attack.

Another issue associated with the choice of language is the notification of nontechnical or offsite personnel. It is important to accurately describe the incident without generating undue alarm or confusion. While it is more difficult to describe the incident to a nontechnical audience, it is often more important. A nontechnical description may be required for upper-level management, the press, or law enforcement liaisons. The importance of these communications cannot be underestimated and may make the difference between resolving the incident properly and escalating to some higher level of damage.

If an incident response team becomes involved, it might be necessary to fill out a template for the information exchange. Although this may seem to be an additional burden and adds a certain delay, it helps the team to act on this minimum set of information. The response team may be able to respond to aspects of the incident of which the local administrator is unaware. If information is given out to someone else, the following minimum information should be provided:

  1. Time zone of logs in GMT or local time

  2. Information about the remote system, including host names, IP addresses, and (perhaps) user IDs

  3. All log entries relevant for the remote site

  4. Type of incident (what happened, why you should care)

If local information (i.e., local user IDs) is included in the log entries, it will be necessary to sanitize the entries beforehand to avoid privacy issues. In general, all information that might assist a remote site in resolving an incident should be given out, unless local policies prohibit this.

5.4.2 Protecting Evidence and Activity Logs

When you respond to an incident, document all details related to the incident. This will provide valuable information to yourself and others as you try to unravel the course of events. Documenting all details will ultimately save you time. If you do not document every relevant phone call, for example, you are likely to forget a significant portion of information you obtain, requiring you to contact the source of information again. At the same time, recording details will provide evidence for prosecution efforts, providing the case moves in that direction. Documenting an incident will also help you perform a final assessment of damage (something your management, as well as law enforcement officers, will want to know), and will provide the basis for later phases of the handling process: eradication, recovery, and follow-up "lessons learned."

During the initial stages of an incident, it is often infeasible to determine whether prosecution is viable, so you should document as if you are gathering evidence for a court case. At a minimum, you should record:

  1. All system events (audit records)

  2. All actions you take (time tagged)

  3. All external conversations (including the person with whom you talked, the date and time, and the content of the conversation)

The most straightforward way to maintain documentation is keeping a log book. This allows you to go to a centralized, chronological source of information when you need it, instead of requiring you to page through individual sheets of paper. Much of this information is potential evidence in a court of law. Thus, when a legal follow-up is a possibility, one should follow the prepared procedures and avoid jeopardizing the legal follow-up by improper handling of possible evidence. If appropriate, the following steps may be taken.

  1. Regularly (e.g., every day) turn in photocopied, signed copies of your logbook (as well as media you use to record system events) to a document custodian.

  2. The custodian should store these copied pages in a secure place (e.g., a safe).

  3. When you submit information for storage, you should receive a signed, dated receipt from the document custodian.

Failure to observe these procedures can result in invalidation of any evidence you obtain in a court of law.

5.4.3 Containment

The purpose of containment is to limit the extent of an attack. An essential part of containment is decision making (e.g., determining whether to shut a system down, disconnect from a network, monitor system or network activity, set traps, disable functions such as remote file transfer, etc.).

Sometimes this decision is trivial; shut the system down if the information is classified, sensitive, or proprietary. Bear in mind that removing all access while an incident is in progress obviously notifies all users, including the alleged problem users, that the administrators are aware of a problem; this may have a deleterious effect on an investigation. In some cases, it is prudent to remove all access or functionality as soon as possible, then restore normal operation in limited stages. In other cases, it is worthwhile to risk some damage to the system if keeping the system up might enable you to identify an intruder.

This stage should involve carrying out predetermined procedures. Your organization or site should, for example, define acceptable risks in dealing with an incident, and should prescribe specific actions and strategies accordingly. This is especially important when a quick decision is necessary and it is not possible to first contact all involved parties to discuss the decision. In the absence of predefined procedures, the person in charge of the incident will often not have the power to make difficult management decisions (like to lose the results of a costly experiment by shutting down a system). A final activity that should occur during this stage of incident handling is the notification of appropriate authorities.

5.4.4 Eradication

Once the incident has been contained, it is time to eradicate the cause. But before eradicating the cause, great care should be taken to collect all necessary information about the compromised system(s) and the cause of the incident, as they will likely be lost when cleaning up the system.

Software may be available to help you in the eradication process, such as antivirus software. If any bogus files have been created, archive them before deleting them. In the case of virus infections, it is important to clean and reformat any media containing infected files. Finally, ensure that all backups are clean. Many systems infected with viruses become periodically re-infected simply because people do not systematically eradicate the virus from backups. After eradication, a new backup should be taken.

Removing all vulnerabilities once an incident has occurred is difficult. The key to removing vulnerabilities is knowledge and understanding of the breach.

It may be necessary to go back to the original distribution media and re-customize the system. To facilitate this worst-case scenario, a record of the original system setup and each customization change should be maintained. In the case of a network-based attack, it is important to install patches for all operating system vulnerabilities that were exploited.

A security log can be most valuable during this phase of removing vulnerabilities. The logs showing how the incident was discovered and contained can be used later to help determine how extensive the damage was from a given incident. The steps taken can be used in the future to make sure the problem does not resurface. Ideally, one should automate and regularly apply the same test as was used to detect the security incident.

If a particular vulnerability is isolated as having been exploited, the next step is to find a mechanism to protect your system. The security mailing lists and bulletins would be a good place to search for this information, and you can get advice from incident response teams.

5.4.5 Recovery

Once the cause of an incident has been eradicated, the recovery phase defines the next stage of action. The goal of recovery is to return the system to normal. In general, bringing up services in the order of demand to allow a minimum of user inconvenience is the best practice. Understand that the proper recovery procedures for the system are extremely important and should be specific to the site.

5.4.6 Follow-Up

Once you believe that a system has been restored to a "safe" state, it is still possible that holes, and even traps, could be lurking in the system. One of the most important stages of responding to incidents is also the most often omitted, the follow-up stage. In the follow-up stage, the system should be monitored for items that may have been missed during the cleanup stage. It would be prudent to utilize some of the tools mentioned in Chapter 7 as a start. Remember, these tools do not replace continual system monitoring and good systems administration practices.

The most important element of the follow-up stage is performing a postmortem analysis. Exactly what happened, and at what times? How well did the staff involved with the incident perform? What kind of information did the staff need quickly, and how could they have gotten that information as soon as possible? What would the staff do differently next time?

After an incident, it is prudent to write a report describing the exact sequence of events: the method of discovery, correction procedure, monitoring procedure, and a summary of lesson learned. This will aid in the clear understanding of the problem. Creating a formal chronology of events (including time stamps) is also important for legal reasons.

A follow-up report is valuable for many reasons. It provides a reference to be used in case of other similar incidents. It is also important to, as quickly as possible, obtain a monetary estimate of the amount of damage the incident caused. This estimate should include costs associated with any loss of software and files (especially the value of proprietary data that may have been disclosed), hardware damage, and manpower costs to restore altered files, reconfigure affected systems, and so forth. This estimate may become the basis for subsequent prosecution activity. The report can also help justify an organization's computer security effort to management.

5.5 Aftermath of an Incident

In the wake of an incident, several actions should take place. These actions can be summarized as follows:

  1. An inventory should be taken of the systems' assets, (i.e., a careful examination should determine how the system was affected by the incident).

  2. The lessons learned as a result of the incident should be included in revised security plan to prevent the incident from recurring.

  3. A new risk analysis should be developed in light of the incident.

  4. An investigation and prosecution of the individuals who caused the incident should commence, if it is deemed desirable.

If an incident is based on poor policy, and unless the policy is changed, then one is doomed to repeat the past. Once a site has recovered from an incident, site policy and procedures should be reviewed to encompass changes to prevent similar incidents. Even without an incident, it would be prudent to review policies and procedures on a regular basis. Reviews are imperative due to today's changing computing environments.

The whole purpose of this postmortem process is to improve all security measures to protect the site against future attacks. As a result of an incident, a site or organization should gain practical knowledge from the experience. A concrete goal of the postmortem is to develop new proactive methods. Another important facet of the aftermath may be end user and administrator education to prevent a reoccurrence of the security problem.

5.6 Responsibilities

5.6.1 Not Crossing the Line

It is one thing to protect one's own network but quite another to assume that one should protect other networks. During the handling of an incident, certain system vulnerabilities of one's own systems and the systems of others become apparent. It is quite easy and may even be tempting to pursue the intruders in order to track them. Keep in mind that at a certain point it is possible to "cross the line," and, with the best of intentions, become no better than the intruder.

The best rule when it comes to propriety is to not use any facility of remote sites that is not public. This clearly excludes any entry onto a system (such as a remote shell or login session) that is not expressly permitted. This may be very tempting; after a breach of security is detected, a system administrator may have the means to "follow it up," to ascertain what damage is being done to the remote site. Do not do it! Instead, attempt to reach the appropriate point of contact for the affected site.

5.6.2 Good Internet Citizenship

During a security incident, there are two choices one can make. First, a site can choose to watch the intruder in the hopes of catching him; or, the site can go about cleaning up after the incident and shut the intruder out of the systems. This is a decision that must be made very thoughtfully, as there may be legal liabilities if you choose to leave your site open, knowing that an intruder is using your site as a launching pad to reach out to other sites. Being a good Internet citizen means that you should try to alert other sites that may have been impacted by the intruder. These affected sites may be readily apparent after a thorough review of your log files.

5.6.3 Administrative Response to Incidents

When a security incident involves a user, the site's security policy should describe what action is to be taken. The transgression should be taken seriously, but it is very important to be sure of the role the user played. Was the user naive? Could there be a mistake in attributing the security breach to the user? Applying administrative action that assumes the user intentionally caused the incident may not be appropriate for a user who simply made a mistake. It may be appropriate to include sanctions more suitable for such a situation in your policies (e.g., education or reprimand of a user) in addition to more stern measures for intentional acts of intrusion and system misuse.

[4]This word of caution actually applies to all external contacts.



 < Day Day Up > 



Critical Incident Management
Critical Incident Management
ISBN: 084930010X
EAN: 2147483647
Year: 2004
Pages: 144

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net