The reality of maintaining a security perimeter built perfectly in tune with the organization's business needs is that at some point it will need to change to accommodate new requirements and emerging threats. The ability of the infrastructure to handle change in a controlled manner is one of the main attestations to the quality of its design, implementation, and maintenance. This section discusses rolling out patches, reconfiguring systems, and detecting environment changes without compromising the security of your network.
Fundamentals of Change Management
The general concept of change management refers to the ability to handle changes in business or technical areas of the organization. In the confines of this chapter, we focus on processes that support changes to network infrastructure and applications. One example of a process that might require change management is the rollout of a critical patch to web server software that needs to take place without causing prolonged downtime, crippling its applications, or exposing new security weaknesses. Similarly, a change to a firewall's rule set should take place without inadvertently denying authorized traffic or exposing internal systems to new threats. To accommodate such adjustments to the infrastructure, a robust change-management process should incorporate the following elements:
The majority of change-management situations benefit from a process that incorporates these steps. However, the extent of each phase might differ, depending on how complicated or risky the change is. For example, migrating a mail server from Microsoft Exchange to Lotus Notes is likely to require more planning and testing than adding a routine security patch to a server. Next, we examine these elements in greater detail so that you are better positioned to set up a change-management program that matches your needs.
Obtaining Buy-in from Relevant Personnel
You should obtain buy-in from relevant parties regarding the reasons for implementing the change before rolling it out. For instance, you might have valid reasons for wanting to block all outbound SMTP access from your screened subnet, but you might not realize that several servers hosted there have valid business reasons for generating email. We recommend formalizing the change-approval process so that management and technical personnel have a chance to voice objections to upcoming modifications. This procedure helps prevent making rushed choices that cause adverse side effects. The process should also define individuals who are authorized to make final decisions regarding proposed changes; this eliminates unending discussions and resolves potential conflicts.
Communicating Proposed Changes
You should keep business and technical personnel apprised of proposed modifications so that they are not caught off guard when the change occurs. Establish a procedure for notifying appropriate parties of scheduled changes and verify that all relevant parties have a way of obtaining necessary information. Also, do not forget to notify end users that systems might be inaccessible for a certain time period. To help alleviate inconvenience caused to end users by potential service disruptions, consider scheduling the change to take place during off-time hours.
To make notification to relevant personnel of planned service disruptions easier, consider preparing template documents that describe some of the more common downtime scenarios. The templates should explain when the event will occur, which services will be impacted, and when systems are scheduled to resume normal operation. You might also want to explain how users or the organization will benefit from upcoming changes so that they are more supportive of your plans.
Preventing and Detecting Unauthorized Changes
You are probably well versed in enforcing access controls by this point in the book. Implementing such controls will help prevent unauthorized or inadvertent changes to your systems. Also, integrity-assessment tools, such as Tripwire, are useful for detecting unapproved changes or for tracking authorized changes that mold your system during its lifespan. To prevent unpleasant situations in which administrators' actions are second-guessed, your security policy should clearly define who is authorized to make changes and what approval and communication process they need to follow.
Testing Changes Before Deployment
All changes to be introduced into a production environment should be tested first on a separate, staging network. Granted, you might not be able to afford to create a full replica of your production infrastructure for this purpose. However, you might still be able to introduce the proposed change into a scaled-down laboratory environment to help you assess the risk of rolling it out to production. When testing, be sure to go through the process of regression testing, which tests the new environment for all bugs. Often, by fixing an open problem, you might reopen an issue that was fixed previously. Regression testing will help you find these problems before they are reintroduced into the production environment. Your effort in testing a change before deployment should be proportional to the impact that the change is likely to have on the production environment. In most cases, adding an entry to a system's hosts file is likely to require less testing than upgrading the version of the firewall software.
Verifying Proper System Operation
Having a detailed plan for verifying that the infrastructure functions properly after the change is rolled out allows you to limit the impact of problems and makes it easier to track down their causes. The plan should incorporate a standard suite of tests that are performed after any change and should include specific tests to verify that the intended change has been implemented. Be sure to test access to relevant services and exercise core aspects of applications after upgrading software, reconfiguring hosts, or tweaking access control rules.
Rolling Back Undesired Changes
Even after all the planning and testing, your change might cause problems that you are unable to correct right away; in that case, you will need to roll back to the previous configuration. You save yourself and those around you a lot of anguish by preparing (and testing) a rollback plan before implementing the change. A time-tested method of reversing the effects of a change to a system is to restore it from the most recent backup. Of course, be sure to verify that such a backup exists and that you are able to restore data from it before implementing the change. A tape backup might come in handy even if you are applying a patch that has an uninstall function because such rollback mechanisms have been known to fail. If the scope of your change is relatively narrow (for instance, if you are just modifying a configuration file), a rollback plan might be simply a matter of creating a copy of the file before editing it.
When making changes to the infrastructure, you are often faced with making tough choices, even if your change-management process incorporates the principles mentioned earlier. In the next section, we go over several mechanisms that can help you make such decisions.
Implementing Change-Management Controls
The process of change management, described in the previous section, focuses on making sure that changes to the environment happen in a controlled and deterministic fashion. The cornerstone of this process is a documented change-management procedure that explains how things should be done, when, and by whom. There are also techniques that can make it easier for your organization to follow this procedure. In this section, we examine two essential ways of using tools and procedures to assist in implementing change-management practices:
Let's start by looking at an age-long problem that you've most likely faced in your career already: applying patches.
On numerous occasions throughout this book, we've emphasized the need for regularly patching your OS and applications to prevent attackers from exploiting known software vulnerabilities. Unfortunately, sometimes this is easier said than done. It is not always easy to keep track of patches that vendors release for all the products you need to maintain. More importantly, some of the more urgent patches do not undergo regression testing, so the vendor does not guarantee that they will work well in all permutations of system configurations.
To make it harder to forget applying necessary patches, clearly designate individuals who are responsible for monitoring patch and vulnerability announcement forums. It often makes sense to assign this responsibility to system administrators who maintain the respective products, but do not assume that they are already subscribed to the relevant mailing lists. Consider handing out a list of applicable announcement forums. As a security professional, you should monitor them as well to help ensure a vulnerability does not go by unnoticed.
As a stream of vulnerability announcements flows through your mailbox, you need to decide whether they apply to your environment and how urgent it is for you to react. You need to weigh the risk of delaying a patch rollout, perhaps to perform more thorough testing, against the risk of having a vulnerable network resource. For instance, if a patch prevents an exploitable compromise of your DNS server, you might want to deploy the fix as soon as it comes out. A patch that is, perhaps, less urgent might be one that aims to correct a theoretical vulnerability in a print server deep within your network. If the print server is not your most critical resource, you might decide to wait until the vendor provides a more comprehensive upgrade or until your peers share their experiences pertaining to this patch. In some cases, patches might not be available or might be too risky to apply, and you will need to consider other workarounds to address critical vulnerabilities.
Whatever decisions you make regarding applying, testing, or delaying the installation of a patch or a workaround, be sure to document them. Such journals will help you remember reasons for making certain decisions, and if you chose not to apply the patch, the journals will allow you to revisit the decision at a later date. We suggest including the following fields in each journal entry:
In large environments, it might be difficult to know versions of applications that are installed throughout your organization, or even which services are running on remotely accessible systems. The next section focuses on techniques for keeping track of new systems and services that come online on your network.
Discovering New Services and Devices
You need to know which systems and services are available on your network to be able to assess risks and correct potential vulnerabilities. To help achieve this awareness, you might establish a policy that new network services have to be registered with and, possibly, approved by information security personnel. However, environments change quickly, and before too long, your network is likely to contain resources that you were not notified about. One of the most effective ways to discover the presence of unauthorized or inadvertent network resources on your network is to compare periodic snapshots of the state of your network.
We already described one mechanism for discovering which services are offered by your systems in Chapter 9, "Host Hardening." There, we mentioned the netstat na command, which lists ports that the system listens on. Consider scheduling this command to run on regular intervals so that you can discover when a critical system begins listening on unexpected ports. You can then create a script that compares the output of each successive run by using tools such as fc.exe under Windows or diff in UNIX. This technique focuses on individual systems, but it might not be practical for monitoring the state of workstations, servers, and devices on a mid-to-large-size network.
Network and vulnerability scanners, which we describe in greater detail in Chapter 22, "Assessment Techniques," offer a way to remotely test for the existence of network services. To locate newly set up systems or unauthorized port listeners, you would compare the output of successive network scans and flag the differences. Nessus, a popular vulnerability scanner, and Nmap, one of the most powerful network scanners, make this a relatively painless task.
Nessus, freely available from http://www.nessus.org, does a great job of scanning networks for accessible hosts or open ports, and it specializes at probing systems for known vulnerabilities. One of the ways to detect changes in your environment is to regularly scan your network with Nessus and compare the results.
If you are not interested in scanning for vulnerabilities or you prefer to use a tool other than Nessus, you can take advantage of the scanning capabilities of Nmap. Nmap (http://www.nmap.org) is a wonderful tool for locating hosts and open ports, and it's invaluable for performing security assessments. You can also schedule Nmap to periodically scan your network and compare differences in the network's state. To perform such differential scans with Nmap, you can compare the results of your scans using a tool such as fc.exe or diff, or you can use an add-on utility called NDiff, available for free from http://www.vinecorp.com/ndiff.
To use NDiff, you first scan your network with Nmap and then save the output into a file by using an m option, like this:
# nmap -m snapshot.nm 192.168.1.0/24
In this example, Nmap port-scans all hosts on the 192.168.1.0/24 network and saves the output into a file called snapshot.nm. The next time you run Nmap, you would save its output into a different filesay, newsnapshot.nm. You would then use NDiff to compare contents of the two files, like this:
# ndiff -b snapshot.nm -o newsnapshot.nm -format minimal new hosts: 192.168.1.100 missing hosts: changed hosts: 192.168.1.204 80/tcp/http (unknown -> open) 443/tcp/https (unknown -> open)
In this example, NDiff detects that a new host appears on the network and that another host begins listening for HTTP and HTTPS connections. To simplify the task of running Nmap, storing its output, and then comparing the results of the scan, NDiff comes with a wrapper utility called nrun.
Using differential scanning, you might detect services that should not be present on your network and react by disabling them. On the other hand, your investigation might show that a new service has a legitimate need, in which case you will need to understand what it is about in order to determine how it should be configured. In Chapter 15, "Software Architecture," we presented a software evaluation checklist with questions that you can ask when assessing how an application fits in with the rest of the security perimeter. We suggest asking such questions not only of your third-party providers, but also of your colleagues. Understanding the application's technical and business requirements will help you assess its risks and devise appropriate mitigations.