Lesson 3: Updating Process

---

Deploying updates involves more than just choosing a technology to install the updates. An effective updating process involves planning, discussion, and testing. Though you should use your organization's existing change management process, if one exists, this section will describe the fundamental components of an update process. Figure 5.6 illustrates an effective updating process.

Figure 5.6: The core updating process

After this lesson, you will be able to

• Describe the various steps in an effective updating process.

• Evaluate updates to determine whether they should be applied, and evaluate their priority.

• Identify the most efficient way to identify and retrieve updates.

Estimated lesson time: 30 minutes

Discovering Updates

The security updating process starts when Microsoft releases or updates a security bulletin. Reissued bulletins that have a higher severity rating should be evaluated again to determine if an already scheduled security release should be reprioritized and accelerated. You might also initiate the security updating process when a new service pack is released.

You can be notified of Microsoft-related security issues and fixes by subscribing to the Microsoft Security Notification Services. You can register for this service from the following Web site: http://www.microsoft.com/technet/security/bulletin/notify.asp. If you subscribe to this service, you will receive automatic notification of security issues by e-mail. Note that you won't ever receive the update as an attachment from Microsoft. E-mail is easy to spoof, so Microsoft includes a digital signature that can be verified. However, it's generally easier to simply check the Microsoft Web site to ensure that the bulletin is officially listed.

Although many organizations start the updating process immediately when Microsoft releases an update, others choose to check for updates on a regular basis. For example, some organizations have a person assigned to check for updates every Thursday, and then initiate the updating process if required. You can retrieve a list of updates available for a product by visiting the Microsoft HotFix & Security Bulletin Service at http://www.microsoft.com/technet/security/current.asp. On this Web site, you can search for updates by product and service pack level.

You can also discover updates by using Automatic Updates. If the service determines that a critical update is applicable, it can notify you that the update is available, download the update and notify you that it is ready to install, or automatically download and install the updates on a schedule you define, as illustrated in Figure 5.7. Using Automatic Updates to discover and/or apply updates is great for organizations with only a handful of servers, but the approach does not scale well enough to be used by enterprises.

Figure 5.7: Notification settings for Automatic Updates

The Microsoft Baseline Security Analyzer (MBSA) is an excellent way to determine whether a specific system is missing updates. MBSA scans for missing security updates and vulnerabilities, reports on a computer's adherence to common security best practices, and identifies any configuration options that leave the computer open to potential security vulnerabilities. By default, MBSA contacts the Windows Update service to determine what security updates and critical updates Microsoft has released that have not been applied to the system.

Evaluating Updates

After you learn of a security update, you need to evaluate the update to determine which computers at your organization, if any, should have the update applied. Read the information that accompanies the security bulletin, and refer to the associated Knowledge Base article after it is released.

Next, look at the various parts of your environment to determine whether the vulnerability affects the computers on your network. You might not be using the software component that the update affects, or you might be protected from the vulnerability by other means, such as a firewall. For example, if Microsoft releases a security update for SQL Server and your company doesn't use SQL Server, you don't need to act. If Microsoft releases a security update for the Windows Messenger service, but you have blocked the vulnerable ports by using Internet Connection Firewall, you don't necessarily need to apply the update. Alternatively, you might decide that applying the update is not the best countermeasure for a security vulnerability. Instead, you might choose to add a firewall or adjust firewall filtering rules to limit the vulnerability's exposure.

Determining whether an update should be applied is not as straightforward as you might think. You could simply apply all critical updates and security updates, but applying an update does have a cost. You will need to dedicate time to testing, packaging, and deploying the update. In larger organizations, applying a software update to a server requires that many hours be dedicated to justifying the update and scheduling the update with the groups who use the server.

Any type of update also carries the risk of something going wrong when the update is applied. In fact, any time you restart a computer, there is a small risk that the server won't start up successfully. There's also the very real risk that the update will break existing applications. Fortunately, this risk can be offset by extensively testing the update before applying it. Naturally, there's a cost to deciding not to apply a security update too: an increased risk of a security vulnerability being exploited.

Besides testing, you can offset the risk associated with an update causing problems by having a plan to roll back the update. When evaluating an update, determine whether the release can be easily uninstalled if it causes a problem that isn't identified during testing. Functionality for uninstalling updates can vary from fully automated uninstall support, to manual uninstall procedures, to no uninstall. If an update cannot be uninstalled, your only option might be to restore the computer from a recent backup. Regardless of the uninstall method required for an update, ensure that there is a defined rollback plan in case the deployment doesn't match the success encountered in the test environment.

To be prepared for the worst, verify that you have recent backups of all computers that will be updated, and that you are prepared to restore those systems if the update cannot be successfully removed. It's not likely that an update will cause your systems to fail completely and require them to be restored from backup, but it is a circumstance you must be prepared to handle.

Off the Record 

Updates really can break applications. Sometimes, after you restart a computer, one of your applications will simply refuse to start. In this case, you can immediately uninstall the update and have the application up-and-running within a few minutes. Other times, the update causes more subtle problems. For example, the update might cause your application to run slightly slower-a performance decrease small enough that nobody would notice. However, a month later when many users are trying to access the application simultaneously, it might crash under a user load that it would have been able to handle before the update was applied. The problem was caused by the update you applied, but identifying the connection between the crash and an update you applied a month ago can be difficult, if not impossible.

Choosing whether to apply an update is such a complicated, yet critical, decision that larger organizations should create a security committee that collectively determines which updates should be applied. The committee should consist of employees that are familiar with the updating requirements of each different type of computer on your network. For example, if you have separate organizations that manage desktop and client computers, both organizations should have a representative in the committee. If separate individuals manage each of the Web, messaging, and infrastructure servers on your network, each should have input into whether a particular update is applied. Ask members from your UNIX, database, and networking groups, and from your internal audit teams, to play an active role-their experience and expertise can be an asset in determining risk. Depending on your needs, the committee can discuss each update as it is released, or it can meet on a weekly or biweekly basis.

If the committee determines that an update needs to be deployed, you then need to determine the urgency. If there is an active attack, you must make every effort to apply the update immediately before your system is infected. If the attack is severe enough, it might even warrant removing vulnerable computers from the network until the update can be applied.

For example, assume that you have an IIS computer connected to the public Internet when a security bulletin is announced. If the vulnerability enables an attacker to take complete control of a computer and an exploit is already known to be spreading outside of your network, your system could be infected at any moment. Alternatively, if a vulnerability only allows an attacker to perform a denial-of-service attack by restarting the computer, your risk is much lower. You can safely choose to delay the update until the update is tested and downtime can be scheduled. If you discover that an attacker is exploiting the vulnerability to restart your IIS computer, you could apply the update immediately without risking the loss of confidential data.

Retrieving Updates

Once you have decided to test and/or deploy an update, you must retrieve it from Microsoft. If you are using Windows Update or SUS as your deployment mechanism, retrieving the update is taken care of by the Automatic Update client. If you are deploying updates by using another mechanism, you should download the update from a trusted Microsoft server.

When manually installing a service pack on a computer, you can choose between a network install and an express install. If you are deploying the service pack to more than one computer in the same location, you should always use the network install. This self-extracting package contains all of the files that are required for any computer running the operating system the service pack was released for. This option is designed for administrators who want to set up a shared network folder for deploying the service pack on multiple computers.

The express install for a service pack is more efficient when installing the update on a single computer or when manually installing the update on multiple computers in various locations. Express installation is available for service packs, but not for other types of updates. This type of installation is generally intended for manually installing a service pack on a single computer; it is not optimal for organizations with more than two or three computers. The express install optimizes Internet bandwidth by detecting the service pack files that are already installed on the computer and installing only those files that must be updated. The installation package includes only the files required to start the installation and connect to a download server: the information (.inf) file, the version (.ver) file, and a URL that points to the Microsoft download server. The remaining files that you need are identified and downloaded when you link to the download server.

Testing Updates

After applying a testing update or group of updates to your test computers, you should test all applications and functionality as described in Lesson 2. In addition to testing within the update test environment, large organizations should conduct at least one pilot deployment before deploying the update or updates into the production environment. When conducting a pilot, you deploy a limited number of computers in a controlled environment, evaluate the results, and fix problems. Deploy successive pilots until you determine that the update is ready for full deployment. Be sure to include a representative cross-section of the computers in your pilot group.

Tip 

The more significant the update, the more important it is to use a pilot program. Service packs, in particular, require extensive testing both in and out of the lab.

In addition to testing your implementation of an update, conducting a pilot provides an opportunity to test your deployment plan and the deployment processes. It helps you to determine how much time is required to install the update, and the personnel and tools needed to do so. It also provides an opportunity to train support staff and to gauge user reaction to the updating process. For example, if a particular update takes an hour for a dial-in user to download, you might have to identify an alternative method for delivering the update to the user.

Installing Updates

After you are comfortable that you have sufficiently tested an update, you can deploy it to your production environment. During the installation process, be sure to have sufficient support staff to handle problems that might arise. Have a method in place to monitor the progress of the updates, and have an engineer ready to resolve any problems that occur in the update deployment mechanism. Notify network staff that an update deployment is taking place, so that they are aware of the cause of the increased network utilization.

For more information about how to deploy updates, refer to Lesson 2.

Removing Updates

Despite following proper planning and testing procedures, problems can arise when you deploy an update to production systems. Before deploying updates, you should have a plan in place to roll back updates from one, many, or all of the target computers. Removing an update from a single computer can often be easily done by using Add/Remove Programs, as shown in Figure 5.8.

Figure 5.8: Using Add/Remove Programs for updates

Though you can remove updates from individual computers by using Add/Remove Programs, you should be prepared to remove the updates by using the same method you used to distribute the updates. For example, if you deploy a service pack by using a Group Policy object, plan to explicitly remove that service pack by using the same Group Policy object. Every Microsoft update can also be removed from the command line, allowing you to remove multiple updates with a batch file.

As part of every update deployment, you will need to define a rollback plan, in case the deployment doesn't succeed as it did in the test environment. The following are the main steps for the rollback and redeployment of updates:

• Stop the current deployment. Identify any steps necessary for deactivating release mechanisms used in your environment.

• Identify and resolve any update deployment issues. Determine what is causing an update deployment to fail. The order in which updates are applied, the release mechanism used, and flaws in the update itself are all possible causes for a failed deployment.

• Uninstall updates if necessary. Updates that introduce instabilities to your production environment should be removed, if possible.

• Reactivate release mechanisms. After resolving update issues, reactivate the appropriate release mechanism to redeploy updates. Security bulletins issued by Microsoft will always indicate whether an update can be uninstalled. Because reverting computers to a previous state is not always possible, pay close attention to this detail before deploying an update that cannot be uninstalled.

When a simple uninstall process is not available for a security update, ensure that the necessary provisions are in place for reverting your critical computers back to their original state in the unlikely event that a security patch deployment causes a computer to fail. These provisions might include having spare computers and data backup mechanisms in place so a failed computer can be rebuilt quickly.

Auditing Updates

After you have deployed an update, it is important to audit your work. Ideally, someone not responsible for deploying the update will perform the actual auditing. This reduces the possibility that the person or group responsible for deploying the update would unintentionally overlook the same set of computers during both update deployment and auditing, in addition to reducing the likelihood of someone covering up oversights or mistakes.

Auditing an update that resolves a security vulnerability can be done in one of two ways. The simplest way to audit is to use a tool such as MBSA to check for the presence of the update. This can also be done by checking the version of files that have been updated by an update, and verifying that the version matches the version of the file included with the update.

Real World

Although the staff responsible for deploying an update should be represented on the security committee, they cannot be solely responsible for deciding which updates are deployed, and their work should be audited by an objective individual or group. I was part of a committee that identified security updates that should be deployed to hundreds of Windows-based servers. Unfortunately, the group responsible for deploying the updates was too busy supporting customers to install the updates-for several months. We on the committee realized that the updates weren't being deployed only after dozens of computers were infected by a worm. I realized the importance of auditing that day, but only after it cost my company hundreds of thousands of dollars.

A more complicated, but thorough, method of auditing is to scan the computers on your network for the actual security vulnerability. This requires a non-Microsoft network scanning and auditing tool, however. To adequately test that the vulnerability has been removed, the tool must be designed specifically to exploit the security vulnerability. Although no such tool exists for every security vulnerability, various companies might release scanners to check for widely exploited vulnerabilities. For example, eEye Digital Security has released a scanner to check for the vulnerability exploited by the Nimda worm. The scanner can be downloaded from http://www.eeye.com.

Practice: Evaluating Your Updating Process

In this practice, you will evaluate the updating process that your current work environment is using. Use the following questions to guide your thoughts.

1. Has your organization formally identified and documented an updating process? If so, what is that process?

2. Has your organization ever had to remove an update after deploying it? If not, are you prepared to quickly remove an update from all computers on your network?

Lesson Review

The following questions are intended to reinforce key information presented in this lesson. If you are unable to answer a question, review the lesson materials and try the question again. You can find answers to the questions in the 'Questions and Answers' section at the end of this chapter.

1. How can you validate that an update is genuine?

2. Which requires more testing, a service pack or a security update?

Lesson Summary

• Every organization should develop an updating process catered to its specific needs. However, every updating process should include steps for discovering, evaluating, retrieving, testing, installing, removing (if necessary), and auditing updates.

• When evaluating security updates, factor in both the cost of deploying the update and the risk associated with not deploying the update. Consider alternative ways to mitigate the vulnerability.

• Always be prepared for the worst when deploying an update. Be sure that all systems receiving the update have a recent backup available.

---