Hack 79 Best Practices for Patch Management

By understanding the different kinds of patches and following a simple regime , you can keep your critical systems free from known vulnerabilities .

Patch management is probably the biggest concern of IT departments these days. With new vulnerabilities being discovered almost every week, keeping systems up-to-date with patches is often a full-time job, especially in large enterprises . In addition, the lag time between when a vulnerability is discovered and when a virus or worm appears in the wild is now measured in weeks rather than months. This puts tremendous pressure on vendors to release patches before they've even been fully regression- tested . The result is that sometimes patches fix the problem they're designed to address but break something else unintentionally in the process. Customers often blame vendors in such circumstances but, let's face it, there's a war going on and, like most wars, it's messy.

Patch Flavors

Before you plan a patch-management strategy, it's important to understand the differences between the various different flavors of patches. Microsoft classifies patches into three basic categories: hotfixes, roll-ups , and service packs .

Hotfixes

Hotfixes are small patches designed to fix a single problem and are developed either in response to a security advisory or by customer request. Hotfixes are typically issued either to plug security holes, such as buffer overflows, or to fix features that don't behave as intended. Not all patches are created equal; hotfixes that address broken functionality are developed by Quick Fix Engineering (QFE) teams at Microsoft Product Support Services (PSS), whereas those that address security vulnerabilities are identified and developed by the Microsoft Security Resource Center (MSRC).

Roll-ups

Occasionally, Microsoft combines several hotfixes together into a single package called a roll-up . This is typically done when several security issues have been identified within a short time interval, and its purpose is to simplify the job of installing hotfixes for administrators. Unfortunately, this is not always a good idea. There have been instances in which installing multiple patches broke applications, and the headache then arises: figuring out which patch in the roll-up actually caused the problem.

Service packs

At pretty regular intervals, Microsoft combines all hotfixes issued for a platform into a single package called a service pack . These service packs are cumulativefor instance, Service Pack 3 includes all hotfixes issued both before and since Service Pack 2 appeared. While service packs undergo more thorough testing than individual hotfixes, there have nevertheless been a few instances in which a service pack caused new problems while solving others.

MSRC Ratings System

Hotfixes that address security vulnerabilities are also called security fixes , and the MSRC rates these according to a four-point scale from high to low. This is a useful scheme for administrators, because it allows them to decide which fixes should be applied as soon as possible and which can be deferred until later or even ignored. The ratings also refer to the types of vulnerabilities they guard against. An example of a critical issue might be a self-propagating Internet worm that can bring servers to their knees and wreak other kinds of havoc, while important means that your confidential business information might be at risk of being lost, stolen, or corrupted. Moderate means you have a properly configured firewall and are following good security practices, so you won't likely to be affected by this problem, though it's still possible. Finally, low means it would take a combination of a genius hacker and a totally negligent system administrator for this exploit to occur (but it's still remotely possible).

Strategies for Patch Management

My own strategy for effective patch management can be summarized as Policy , Process , Persistence (PPP). Let me unravel this, along with some helpful recommendations from Microsoft.

Policy

The first step in developing a patch management strategy is to develop a policy that outlines the who, what, how, when, and why of patching your systems. That takes planning, and with administrators being as busy as they are these days, it's difficult to allocate time for proper planning. Still, planning is essential. My view is that the difference between planning and an ad hoc fix-it-when-it's-broke approach is the difference between peace of mind and success, and constant anxiety and a disaster waiting to happen.

It all boils down to being proactive instead of reactive. Proactive management anticipates problems in advance and develops policies to deal with them; reactive management adds layer upon layer of hastily thought-up solutions patched together using bits of string and glue. It's easy to see which approach will unravel in the event of a crisis. Once you have a patch-management policy in place (usually it's part of your overall security policy) and a notification arrives of a critical vulnerability in some product, you immediately know who will deal with it, which tools will be used to deploy the patch, whether it needs to be done sooner or later, and so on. For example, a simple element of a patch-management policy might be that critical or important patches should be applied immediately, while moderate or low patches should be submitted to a team member for further study. Another example is proactively scheduling a specific day of the week or month for installing patches (usually weekends, in case something breaks), as opposed to the drop-everything, the-sky-is-falling approach common in a reactive environment. Making a decision tree that addresses these issues ahead of time reduces anxiety and speeds response when the time comes to patch something.

Process

The detailed procedure you will use to respond to vulnerabilities and deploy patches should be explicit within your security policy. In this regard, we have some help from Microsoft, which recommends following a six-step process.

1. Notification

Information comes to you about a vulnerability, including a patch meant to eliminate it. Notification might be sent via email from the Microsoft Security Notification Service, a pop-up balloon when you're using Automatic Updates, a message displayed in the Software Update Services (SUS) web console, or some other method. It all depends on which tools you use to keep your systems patched and up-to-date (we'll summarize these tools in a moment).

2. Assessment

Based on the patch rating and the configuration of your systems, you need to decide which systems need the patch and how quickly they need to be patched to prevent an exploit. Obviously, having an accurate inventory of systems and applications running on your network is essential if you want to keep your network secure against intrusion.

3. Obtainment

How you get the patch you need depends on which patch-management tools you choose to deploy. In general, such tools range from completely manual (e.g., visiting the Windows Update web site) to almost entirely automatic (e.g., via Automatic Updates or SUS). Like everything in security, there is a tradeoff : the manual approach is slower, but it gives you more control.

4. Testing

Testing should always take place before you apply patches to production systems. Test your patches on a testbed network that simulates your production network. Remember that Microsoft can't test all possible effects of a patch before releasing it, because there are thousands of applications that can run on servers and millions of combinations of applications. So, make sure you test patches before deploying them, especially if you have custom code running on your machines. If you need a way to justify the cost of purchasing duplicate equipment for a testbed network, tell the boss it's like insurance.

5. Deployment

Deploy a patch only after you've thoroughly tested it. You are then ready to apply it, but do so carefully . Don't apply it to all your systems at once, just in case your testing process missed something. A good approach is to apply patches one at a time, testing your production servers after each patch is applied to make sure applications still function properly. That's the problem with security roll-ups: by combining several fixes into a single package, the probability of a patch going wrong and breaking something is multiplied. Again, it's a tradeoff: roll-ups speed up patch deployment but give you less control over the result. Fortunately, even a tool like Automatic Updates can be hacked to apply one patch at a time [Hack #86].

6. Validation

This final step in the process is often forgotten: making sure that the patch has actually been installed on the targeted systems. Fortunately, there are tools available to scan your network to see whether your systems are properly patched by looking for changes in the server's filesystem and Registry to verify that a patch has been installed properly (see [Hack #80])

As far as notification is concerned , never install a patch that is attached to an email message purportedly sent to you by Microsoft. Microsoft doesn't send out patches by email (it sends out notification bulletins only). Such attachments are most likely spam or possibly even viruses, so don't open them!

Persistence

Policies are useless and processes are futile unless you persist in applying them consistently. Network security requires constant vigilance , not just because of the new vulnerabilities and patches that appear almost daily, but also because new tools are constantly being developed to handle the growing problem of keeping systems patched. At the time of this writing, Microsoft's whole patch-management strategy is in a state of flux.

So, we are on the horns of a dilemma. If you assert that Microsoft is responsible for ensuring that Windows systems are patched and up-to-date, then you should agree that Microsoft should have the right to package their products with automatic patching turned-on, so that patches are downloaded and installed automatically whether or not administrators want them. However, most administrators won't agree to this, because they want to maintain control and don't trust Microsoft. In that case, you should agree that the administrators who deploy and configure Windows systems should be considered responsible for keeping them patched properly.

Unfortunately, incidents like the Slammer worm, which propagated using unpatched Microsoft SQL 2000 servers, clearly indicate that not all administrators act responsible when it comes to keeping their systems up-to-date with patches. To be fair, though, poorly patched systems are not always the fault of administrators; sometimes, they are the fault of tight-fisted CEOs who refuse to budget adequate funds for hiring IT staff or procuring patch-management tools and test systems.

The point is that if Microsoft can't control the patching process, then it's pushed back onto the users. And a few irresponsible users can wreak havoc on the systems of responsible ones through the flood of worm traffic they unleash through their unpatched systems. Responsible users then cry out, "Microsoft should stop this from happening!," when perhaps they should be suing the companies that don't keep their systems properly patched.

I might add another P here for Practice . Once you've developed your patch-management policy, you should periodically have your staff practice the procedures so that the procedures become second nature. Mind you, with the number of patches coming out of Redmond these days, who needs to practice?

Patch-Management Tools

Once you have a policy in place and have outlined a detailed process for handling patches, what tools can you use to deploy patches to your systems? Once again, various tradeoffs are involved, including power versus simplicity and risk versus control. Here's a quick summary of what's currently available from Microsoft.

Windows Update

The granddaddy of all patch-management tools, Windows Update is a web site (http://windowsupdate.microsoft.com) that allows users to scan their computers manually to see which hotfixes, roll-ups, or service packs need to be installed. Windows Update also offers add-ons and enhancements that Microsoft develops for Windows.

The advantage of this approach is that users have complete control over which patches are installed on their system. The disadvantages, however, are numerous . First, your computer must be connected to the Internet, which is where most threats come from. Second, you must have cookies enabled; there goes your privacy, some might say. Third, you must allow ActiveX controls to run, which is another potential source of vulnerability. Finally, you must be a member of the local administrators group when you use Windows Update. This one is serious; in a corporate environment, it means you have to give employees administrative privileges so that they can keep their machines up-to-date.

Clearly, Windows Update is suited only for small offices and home networks as a patch-management solution.

Automatic Updates

Starting with Service Pack 3 for Windows 2000, Microsoft includes a feature called Automatic Updates on all subsequent versions of Windows. This feature has some of the security weaknesses of the Windows Update approachnamely, your machines must be connected to the Internet and Internet Explorer must be configured to allow ActiveX controls to run. But on the plus side, Automatic Updates doesn't require that users have administrative privileges, as Windows Update does. The main advantage of Automatic Updates is that it enables systems to download new patches automatically when they become available on the Windows Update web site and install them according to a schedule the administrator can specify. For more information on how this tool works, see [Hack #86].

Software Update Services (SUS)

The Software Update Services (SUS) tool is available as a free download from Microsoft and takes Automatic Updates several steps further. Instead of requiring each system to be connected to the Internet, SUS downloads and stores patches on one or more SUS servers, where administrators can review them and either approve or decline their installation. Client computers then have their Automatic Updates component configured to point toward the SUS servers instead of the Windows Update web site as the source for their patches. This approach has all the advantages of Automatic Updates, without the disadvantages of requiring every machine to be exposed to the Internet.

SMS Software Update Services Feature Pack

At the high end of things is Microsoft Systems Management Server (SMS), a powerful but complex tool for deploying, configuring, and maintaining large numbers of systems. The SUS Feature Pack enables SMS to leverage SUS technology to determine which systems need which patches, push the patches out and install them, and report the results. The Feature Pack gives you more granular control than SUS over which systems receive which patches, lets you build an inventory of installed patches for each system, has better reporting tools, and overcomes SUS's limitation of 15,000 client computers (though, in reality, SUS starts to become unmanageable around 5,000 clients ). For further information, see http://www.microsoft.com/smserver/downloads/20/featurepacks/suspack/.

Third-party tools

Finally, there are a number of third-party patch-management tools available. GFI LANguard Network Security Scanner (N.S.S.) from GFI (http://www.gfi.com/) is a good one. In addition to identifying and deploying patches each system needs, N.S.S. can also scan for other vulnerabilities, such as weak password policies and ports that shouldn't be open, and inform you how to harden your systems better. There are also other patch-management systems available from third-party vendors; a quick search on Google will turn up several.