Patch Management Is Risk Management

Patch Management Is Risk Management

The vendor usually does not publicize the exact details of every security update. We have heard arguments that vendors should make public all the information about a security update, including sample exploit code so that users can evaluate whether they are vulnerable, and then let the customer make the case for whether to apply the fix. However, this argument does not hold water when you consider that the Heisenberg uncertainty principle applies here. By measuring somethingin this case, riskyou change it. What is risk? There are a couple of ways to measure risk. Risk is a relationship:

Risk = Damage potential * Likelihood of damage

The likelihood itself is actually another relationship. It can be broken into the level of access necessary to exploit a problem, divided by the degree of difficulty to actually exploit the problem. Whichever way to show the relationship, however, it is crucial to realize that we need to analyze risk.

NOTE: Everything we do in information security and network protection boils down to risk management.

What this means is that your (or your organization's) propensity for risk must govern the decision as to which risks you are willing to accept, and which you need to mitigate. This risk profile needs to be articulated in the organizational security policy, which is the subject of Chapter 4, "Developing Security Policies."

The argument for full disclosure goes that in order for customers to be able to ascertain the risk, all details of the problem must be disclosed. Since this information is made public, the people who would use that vulnerability to attack systems also get it. This increases the likelihood that they will discover exactly how to exploit the problem, thus increasing the risk. The vast majority of issues we see exploited are those where exploit code or proof-of-concept code was posted publicly . In fact, most of the extremely damaging worms, including Code Red, Nimda, Slammer, Blaster, and Sasser, were all based on publicly posted sample exploits. ^[1] This lends credence to the argument that the more information is available about a security issue, the higher the likelihood that potential criminals will be able to exploit the problem. This may change going forward, but to date, it still holds and that is why vendors do not publish sample exploits.

^[1] It is also worth noting that all of these worms would also have been mitigated had the administrators configured the systems properly. Even an unpatched system could have been protected with some security hardening.

Risk Ratings

To assist you in determining how soon you need to deploy a security update, some vendors, including Microsoft, apply a risk rating to individual vulnerabilities in a security update. Microsoft's ratings are based on the International Standards Organization's (ISO) risk management ratings. The ratings, and the associated recommendations for how fast to deploy them, are shown in Table 3-1.

Table 3-1. Microsoft Security Risk Ratings

Severity Rating	Definition	Recommended Patching Timeframe
Critical	Exploitation could allow the propagation of an Internet worm such as Blaster or Slammer without user action.	Within 24 hours
Important	Exploitation could result in compromise of the confidentiality, integrity, or availability of users' data or in the integrity or availability of processing resources.	Within 1 month
Moderate	Exploitation is serious, but has been mitigated to a significant degree by factors such as default configuration, auditing, need for user action, or difficulty of exploitation.	Depending on expected availability, wait for next service pack or update rollup that includes the security update, or deploy the update within 4 months
Low	Exploitation is extremely difficult or impact is minimal.	Depending on expected availability, wait for next service pack or update rollup that includes the security update, or deploy the update within 1 year

Security Update Testing

Every time someone tells us to install security updates, they will also end with "and be sure to test this in your environment prior to rolling it out in production." Security updates have a bad reputation, not entirely deservedly so, for breaking things. It is impossible for the vendor to test patches in every configuration, so breaks will always be missed. One of the patches issued with Microsoft Security Bulletin MS04-011, for instance, caused a significant number of problems. Some were kernel crashes due to third-party VPN clients , and others were significant incompatibilities with extended disk partitions on Windows NT 4.0. The fact of the matter is that the vast majority of installations see no stability issues with the majority of security updates. However, for the few that do, the impact can be disastrous. If an update is installed on 20 million computers (not a very high number on a widely distributed platform such as Windows ) and 0.01 percent of those computers had problems with the update, we have 2,000 machines that had problems. That number explains why we often hear about problems with security updates. Even if a problem only affects a very small fraction of systems, it is enough to generate media attention. Unless your systems have very common setups, running only common software from Microsoft and other large vendors, you should test any patches before rolling them out network-wide.

WARNING: By no means do we mean by this that you should delay testing or installing patches based on unconfirmed reports in the media and elsewhere of problems. Nor should you give up and not install a patch if a test reveals problems with it. You still need to evaluate the patch and see if there is some way to get it installed with minimal adverse impact. Far too many people get attacked after patches are available because they heard from their wife's second cousin's hamster's boyfriend that there is some unspecified problem with a patch. You walk a thin line with patches, and it behooves you to know your network so you know which systems you can patch with minimal testing and which need more testing.

Occasionally, there are intrinsic problems in security updates, but these are relatively rare. The majority of problems experienced with security updates are due to third-party applications or modifications to default configuration settings. This is the reason behind the "test in your environment" comment. Given the extremely heterogeneous environments computer software is deployed in, it is impossible to test all permutations . Just because a security update did not cause problems in testing, does that mean it will not cause problems when applied to a gas pump running Windows NT 4.0 Workstation with an extended disk partition? How about a cash register running Windows XP Professional? It is impossible to tell. However, some general rules of thumb apply.

You can expect that all security updates are tested with common configurations. For example, it is usually safe to assume that Windows operating system (OS) security updates, and service packs in particular, are tested with supported versions of Microsoft Office and vice versa. For application security updates, it is safe to assume that it has been tested on all supported platforms. It is a good bet, too, that the update has been tested and will work correctly with hardware listed on the Windows Hardware Compatibility List (HCL)hardware that displays the Windows logo for the OS the update applies to. Many problems with patches stem from hardware, and are often caused by hardware that was never tested and approved for the platform (hardware that does not appear on the HCL) or hardware that was about to fail anyway.

However, what about other, less widely used, third-party software? Are SQL Server security updates tested on a system that also runs SAP, PeopleSoft, or Siebel? Are IIS security updates tested with that really cool shareware doodad that makes your server play "Yankee-Doodle Dandy" through the PC speaker when it is rebooted? Generally speaking, it is up to the third-party vendor (ISV) to test security updates with their product. In some cases, the vendor has a specific policy that states that their product is only supported if it is running on a platform configured a particular way, including service pack level. In this case, you risk voiding your support contract if you install patches. What to do?

The course of action must depend on the security update in question and its interaction with the rest of your environment. For a critical update, you have four options:

1.	Follow the ISV's recommendation and wait for them to accredit the update with their product, possibly running the risk of getting your network hacked in the meantime.
2.	Perform your own testing, and if everything seems OK, install the update anyway, running the risk of losing your support options.
3.	Just install the update, with the same consequences as Option 2.
4.	Press the ISV to speed up testing and accreditation, which runs the same risks as Option 1, but may expose you for a shorter time window depending on the vendor's response.

The answer is not as clear as it may seem. Follow Option 1, and you run a high probability of getting yourself hacked. Option 2 is a bit risky, just in case you missed anything. However, it is really the best option, particularly if you combine it with Option 4. Many ISVs lag significantly behind in security updates. In some cases, entire industry-niche ISVs have these "no-patch" policies. In those situations, you may be able to work through industry groups to apply pressure to the ISVs to change their policies. Quite frankly, if you were to only run some products on the platforms they are accredited for, your operation will probably be taken down by attackers and bots in short order. In addition, industry regulations may require due diligence in information security, meaning you have a legal incentive to patch as well.

Building a Patch Test Bed

Given the propensity of vendors not to explain a lot about the testing they have done on particular updates, and in some cases, even overtly refusing to support their product on a patched system, it is critically important for you to test patches properly. Software testing, in general, is a complicated process, but a few basic principles apply. First, you need a test bed where you can actually perform the testing.

TIP: There is a neat way to get a near-perfect replica of a production system that you can use for testing. Microsoft Virtual PC 2004 (http://www.microsoft.com/windowsxp/virtualpc/) enables you to make a new virtual disk based on an existing hard disk. Using this functionality, you can install Virtual PC on a production system (mind you, probably not one that is currently serving clients) and make a replica of its disk. Then you can copy that disk image off to another computer. Now install Virtual PC on that other computer and create a new virtual machine pointing to the disk image you just created. On the first boot, you need to remove Virtual PC from the virtual machine, and possibly install a few drivers. However, you can now set up the disk in undoable mode and have a copy of a production system on which you can test new security updates. Obviously, this does not work for all types of testing, such as testing with particular hardware that is not emulated by Virtual PC, but it works pretty well for testing interoperability with software. Cost-wise it is far more efficient than purchasing and managing additional hardware because Virtual PC only costs about $150. See Microsoft Knowledge Base article 833141 for details on how to perform these steps in Virtual PC.

Using Virtual PC to create a replica of your environment is primarily suitable for nonstandard systems. If you have a set of standardized configurations, which is a good idea anyway, you can simply build out a few more systems and use those as the test bed. You can, of course, use virtual machines for this type of test bed as well. Just be a bit careful when using virtual machines as patch test beds. The emulated hardware will not be identical to the real systems. This is usually not a problem with patches for software, but obviously will not work with driver updates, for example. Many organizations have remote installation services or Ghost images of their standard builds as well. These could prove invaluable for patch testing, and are used with the proper hardware, but require more computers to run the tests on.

After you have the test bed built out, you need to generate a test rig. The test rig tells you what kinds of tests to run, and how. The test rig should be realistic in the sense that it needs to reflect the tasks the systems you are testing would normally perform. On a Web server, for example, you need a test rig that reflects the ordinary types of requests made to that Web server. For an end-user system, the test rig needs to contain tests using the standard set of productivity applications you would normally use on one of those systems. One way to create such a test rig is to analyze the logs on a system. For example, you can capture Web logs, parse requests out of them, and then reproduce those requests . For end-user systems, you can simply observe users performing ordinary tasks.

There is software you can use to generate automated test rigs. Although we have not tested it ourselves , several people have recommended Seque SilkTest (www.seque.com) for this purpose.

Another option is to make use of your users in patch testing. If you have a set of users who are more advanced than others, you should consider using them in an internal beta program. When a patch is released, roll it out to those users immediately. As long as they perform all or most of the tasks that the remainder of the user base would perform, you would relatively quickly be able to ascertain whether the patch has any adverse side-effects. Note, however, that you will never find everything with a test bed. You need to do a staggered rollout no matter what. Start by rolling patches out to those users who are used to pain and who can report on what went wrong. Think of it as widening the beta program. Of course, you cannot do this with all patches. If the world is melting around you, you have to start rolling things out now.

For server systems, performance is perhaps the most important aspect of all. Most systems come with some form of performance monitoring tool. For example, Windows NT-based systems come with the Performance Monitor (see Figure 3-1).

Figure 3-1. You can use Performance Monitor to gauge the impact of a patch on performance of your system.

You can use Performance Monitor to, for example, gauge the relative impact of a patch on performance. A complete discussion of Performance Monitoring is far beyond this book. For more details, refer to titles such as Windows NT Performance Monitoring , by Edmead et al. (New Riders, 1998), or Windows 2000 Performance Guide , by Friedman et al. (O'Reilly, 2002). However, a couple of points are important to note.

First, you must have a baseline of the system's performance prior to installing the patch. This benchmark needs to include items such as normal processor load, the committed bytes to various processes, disk usage, and so on. When you know how a production system behaves, you can then apply the patch to one production system and log the same counters. After a warm-up period of a few hours, track enough data for a meaningful comparison. Then you can compare the performance of a patched system to an unpatched one. Problems such as memory leaks become relatively obvious using this technique.

One final note on performance monitoring is important. Every time you measure something, you change it. An instrumented system will not perform as well as one without the instrumentation. That is not particularly important to our objective, however, because we are primarily interested in relative performance pre- and post-patch.