Hack 80 Beginners Guide to Enterprise Patch Management

< Day Day Up >

Here's another take on managing the patch-management cycle effectively in a large enterprise environment, written by an expert on the subject .

One of the most heated and wildly debated subjects in many organizations today is the subject of desktop security. When large IT organizations have an enterprise product like Systems Management Server (SMS) deployed, security teams usually push to the desktop teams the task of ensuring that the latest security updates released by Microsoft are installed. This means that the responsibility of ensuring that a high percentage of clients in the environment are patched rests on the shoulders of desktop support personnel members or SMS team members . In order to better distribute this responsibility, it's best to understand the functionality that can be extended in the following steps:

Identify vulnerable systems
Assess the business impact of patching
Package patches for distribution
Test patches
Evaluate successes and failures
Finish up

Before we discuss each of these steps in detail, if you are currently looking to evaluate which tools would be best for your organizations patch-management strategies, take a look at these helpful links from Microsoft's two heaviest hitters:

Patch Management Using Microsoft Software Update Services: http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msm/swdist/pmsusog.asp
Patch Management Using Microsoft Systems Management Server: http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msm/swdist/pmsmsog.asp

Identifying Vulnerable Systems

SMS stands out from SUS the most in its ability to report on the current client state. In the past, SMS_DEF.MOF updates accomplished this by pulling data from the Registry in HKLM\SOFTWARE\Microsoft\ Windows NT\CurrentVersion\ HotFix and basing distributions on that data. However, with the advent of the SUS Feature Pack for SMS (SUSFP), this is no longer necessary. A more robust security hotfix and reporting mechanism can now be added to your infrastructure. Installed and missing updates are reported through normal SMS inventory, so collections and queries can be based on of this information. Microsoft also supplies a web-based reporting package with built-in reports to give high-level overviews of an organization's patch state. The job of an IT administrator to manage the environment will be significantly easier if she is able to view which updates are already installed and which are a priority to be deployed.

Assessing the Business Impact of Patching

The business impact of patching is mostly comprised of developing patch distribution schedules and policies. Each company's needs are different and patch distribution should conform to those specific needs. Many desktop IT divisions let mandates from their security teams determine when and how patches should be deployed; generally , this causes more harm than good. Do your customers have schedules to meet as well? Shipping dates for when the product must be out the door? What if you send a package with 10 critical system updates to 100 workstations? Problems might occur through no fault of your own, no matter how much testing is done prior to deployment. Optimal success is gained by talking to your customer base to determine the best dates for distribution. If a major product order is being shipped on the 15th, you should probably wait until the 20th to send out updates.

What, then, guarantees the least amount of interruption to users, while still ensuring success? Most of the time, this is encapsulated in your company's workstation reboot policy. If most IT administrators had their druthers, every workstation would get restarted on a daily basis or the administrators would be allowed to force system restarts whenever they want to during distributions. But in the real world, especially for engineering or R&D-driven companies, this is not a reality. Even if you are using the SUS Feature Pack (SUSFP) to give your user base the ability to install their updates early, it is still difficult to manage workstations that might not get logged onto for weeks at a time or workstations that run test simulations and cannot get restarted even while a distribution is happening. There is no easy answer to this, and many different internal solutions have been developed by various organizations. If you are faced with this dilemma and are not able to use the built-in tools (such as the SUSFP), the best option is to speak with other members of the IT community in forums, newsgroups, or mailing lists and learn which methods they use to overcome this problem. However, it is crucial to keep two ideas in mind: the impact to your user base and whether the solution still ensures successful installation.

Packaging Patches for Distribution

If your organization is already using the SMS SUS Feature Pack's patch-installation agent (i.e., the Distribute Software Updates Wizard functionality) for distribution, you can pretty much ignore this section because everything you need is already done for you. Microsoft has developed a common template in the SUSFP that consistently applies patches in a safe fashion, so the SMS administrator no longer needs to perform constant scripting. Most companies using SMS that have not made the transition to SUSFP to distribute patches haven't done so either because their organizations have either specific user-base needs (such as workstations in labs that might go long periods without being looked at) or a scattered hierarchy of sites and multiple administrators that would need to replicate the same settings repeatedly (this scenario introduces an increased possibility for human error, along with other issues).

If you decide to write your own package to suit your company's needs, make sure you perform the following major steps in your code repeatedly for each patch:

Determine if the patch is already present by checking for a Registry key's existence, file version, and so on. You'll spend a lot of time reading through Microsoft security bulletins to figure out which data to look for.
Detect which operating system your package is running on and install the correct version of that patch for each bulletin.
Verify successful installation by again checking for a Registry key's existence, file version, and so on. But this time you will need to code in failure messages (MIFs for SMS) and logging so that you can troubleshoot issues that might arise later.
QChain the patches. Many people forget to do this step.

This process is a simplified flow of what needs to be done for a solid patch-distribution package. It might look easy at first glance, but when you are dealing with over 15 hotfixes, the package size can grow quite considerably, and the time the administrator spends coding and working out bugs greatly increases as well. In the past, packages that do just these simple tasks have been known to grow to a few thousand lines of code!

It's easy to see why using the SUSFP Patch Installation agent would make an SMS administrators job easier, since these steps are already done. Proper hotfix command-line switches are really all you need to research.

Testing Patches

Proper testing procedures should be at the foundation of every software distribution, regardless of weather they are hotfixes. Your company's client base might have any number of different configurations that could effect distribution. I won't discuss package testing in detail here, because entire books can be written on the subject, but I will add a couple of notes.

First, test the patch in its raw installer form from Microsoft to eliminate any chance of causing problems on the system. Next, when testing configurations on workstations that are designated for testing, do the majority of your tests via SMS once you are confident that the package functions correctly. Many problems can be uncovered when distributing via SMS. Because the majority of your enterprise will execute the installation in this method, the majority of your testing should mirror that use.

Also, note that there is no substitute for beta testers among your customer base. Not only will you get more feedback if a problem arises, but they are also using the tools that could be affected. Finally, base distributions on testing; don't base testing on distributions

Evaluating Successes and Failures

Verifying that the client base is up-to-date after distribution might be the most important phase in this entire process. Managing the patch state in your enterprise will never be finished as long as there is person out there who is smarter than the Microsoft developers, which is why companies need to patch in the first place. Unfortunately, many organizations distribute an update and never look at reports to make sure that a high-enough percentage of machines are updated. If your infrastructure is functioning properly, the reporting methods you used earlier to identify vulnerable systems (such as the SUSFP) should show that the patch-installation state is higher after distribution. If this is not the case (if installations are failing but notifications of the failure are not sent back to the IT administrator), you will need to look into either your reporting mechanism or your packaging technique.

Finishing Up

Thoroughness and attention to detail are the most important aspects to managing a company's patch state. Although there are many delivery machines (such as SMS) that can be used to apply patches to your workstations, the same basic premises should be followed throughout the entire process. Even in this heightened time of hacking, viruses, and worms, a solid patch-management process and attentive IT administration can avoid almost any vulnerability by actively keeping workstations' patch state updated.

Do some research on Microsoft patch release dates and virus/worm release dates. You'll find that in almost every instance a patch has been released far ahead of time. When administrators fail to apply these updates, frantic patching is often required when an attack happens. Do you and your customers a favor: avoid these emergencies by having a stable enterprise patch-management solution in place.