So, what about your mission-critical servers? All the solutions mentioned previously are basically for clients and noncritical servers. How do you update your servers? The first step is to find the updates. You can find links to all Microsoft security updates at the security bulletin search site at http://www.microsoft.com/technet/security/current.aspx. (A few updates cannot be downloaded from TechNet. For example, as of this writing, Microsoft is legally prevented from distributing patches for the Microsoft Java Virtual Machine [JVM] except under very strict rules and only to systems that already have the Microsoft JVM. Therefore, patches for the JVM are only available on Windows Update.) Next you need to apply them. There are some tricks for that. Keep in mind that these are advanced techniques . They should only be used by experts, and only after careful study and practice of the technique, the target application, and the update. These techniques are not for use by end users.
One new (to Microsoft) technique is hot-patching, which updates running code in memory. Hot-patching does not replace the actual executable on disk until after the service or application has shut down. This typically happens on the next reboot. This technique requires the user that installs the update to have the Debug Programs privilege. If you have removed that privilege from Administrators, for example, you need to grant it back at least to the accounts that will install updates. Unfortunately, hot-patching comes at a costhot patches cannot currently be integrated into an installation point by slipstreaming (see below).
Minimizing reboots on mission-critical servers is obviously of great value. Often a reboot is caused by the fact that some component that needs to be updated is currently running. If a hot patch is not available for that component, the patch will reboot the system after patching. This may be avoidable with some patches by performing some careful investigation of the update and then turning off the items that are using the files in it before running the update. For example, suppose you need to roll out an IIS security update on a large server farm. The first step is to unpack the update itself and get a list of the files in it. You can do that by running the update executable with the /x switch. Now you have a directory with the files in the update. Depending on the update type, you can ignore update.exe, update.inf, and most of the rest of the files in the root directory of the update. Those are part of the update installer, and are not actually installed. The next step is to find out which applications are holding these files open . Start out by listing the files. Next download a copy of System Internals excellent Process Explorer (http://www.systeminternals.com). Process Explorer, as you can see in Figure 3-2, is much like the Windows Task Manager, on serious steroids!
Process Explorer not only shows you the executables that are running, it also shows you the handles they have open. A "handle" here means that the program holds some object open. For example, an IIS security update may include w3svc.dll. Click Find:Find DLL, and then type in w3svc.dll and press Enter. You will see inetinfo.exe listed. Now, double-click inetinfo.exe in the results window. That switches the main window into DLL view (you can also change into DLL view by hitting Ctrl+D), and shows you not only w3svc.dll, but all the DLLs loaded into the inetinfo.exe process. Double-click any of the DLLs and you get an enhanced properties dialog for it, which gives you a wealth of information about the file. When you are done playing with Process Explorer, write down all the processes that hold the binaries in the update open.
Now you need to decide whether you can avoid a reboot. Frankly, there is no easy answer here, but if you have a lot of servers to patch, it is well worth some testing. If you have a vulnerability or patch scanner, use it at this stage to ensure the patch was applied. Generally speaking, if you can shut down any processes that use the files in the update, you can avoid the reboot. However, keep in mind that some services, such as Windows Management Instrumentation (WMI), automatically restart if you try to just terminate them. Thus, any custom scripts you use to install the updates will have to either perform a graceful shutdown, which the service may prevent, or disable it first, and then terminate it. You can use the sc.exe command-line tool to manage services in a script. After you have figured out which services need to be stopped and how, all that is left is to write a script that does some or all of the following:
Stop the service
Install the update
Re-enable the service
Start the service
Although this will still result in some amount of downtime, it is highly preferable to a full reboot.
In load-balanced environments, you can usually take boxes offline one at a time during nonpeak hours, patch them, and then re-enter them into service. In a cluster environment, you can accomplish this by failing over services to other systems in the cluster temporarily. If you cannot do this for some reason, using the same technique outlined above will still help you minimize the time you are running without a full complement of systems, however. Notice also that using clusters and load balancing to minimize downtime is relatively complex and needs to be carefully planned before execution.
You can also batch update installation so that you can install several updates at one time. This also helps avoid multiple reboots, and can minimize downtime. Most Windows security updates, and some other updates, take various command-line switches that you can use in batch files. For example, the /z switch of a typical Windows security update means "do not reboot the system after patching." This means that you can create a batch file that installs multiple security updates, all with the /z switch, and then reboot afterward. Switches accepted by the update.exe tool used for Windows security updates include the following:
Switch Description ------ ----------- /f Forces other applications to close at shutdown. /n Does not back up files for removing hotfixes. /z Does not restart the computer after the installation is completed. /q Uses Quiet mode; no user interaction is required. /m Uses unattended Setup mode (Windows 2000). /u Uses unattended Setup mode (Windows XP). /l Lists installed hotfixes.
Note that not all updates use this installer and future versions of the installer may have additional switches. For example, Windows Media security updates and some Internet Explorer security updates use a different syntax that does not implement the equivalent of the /z switch. If in doubt, run the update with the /? switch to check the syntax. Even if a patch does not have a /z switch, it usually will not reboot the system without asking first. That means you can run the patch installer asynchronously in a script, using the start command, and just not click Yes in the dialog asking to reboot the system.
In some cases, several security updates update the same files. In this case, it is important to ensure that the installation order is correct. You can do that with the qchain tool. The tool is discussed in Knowledge Base article 296861. Generally speaking, however, qchain is not needed in fixes released after December 2002, because they include the functionality in qchain already. See the Knowledge Base article for more details.