The Importance of Keeping Your Software Up to Date

If you haven't already picked up on the importance of keeping your system properly patched and up to date from all the stories constantly in the media and on various mailing lists, then let us join the chorus. You must get into the habit of monitoring the latest patch releases from your vendors, and you must keep up to date with the various security lists for your products. There is a continuous daily deluge of new vulnerabilities being discovered, which might affect you. If you want to keep the probability of intrusion down, then you really need to make this a top priority in your Information Technology plan. As we already stated, computer products, hardware, and software are created by people, and people are not perfect creatures. They make buggy software, and they keep making buggy software, revision after revision. Hardware is no different. It's critical that you install vendor security patches, new security flash updates for your hardware, and other fixes as quickly as you can after they come out. In some cases we recommend installing them the day they are released. This isn't always practical, so you will have to weigh the risks associated with each new software change in your enterprise against what vulnerabilities they protect you against, if those vulnerabilities are exposed in your enterprise, the problems they fix, and if they introduce any new problems.

This isn't call to procrastinate. Waiting for your peers to show you the way is a recipe for disaster. The really bad news is that by the time a patch comes out, the bad guys are definitely on their way to writing an exploit for it that will be crawling around the Internet within days, if not hours. In some cases, the patch might fix a vulnerability that the bad guys found out about before the good guys did, and are already actively exploiting it. Every day the window of safety between when a patch is released and when it starts to become actively exploited shrinks, and, as stated, sometimes that gap does not exist at all. There are several security approaches that seek to address this "zero day" problemthat is, the notion that you have no time to keep your systems up to date and the assumption that your systems have additional, as yet undiscovered vulnerabilities. We will touch on some technologies available to Linux that can assist in this area. One of these is the new SELinux extensions in the 2.6 kernel. SELinux gives you the ability to define policies for software running on your systems and roles for users. Again, we will cover SELinux briefly in other chapters in this book. We also have a lot of material on SELinux and links to other sites about SELinux at our website (www.gotroot.com).

Returning to the issue of patching your systems and keeping your software up to date, outside of tools such as SELinux (and even with them), patching your systems rapidly is the best way to manage this risk. Even with SELinux and other hardening tools, you are still dependent on those tools to be as bug free as possible, and we already know that bugs are a fact of life. So you will still want to keep your systems up to date even with those tools.

The good news is that there are excellent tools for Linux to help automate the process of patch maintenance. Some of the more well-known ones are apt-get, emerge, red carpet, open carpet, yum, up2date, autorpm and others. Most of these tools can be set up as cronjobs or are already automated to download and to verify the cryptographic signature of and, optionally, install the latest updates. We have found all of these to be excellent tools that in most cases can be trusted to do the right thing automatically, but bewaresometimes a bad patch will make it into the mix. It's extremely rare that this happens. However, it's fair to point out this risk as well. You will need to be prepared for what to do if your patch management system installs an update that breaks something. We can recall one incident when a new version of glibc had a bug in it that prevented systems from booting. Bad things happen. Even patches can have bugs.

If you are unsure, back up your system first before running any of these tools in automated mode. As part of your risk management program, you should be backing your systems up on a nightly basis anyway. In the aforementioned glibc incident, we had a few systems affected by this bug, but fortunately all of them had been backed up the night before, and we simply rolled back to an old version and installed the fixed glibc.

Here are some instructions for using some of the more popular patch and software management tools for Linux:

Yum

yum is an automated package management tool that utilizes the system's existing rpm system to do the package management but provides for automated package downloading, integrity testing, fail over, and other useful networking features. yum was originally part of Yellow Dog Linux and is now used for many Linux distributions, including Redhat, Suse, Fedora and others. We especially like yum because it is very easy to use and to add new software repositories to its configuration. yum also includes gpg/pgp signature verification and download site fail over methods to ensure that patches are downloaded, in case a site is down, and that the patches have not been tampered with. You can run yum out of cron, or you can use some of the init.d scripts that some yum distributions include to add yum to the cron process automatically.

Here is an example yum cron process that would attempt to download and install any new updates to the system at 3:00 a.m. every night of the week. The -y switch simply tells yum to answer "yes" to any questions it might ask. The upgrade option tells yum to only update existing software and not to install any new packages that have been released.

 0 3 * * *     /usr/bin/yum upgrade -y 

You can find yum at the following websites: http://www.gotroot.com and http://www.fedora.us.

Red Carpet

red carpet was Ximian's, now Novell's, package management system. red carpet is a truly advanced package management system, leveraging the systems local package management system (rpm for instance) along with remote host management capabilities, a slick GUI, integrity verification, site fail over, and a true client/server model. This wonder package allows you to remotely control all your hosts running red carpet and to push/remove/install software on all those hosts. You can also run it in standalone mode if you do not need the remote control features, and it also works just as well from the command line as it does from the GUI.

For instance, from the command line you can instruct red carpet to upgrade all the software on a system:

 rug up -y 

or for the older red carpet client:

 rc up -y 

It's important to note though that red carpet, like yum, uses channels to control the software it will manage. You will need to configure red carpet to download and watch for software changes to the channels that matter to your system.

Also red carpet will not automatically install updates. You will need to setup a cronjob to do that for you. Here is an example of a cronjob we have used successfully:

 0 */12 * * *     /usr/bin/rug up -y 

This tells red carpet to run the update process every 12 hours and to download and automatically install all updates. This does not tell red carpet to remove anything; if anything needs to be removed, red carpet will not install any of the updates, so it's important to watch your e-mail for these sorts of errors. If you want to allow removals, just add the -r flag:

 rug up -y -r 

Up2Date

up2date is part of Redhat's patch management system. It too utilizes the underlying systems package management system, in this case rpm, to do the actual package management. up2date provides the networking infrastructure and channel management to download the necessary patches for your system. up2date is also a subscription-based system, and with previous versions of Redhat, the system will stop working, without a subscription, after a short number of days.

The syntax for up2date is very similar to yum and red carpet. To install updates, you would execute this command:

 up2date update 

And as with the previous tools, you can also run this one from cron:

 0 3 * * *     /usr/bin/up2date upgrade -y 

Emerge

emerge is part of the Gentoo system and is part of Gentoo's ports system, a system very similar to BSD's ports system. emerge is the package management, downloading, integrity checking, compiling, installing, whole ball of wax system for Gentoo. At present, it is only used on Gentoo systems, but we don't see why it couldn't be used on others. Also Gentoo currently works by downloading source files and compiling the updates on the fly for that specific system, so with emerge you will not be downloading precompiled binaries; you will have to compile everything. We do not recommend fully automating emerge unless you are comfortable with the ports system and your system is using up considerable cycles to compile the new updates. For our Gentoo systems, we only automate the synchronization phase of the emerge process so that we are notified of any changes in the Gentoo source code repositories. We then manually select the packages we want our Gentoo systems to download, compile, and install. This is not nearly as "hands off" as the other package management systems, so you will need to allocate resources and time to monitor any updates to the Gentoo source code repositoriesand to then have someone review the updates and oversee the compiling process.

When using Gentoo, there are three critical commands for emerge that you will want to familiarize yourself with. The first is sync.

 emerge sync 

This tells emerge to synchronize the local portage tree to the remote source repositories tree. This will tell you if anything has changed and download emerge files that describe the patches to the emerge system and other information needed to download these updates. The sync process will also tell you about new code in the tree. Keep that in mind as welljust because it was added does not mean that it's worth upgrading. Some of the additions many not be stable patches or might be brand new higher revision code that you might not want to use (apache 1.x versus 2.x for instance).

 emerge -up world 

In this example, there are two new switches, -u and -p. -u tells the system to upgrade whatever the object is. In this case the object is the "world" or the entire system. The second switch, -p, tells emerge to upgrade the entire system. We like to use this step before upgrading anything so we can see all the new patches and software related to our systemand then make decisions about what we want to upgrade and/or install.

The final command is to actually tell emerge to upgrade/install the new package.

 emerge <package_name> 

For instance:

 emerge openssh 

emerge would then check its local portage tree, find the openssh entry, locate the latest revision, download it, check for dependencies in the system that might also need to be updated and downloaded, compile all the parts, remove the old ones, and install all the updated code. Unlike binary only installations, this process can be very time-consuming and bandwidth intensive.

There are, of course, many more things you can do with emerge and numerous commands, but these are the big three. As we already stated, we do not recommend automating this entire process. If you are going to automate anything, the sync is the only one we recommend. Also please do not set your emerge sync to run more than once a day, unless you have set up your own mirror of the latest Gentoo repositories.

 0 4 * * *    /usr/bin/emerge sync 

Apt-Get

apt-get is part of Debian but is available for other platforms such as Redhat and others. It's also very similar to yum, up2date and to a lesser extent, red carpet. It's network aware, will check gpg/pgp signatures, provides for fail over and other useful services.

You will notice that the syntax of many of these tools is very similar. With apt-get, if you want to update you system, you simply pass apt-get the update command:

 apt-get update 

And you can automate the process with cronjobs as well:

 0 3 * * *    /usr/bin/apt-get update -y 

As with red carpet, the -y switch tells apt-get to assume yes to any questions, such as should apt-get install a new update. apt-get comes with Debian, and rpms for apt-get are available for Redhat, Fedora, and Yellow Dog Linux at http://apt.freshrpms.net/ and http://www.gotroot.com.

There might be other patching tools, no doubt fantastic ones, which we might have failed to mention. If you have a patching system that works for you, then by all means stick with it. The intent is to make updating your system a regular part of your system's life cycle, and one easy way to do that is to automate the process. Computers don't forget, but people sometimes do. Use a cronjob to keep up with the latest updates, and you won't have to worry about missing anything.