Implementing a Patch and Update Policy

As we have established, new threats and exploits are constantly coming out that impact your network infrastructure. Some reports have stated that the number of new exploits has increased 90 percent a year, every year since 1992. No matter how much we wish that vendors would release bug-free code, it is never going to happen. We have to accept that we will need to patch and upgrade our systems regardless of which vendor or product we are using, and each of these patches or updates represents a change we will need to make in our organization.

A lot of folks like to focus on Microsoft as being the problem with buggy code and patches, and although no doubt Microsoft has a cross to bear here, anyone who seriously pitches Linux or anything else as a better solution solely due to patches or security is approaching the subject with entirely too much naivety. The same can be said with your network infrastructure devices. Do not think that by using a certain vendor you will be protected from having to routinely patch and upgrade your systems. As times change and other vendors and new products displace the market share of the leading vendors and products, those new products will become more and more commonly exploited. You have to accept and be prepared to apply patches and updates for all software and hardware you receive from all your vendors.

Now I have to be careful here, because I am not saying to only buy Microsoft or only buy Cisco. I m also not saying not to buy Microsoft or not to buy Cisco. Instead, I am saying that you must be aware of all the conditions that lend themselves to a particular product having an exploit before you decide whether to use that product. If all you are doing is saying, Well, they seem to have fewer exploits than product X, you probably are not looking at enough information. Information is the key here, and you have to cut through the FUD (fear, uncertainty, and doubt) and religion (see the upcoming sidebar) to make an intelligent and informed decision that weighs not only how buggy a product might be, but also the functionality that product contains, the needs it addresses, the level of support the product has (including responsiveness to bug reports), the interoperability with your existing systems, and the usability of the given product or solution. Simply put, look at the totality of the circumstances before you make a decision on how to proceed.

The Religious Layer of the OSI Model

I often refer to the top three layers of the OSI model as being the political, religious, and monetary layers, because those are the three layers that dictate most decisions an organization makes. The debate over bugs and patches is largely a religious debate. Simply put, you have advocates on all sides who believe with an almost religious fervor in a particular solution, and there is often nothing you can say or do that can change their thinking. They bleed IBM blue, Novell red, or Microsoft rainbow, and that is just the way it is. Fortunately, we can all choose to remain as agnostic as possible and judge a particular solution on its technical merits. If you do this, you will often find that your organization's security policy and posture will be greatly enhanced by your openness to selecting best-of-breed solutions that meet your needs.

 

When to Use a Workaround, Hotfix , Patch, or an Upgrade

Vendors employ four major methods to deal with a particular bug, exploit, or issue. These are to use a workaround, employ a hotfix, release a patch, or release an upgrade. Each of these terms carries a distinct meaning that you need to be aware of so that you can decide which solution is appropriate for your circumstances.

A workaround is something the vendor will recommend to address a problem without the vendor needing to release any new code to the public. Workarounds generally require you to make configuration changes to the product so as to protect against a given exploit. One of the major drawbacks of many workarounds is that often the workaround will interfere with needed functionality. An example of this might be a vendor stating that in order to protect yourself from an exploit in their web server, you should disable the web server. Although this may be a very effective workaround, if you require the use of the web server for your remote management functionality, the workaround is probably no longer valid for your environment.

Hotfixes and patches are often treated as the same thing, though there is (or should be) a subtle difference between the two. Hotfixes are generally made available to vendor technical support without going through exhaustive or detailed quality assurance testing. Consequently, many companies require that customers explicitly request a hotfix so that they can track who has this relatively untested and possibly unstable code. You should avoid hotfixes of this nature unless you are susceptible to the problem the hotfix addresses, and even then I would only apply it to my high-risk resources.

Patches, on the other hand, generally go through a more detailed and exhaustive quality assurance testing process before they are made available to the general public. The benefit of this is that there is a much smaller chance of a patch causing a problem on your systems compared to a hotfix. The drawback, however, is that patches often take more time to be made available to customers, which can leave your systems vulnerable to an exploit for a longer period of time compared to a hotfix or workaround. Even with this drawback, it is a best practice to not widely deploy an update to your systems unless that update is at least of patch quality.

Whereas workarounds, hotfixes, and patches generally address a specific issue, upgrades are designed to address multiple issues. In fact, upgrades are often a combination of multiple hotfixes and patches, potentially with some new functionality included as well. The big benefit of an upgrade is that it has often undergone significant quality assurance testing to ensure as best as can be done that the upgrade will not adversely affect an environment. The drawback of upgrades is twofold. On one hand, it generally takes months for upgrades to be released due to the development and testing that must be performed. On the other hand, upgrades are often not free like many hotfixes and patches are.

Staying Informed of Workarounds, Hotfixes, Patches, and Upgrades

One of the most difficult processes of dealing with patches and upgrades is knowing when patches and upgrades are available. If anyone is unsure of this fact, consider that Microsoft had a hotfix for Code Red almost two months before Code Red was released in the wild. Code Red was 100-percent preventable, yet so many companies and users had not applied a fix that was two months old that Code Red was still able to devastate numerous networks and the Internet in general. Using many of the same resources I mentioned in Chapter 13 is the best method for staying aware of the patches and upgrades your vendor has:

• BugTraq BugTraq is a mailing list maintained by SecurityFocus (http://www.securityfocus.com/archive/1) that tracks bugs and vulnerabilities for virtually every product and vendor on the planet. Subscribe to this mailing list.

• NTBugTraq Modeled after BugTraq, NTBugTraq, according to its website (http://www.ntbugtraq.com/) is a mailing list for the discussion of security exploits and security bugs in Windows NT, Windows 2000, and Windows XP plus related applications. Subscribe to this mailing list.

• Full Disclosure Run by Netsys, Full Disclosure is a mailing list that grew because of the perception that other mailing lists delay information until the vendors have had a chance to build a response, thus delaying the ability for you and me to know about potential threats. Although there is much bluster and hot air behind this, there is a grain of truth to it. You can subscribe to this mailing list at http://lists.netsys.com/mailman/listinfo/full-disclosure.

• Vendor mailing lists Many of your vendors have mailing lists or notification processes to inform their customers of new security-related issues.

• The CERT Coordination Center Located at http://www.cert.org, the CERT/CC serves as a central information reporting center and clearinghouse of security-related information. You should make it a habit to check this website a couple of times a day for new events and information.

• Virus software vendor websites Many virus scan vendors maintain websites with up-to-date information regarding new viruses and worms. Because so many of these programs can affect your network infrastructure, you should make it a point to check your favorite virus software vendor s website a few times a day. Personally, I check with Symantec at http://www.sarc.com/ or Network Associates at http://www.nai.com/us/security/vil.htm.

If you are not able to stay on top of when vendors release a hotfix, patch, or upgrade, you can almost pack up the shop and call it a day because your regular routine is going to consist largely of dealing with the fallout related to an exploit or frantically implementing a patch when you could have worked it into the regular change management schedule instead and thus preventing the exploit from affecting you in the first place. Make sure you stay informed of patches and updates at all costs.

Purchasing Maintenance and Support Agreements

An unfortunate reality with many corporate hardware and software vendors is the use of paid maintenance and support agreements to allow their users to download hotfixes, patches, and upgrades. Although in a perfect world all vendors would make available for free all patches and fixes for the software and hardware you have purchased, many vendors do not do this. The level of support without paying for a maintenance or support agreement differs from vendor to vendor, but in many cases upgrades are only provided to customers who either have purchased a maintenance and support agreement or buy the upgrade outright . Patches and hotfixes are generally made available for free in many, but not all, cases. The more high profile a particular threat is, the greater the likelihood that the hotfix, patch, and sometimes even the upgrade is made available for free.

Another issue that maintenance and support agreements address, in addition to providing access to software updates, is the ability to contact the vendor regarding potential security or technical support issues. Without a valid maintenance and support agreement, many vendors require you to pay for each support call you place. In some cases, they will refund the call if you are the first person to report a valid vulnerability, but that is not always the case. Although maintenance and support agreements on the surface may seem to be just another method vendors use to make more money off of you, when you consider the access to vendor technical support and free software updates, it is probably a good investment to make for your critical business resources at the very least.

Defining a Change Control Patch Policy

It is paramount that you define a policy and procedure that will ensure the application of updates in a rapid fashion, while leaving the network insulated from untested updates as much as possible. Your patch update policy should be defined in accordance with your change control policy, and the process for applying a patch should follow your change control process flow. They should complement each other, with your patch policy addressing issues that are unique to addressing patches and your change control policy dictating the process for applying the patch in your environment. To do so, follow these steps:

1. Define a timely method for identifying and applying patches and updates. If you are waiting months between when updates are released and when you apply them, you are inviting problems. Shortening the time period between availability of a patch and the time you implement it is even more critical when updates are high risk. Remember, in many cases, the patch that corrected the vulnerability exploited by a major worm was available many months before the worm was released. For example, the vulnerability that Code Red exploited was patched two months before Code Red was released. Apply updates in a timely fashion.

Heads Up

It is just a matter of time before we have a zero-day exploit (in other words, an exploit that occurs within a day of a particular vulnerability being discovered ). Because you will likely not be able to patch your systems fast enough to address a zero-day exploit, you have to build a security policy on your network that follows the security principles and practices in this book. For example, if a zero-day exploit comes out that uses port 1433 to spread, but you are blocking all unnecessary ports (including 1433) at your firewall, you can reduce the likelihood that the exploit can infiltrate your network.

 

1. Specify how to determine which workarounds, hotfixes, patches, and upgrades are relevant to your environment. You are not going to be susceptible to every exploit that is out there, so you do not want to spend time worrying about issues that aren t relevant to your environment. For example, if a Microsoft Outlook exploit is made public but you don t use Microsoft Outlook in your environment, you don t need to worry about patching your systems.

2. Require explicit identification of the systems that need to be upgraded. This serves a couple of purposes. First, it allows you to identify whether any critical resources are susceptible to a given exploit. Second, it allows you to identify the scale of the task that it will take to update your systems. If you only have five systems that the update is applicable to, applying the update will probably be a relatively easy process. However, if you have identified that all 20,000 desktops in your environment are susceptible, the scale of the task just got much more difficult because you will probably need to rely on scripting or deployment software to apply the update.

3. Detail the implementation process. Once you have identified the relevant upgrades and the systems affected, you should proceed through the change control process to begin the planning and implementation of the patch or update. Depending on the severity of the issue with the patch or update, you might follow your standard change control process or you might need to invoke your emergency change control process. Regardless of which process you follow, you should always adhere to your change control process.

Writing Patch and Update Procedures

The most important aspect of patching and upgrading your systems is to define the patch and upgrade process and procedures. For the vast majority of your network infrastructure products, you may need to make three types of changes as part of your patch and update process:

• The first type of change is to upgrade the actual system image. For example, Cisco may release a new version of the IOS that addresses the relevant security issues or provides new features that you require. In turn , this requires you to upgrade your existing IOS version to a new version. This type of change is commonly applied to the actual network infrastructure hardware, such as routers, switches, hardware-based firewalls, and intrusion detection/prevention systems.

• The second type of change is to upgrade or change the configuration of the network device. During a configuration change, the actual system image is not going to be touched in any fashion, but rather you will simply reconfigure the device to address the security issue you are concerned about. This type of change can be applied to all your network infrastructure devices because they all support some kind of configuration mechanism.

• The third type of change involves upgrading an application that is running as part of your network infrastructure. For example, your content-filtering system typically is installed as an application on an existing network operating system such as Microsoft Windows or Linux. You may need to install a vendor-supplied patch to upgrade or replace the application to address the security issue you are concerned about.

Although each method of change has unique characteristics, there are some common elements that we should consider to allow us to safely and effectively apply these patches and upgrades in our environment. They are listed here:

• Plan the change. You have to plan the change in order to identify any potential problems before you decide to deploy the change. Proper planning will ensure the success of your change.

• Make backup images before and after a change. You should have a central TFTP server to send images and configuration files to and from. When you make backups , make them of the running and startup configuration both before and after the changes. In addition, back up any old OS images prior to upgrading by whatever means that device has. Many times, the running configuration and startup configuration don t match, which means when the device gets rebooted, something is broken or undone because someone made changes to the running configuration and didn t save them.

• Test the upgrade . You have to test the change as much as possible. You do not want to deploy a change to your environment that has not been tested . Where possible, you should test the change in a controlled environment, such as a lab. If you lack those resources, however, you should at least test the change against noncritical production resources, where the negative impact of the change would be minimal, before you deploy it widely in your environment.

• Adhere to your change control process. Above all else, you must adhere to your defined change control process. It exists for a reason, and this is that reason. By strictly adhering to your change control process, you gain a methodical and structured approach to the change, which will ultimately cause the change implementation to occur much smoother.

Changing the System Image

One of the most delicate changes you can make in your environment is changing the system image of a device. This is due in large part to the fact that if you incorrectly or improperly change the system image, you could be looking at a very laborious and time-consuming remedy that may even require travel to a remote location. At the same time, however, if you have planned and tested your system image change, the upgrade process can be very simple and straightforward.

Because the system image typically is stored in flash (non-volatile memory), you can change the system image while the device is still running, without impacting the currently running system image. You can then reboot the device at your leisure, and upon reboot the device will read and load the new system image into RAM and begin functioning based on the new system image.

Planning Your Patch Response

I worked at a company that did not plan before they reacted to the Nachia/Blaster worms. When they realized that it was using port 135 to spread, they blocked port 135 at all their internal and external routers within their infrastructure. On the surface, this sounded like a good way to isolate the worm on subnets; however, they did not consider the fact that virtually all Microsoft communications require the ability to communicate on this port; therefore, because their Exchange server was on a dedicated segment away from the user segments, all e-mail, in addition to file and print and many database functions, ceased working. They had effectively implemented a denial of service against their users in an attempt to control the spread of the worms. Had they done even the most basic of planning and testing before distributing this change in their environment, they would not have spent hours trying to undo a change in the middle of a security incident, and they would have been able to spend more time focusing on more effective methods of controlling the worm.

 

So how exactly can we change the system image? A few methods are available that work for most of the network hardware that is out there:

• Use vendor-supplied tools to change the system image. Tools include Cisco s CiscoWorks and Nortel Networks Optivity network management software.

• Manually change the system image. You can manually connect to and run the commands to perform the change.

• Automate the manual process for changing the system image. You can use Perl, VBScript, or another scripting language as well as SNMP to make the changes.

Heads Up

Many times people make changes to the running configuration and never save those changes to their startup configuration, which causes the device to lose whatever changes had been made when the device is rebooted. This happened at a location where I used to work. They had changed their routing to use an Internet gateway at a new location, but they didn't save that change in their core router. They had to reboot the router due to a hardware problem, and when they did, suddenly no one could get to the Internet anymore. This was due to the fact that the router was trying to send Internet-bound traffic to the old gateway, which was trying to send it right back to the core router.

 

Because the vendor-supplied tools provide an abundance of documentation regarding how to use them to make changes in your environment, I am going to focus on the second and third methods.

Manually Changing the System Image

Manually making changes to the system image is an effective, albeit time-consuming, method of updating the system image on your network devices. Virtually all vendors use the same method of upgrading their systems through the use of Trivial File Transfer Protocol (TFTP). The obvious drawback to TFTP is the lack of any encryption or authentication mechanism, and although some vendors can allow you to define the TFTP server that the device is permitted to use to obtain software and updates from, this is still a highly suspect security practice because of the underlying lack of encryption or authentication.

The actual implementation process tends to be rather straightforward. Most vendors use a command-line interface (CLI) that is very similar to the Cisco IOS CLI, or they use a menu-driven CLI. For a menu-driven CLI, the upgrade process is as simple as navigating to the appropriate menu and entering the IP address of the TFTP server and the full path and name of the file to retrieve. The file is then transferred and will typically overwrite the previous system image. The final step is to simply reboot the device, allowing the device to read and process the new system image.

Note  

Although this procedure focuses on the actual commands and uses command examples from a Cisco device running IOS, it is largely the same for all IOS-based CLIs, such as those by Extreme Networks, Nortel Networks, Dell, and Foundry Networks.

From the IOS CLI, the process is very similar, although you often have additional options to consider. The most noteworthy difference is the ability to decide whether you are going to overwrite the existing system image or install the new system image in addition to the existing system image. The biggest benefit of this choice is that if you decide not to overwrite the previous system image, you can quickly and easily roll back to the previous system image in the event of a problem. You can do this by specifying the boot file to use in the device configuration and setting the configuration register appropriately:

 local-rtr#configure terminal Enter configuration commands, one per line.  End with CNTL/Z. local-rtr(config)#boot system flash c2500-jk8os-l.122-1d.bin local-rtr(config)#config-register 0x010F local-rtr(config)#exit local-rtr#copy running-config startup-config 

In this case, I have specified to boot using the file c2500-jk8os-l.122-1d.bin. If I wanted to boot using the file c2500-jk8os-l.113-1.bin, I would simply replace that value in the configuration, and my device would roll back to booting on the previous system image.

The decision to overwrite your previous system image is generally going to be defined by whether you have enough storage for both system images in flash. If you do, I recommend that you keep the most recent running system image and the new system image at a minimum for backup purposes. If you don t have enough storage for both, it is OK to overwrite the existing system image if you are sure the new system image will work. Although you can usually recover from an incorrectly installed system image, the process is laborious and may require physical access to the device in order to be performed.

The first step of manually upgrading your system image is to make sure the system image file exists on your TFTP server, as shown here:

The next step is to ensure that your TFTP server is running and operational. If you are using a TFTP server application, make sure that the application is running and that the TFTP server is using the same directory your system image file exists in, as shown here:

At this point, it is time to connect to the network device and run the commands that will upgrade the system image. You should first verify that you can connect to the TFTP server by running a ping from the network device. The next step is to simply copy the system image file to the flash on the target device. When doing that, you can choose whether to erase the contents of flash (to not erase the contents of flash, type n when prompted). The file will be copied to the device, and upon completion and checksum verification, you can simply reboot the device for the new system image to be loaded.

Here s an example of upgrading the system image while erasing the contents of flash:

 Local-rtr#copy tftp://192.168.1.254/c1721-y-mz.122-1.bin flash Destination filename [c1721-y-mz.122-1.bin]? <  press enter  > Accessing tftp://192.168.1.254/c1721-y-mz.122-1.bin... Erase flash: before copying? [confirm] <  press enter  > Erasing the flash filesystem will remove all files! Continue? [confirm] <  press enter  > Erasing device... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee ...erased Erase of flash: complete Loading c1721-y-mz.122-1.bin from 192.168.1.254 (via Serial0): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [OK - 3491304/6981632 bytes] Verifying checksum...  OK (0xB352) 3491304 bytes copied in 430.392 secs (8119 bytes/sec) 

Heads Up

If you select to erase the contents of flash, you will erase all the contents of flash. Make sure you do not have any other files in flash that you need to save. You can check this by running the command show flash at the privileged mode to view the contents of flash.

 

Here s an example of upgrading the system image without erasing the contents of flash:

 Local-rtr#copy tftp://192.168.1.254/c1700-y-mz.122-1.bin flash:c1700-y- mz.122-1.bin Destination filename [c1700-y-mz.122-1.bin]? <  press enter  > Accessing tftp://192.168.1.254/c1700-y-mz.122-1.bin... Erase flash: before copying? [confirm]  n  Loading c1700-y-mz.122-1.bin from 192.168.1.254 (via Serial0): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [OK - 3491304/6981632 bytes] Verifying checksum...  OK (0xB352) 3491304 bytes copied in 439.576 secs (7952 bytes/sec) 

Automate Changing the System Image

The obvious drawback to manually changing the system image is the time it takes to manually connect, log into, and run the commands required to update the system image. The alternative is to automate the change. There are two predominant methods for automating the upgrade procedures of the system image on your network infrastructure devices, depending on what your vendor supports:

• Use SNMP and scripting to perform the upgrade. (Not all vendors support using SNMP for performing this change.)

• Use a management client, such as Telnet or SSH, that supports scripting to automate the manual commands. (Not all management clients support scripting.)

If your network device supports using SNMP to upgrade the system image, you can script the necessary SNMP commands that will enable this functionality. Foundry Networks supports doing this on both their management processors and their switching processors. The steps to using SNMP to upgrade the system image are as follows:

1. Configure the device with a read-write community string. You can run the command snmp-server community < string > rw to configure a read-write string.

2. Disable the password checking for SNMP set requests . This is a proprietary measure that must be done only if you are using a third-party (non-Foundry) SNMP management application. The command to run is no snmp-server pw-check .

   Heads Up

   Disabling the password-checking option will reduce a proprietary security measure that Foundry uses to protect against unauthorized SNMP set requests. When you attempt to issue an snmpset request to a Foundry device, it expects you to provide the privileged password in the SNMP request. If you use a Foundry SNMP tool, you can enter the password and you don't need to disable the password checking functionality. However, if you use a third-party SNMP tool that does not support passing this data, you have to disable this functionality. If you do disable this functionality, I strongly recommend that you reenable it after you have performed the required functions.

    

3. For your management process, run the following command:

    snmpset c <rw-community-string> <device-ip-address> 1.3.6.1.4.1.1991.1.1.2.1.5.0 ipaddress <tftp-ip-address> 1.3.6.1.4.1.1991.1.1.2.1.6.0 octetstringascii <image-file-name> 1.3.6.1.4.1.1991.1.1.2.1.7.0 integer <command-integer> 

The supported command integers are the following:

20	Download the flash code into the device s primary flash area.
22	Download the flash code into the device s secondary flash area.

For your switching processors on a chassis device, run the following command:

 snmpset c <rw-community-string> <device-ip-address> 1.3.6.1.4.1.1991.1.1.2.1.5.0 ipaddress <tftp-ip-address> 1.3.6.1.4.1.1991.1.1.2.1.6.0 octetstringascii <image-file-name> 1.3.6.1.4.1.1991.1.1.2.1.56.0 integer <module-type> 1.3.6.1.4.1.1991.1.1.2.1.57.0 <slot-number> 1.3.6.1.4.1.1991.1.1.2.1.7.0  integer <command-integer> 

<module-type> is one of the following values:

2	VM1 module
3	OC-3, OC-12, and OC-48 non “Network Processor Architecture (NPA) POS modules
4	OC-48 NPA POS modules
5	ATM module

<slot-number> is the slot that contains the module you are upgrading. To upgrade all slots, you can enter (zero).

For <command-integer>, enter one of the following values:

24	Download the flash code into the device s primary flash area.
25	Download the flash code into the device s secondary flash area.

You could then write a script that runs the appropriate snmpset commands for each device in your network that you want to change.

If your device does not support using SNMP to automate upgrading the system image, you can attempt to script the manual commands, provided you have a Telnet or SSH application that supports it. For this purpose, I recommend SecureCRT by VanDyke Technologies (http://www.vandyke.com). SecureCRT supports using a variant of VBScript as their scripting language. The steps to script an upgrade of the system image on a Cisco IOS “based router are as follows:

1. Create a new session in SecureCRT for each device you will be connecting to. The script will use these session names to define which devices to connect to. Do not use spaces in the folder or session name. For example, in the following screen I have created a folder named Routers and created seven sessions to connect to router1 through router7.

   

2. Write a script that performs the following functions:

   • Provides the required login information. You can either script the login as part of your session creation, as shown next, or you can have the script prompt for the appropriate login information. For simplicity, I placed the login information in the session creation.

     

   • Defines the TFTP server that the system image is located on.

   • Defines the system image source filename.

   • Defines an array containing all the devices you want to connect to. In the example, I have defined an array of seven values to connect to router1 through router7.

   • Defines a subroutine that contains the input and output you will pass through SecureCRT to the device.

   • Provides a logging mechanism so that you can verify the success of all functions.

   • Provides error-handling capabilities as required.

In the following sample script, I have created seven sessions for seven different routers in SecureCRT. I am using the SecureCRT session login scripts to log into the routers. The script will connect to a TFTP server located at 192.168.1.45 and copy a new system image file named c1700-y-mz.122-1.bin. If the copy result is successful, it will simply be logged into a file and will proceed to the next router in the array until all routers have completed. If there is an error with the TFTP server or system image name, it will prompt the user to find out whether they want to enter a new TFTP server name or IP address and system image name. If the user selects no, the script logs the error and exits. If the user selects yes, the script will attempt to reconnect using the new information.

Note  

Due to page width limitations, some lines in this code will be wrapped.

 #$language = "VBScript" #$interface = "1.0" '========================================================================== ' VBScript Source File -- Created with SAPIEN Technologies PrimalSCRIPT(TM) ' NAME: IOS_update.vbs ' AUTHOR: Wes Noonan , WJN Consulting, LLC. [e-mail] ' DATE  : 01/14/2004 ' COMMENT: This script is used by SecureCRT to automate a process. While it ' is based on VBScript, it is customized to run in SecureCRT as evidenced ' by the first 2 lines above. See www.vandyke.com for more info. ' This script will automatically login to a series of Cisco switches/routers ' as defined in an array and attempt to upgrade the system image file from a ' TFTP server. If the copy fails, the user will be prompted whether to  ' continue updating the remaining systems or exit. All transactions will be ' logged to a file so that the admin can verify the status of the copy. '========================================================================== ' dim variables ' change the number to represent the number of values in the array below dim SwitchArray(4) dim SessionName, DisconnectString, Switch, Start, Append, loginpass dim enablepass, tftpservername, sourcefilename, result, errorresult dim exitresult, logfile, errorornot, screenrow, datetime ' Define constants Const ICON_STOP = 16                 ' display the ERROR/STOP icon. Const ICON_QUESTION = 32             ' display the '?' icon Const ICON_WARN = 48                 ' display a '!' icon. Const ICON_INFO= 64                  ' displays "info" icon. Const BUTTON_OK = 0                  ' OK button only Const BUTTON_CANCEL = 1              ' OK and Cancel buttons Const BUTTON_ABORTRETRYIGNORE = 2    ' Abort, Retry, and Ignore buttons Const BUTTON_YESNOCANCEL = 3         ' Yes, No, and Cancel buttons Const BUTTON_YESNO = 4               ' Yes and No buttons Const BUTTON_RETRYCANCEL = 5         ' Retry and Cancel buttons Const DEFBUTTON1 = 0        ' First button is default Const DEFBUTTON2 = 256      ' Second button is default Const DEFBUTTON3 = 512      ' Third button is default ' Possible MessageBox() return values Const IDOK = 1              ' OK button clicked Const IDCANCEL = 2          ' Cancel button clicked Const IDABORT = 3           ' Abort button clicked Const IDRETRY = 4           ' Retry button clicked Const IDIGNORE = 5          ' Ignore button clicked Const IDYES = 6             ' Yes button clicked Const IDNO = 7              ' No button clicked ' set variables to allow logging. Change as needed. Start = true Append = true Logfile = "C:\Test.log" ' This variable sets the date/time variable to ensure that the config file  will be unique Datetime = Month(Date) & Day(date) & Year(Date) & "-" & Hour(time) &  Minute(time) & Second(time) ' prompt for variables to be used in the script ' remove the comment on the next 4 lines if you want to be prompted for the  information ' and comment the last two lines out. Otherwise leave as is for logging on using session ' login scripts and statically defined tftp server and source system image  names 'loginpass = crt.Dialog.Prompt("Enter the login password:", "Login  Password", "", True) 'loginpass = "" 'enablepass = crt.Dialog.Prompt("Enter the enable password:", "Enable Password", "", True) 'enablepass = "" 'tftpservername = crt.Dialog.Prompt("Enter the name or IP address of the  TFTP server:", "TFTP Server", "Enter the TFTP Server Name or IP Address") 'sourcefileame = crt.Dialog.Prompt("Enter the name of the system image  source file:", "Source File Name", "Enter the name of the system image  source file") tftpservername = "192.168.1.45" sourcefilename = "c1700-y-mz.122-1.bin" ' define the array of devices to configure. ' These should be the hostnames of the devices you want to update ' you will need to change the array values as you add/remove devices SwitchArray(1) = "Routers\Router1" SwitchArray(2) = "Routers\Router2" SwitchArray(3) = "Routers\Router3" SwitchArray(4) = "Routers\Router4" ' change the numbers to reflect the min/max value of the array defined above For Switch = 1 to 4       SessionName = SwitchArray(Switch)       for each object in SwitchArray             If SessionName <> "" then                   ' turn on synchronous mode so we don't miss any data                   crt.Screen.Synchronous = True                   ' connect using SecureCRT session names                   crt.session.Connect("/s " & SessionName)                   ' log the results to review for success                   crt.Session.LogFileName = LogFile                   crt.Session.Log Start, Append                   ' run login subroutine do not use if you are using session login scripts                   'login                   ' Send a command to display the date/time for the log                   crt.Screen.WaitForString "#"                   crt.Screen.Send "show clock" & vbCr                   ' run tftpconnect subroutine                   tftpconnect                   ' Check to see if there was an error updating. If so, give  the option to exit the script                   crt.Screen.WaitForString "#"                   screenrow = crt.screen.CurrentRow - 1                   errorornot = crt.Screen.Get(screenrow, 1, screenrow, 6)                   If errorornot = "%Error" Then                         errorresult = crt.Dialog.MessageBox("There was an  error updating the system image for: " & VbCr & VbCr & SessionName & VbCr & VbCr & "Check the log at " & LogFile & " for the error output." & VbCr &  VbCr & "Select OK to continue, CANCEL to end the script.", "Error Backing Up  Configuration!!", ICON_WARN or BUTTON_CANCEL)                         If errorresult = IDCANCEL then                              exitresult = "1"                              ExitDialog                         End if                   End If                   ' Wait for copy to end then disconnect from device                   crt.session.Disconnect                   ' Set the SessionName variable to nothing to cause the                   ' script to run through each session in the array                   SessionName = ""                   ' turn off synchronous mode to restore normal input  processing                   crt.Screen.Synchronous = False             end if       next next ExitDialog ' subroutines used in script are listed below ' This subroutine contains the logon functions Sub login       ' Wait for a string that looks like "password: " or "Password: "       'crt.Screen.WaitForString "Password: "       ' Send your username followed by a carriage return       'crt.Screen.Send loginpass & VbCr       ' Send the command to enter priviledged mode       'crt.Screen.Send "ena" & VbCr       ' Wait for a tring that looks like "password: " or "Password: "       'crt.Screen.WaitForString "Password:"       ' Send your password followed by a carriage return       'crt.Screen.Send enablepass & VbCr End Sub ' This subroutine contains the commands to connect to the TFTP server ' and copy the system image to flash Sub tftpconnect       ' Copy config from TFTP       crt.Screen.Send "copy tftp flash" & VbCr       ' wait for the prompt for the tftp server name       crt.Screen.WaitForString "]?"       crt.Screen.Send tftpservername & VbCr       ' wait for the prompt for the source image name       crt.Screen.WaitForString "]?"       crt.Screen.Send sourcefilename & VbCr       ' this line accepts the default name (the original source image name)       crt.Screen.WaitForString "Destination filename [" & sourcefilename &  "]?"       crt.Screen.Send VbCr       ' this line confirms to erase the flash       ' this will also detect if there was a tftp error       If crt.Screen.WaitForString ("[confirm]",20) = True Then             errorornot = ""             crt.Screen.Send VbCr       else             tftperror       End if       ' this line reconfirms to erase the flash       crt.Screen.WaitForString "[confirm]"       crt.Screen.Send VbCr End Sub ' This subroutine checks for an error connecting to the TFTP server Sub tftperror       screenrow = crt.screen.CurrentRow - 1       errorornot = crt.Screen.Get(screenrow, -1, screenrow, 1) Do While errorornot = "%"       ' If there was an error, see if the user wants to continue or exit       If errorornot = "%" then             errorresult = crt.Dialog.MessageBox("The server name " &  tftpservername & " or system image name " & sourcefilename & " was invalid."  & VbCr & VbCr & "Do you want to enter a new server name and source system  image name?" & VbCr & VbCr & "Select YES to enter a new name and continue."  & VbCr & VbCr & "Select NO to end the script.", "TFTP Server Name  Unknown!!", ICON_WARN or BUTTON_YESNO)             if errorresult = IDNO then                   exitresult = "1"                   Exitdialog                   else if errorresult = IDYES then                         tftpservername = crt.Dialog.Prompt("Enter the name  or IP address of the TFTP server:", "TFTP Server", "Enter the TFTP Server  Name or IP Address")                         sourcefilename = crt.Dialog.Prompt("Enter the name  of the source system image file name:", "System Image Name", "Enter the name  of the source system image name")                         tftpconnect                   End if             End if       End if Loop End Sub ' This subroutine exits the script and closes SecureCRT Sub ExitDialog       If exitresult = "1" then             result = crt.Dialog.MessageBox("There was an error and the  script was stopped." & VbCr & "Please check the log at " & LogFile & " for  more details.", "Copy Failed!!", ICON_STOP or BUTTON_OK)             Else             result = crt.Dialog.MessageBox("Please check the log at " &  LogFile & VbCr & "to verify that all the devices have upgraded.", "Upgrade  Complete", ICON_INFO or BUTTON_OK)       End If       crt.quit End Sub 

Changing the System Configuration

System configuration changes are perhaps the most common patch/upgrade procedure that you will do. This is due to the fact that if a new system image that addresses the problem does not exist or if you cannot upgrade the system image for some reason, you may have a workaround available by simply changing the system configuration to address the specific concern.

The methods of changing your system configuration are virtually the same as changing your system image. Changing your system configuration can be performed via a vendor-specific configuration management tool such as CiscoWorks or Nortel Optivity. This can be manually performed or can be automated in some fashion.

Manually Changing the System Configuration

Manually changing your configuration is as simple as connecting to your device via a GUI, Telnet, or SSH and running the appropriate commands. In the case of CLI-based products, you can often simply copy and paste the configuration commands from a text file into the CLI. For example, if I wanted to change the SNMP community strings and add an access list entry that restricted the systems that could connect via SNMP, I would need to run the following commands after logging into a Cisco IOS “based router and accessing the privileged mode:

 configure terminal access-list 10 remark Access List for SNMP Access access-list 10 permit host 192.168.1.45 snmp-server community readonlystring RO 10 snmp-server community readwritestring RW 10 exit write memory 

So what I could do is simply list the commands in order in a text file, copy the contents of the file as shown here, and connect to each device I needed to update. Then, as soon as I logged in, I could simply paste the commands into the CLI. The router would run each command and finish by saving the active configuration. In a sense, you somewhat automate the configuration update process by doing this in this manner. If you want to fully automate the process, you must either use SNMP and scripting or script the commands necessary to upgrade the configuration in a fashion similar to upgrading the system image.

Automate Changing the System Configuration

Updating your configuration is much simpler to do than upgrading your system image. This is due to the fact that your configuration is really just a batch file of commands to run. Consequently, you can build a configuration file that contains only the commands that you want to run and then load it on your device. This will cause the router to parse the new configuration file, making it part of its active configuration, at which point you can choose to either save the configuration or not. For example, let s say that I wanted to make the previously mentioned changes on a Cisco IOS “based router using SNMP. I would need to perform the following steps:

1. Build a text file containing the commands I want to run on my router. For example, I could put the following commands in a text file:

    access-list 10 remark Access List for SNMP Access access-list 10 permit host 192.168.1.45 snmp-server community readonlystring RO 10 snmp-server community readwritestring RW 10 

2. Store the configuration file on a TFTP server (for example, as snmpupdate).

3. Run the following command to tell the router to copy the source file from the network:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  1.3.6.1.4.1.9.9.96.1.1.1.1.3.12 int 1 

4. Run the following command to tell the router to copy the destination file to the running configuration:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.4.12 int 4 

5. Run the following command to tell the router the IP address of the TFTP server:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.5.12 a <tftp-ip-address> 

6. Run the following command to tell the router the name of the configuration file to use from the TFTP server:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.6.12 s <config-file-name> 

One Step Further

Cisco uses the following integer values for copying from and copying to:

1	Network File (from a TFTP server, for example)
2	iosFile
3	Startup Configuration
4	Running Configuration
5	Terminal

You can change the integer to reflect the kind of copy that you want to perform. For example, if you wanted to copy from the running configuration to the startup configuration, you would specify integer 4 for the source file and integer 3 for the destination file.

 

1. Run the following command to tell the router to initiate the copy:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.14.12 int 1 

2. Because I updated the running configuration, the next step is to copy the running configuration to the startup configuration to make the changes permanent. First, however, I must clear the row status of the previously entered values:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.14.12 int 6 

3. Run the following command to tell the router to copy the source file from the running configuration:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.3.12 int 4 

4. Run the following command to tell the router to copy the destination file to the startup configuration:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.4.12 int 3 

5. Run the following command to tell the router to initiate the copy:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.14.12 int 1 

6. The final step is to clear the row status one final time to remove the existing values:

    snmpset -v 2c -c <read-write-community> <router-ip-address>  .1.3.6.1.4.1.9.9.96.1.1.1.1.14.12 int 6 

I could then write a script that ran the appropriate snmpset command for each device in my network that I wanted to change.

I can also script the updating of my configuration by using the manual commands at the CLI using SecureCRT. The steps to script an upgrade of the running configuration on a Cisco IOS “based router are as follows:

1. Build a text file containing the commands that I want to run on my router. For example, I could put the following commands in a text file:

    access-list 10 remark Access List for SNMP Access access-list 10 permit host 192.168.1.45 snmp-server community readonlystring RO 10 snmp-server community readwritestring RW 10 

2. Store the configuration file on a TFTP server (for example, as snmpupdate).

3. Create a session in SecureCRT for each device that I want to connect to, as previously described.

4. Write a script that performs the following functions:

   • Provides the required login information. I can either script the login as part of my session creation, as previously described, or I can have the script prompt for the appropriate login information. For simplicity, I place the login information in the session creation.

   • Defines the TFTP server that the configuration file is located on.

   • Defines the configuration file source filename.

   • Defines an array containing all the devices I want to connect to. In the example, I have defined an array of seven values to connect to router1 through router7.

   • Defines a subroutine that contains the input and output that I will pass through SecureCRT to the device.

   • Provides a logging mechanism so that I can verify the success of all functions.

   • Provides error-handling capabilities, as required.

In the following sample script, I have created seven sessions for seven different routers in SecureCRT. I am using the SecureCRT session login scripts to log into the routers. The script will connect to a TFTP server located at 192.168.1.45 and copy the configuration file snmpupdate to the running configuration. If the copy result is successful, the router will save the running configuration to the startup configuration, log the success into a file, and proceed to the next router in the array, until all routers have completed. If there is an error with the TFTP server or configuration filename, it may prompt the user to determine whether they want to enter a new TFTP server name or IP address and configuration file. If the user selects no, the script logs the error and exits. If the user selects yes, the script will attempt to reconnect using the new information.

Note  

Due to page width limitations, some lines in this code will be wrapped.

 #$language = "VBScript" #$interface = "1.0" '========================================================================== ' VBScript Source File -- Created with SAPIEN Technologies PrimalSCRIPT(TM) ' NAME: tftpupdate.vbs ' AUTHOR: Wes Noonan , WJN Consulting, LLC. [e-mail] ' DATE  : 01/11/2004 ' COMMENT: This script is used by SecureCRT to automate a process. While it ' is based on VBScript, it is customized to run in SecureCRT as evidenced ' by the first 2 lines above. See www.vandyke.com for more info. ' This script will automatically login to a series of Cisco switches/routers ' as defined in an array and attempt to update them by using TFTP. If the ' update fails, the user will be prompted whether to continue updating the ' remaining systems or exit. All transactions will be logged to a file so ' that the admin can verify the success/failure of the updates. '========================================================================== ' dim variables dim SessionName, DisconnectString, SwitchArray(4), Switch, Start dim Append, loginpass, enablepass, tftpservername, configfilename dim result, errorresult, exitresult, logfile, errorornot, screenrow ' Define constants Const ICON_STOP = 16                 ' display the ERROR/STOP icon. Const ICON_QUESTION = 32             ' display the '?' icon Const ICON_WARN = 48                 ' display a '!' icon. Const ICON_INFO= 64                  ' displays "info" icon. Const BUTTON_OK = 0                  ' OK button only Const BUTTON_CANCEL = 1              ' OK and Cancel buttons Const BUTTON_ABORTRETRYIGNORE = 2    ' Abort, Retry, and Ignore buttons Const BUTTON_YESNOCANCEL = 3         ' Yes, No, and Cancel buttons Const BUTTON_YESNO = 4               ' Yes and No buttons Const BUTTON_RETRYCANCEL = 5         ' Retry and Cancel buttons Const DEFBUTTON1 = 0        ' First button is default Const DEFBUTTON2 = 256      ' Second button is default Const DEFBUTTON3 = 512      ' Third button is default ' Possible MessageBox() return values Const IDOK = 1              ' OK button clicked Const IDCANCEL = 2          ' Cancel button clicked Const IDABORT = 3           ' Abort button clicked Const IDRETRY = 4           ' Retry button clicked Const IDIGNORE = 5          ' Ignore button clicked Const IDYES = 6             ' Yes button clicked Const IDNO = 7              ' No button clicked ' set variables to allow logging. Change as needed. Start = true Append = true Logfile = "C:\Test.log" ' prompt for variables to be used in the script 'loginpass = crt.Dialog.Prompt("Enter the login password:", "Login  Password", "", True) 'loginpass = "" 'enablepass = crt.Dialog.Prompt("Enter the enable password:", "Enable  Password", "", True) 'enablepass = "" 'tftpservername = crt.Dialog.Prompt("Enter the name or IP address of the  TFTP server:", "TFTP Server", "Enter Server Name or IP Address") tftpservername = "192.168.1.45" 'configfilename = crt.Dialog.Prompt("Enter the name of the config file to  download from the TFTP server:", "Config File", "Enter Config File Name") configfilename = "snmpupdate" ' define the array of devices to configure. ' These should be the hostnames of the devices you want to update ' you will need to change the array values as you add/remove devices SwitchArray(1) = "Routers\Router1" SwitchArray(2) = "Routers\Router2" SwitchArray(3) = "Routers\Router3" SwitchArray(4) = "Routers\Router4" ' change the numbers to reflect the min/max value of the array defined above For Switch = 1 to 4       SessionName = SwitchArray(Switch)       for each object in SwitchArray             If SessionName <> "" then       ' *** If you want the config file to change on a per device basis  remark the configfilename       ' *** variable on line 79 and unremark the lines below. This will  allow you to save the       ' *** config for each router/switch in a unique name based on the  session/host name.       ' *** The msgbox line is for debugging purposes       'configfilename = SessionName & "-config"       'msgbox configfilename                   ' turn on synchronous mode so we don't miss any data                   crt.Screen.Synchronous = True                   crt.session.Connect("/s " & SessionName)                   ' log the results to review for success                   crt.Session.LogFileName = LogFile                   crt.Session.Log Start, Append                   ' run login subroutine                   'login                   ' Send a command to display the date/time for the log                   crt.Screen.WaitForString "#"                   crt.Screen.Send "show clock" & vbCr                   ' run tftpconnect subroutine                   tftpconnect                   ' Check to see if there was an error updating. If so, give  the option to exit the script                   crt.Screen.WaitForString "#"                   screenrow = crt.screen.CurrentRow - 1                   errorornot = crt.Screen.Get(screenrow, 1, screenrow, 6)                   If errorornot = "%Error" Then                         errorresult = crt.Dialog.MessageBox("There was an  error updating system: " & VbCr & VbCr & SessionName & VbCr & VbCr & "Check  the log at " & LogFile & " for the error output." & VbCr & VbCr & "Select OK  to continue, CANCEL to end the script.", "Error Updating System!!", ICON_WARN or BUTTON_CANCEL)                         If errorresult = IDCANCEL then                               exitresult = "1"                               ExitDialog                         End if                   End If                   ' run writechange subroutine                   writechange                   ' Wait for copy to end then disconnect from device                   crt.session.Disconnect                   ' Set the SessionName variable to nothing to cause the                   ' script to run through each session in the array                   SessionName = ""                   ' turn off synchronous mode to restore normal input  processing                   crt.Screen.Synchronous = False             end if       next next ExitDialog ' subroutines used in script are listed below ' This subroutine contains the logon functions 'Sub login       ' Wait for a string that looks like "password: " or "Password: "       'crt.Screen.WaitForString "Password: "       ' Send your username followed by a carriage return       'crt.Screen.Send loginpass & VbCr       ' Send the command to enter priviledged mode       'crt.Screen.Send "ena" & VbCr       ' Wait for a tring that looks like "password: " or "Password: "       'crt.Screen.WaitForString "Password:"       ' Send your password followed by a carriage return       'crt.Screen.Send enablepass & VbCr 'End Sub ' This subroutine contains the commands to connect to the TFTP server Sub tftpconnect       ' Copy config from TFTP       crt.Screen.Send "copy tftp run" & VbCr       crt.Screen.WaitForString "Address or name of remote host []?"       crt.Screen.Send tftpservername & VbCr       If crt.Screen.WaitForString ("Source filename []?",3) = True Then             errorornot = ""             entertftpfile       else             tftperror       End if End Sub ' This subroutine contains the commands to enter the TFTP file to use Sub entertftpfile       crt.Screen.Send configfilename & VbCr       crt.Screen.WaitForString "Destination filename [running-config]?"       crt.Screen.Send VbCr End Sub ' This subroutine checks for an error connecting to the TFTP server Sub tftperror       screenrow = crt.screen.CurrentRow - 1       errorornot = crt.Screen.Get(screenrow, -1, screenrow, 1) Do While errorornot = "?"       ' If there was an error, see if the user wants to continue or exit       If errorornot = "?" then             errorresult = crt.Dialog.MessageBox("The server name " &  tftpservername & " was invalid. Do you want to enter a new server name?" &  VbCr & VbCr & "Select YES to enter a new name and continue." & VbCr & VbCr &  "Select NO to end the script.", "TFTP Server Name Unknown!!", ICON_WARN or  BUTTON_YESNO)             if errorresult = IDNO then                   exitresult = "1"                   Exitdialog                   else if errorresult = IDYES then                         tftpservername = crt.Dialog.Prompt("Enter the name  or IP address of the TFTP server:", "TFTP Server", "Enter the TFTP Server  Name or IP Address")                         tftpconnect                   End if             End if       End if Loop End Sub ' This subroutine writes the changes to the startup-config Sub writechange       ' Copy the running config to startup config       If errorornot <> "%Error" then             crt.Screen.Send VbCr             crt.Screen.Send "copy run star" & VbCr             crt.Screen.WaitForString "[startup-config]?"             crt.Screen.Send VbCr             crt.Screen.WaitForString "#"             crt.Screen.Send VbCr       End if End Sub ' This subroutine exits the script and closes SecureCRT Sub ExitDialog       If exitresult = "1" then             result = crt.Dialog.MessageBox("There was an error and the  script was stopped." & VbCr & "Please check the log at " & LogFile & " for  more details.", "Update Failed!!", ICON_STOP or BUTTON_OK)             Else             result = crt.Dialog.MessageBox("Please check the log at " &  LogFile & VbCr & "to verify that all the devices have been configured.",  "Update Complete", ICON_INFO or BUTTON_OK)       End If       crt.quit End Sub 

Changing the Application

Although the previous sections defined managing changes and applying patches and updates to your network infrastructure s hardware (such as routers and switches), some of our network infrastructure is actually an application running on a server. For example, Check Point Firewall-1 is an application that can run on Solaris, Windows, or Linux. Content-filtering software may run on Windows or Linux. In the case of applications, our options available for performing and automating patches and updates are limited largely by what the vendors provide as an installation mechanism. Some vendors support using tools such as Microsoft SMS for the deploying of updates and patches. Others may support scripting the installation routing through the use of InstallShield or other installation mechanisms. Unfortunately, however, the majority of your application changes will likely have to be manually performed by you on each system that needs to be updated.