Section 7-5. Upgrading Firewalls in Failover Mode

7-5. Upgrading Firewalls in Failover Mode

Upgrading the operating system on a single standalone firewall is straightforward. You download a new image to the firewall, save the running configuration, and reload the firewall. Obviously, this should all be done during a scheduled maintenance time in your network, because the reload interrupts network connectivity.

A failover pair of firewalls is slightly more complicated, because both firewalls must be running exactly the same release of code at all times. If the code release differs between the two firewalls, failover is disabled. This causes each firewall to run independently, each thinking that the other has either failed or is incompatible for failover.

Firewall platforms running the FWSM 2.x or PIX 7.x operating system releases are exceptions. These versions offer a "hitless upgrade" feature that allows failover to continue operating even if the two firewalls are running different maintenance releases of the software image, such as 7.0(1) and 7.0(3). However, failover is disabled if different major or minor releases, such as 7.0(3) and 7.1(0), are running.

Upgrading an Active-Standby Failover Pair

With PIX 6.3, for example, the firewalls always operate as an active-standby pair. Only one firewall is active, and both units must be running identical software releases. In a nutshell, the idea is to upgrade the image on the primary unit first. The primary firewall is removed from the network and rebooted, and its new software image is verified. After it is powered off, it is added back into the network and powered on. At the same time, the secondary unit is powered down and removed from the network, to receive a similar upgrade.

The firewalls are upgraded and checked one at a time to make sure each will boot up and run the new image correctly before being added back into the network. This prevents the worst-case scenario from happening, in which a hasty upgrade to both firewalls might result in both of them trying to boot a corrupt image. Having both units of a failover pair "broken" at the same time can create a long outage in a network.

To upgrade the operating system on a failover pair, follow these steps:

1.

Upgrade and test the primary unit first. a. Open a session to the primary firewall unit. While the two firewalls are operating as a failover pair, you can't make configuration changes to each one. Configuration is allowed only on the active unit. However, you can download a new operating system (OS) image without disrupting the failover status. Remember that both firewalls have the same host name, because it is replicated between them. Therefore, you can't recognize the primary unit by its host name in the prompt. To determine which failover unit you're connected to (primary or secondary), use the show failover command on each. For example, the primary firewall shows something like this: Firewall# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds Last Failover at: 04:57:40 EST Sun Oct 26 2003 This host: Primary - Active Active time: 245010 (sec) Interface stateful (192.168.199.1): Normal Interface dmz2 (127.0.0.1): Link Down (Shutdown) Interface outside (192.168.110.65): Normal Interface inside (192.168.254.1): Normal Other host: Secondary - Standby Active time: 0 (sec) Interface stateful (192.168.199.2): Normal Interface dmz2 (0.0.0.0): Link Down (Shutdown) Interface outside (192.168.110.71): Normal Interface inside (192.168.254.9): Normal [output omitted] Firewall# It doesn't really matter whether the primary unit is in active or standby mode. If it is active, the secondary unit takes over the active role when the primary is powered down and rebooted in later steps. b. Download the new OS image into the primary firewall. You can use any supported image transfer method, such as TFTP or FTP. Be certain to monitor the image file download to be certain that the firewall has written a complete new image into its Flash memory. As soon as the OS image is stored in Flash, you cannot view or verify it; you can either watch as it is being downloaded and written or watch as it is being run after a reload. c. Save the running configuration on the primary unit: Firewall# write memory or Firewall# copy running-config startup-config Always make sure you have saved the current running configuration before a reload. The running configuration is dynamically updated from the active unit to the standby unit as commands are entered. However, you must manually save the running configuration to Flash with either of the preceding commands. Doing this causes the same command to be run on the standby unit as well. Copying the running configuration to an external location is also a good idea. For example, you can copy it to a TFTP server with the copy running-config tftp:[[//location][/pathname]] command. In case the primary firewall has a catastrophic failure, you have a copy of the configuration to load into a replacement unit. d. Power off the primary unit and remove its network connections. e. Restore power to the primary unit. Connect to the firewall console and watch the primary unit boot up. Verify that the new operating system release is loaded and that it appears to run correctly. f. Power the primary unit off again and reconnect its interface connections.

2.

Upgrade and test the secondary unit. a. Download the new OS image into the secondary unit. b. Power off the secondary unit and restore power to the primary unit. The upgraded primary unit operates as the active unit as soon as it boots up. Because the primary unit has an "unrestricted" license, it can boot up and function alone. During the bootup process, the network does not have a functioning firewall and experiences a brief downtime. c. Remove network connections to the secondary unit. d. Restore power to the secondary unit. Connect to the firewall console and watch the secondary unit boot up. Verify that the new operating system release is loaded and that it appears to run correctly. e. Power off the secondary unit again, and then reconnect its interface connections. f. Restore power to the secondary unit again. The secondary unit should boot up and assume the standby failover role.

Running with Mismatched Image Releases

Suppose you decide to preload both units of a failover pair with an upgraded operating system image. You might do this to save time, thinking you will be able to carefully reload each unit in succession during a scheduled maintenance window. Suddenly, one of the firewalls is power-cycled or reloaded for some unexpected reason. Now, one firewall is running the "old" image and the other is running the "new" image. This doesn't seem bad, because half of the upgrades have already completed.

However, this situation has the potential to be bad, because the failover function is now broken. Because the two firewalls aren't running an identical OS version, failover is automatically disabled on the firewall that reloads first.

An example of a failover pair in this scenario follows.

Secondary Unit, Older Image	Primary Unit, Newer Image
Firewall# show version Cisco PIX Firewall Version 6.3(1) Cisco PIX Device Manager Version 3.0(1) Compiled on Wed 19-Mar-03 11:49 by morlee Firewall up 49 days 18 hours Hardware: PIX-525, 256 MB RAM, CPU Pentium III 600 MHz [output deleted] Licensed Features: Failover: Enabled VPN-DES: Enabled VPN-3DES-AES: Enabled Maximum Interfaces: 8 Cut-through Proxy: Enabled Guards: Enabled URL-filtering: Enabled Inside Hosts: Unlimited Throughput: Unlimited IKE peers: Unlimited This PIX has a Failover Only (FO) license. Firewall# Firewall# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds This host: Secondary - Active Active time: 862350 (sec) Interface stateful (192.168.199.1): Normal (Waiting) Interface dmz2 (127.0.0.1): Link Down (Shutdown) Interface outside (192.168.110.65): Normal (Waiting) Interface inside (192.168.254.1): Normal (Waiting) Other host: Primary - Standby Active time: 3418080 (sec) Interface stateful (192.168.199.2): Unknown Interface dmz2 (0.0.0.0): Unknown (Shutdown) Interface outside (192.168.110.71): Unknown Interface inside (192.168.254.9): Unknown	Firewall# show version Cisco PIX Firewall Version 6.3(3) Cisco PIX Device Manager Version 3.0(1) Compiled on Wed 13-Aug-03 13:55 by morlee Firewall up 1 days 19 hours Hardware: PIX-525, 256 MB RAM, CPU Pentium III 600 MHz [output deleted] Licensed Features: Failover: Enabled VPN-DES: Enabled VPN-3DES-AES: Enabled Maximum Physical Interfaces: 8 Maximum Interfaces: 12 Cut-through Proxy: Enabled Guards: Enabled URL-filtering: Enabled Inside Hosts: Unlimited Throughput: Unlimited IKE peers: Unlimited This PIX has an Unrestricted (UR) license. Firewall# Failover Warning: peer pix os version is different Failover Warning: peer pix os version is different Failover Warning: peer pix os version is different WARNING: Failover disable but failover cable connected To enable failover, in config, type failover Firewall# show failover Failover Off Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds
Firewall# show interface interface ethernet0 "stateful"is up, line protocol is up Hardware is i82559 ethernet, address is 0030.8587.546e IP address 192.168.199.1, subnet mask 255.255.255.0 interface gb-ethernet0 "outside"is up, line protocol is up Hardware is i82542 rev03 gigabit ethernet, address is 0003.4725.2f97 IP address 192.168.110.65, subnet mask 255.255.255.224 interface gb-ethernet1 "inside"is up, line protocol is up Hardware is i82542 rev03 gigabit ethernet, address is 0003.4725.2e32 IP address 192.168.254.1, subnet mask 255.255.255.0 [output omitted]	Firewall# show interface interface ethernet0 "stateful"is up, line protocol is up Hardware is i82559 ethernet, address is 0030.8587.546e IP address 192.168.199.1, subnet mask 255.255.255.0 interface gb-ethernet0 "outside"is up, line protocol is up Hardware is i82542 rev03 gigabit ethernet, address is 0003.4725.2e23 IP address 192.168.110.65, subnet mask 255.255.255.224 interface gb-ethernet1 "inside"is up, line protocol is up Hardware is i82542 rev03 gigabit ethernet, address is 0003.4725.2e94 IP address 192.168.254.1, subnet mask 255.255.255.0 [output omitted]

The primary unit has reloaded for some reason, causing it to run the newer image. As it boots, the primary unit notices that the OS images are different and disables failover on itself. The secondary unit, running the older image, has only noticed that the standby failover unit is no longer responding. Each of the standby unit's interfaces are shown to be in the "unknown" state.

Notice also that, in effect, each firewall now thinks it must be the primary active unit. According to their configurations, they should use identical IP addresses on their corresponding interfaces when they are active. Obviously, having two live firewalls with identical IP addresses could be bad.

To recover from this condition, you should reload the firewall that is still running the "old" image. This causes it to boot up with the newer image that has already been downloaded into its Flash memory.

The two firewalls should then be up and running identical software releases, making them compatible for failover. However, notice in the example that the primary unit originally detected the release mismatch and disabled its own participation in failover. You should connect to the primary firewall and manually re-enable failover with the failover configuration command.

Upgrading an Active-Active Failover Pair

With an active-active failover pair of firewalls, the operating system upgrade process is much easier. The "hitless upgrade" feature allows the firewalls to run different maintenance releases without complaining or interfering with each other. That means you can basically upgrade each of the units on-the-fly, with no network downtime.

To upgrade the operating system on an active-active failover pair, follow these steps:

1.

Upgrade the primary unit first in case the secondary unit doesn't have a license that allows it to operate alone after a reload. a. Open a session to the primary unit. Make sure you are in the system execution space with the following command: Firewall/admin# changeto system Firewall# b. Confirm that it is the primary unit with the show failover command: Firewall# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Primary Failover LAN Interface: Failover Ethernet2 (up) Unit Poll frequency 3 seconds, holdtime 9 seconds Interface Poll frequency 15 seconds Interface Policy 2 Monitored Interfaces 3 of 250 maximum Group 1 last failover at: 10:29:18 EST Jan 30 2005 Group 2 last failover at: 16:18:28 EST Mar 9 2005 This host: Primary Group 1 State: Active Active time: 3343323 (sec) Group 2 State: Standby Ready Active time: 3304092 (sec) c. Download the new OS image into the primary firewall. You can use any supported image transfer method, such as TFTP or FTP. Be certain to monitor the image file download to be certain that the firewall has written a complete new image into its Flash memory. You can see the file by using the show flash:/ command. d. Make sure the primary unit is configured to boot the new OS image. Use the following command to identify the image file: Firewall(config)# boot system device:/path The firewall allows you to enter multiple boot system commands; if the configuration has more than one command, they are evaluated in sequence. Therefore, you should make sure the new OS image is identified in the first boot system command. You might have to delete other entries by using the no boot system device:/path commands. You have to issue the boot system commands on the firewall unit that is currently active for the system execution space; otherwise, the configurations become unsynchronized, and the changes are not replicated. Before you proceed, verify the current boot configuration with the show run boot command, as in the following example: Firewall# show run boot boot system flash:/image.bin Firewall# e. Save the running configuration on the primary unit: Firewall# write memory or Firewall# copy running-config startup-config Always make sure you have saved the current running configuration before a reload. The running configuration is dynamically updated from the active unit of a context to the respective standby unit as commands are entered. However, you must manually save the running configuration to Flash with either of the preceding commands. Doing this also causes the same command to be run on the standby unit. f. Reload the primary unit with the reload command. If it is running in multiple-context mode, you are prompted to save any of the individual context running configurations that have changed since the last save. The firewall prompts you to save configurations only for the contexts it is currently serving in the active role. While the primary unit is reloading, the secondary unit should assume the active role for every configured context. You can verify this by using the show failover command on the secondary unit, as in the following example: Firewall# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Secondary Failover LAN Interface: Failover Ethernet2 (up) Unit Poll frequency 3 seconds, holdtime 9 seconds Interface Poll frequency 15 seconds Interface Policy 2 Monitored Interfaces 3 of 250 maximum Group 1 last failover at: 03:25:22 EST Mar 11 2005 Group 2 last failover at: 15:40:14 EST Mar 10 2005 This host: Secondary Group 1 State: Active Active time: 3 (sec) Group 2 State: Active Active time: 42312 (sec) As soon as the primary unit comes back up, it assumes the active role on any failover group and context that is configured with the preempt command. Otherwise, the secondary unit retains its active roles until a future failure or manual intervention.

2.

Upgrade the secondary unit. First, confirm that the primary unit is up and failover is functioning with the show failover command. All the monitored interfaces should be in the "Normal" state, indicating that the two units are communicating over the interfaces. Repeat Steps 1a through 1f for the secondary unit rather than the primary unit. When you complete these steps, both units should be running identical OS image releases, and failover should be operational again.