Analysis of Problem Areas

FWSM problem areas can be broken down into the following areas:

• Licensing issues

• Hardware issues

• Firewall module administration issues

• Connection issues

• AAA issues

• Virtual and transparent firewall

• High CPU issues

• Intermittent packet drops issues

• Failover issues

A more detailed discussion of the previously listed items follows in the coming sections.

Licensing Issues

You do not need a license to run FWSM in single firewall mode. If you use the following command to convert FWSM to multiple mode, you can use two contexts in addition to an admin context:

FWSM# configure terminal FWSM(config)# mode multiple FWSM(config)# show mode Firewall mode: multiple The flash mode is the SAME as the running mode. FWSM(config)# 

Every context is mapped to a virtual firewall. If you require more than two virtual firewalls (contexts), you will need an activation key. Activation keys can be obtained from [e-mail], once you purchase the license. For security contexts, you can purchase up to 100 licenses in increments of 20.

The activation key comes in number form with four octets that can be entered into the FWSM with the activation-key key command as follows:

FWSM(config)# activation-key 0xaaaaaaaa 0xbbbbbbbb 0xcccccccc 0xdddddddd FWSM(config)# 

Then you can verify the number of contexts available with the show version command or the show activation-key command as follows:

FWSM(config)# show activation-key Running Activation Key: 0xaaaaaaaa 0xbbbbbbbb 0xcccccccc 0xdddddddd Licensed Features: Failover:           Enabled VPN-DES:            Enabled VPN-3DES:           Enabled Maximum Interfaces: 256 (per security context) Cut-through Proxy:  Enabled Guards:             Enabled URL-filtering:      Enabled Throughput:         Unlimited ISAKMP peers:       Unlimited ! Following line indicates how many Virtual FW can be configured Security Contexts:  20 This machine has an Unrestricted (UR) license. The flash activation key is the SAME as the running key. FWSM(config)# 

You can simply override an activation key with the activation-key command, or you can clear the old activation key with the clear activation-key command and enter the new one.

Hardware Issues

Troubleshooting an FWSM blade hardware problem on the switch is very similar to troubleshooting an IDSM-2 blade. Hence, the troubleshooting techniques discussed in the "Hardware Issues" section of Chapter 15, "Troubleshooting IDSM-2 Blade on Switch," are not repeated here. Only information specific to the FWSM blade is discussed. If you cannot session into the FWSM blade, first execute the show module command to determine the status of the module. The module can be in any of the following conditions:

• Module is not recognized

• Module status shows faulty/other

• Module shows OK but you cannot session into it

To troubleshoot issues with any of these three conditions or more, follow these troubleshooting steps:

Step 1.	Ensure that you are running a supported version of code on your switch for FWSM. You can find this information in the Release note of the corresponding version. For example, for FWSM Version 2.3(2), you can find the "Chassis System Requirements" from the following link: http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/mod_icn/fwsm/fwsm_2_3/fwsm_rn/fwsmrns.htm#wp73044
Step 2.	Ensure that the FWSM can coexist with other blades in the same chassis. Refer to the Catalyst 6500 Release note in the following location: http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/relnotes/index.htm In addition, you can look at the Software Advisor for additional information (only available to registered customers): http://www.cisco.com/pcgi-bin/Support/CompNav/Index.pl Beginning with Native IOS Version 12.2(14)SY, FWSM, VPN, IDSM-2, and NAM modules can coexist in the same chassis. There is no CatOS available to support all these modules in the same chassis.
Step 3.	If you are running CatOS (Hybrid) code on your switch, reset the configuration for the slot that is occupied by the FWSM module. To do this, use the following commands: Type set module power down mod to power down the FWSM. Clear the switch's configuration associated with that slot and to power up the module.
Step 4.	For additional troubleshooting refer to Chapter 15, "Troubleshooting IDSM-2 Blade on Switch," and consult the following links: For Hybrid Mode, refer to the following link: http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a0080134734.shtml For Native IOS, refer to the following link: http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801751d7.shtml

Firewall Module Administration Issues

To administer the Firewall module effectively and efficiently, it is important to be familiar with the following items below:

• Flash

• Setting the boot device

• Maintenance partition

• Password recovery procedure

• Upgrading a new image

The following section explores each these items in more detail.

Flash

FWSM contains a 128 MB Compact Flash, which is partitioned into six sizes for different tasks as shown in Table 4-1.

Table 4-1. The Partitions of Compact Flash Memory

Partition Number	Size (MB)	Description
First (cf:1)	16	Maintenance Partition
Second (cf:2)	7	Network Configuration
Third (cf:3)	1	Crash Dump Partition
Fourth (cf:4)	20	Application Partition1FWSM file systemimage, configuration, and PDM file
Fifth (cf:5)	20	Application Partition2FWSM file systemimage, configuration, and PDM file
Sixth (cf:6)	64	Reserved

Typically, you will be working with cf:1, cf:4 or cf:5 partitions. If you set the boot device for cf:4 or cf:5, the FWSM will boot up in normal mode (application partition). However, if you set the boot device for cf:1, the module will boot up as Maintenance, which can be used to re-image or perform the password recovery for the Application partition (cf:4 or cf:5). More details on setting up the Boot Device follow in the next section.

Setting the Boot Device (Route Processor)

In native IOS mode, to boot the Firewall to Application Partition (Firewall module), set the Compact Flash Partition to 4 or 5 with the following command:

Cat6509(config)# boot device  module mod-num cf:4 

To boot from the Maintenance Partition, set the Compact Flash partition to 1 with the following command:

Cat6509(config)# boot device  module mod-num cf:1 

To display the Boot Partition that is configured, execute the following command:

Cat6509# show boot device module mod-num 

Example 4-7 shows the module 6 boot to Application Partition.

Example 4-7. FWSM Boot to Application Partition Setting

Router# show boot device module 6 [mod:6 ]: cf:4 Router# 

By default, the FWSM module is set to boot to Application partition. To change this to maintenance partition, change the boot variable, then reset the module.

To work on a specific partition temporarily, regardless of the boot variable setup, use the following command for Native IOS (to reset the module for Maintenance partition):

Router# hw-module module mod_num reset cf:1 

For Catalyst operating system software, execute the following command to go to the Maintenance partition:

Console> (enable) reset mod_num cf:1 

For Application partition, use either cf:4 or cf:5 instead of cf:1.

Then to session into the FWSM, enter the following command in Native IOS:

Router# session slot mod_num processor 1 

For Catalyst operating system software, enter the following command:

Console> (enable) session mod_num 

A more detailed discussion follows in the next section for the Maintenance partition.

Maintenance Partition

You can use the Maintenance Partition for the following purposes:

• If the modules application partition is corrupted, use the maintenance partition to recover and reprogram the application partition.

• Use the show crashdump command from the Maintenance Partition to obtain the crash dump, for debugging purposes.

• The application module can be upgraded from the Maintenance Partition.

• The enable password for the module can be cleared from the Maintenance Partition command line interface (CLI).

To get into the Maintenance partition, use either the session command or telnet* command from the switch. It can be accessed by Telnetting to 127.0.0.x1 (x = slot # of the FWSM module). Example 4-8 shows a sample output of how to connect to the FWSM if it is on slot 3 on the switch.

Example 4-8. Connecting to the Maintenance Partition

System> telnet 127.0.0.31 Maintenance image localhost.localdomain login: root Password: cisco Maintenance image version: 1.1(0.4) [e-mail]# 

When asking for login information, you can use either a root or guest user account. Table 4-2 summarizes the purpose of these two users' accounts, and what rights each user account has.

Table 4-2. User Accounts for Maintenance Partition Access

Users/Password	Description
root/Cisco	Used for normal access. It provides a restricted shell and allows users to upgrade Application Partition images. It also allows users to display version/image information.
guest/cisco	Guest account is meant for Cisco Support team operations, such as retrieving the diagnostic results and upgrading the BIOS image. Guest account access can be enabled/disabled from the root account. However, guests cannot change the root password. So, a guest is a superset of root level commands.
The default password for root and guest accounts is Cisco, and this can be reset by the passwd command from the maintenance partition. This can also be reset on the Application partition.

Once you are in the maintenance partition, you need to set up the networking parameters, so that the FWSM blade on the Maintenance partition can communicate with the rest of the network or the TFTP/FTP server. Remember that the IP address that you set on the MP partition is always part of VLAN 1 on the switch. So, you need to ensure that the TFTP/FTP server for the image download has a proper route from the VLAN1 on the switch. Table 4-3 summarizes the configuration commands available on the Maintenance partition.

Table 4-3. Maintenance Partition Commands

Commands	Description
ip	To set IP parameters (root and guest)
show	To show system parameters (root and guest)
passwd	To set the password for the current user (root and guest)
upgrade	To download and install new application image (root and guest)
clear ip	To clear network configuration for the interface (root and guest)
clear log upgrade	To clear the log file for upgrade operation (root and guest)
logout	To log out of the shell (root and guest)
passwd-guest	To set password for guest account (root)
enable-guest	To enable the guest account (root)
Disable-guest	To disable the guest account (root)
clear passwd	To clear the enable password for application (root and guest)
upgrade-bios	To install new BIOS image (guest)
debug	To use Network Processor (NP) debug utility (guest)

Table 4-4 lists and describes the different show commands available on the Maintenance Partition.

Table 4-4. show Commands Available in MP

Commands	Description
show ip	Shows network configuration for the interface (root and guest)
show images	Shows images on the application partition (root and guest)
show version	Shows system parameters (root and guest)
show log upgrade	Shows upgrade log file (root and guest)
show diaglog	Shows diagnostics log file (guest)
show ethif	Shows Ethernet interface info (guest)
show crashdump	Shows the contents of crashdump file (guest)

Based on the command syntax listed in Tables 4-3 and 4-4, configure your FWSM Maintenance partition to set initial networking parameters as shown in Example 4-9.

Example 4-9. Initial Networking Configuration for Maintenance Partition

Cisco Maintenance image login: root Password: Maintenance image version: 1.1(0.3) root@localhost# ip address 10.1.1.1 255.255.255.0 root@localhost# ip broadcast 10.1.1.254 root@localhost# ip host FWSM root@localhost# ip gateway 10.1.1.100 root@localhost# ip domain cisco.com root@localhost# ip nameserver 10.1.1.150 root@localhost# show ip IP address: 10.1.1.1 Subnet mask: 255.255.255.0 IP Broadcast: 10.1.1.254 DNS Name: FWSM.cisco.com Default Gateway: 10.1.1.100 Nameserver(s): 10.1.1.150 root@localhost# 

Password Recovery Procedure

If you forget or lose the passwords for the FWSM application partition, you can reset the password to default values from the Maintenance Partition. Clearing the password with the following command from the Maintenance partition will reset the Telnet password to cisco and will clear the enable password:

guest@localhost# clear passwd cf:partition_number> 

Here partition_number refers to the number of the application partition whose password has to be reset. The following example shows the resetting of the password for the application partition in slot 4.

guest@localhost# clear passwd cf:4 Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration:        enable password 8Ry2YjIyt7RRXU24 encrypted        passwd 2KFQnbNIdI.2KYOU encrypted Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. guest@localhost# 

Upgrading a New Image

The Application partition and Maintenance partition use two separate images. This section discusses the new image upgrade on both the Maintenance and Application partitions:

• Upgrading the Maintenance partition

• Upgrading the Application partition

Upgrading the Maintenance Partition

To upgrade MP, you must use the Application partition using the following command:

upgrade-mp   tftp-url 

Here tftp-url is the Trivial File Transfer Protocol (TFTP) server location and name of the FWSM Maintenance software image file. You can download the Maintenance image from the following link:

http://www.cisco.com/cgi-bin/tablebuild.pl/cat6000-fwsm

Example 4-10 shows the upgrade procedure for Maintenance partition.

Example 4-10. MP Upgrade from the Application Partition

FWSM# upgrade-mp Address or name of remote host [10.1.1.150]? 10.1.1.150 Source file name []? mp-1.0.1-bin.gz copying tftp://192.168.253.79/mp-1.0.1-bin.gz to flash [yes|no|again]?y !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!! Received 7700916 bytes. Maintenance partition upgraded. FWSM# 

Upgrading Software Images

You can download the Application partition image from the same location as the Maintenance partition:

http://www.cisco.com/cgi-bin/tablebuild.pl/cat6000-fwsm

You can upgrade the Application partition in either of the following ways:

• From the Application partition The usual upgrade procedure performed on the Application partition is by the Application partition itself. File transfer for the upgrade on the Application partition is possible with the TFTP protocol only. Use this command for upgrade:

  FWSM# copy tftp[:[[//location][/pathname]]] flash[:[image | pdm]] 

  The following example shows how to upgrade the Application partition from the Application partition itself:

  FWSM# copy tftp://10.1.1.150/cdisk flash:image 

• From the Maintenance Partition If the Application Partition is corrupted, use the Maintenance partition to re-image the Application partition. You can also upgrade the Application partition from the MP. The following command is used in both the re-imaging and upgrading of application partition from the Maintenance partition.

  Guest@localhost# upgrade ftp://username@server/path/image device 

  The following example shows how to upgrade the Application partition from the Maintenance partition:

  FWSM# upgrade ftp://username@10.1.1.150/tftpboot/c6svc-fwm-k9.1.1.0.127.bin cf:4 

Connection Problems

A typical FWSM is depicted in Figure 4-5, in which MSFC is situated on the outside interface of the FWSM.

Figure 4-5. A Typical FWSM Deployment

As shown in Figure 4-5, VLAN 10 is sitting on the inside, and VLAN 50 is the SVI interface on MSFC. The outside interface of the FWSM is configured to be part of VLAN 30 on the switch. Hence, this interface VLAN 30 on the MSFC is used by the FWSM to communicate with MSFC and any other VLANs configured on the MSFC.

This section examines the following two topics in detail based on Figure 4-5.

• Configuration steps

• Troubleshooting steps

Configuration Steps

Work through the following steps to configure the switch and the FWSM with two interfaces (inside and outside) based on Figure 4-5:

Step 1.	Configure the VLAN, and the ports on the switch. First, configure the interface and the VLAN on the switch. If you are running Native IOS, configure the interfaces and VLANs as shown Example 4-11. Example 4-11. VLAN and Interfaces on the Native IOS Switch CAT6509# configure terminal Enter configuration commands, one per line. End with CNTL/Z. ! Create the VLANs 10, 30, and 50 on the switch with the following command CAT6509(config)# vlan 10,30,50 CAT6509(config-vlan)# exit ! Following interface is connected to the inside network VLAN 10 CAT6509(config)# int g2/1 CAT6509(config-if)# switchport CAT6509(config-if)# switchport mode access CAT6509(config-if)# switchport access vlan 10 CAT6509(config-if)# spanning-tree portfast CAT6509(config-if)# no shut CAT6509(config-if)# exit ! The following interface is connected to the outside VLAN 50. Note that this is ! not SVI interface CAT6509(config)# int g2/3 CAT6509(config-if)# switchport CAT6509(config-if)# switchport mode access CAT6509(config-if)# switchport access vlan 50 CAT6509(config-if)# spanning-tree portfast CAT6509(config-if)# no shut CAT6509(config-if)# exit CAT6509(config)# ! The following layer III VLAN interface is the SVI interface which is connected ! with the outside interface of the FWSM. You must have already created the layer ! II VLAN for the VLAN interface 30. CAT6509(config)# interface vlan 30 CAT6509(config-if)# description To FWSM's outside interface CAT6509(config-if)# ip address 30.1.1.2 255.255.255.0 CAT6509(config-if)# no shut CAT6509(config-if)#^Z Example 4-12 shows the same configuration in the Hybrid mode as shown in Example 4-11 in the Native IOS mode. Example 4-12. Showing VLAN and Interface Configuration on the CatOS ! Create the VLANs on the Supervisor first (make sure your VTP mode allows it) cat6509> (enable) set vlan 10,30,50 ! Put physical ports 2/1 and 5/6 in their vlans; set ports as host ports cat6509> (enable) set vlan 10 2/1 cat6509> (enable) set vlan 50 2/3 cat6509> (enable) ! Session into the MSFC and create VLAN 30 interface for the SVI interface for ! FWSM cat6509> (enable) session 15 Trying Router-15... Connected to Router-15. Escape character is '^]'. MSFC> enable Password: MSFC# configure terminal ! create a "point-to-point" vlan (30) between the MSFC and the FWSM MSFC(config)# interface vlan 30 MSFC(config-if)# ip address 30.1.1.2 255.255.255.0 MSFC(config-if)# no shut ! create VLAN 50 interface so that this VLAN can communicate with VLAN 10 through ! the FWSM MSFC(config-if)# int vlan 50 MSFC(config-if)# ip address 50.1.1.1 255.255.255.0 MSFC(config-if)# no shut Note Always create the firewall VLANs on the Supervisor first. Then session to the MSFC and bring up the VLAN interface between the MSFC and the FWSM. If you perform the operation in the reverse order, you might run into a situation in which the VLAN will fail to come up. If so, clear all firewall VLANs on the Supervisor, remove the VLAN interface on the MSFC, and create the firewall VLANs on the Supervisor again.
Step 2.	Map the VLANs on the switch to the FWSM. Once the VLANs and interfaces are configured on the switch, configure the switch to map the VLANs to the FWSM blade. The syntax to map the VLANs is different in Native IOS and Hybrid modes on the switch. If you are running Native IOS, use the following command to group VLANs with the first line and map the VLAN group to the FWSM that is seated on slot 3 of the FWSM: CAT6509(config)# firewall vlan-group 1 10,30 CAT6509(config)# firewall module 3 vlan-group 1 If you are running the switch in Hybrid mode, use the following command: cat6509> (enable) set vlan 10,30 firewall-vlan 3 Vlans 30,40 declared secure for firewall module 3 CAT6509> (enable)
Step 3.	Configure the interfaces on the FWSM. By now, VLANs 10 and 30 should be downloaded to the FWSM. The download can be verified by using the following command: FWSM# show vlan 10, 30 FWSM# Map the downloaded VLANs on the FWSM to the corresponding interfaces of the FWSM with the following command: FWSM(config)# nameif vlan_number if_name security_level VLAN10 is mapped to inside interface, and the VLAN30 is mapped to the outside interface as follows: FWSM(config)# nameif vlan10 inside security100 FWSM(config)# nameif vlan30 outside security0 After defining the interface, assign the IP address to the interfaces with the following command: FWSM(config)# ip address if_name ip_address [netmask] The following commands show the IP address configuration of the inside and outside interfaces: FWSM(config)# ip address inside 10.1.1.1 255.255.255.0 FWSM(config)# ip address outside 30.1.1.1 255.255.255.0
Step 4.	Define routes on the FWSM. By default you can create a single Switched Virtual Interface (SVI) on the MSFC and point to that as the default gateway on the FWSM. The syntax for configuring the static route is as follows: FWSM(config)# route if_name ip_address netmask gateway_ip metric The following command shows a default route pointing to the SVI interface (VLAN interface 30): FWSM(config)# route outside 0 0 30.1.1.2
Step 5.	Configure NAT/PAT/Static. For outbound traffic (traffic from inside to outside, for instance), the source address translation is required by default. You may configure nat/ global or static NAT to translate the source address. Alternatively, you may bypass NAT checking with a nat zero ACL command. Nat/global is used for dynamic translation, but static NAT is used to create permanent mapping. Static is used mainly for inbound connections to perform the destination address translation. Dual nat/alias is used to support bi-directional NAT. The following commands are used to configure nat/global and static: FWSM(config)# nat [(if_name)] nat_id local_ip [netmask [max_conns [em_limit]]] [norandomseq] FWSM(config)# global [(if_name)] nat_id {global_ip [-global_ip] [netmask global_mask]} | interface FWSM(config)# static [(internal_if_name, external_if_name)] global_ip local_ip [netmask network_mask] [max_conns [em_limit]] [norandomseq] The following commands show the PAT configured for the outbound connection: FWSM(config)# nat (inside) 1 10.1.1.0 255.255.255.0 FWSM(config)# global (outside) 1 interface To allow an inbound connection, either exempt the NAT with the nat zero acl command or with the static command. The following command shows the static translation for the inbound connection to the VLAN10. FWSM(config)# static (inside,outside) 30.1.1.110 10.1.1.110 netmask 255.255.255.255 0 2
Step 6.	Configure an access-list. Unlike the PIX firewall, you must allow the outbound traffic with an access-list applied on the higher security interface (for example, inside interface). Otherwise, packets are denied in both directions (inbound and outbound). The syntax for an access-list configuration is as follows: FWSM(config)# access-list acl_ID [deny | permit] protocol {source_addr | local_addr} {source_mask | local_mask} operator port {destination_addr | remote_addr} {destination_mask | remote_mask} operator port Then, apply the ACL to the interface with the following command. FWSM(config)# access-group acl_ID in interface interface_name Example 4-13 shows the access-list configuration and how to apply it on the interface. Example 4-13. ACL Configuration and How to Apply it on the Interface ! Following ACL will allow all traffic from inside to outside network FWSM(config)# access-list 101 permit ip any any ! Following ACL is to allow only web traffic to 30.1.1.110 which is translated to ! inside IP address of 10.1.1.110 (actual web server IP on the inside network. FWSM(config)# access-list 102 permit any host 30.1.1.110 eq 80 ! ACL 101 is applied on the inside interface FWSM(config)# access-group 101 to the inside interface ! ACL 102 is applied to the outside interface FWSM(config)# access-list 102 in interface outside FWSM(config)#
Step 7.	Turn on fixups. To turn on fixup for a specific protocol, you can use the fixup protocol command. For example, to turn on fixup for the ICMP, which is off by default, use the following command: FWSM(config)# fixup protocol icmp

Troubleshooting Steps

If you follow the configuration steps carefully, you usually will not run into any problem with connections across the FWSM. If you run into connection problems, work through the following steps to correct the problems:

Step 1.	Be sure you can ping to the directly connected interface of the FWSM. If you cannot pass traffic across the FWSM, you must ensure that you can ping to the interface of the FWSM from a host in the same VLAN. For example, if you mapped the inside interface to VLAN 10, a host (10.1.1.10) on VLAN 10 must be able to ping to the FWSM inside interface (10.1.1.1). If you cannot ping, work through the following steps to correct the problem; otherwise proceed to the next step: (a). Make sure the ICMP is allowed on the interface of FWSM by using the icmp command. You can execute the show icmp command to verify whether the icmp is allowed on the interface. To enable ICMP on the inside interface, use the following command: FWSM(config)# icmp permit 0 0 inside (b). If the ping is still unsuccessful, run the debug icmp trace command and see if FWSM displays any debug output. If there are no debug messages on ICMP on the FWSM, go to the next step. (c). Ping to other devices in the same VLAN and subnet. If you are not successful, check the port that is connected to the device. If the ping is successful, move to the next step. (d). Execute the show interface command and ensure that the interfaces are shown up/up. If they are shown in any other state, verify that the nameif command used the proper VLAN number with the commands that follow: FWSM# show nameif nameif vlan10 inside security100 nameif vlan30 outside security0 FWSM# (e). If you still have problems with the interface, execute the show vlan command and be sure the correct VLANs are downloaded properly on the FWSM. FWSM# show vlan 10, 30 FWSM# (f). If you have FWSM set up with Hybrid mode, be sure to configure the VLANS on the switch before defining the SVI on the MSFC. If you have configured an SVI interface on the MSFC before configuring the VLAN on the switch, you must remove the SVI from the MSFC and VLANs from the switch. Reconfigure VLANs on the switch first and then configure the SVI interface on the MSFC. (g). If the packets still do not flow, change the Ether channel algorithm on the switch.
Step 2.	If you can successfully ping the inside and outside interface from the corresponding VLANs, the next step is to make sure that you can ping across the FWSM. If you cannot do so, follow the next step.
Step 3.	By default, FWSM denies all packets from higher to lower security, and lower to higher security networks. So you must configure the ACL to allow this traffic on both directions. You can execute show access-list in conjunction with show access-group commands to verify whether the ACL is configured, and applied to the correct interface. You must also allow the return packet from the opposite direction. For example, to ping the outside from the inside, you must allow the ICMP on the inside interface and on the outside interface. If you do not want to allow ICMP packets on the outside interface, you can configure fixup protocol icmp. If the packets are dropped by the ACL, this information will be logged in the syslog, if you turn on syslogging to level 4. If with the show access-list command packet, counters are incrementing, you know that the ACL is allowing the packets that the NAT may not be configuring or may be configuring incorrectly. So, go to the next step.
Step 4.	Check to see if the translation is not being built. If the hit counters are incrementing on the ACL, make sure you are not running with a translation issue. As the NAT is taken care of by the NP3 processor, you can execute the show np3 nat|global|static|alias command to verify the configuration. You can execute show xlate detail command to see if the translation is being built up as follows: FWSM(config)# show xlate detail Flags: D - DNS, d - dump, I - identity, i - inside, n - no random, o - outside, r - portmap, s - static 1 in use, 2 most used ICMP PAT from inside:10.1.1.10/45 to outside:30.1.1.1/1036 flags r FWSM(config)#
Step 5.	Verify that the route exists in the routing table for the destination network. If the ACL and translation are fine, execute the show route command to verify that the routing table is in the firewall module. Example 4-14 is an example of a routing table. Example 4-14. Showing a Routing Table on the FWSM FWSM(config)# show route S 0.0.0.0 0.0.0.0 [1/0] via 30.1.1.2, outside C 10.1.1.0 255.255.255.0 is directly connected, inside C 30.1.1.0 255.255.255.0 is directly connected, outside C 127.0.0.0 255.255.255.0 is directly connected, eobc FWSM(config)#
Step 6.	Verify whether the route exists on the outside router or MSFC for the return traffic. The MSFC or the outside router must have the route pointing back to the outside interface of the FWSM. You can run the debug ip packet detail ACL on the MSFC or the router to make sure that a route is defined on the MSFC or router pointing back to the outside interface of the FWSM. Note that if you want the communication to take place between these VLANs and outside network, you must have a route for every VLAN defined on the FWSM. The following commands show how to run the debug ip packet detail ACL command. Assume that you want to verify the connection from the source address of 30.1.1.1, and destination address of 50.1.1.50 on the MSFC. You can configure the ACL as follows and turn on debug using this ACL. MSFC(config)# access-list 101 permit ip host 30.1.1.1 host 50.1.1.50 MSFC(config)# access-list 101 permit ip host 50.1.1.50 host 30.1.1.1 MSFC(config)# exit MSFC# debug ip packet detail 101
Step 7.	Finally, verify the connection establishment with the following command: FWSM(config)# show local 10.1.1.10 local host: <10.1.1.10>, tcp conn(s)/limit = 1/0, embryonic(s)/limit = 0/0 udp c onn(s)/limit = 0/0 Xlate(s): PAT Global 30.1.1.1(1042) Local 10.1.1.10(11003) FWSM(config)# To determine the details of the connection, use the following commands: FWSM(config)# show conn 1 in use, 2 most used Network Processor 1 connections TCP out 50.1.1.50:23 in 10.1.1.10:11003 idle 0:00:42 Bytes 4672 FLAGS - U Network Processor 2 connections If you want more details, you can use the show conn long 3 command.

AAA Issues

AAA implementation on the FWSM is the same as on PIX Firewall. For Version 1.x, AAA implementation is based on PIX OS 6.0 and 6.2. However, for Version 2.x, AAA implementation is based on PIX 6.3.x. The implementation difference between the FWSM and PIX Firewall varies for where the authentication information caches. If there are new connection requests, and the cut-through proxy is turned on, the initial connection will be processed by the Control Plane. Once the user authentication is successful, it will be cached in the NP processors. So, subsequent requests will not go to the PC complex. For troubleshooting AAA on the FWSM, you can refer to Chapter 10, "Troubleshooting AAA on PIX Firewalls and FWSM."

Virtual and Transparent Firewall

Virtual Firewall and Transparent Firewall work the same way on the FWSM as does PIX Firewall. So, for details on configuration, and on troubleshooting on Virtual and Transparent firewalls, refer to Chapter 3.

High CPU Issues

FWSM may experience high CPU utilization for many reasons: misconfiguration, fixup of certain protocols, and too much traffic going across the FWSM. Work through the following steps to correct the high CPU utilization problems on the FWSM:

Step 1.	Take two "show processes" command outputs, about one minute apart. This will give you the difference in CPU utilization at intervals of one minute.
Step 2.	Calculate the difference in the number of processes taking the maximum amount of CPU in a one-minute interval. The values displayed are in milliseconds (ms). Be sure to exclude any polling thread.
Step 3.	Because the CPU shown is what is running on the CP (not the NPs), this limits the scope to: - Traffic sourced from the FWSM (mgmt, routing protocols, AAA, websense, syslog, and so on) - Traffic requiring L7 fixups (VoIP, PortMapper, ils, and so on)
Step 4.	Find out what and how many connections are on the CP by issuing the show pc conn command.
Step 5.	Typically high CPU utilization is caused by syslog, or the type of traffic hitting the CPU. Disable syslog or disable individual fixups until you uncover the utilization problem.

Remember that if the CP is experiencing high CPU utilization, this should not affect the flow of traffic across the FWSM that is processed by the NPs. However, if the NPs become busy (high CPU), then you could experience packet drops. There is no command available to show the CPU utilization on the NPs. However, if the NPs start sending PAUSE frames towards the two Pinnacle ASICs, that indicates how busy they are. You can get this information by using the following command:

FWSM(config)# show np 1 stats | include pause ! The following number should stay 0 PF_MNG: pause frames sent (x3)                     : 0 FWSM(config)# sh np 2 stats | include pause PF_MNG: pause frames sent (x3)                     : 0 FWSM# 

Ideally, you should see zero in the output of the previous command. If you see a number higher than zero, you should check the command output a few times to see if the number is incrementing. If it is incrementing, you might consider sniffing the traffic to analyze what is causing this extra overload.

Additionally, you can check the ingress queue on NP1 and NP2, as shown in Example 4-15, to see if they are filling up (another indication of high CPU utilization).

Example 4-15. Check the Ingress Queue for the NP1

FWSM(config)# show np 1 ingress | include count ! The column for the following three lines should be zero    bcb_fq_th_0_count      0x10080040   0x00000000  --> here    bcb_fq_th_1_count      0x10080020   0x00000000  --> here    bcb_fq_th_2_count      0x10080010   0x00000000  --> here    bcb_fq_arr_count       0x10100400   0x000c0000    dmu_rx_counters        0x00000000   0x00000000 FWSM(config)# 

If you see non-zero values in the last column, it means that the NPs are dropping incoming frames.

Intermittent Packet Drops Issues

Intermittent packet drops can occur for various reasons. You need to go through the "High CPU Issues" troubleshooting steps to make sure that you are not encountering any performance issues. Additionally, you might need to check the following to troubleshoot the intermittent packet drops issue:

1.

Fragmentation issues By default, fragmented packets cannot traverse the FWSM if you are running a version earlier than FWSM 2.2. You can use the fragment command to configure this feature as described in this link: http://cco/en/US/products/sw/secursw/ps2120/products_command_reference_chapter09186a00801727a8.html#wp1029667 http://www.cisco.com/en/US/products/sw/secursw/ps2120/products_command_reference_chapter09186a00801727a8.html#wp1029667 This behavior differs from that of the PIX firewall. Common protocols that use fragmented packets are Open Shortest Path First (OSPF) and Network File System (NFS).

2.

If you are running IP URL filtering, be sure that the policy defined in the URL server is not denying the packet.

Failover Issues

Failover helps to avoid the single point of failure in your network and provides the constant security to your network. The stateful failover feature provides a seamless connectivity experience through the firewall for the end users. As of the writing of this book, FWSM operates in Active/Standby mode, which means that at any point in time, one unit will be active and the other unit will be standby. Only one unit can be a standby unit for an active unit. Unlike the PIX firewall, FWSM can function only in LAN-based Failover mode, in which a LAN interface carries the failover messages from one unit to the other. As there is no dedicated serial cable available for the LAN-based Failover, the active unit is selected based on configuration of the FWSM. You can configure one unit as primary and other unit as secondary. When both units boot up at the same time, the primary will become active and the secondary will becomes standby. However, if the secondary boots up before the primary, the secondary becomes active, and stays as active until it fails or you manually make the active unit primary. There is no automatic preemption.

Failover for FWSM can be deployed in one of the following two ways, as shown in Figure 4-6:

• Intra Chassis Redundancy

• Inter Chassis Redundancy

Figure 4-6. Intra/Inter Chassis Redundancy

• Intra Chassis Redundancy (Single Chassis) For Intra Chassis Failover setup, two FWSMs are inserted into the same chassis. One of the FWSMs acts as the primary and the other FWSM acts as the secondary unit. From a network point of view, there is no issue in supporting Active and Standby Firewall blades in the same chassis, but the risk is the single point of failure. Note that there is a dedicated failover interface between active and standby units.

• Inter Chassis Redundancy (Multi Chassis) As shown in Figure 4-6, for Inter Chassis setup (the figure on the right), the Catalyst Switch on the left has the Active Firewall, whereas the one on the right has the Standby Firewall unit. All Firewall interfaces between active and standby Firewall are Layer 2 apart, which requires a 6 Gb dot1q Etherchannel link between the two switches. The 6 Gb channel is not mandatory, because all Firewall interfaces are virtual. It also can be of smaller bandwidth, with the obvious side effect of degradation in bandwidth after switchover. In this case, this link can be a bottleneck after switchover. Also it is not mandatory to have an Etherchannel link between Active and Standby Firewall modules. The only requirement is to have all corresponding interfaces of Active and Standby Firewall Layer 2 of OSI model apart.

  Both FWSMs (with Active and Standby Firewall modules) should have identical definitions of the firewall and normal router interfaces on MSFC. After switchover, that is, when the Active Firewall fails and the Standby unit becomes the new Active Firewall, all (and only) the Firewall traffic is bridged to the active Firewall over the Etherchannel between the two switches. Traffic coming out of Firewall will have to cross Etherchannel (or any other layer 2 connectivity to the other switch for Firewall interfaces) to go to the actual hosts.

It is extremely important to understand how failover operates, before delving into the details of configuration and troubleshooting of failover on FWSM. This section covers the following three items:

• Failover operations

• Configuration steps

• Troubleshooting steps

Failover Operations

Once two FWSMs are configured for failover, one will act as an active unit and the other one will act as standby. The decision-making process is dictated by the configuration and sequence of reboot. You can configure one of the units to be primary and other to be secondary. Prior to FWSM Version 2.x, FWSM worked in single firewall mode, but beginning with FWSM Version 2.x, a single FWSM can be converted to multiple virtual firewall (by configuring a security context). If you run failover in multiple mode, failover is not virtualized in release 2.x, which means that failover is not determined on a per-context basis, but rather as a whole for the module. A single context cannot be active when another context is standby on the same FWSM; rather all contexts are either active or standby. During the initialization phase, active and standby roles are decided. It is, however, possible to change the role of failover manually after the initialization process.

Initialization Phase

After both FWSM modules have booted and failover is enabled, the modules will attempt to communicate with each other. Once the communication is established, the primary will become active and the secondary will become standby. The active will then synchronize its configuration to the standby. The system is now in a steady state with both active and standby modules.

Failover Conditions

The standby unit can become active under one of the following conditions:

• If the active FWSM is removed or shut down

• If the active FWSM reboots due to manual or forced reset

• If communication over failover and firewall interfaces is lost

• If the number of healthy firewall interfaces on active falls to a number that is below half the number on standby

• If you force failover through CLI

Forced Reboot Conditions

An FWSM will self-reboot under one of the following three conditions:

• If the standby is unable to sync interfaces from Active properly

• If the standby is unable to complete the configuration sync from Active

• Either on active or standby, if a pool of 1550 size blocks is exhausted

Monitoring

As you have seen, failover can trigger either because one FWSM unit fails or one standby unit becomes healthier than the active unit. An FWSM unit's health depends on the interface's health. So to detect unit failure, or to decide which unit is healthier, both unit monitoring and interface monitoring needs to be performed as follows:

• Unit Monitoring Each FWSM will monitor the unit health of its failover peer by sending a Hello message to the other unit. If a unit has not received a Hello message from the other unit for 30 seconds, it will perform the following test:

  - Send ARP request for failover interface and each Firewall interface

  If a reply is heard on a Firewall interface, but not the failover interface, no failover will take place, and the Active/Standby state will remain as it is. If no replies are heard on either of the Firewall interfaces and on the failover interface, the other unit is marked as "Failed" and this unit will become Active, if it is not already. Conditions which can cause this type of failure include:

  - Removal of peer FWSM module

  - Reboot of peer FWSM module

  - Removal or failure of the physical medium carrying failover interface and Firewall interface traffic for FWSMs in separate chassis

  - Shutdown of failover interface and Firewall interface VLANs

  - Traffic overload on the failover interface and Firewall interfaces causing packet loss

• Interface Monitoring Each unit monitors the health of its own Firewall interfaces and those of its failover peer by sending Hello messages on the Firewall interfaces. If a unit does not receive any messages on a particular interface for three consecutive poll intervals, the unit will run the following tests on that interface for 30 seconds:

  1. Check for link status of that interface

  2. Check for any incoming traffic on that interface

  3. Send the ARP for the most recent hosts (up to 2) learned from that interface

  4. Broadcast ping to the interface's subnet

  If all of these tests fail, and the interface on the other FWSM is receiving traffic, or is able to ARP/PING a host on that interface, this unit's interface is marked as Failed.

  Remember that in FWSM version 1.x, if a unit finds that it has fewer than half the number of healthy interfaces as the other unit, it will mark itself as Failed. The Standby will take over if it is determined that the Active has failed. This is referred to as the 50 percent rule.

  The biggest drawback with the 50 percent rule is that that the failure of an important link does not necessarily trigger a unit failover. To understand this point, look at an example. Assume that you have three interfacesinside, outside, and DMZ for FWSM, which is set up as a failover. If the outside link is down, you will lose all connectivity to the outside world. Although this is an important link, due to the 50 percent rule, the Primary unit that is currently active will not failover, even though the secondary unit is healthier (as all three interfaces are up on the secondary unit). So you need to rely on Layer 1 or 2 redundancy schemes, such as Etherchannel and the Spanning-Tree protocol, to avoid this type of failure.

  To address this 50 percent rule shortcoming, FWSM release 2.x introduces the capability to modify the fixed 50 percent rule that applies to the 1.1(x) release. This is achieved using interface tracking. You can designate monitored interfaces across contexts. Each time a monitored interface fails, a counter (N) is incremented. It is compared against a global counter (M) that is specified by the user. Whenever N exceeds M, a module failover is triggered. Using this property, a module failover could be initiated when, for instance, all the interfaces of a context fail, or when the interface leading to the ISP router fails. This feature is available both in single and multiple modes.

  Interface failure might occur for one of the following reasons:

  - The VLAN interface is shut down or the VLAN is cleared from the switch.

  - The port/ports or the cable carrying the interface's VLAN between a pair of FWSMs in separate chassis is removed or becomes faulty.

  - The interface is overloaded with traffic and experiencing packet drops.

  - No failover IP address is configured.

  Interface failover can be determined either by using the show failover or show interface command.

Configuration Steps

Two protocols that help in attaining redundancy between two FWSMs are a failover protocol, and a logical update (LU) protocol. The failover protocol monitors the health of both FWSMs and their interfaces at fixed intervals, whereas the LU protocol ensures the replication of the connection table to the Secondary unit, to maintain the data flow of existing connections upon failover. Work through the following steps to configure Failover:

Step 1.	Be sure to fulfill the minimum requirements. Before attempting to configure for Failover, you must fulfill the following minimum requirements for it to operate correctly: Both FWSMs must be running the same version of FWSM software. Both must have the same number of VLANS mapped from the switch. Also be sure to configure the same number of interfaces on both units. Both units must be Layer 2 adjacent on all their interfaces. In other words, all interfaces must be capable of exchanging Layer 2 broadcast packets (Address Resolution Protocol [ARP] and so on) between each other, as the failover protocol packets cannot be routed. When running multiple modes, both modules must have the same licensing characteristicsfor example, the number of contexts, and so on. (This is not applicable for FWSM 1.x.) Both modules must run in the same mode; a single mode unit cannot be paired with a multiple mode unit. (This is not applicable for FWSM 1.x.) Both firewalls must agree on operating either as routed or transparent. (This is not applicable for FWSM 1.x.)
Step 2.	Configure two additional VLANs on the switch and map them to FWSM for the failover interface and the stateful failover link. You must create a VLAN on the switch and map this VLAN to the failover interface of both primary and secondary FWSMs. If your FWSMs are in different chassis, you must configure a trunk link to carry the VLAN to the other switch. If the FWSM modules are in the same chassis, be sure to map the VLANs that you have created to both of the FWSMs. The same thing applies for the stateful failover link. It is strongly recommended to create two separate VLANs for the failover interface and the stateful failover link. In the native IOS, you can create a layer 2 VLAN with the vlan command in configuration mode. In hybrid mode, you can do the same with the set vlan command. The LU protocol can generate large (up to 700Mbps) amounts of traffic. If the LAN and link interfaces are not dissociated, providing guaranteed output queue servicing to the failover protocol can be difficult when there is congestion. For instance, VLAN 100 could be dedicated to carrying the LU traffic, while VLAN 101 takes care of the failover protocol. Both VLANs can be trunked using 801.Q between the two switches.
Step 3.	Configure the failover LAN interface and the failover link. For failover to operate on the FWSM, a failover communication interface must be configured. This is the LAN interface over which failover protocol packets will travel. To configure stateful failover, use the same interface or a different one to carry LU traffic. With the previous steps, be sure to configure the VLANs for the failover interface and stateful failover link. Execute show vlan on both FWSMs to ensure that both VLANs (among other VLANs) are downloaded to the FWSM. Once the VLANs are downloaded, you can configure the failover interfaces with the following commands if you are running a FWSM version earlier than 2.x: nameif VLAN_NUMBER interface_name security_level ip address interface_name IP_address netmask failover lan interface <if_name> failover link <if_name> Example 4-16 shows the primary FWSM's configuration in single mode with the inside VLAN10, the outside VLAN30, and the failover VLAN 400 with FWSM version 1.x. Example 4-16. Failover Configuration for the Primary FWSM FWSM(config)# nameif 400 fover 50 FWSM(config)# nameif 450 flink 60 FWSM(config)# ip address fover 1.1.1.1 255.255.255.0 FWSM(config)# ip address flink 2.1.1.1 255.255.255.0 FWSM(config)# failover ip address fover 1.1.1.2 255.255.255.0 FWSM(config)# failover ip address flink 2.1.1.2 255.255.255.0 FWSM(config)# failover lan interface fover FWSM(config)# failover link interface flink FWSM(config)# On the secondary unit, if you are running FWSM Version 1, all you need is the failover interface configured as shown in Example 4-17. (You can simply copy and paste the failover interface configuration from the primary to the secondary unit.) Example 4-17. Failover Configuration for the Secondary FWSM FWSM(config)# nameif 400 fover 50 FWSM(config)# nameif 450 fover 60 FWSM(config)# ip address fover 1.1.1.1 255.255.255.0 FWSM(config)# failover ip addr fover 1.1.1.2 255.255.255.0 FWSM(config)# failover lan interface fover FWSM(config)# failover link interface flink FWSM(config)# Beginning with Version 2.x, this is configured with the following command: failover lan interface <if_name> vlan <vlan> failover link <if_name> [vlan <vlan>] Unlike in Version 1.x, you do not need to first nameif the VLAN interfaces and then associate them with failover functions. This is now performed using a single command that was described previously. If the firewall is operating in multiple mode, the failover interface and the stateful link interface are configured at the system execution space as shown in Example 4-18 (in single mode, the configuration is the same). Example 4-18. Two VLANS for the LAN Interface and the LU Interface FWSM(config)# failover lan interface fover vlan 400 FWSM(config)# failover link flink vlan 450 FWSM(config)# Assign IP addresses to these interfaces. The configuration is performed on both units (copy and paste from the primary unit to the secondary unit) as shown in Example 4-19. Example 4-19. Failover Interface IP Address Configuration FWSM(config)# failover interface ip fover 1.1.1.1 255.255.255.0 standby 1.1.1.2 FWSM(config)# failover interface ip flink 2.1.1.1 255.255.255.0 standby 2.1.1.2 FWSM(config)#
Step 4.	Assign IP addresses for the failover interfaces. You must assign standby IP addresses for each failover and Firewall interface. This is configured with the following command in FWSM Version 1.x: FWSM(config)# failover ip address <if_name> <IP_Address> The Standby FWSM will use these IP addresses as the addresses for its interfaces. The Active will use the IP addresses specified by the ip address command. The only exception to this is the failover communication interface, for which the primary will always use the IP address specified by the ip address command, and the secondary will always use the IP specified by the failover ip address command as shown in Example 4-20. Example 4-20. IP Address Configuration in Single Mode Running FWSM 1.x FWSM(config)# nameif 30 outside 0 FWSM(config)# nameif 10 inside 100 FWSM(config)# ip address inside 10.1.1.1 255.255.255.0 FWSM(config)# ip address outside 10.30.1.1 255.255.255.0 FWSM(config)# failover ip addr inside 10.1.1.2 255.255.255.0 FWSM(config)# failover ip addr outside 10.30.1.2 255.255.255.0 You do not need to configure the interfaces shown in Example 4-20 on the secondary. This information will be replicated to the secondary unit. If you are running FWSM 2.2, the same interface is configured with the following command: failover interface ip <if_name> <ip_address> <mask> standby <ip_address> The failover interface ip command now assigns an IP address to both the active and standby units, deprecating the 1.1(x) failover ip address syntax. Example 4-21 shows the interface configuration on the Primary unit. Example 4-21. Primary Unit's Interface IP Address Configuration FWSM(config)# failover interface ip inside 10.1.1.1 255.255.255.0 standby 10.1.1.2 FWSM(config)# failover interface ip outside 10.30.1.1 255.255.255.0 standby 10.30.1.2 FWSM(config)# If you are running multiple firewall mode, within a context, the same rule applies. The failover ip address command is deprecated in favor of an extension to the regular ip address command: ip address <if_name> <ip_address> <mask> [standby <ip_address>] In multiple firewall mode, create contexts, assign interfaces to them, and associate VLANs to interfaces within those contexts. Use the new ip address CLI when assigning IP addresses to interfaces. These steps are only performed on the primary unit. Example 4-22 shows a single context configuration in multiple mode. Example 4-22. Context Configuration on Multiple Mode FWSM(config)# context engineering Creating context 'client'... Done. (4) FWSM(config-context)# logical-interface vlan10 FWSM(config-context)# logical-interface vlan30 FWSM(config-context)# config-url tftp://10.10.1.101/engineering.cfg FWSM(config-context)# changeto context engineering FWSM/client(config)# nameif vlan10 inside 100 FWSM/client(config)# ip address inside 10.1.1.1 255.255.255.0 standby 10.1.1.2 FWSM/client(config)# nameif vlan30 outside 0 FWSM/client(config)# ip address outside 10.30.1.1 255.255.255.0 standby 10.30.1.2 ! Monitor all contexts' interfaces FWSM(config)# changeto context engineering FWSM/client(config)# monitor-interface inside FWSM/client(config)# monitor-interface outside ! Switch back to system execution space and enable failover on the primary unit: FWSM-Prod(config)# changeto system
Step 5.	Designate primary and secondary units. Unlike serial cable-based Failover on PIX Firewall, in LAN-based failover there is no physical cable to distinguish the Primary and Secondary units. This is determined through configuration. You can use the following command to elect which FWSM should become Primary and which one should be the Secondary unit: FWSM(config)# failover lan unit {primary|secondary} On the primary FWSM, execute the following command: FWSM(config)# failover lan unit primary On the secondary unit, execute the following command to turn it to a standby unit: FWSM(config)# failover lan unit secondary The FWSM configured as primary will become Active, and the secondary will become Standby if both units are booted up at the same time. In a running system, there will be no preemption. In other words, if a secondary is currently the Active module and a primary has finished booting, the primary will become Standby, rather than forcing a failover and becoming Active.
Step 6.	Configure interface policy (optional). Beginning with FWSM Version 2.x, the global interface policy can be specified either as a percentage or as an absolute number. When the number of monitored interfaces that have failed reaches or exceeds the interface policy, a module failover is triggered. The failover interface policy is configured within the system execution space using the following command: failover interface-policy <n> [percent] Assuming a failover interface policy of 1 interface, here is the sequence of events that occurs as soon as one monitored interface fails: Switching to Standby 104002: (Primary) Switching to STNDBY - Interface check Remember that a set of criteria needs to be met to fail an interface.
Step 7.	Configure poll interval (optional). A poll interval is the amount of time that elapses between each failure detection attempt. That interval is specified using the failover poll <interval> command. The poll interval is used for both unit and interface health monitoring: failover polltime [unit|interface] [msec] <x> [holdtime [msec] <y>] For example, failover poll time 15 can be configured with the following command: FWSM(config)# failover poll 15 In release 1.1(x) the failover polltime command is used to configure the frequency of failover Hello messages exchanged between two units. The value in 1.1(x) can range from 3 to 15 seconds. The hold timer is always fixed to three times the poll time value. In release 2.1(x), the poll interval can be adjusted independently for the interface and the unit. The unit poll interval is fixed to 1 second by default, but can be modified to any value. The interface poll time can be set between 3 to 15 seconds and does not have a configurable hold time.
Step 8.	Turn on stateful failover for http traffic (optional). HTTP replication is not on by default. The Failover replication http command turns on HTTP replication. Enabling HTTP replication reduces performance dramatically. It is not recommended in FWSM. The failover LAN interface should be dedicated.
Step 9.	Turn on the failover. You should turn on the failover on the primary unit first with the following command: FWSM(config)# failover ! No Response from Mate FWSM(config)# Once the no response from mate message appears, enable failover on the secondary unit by using the failover command with no parameters. From here on, no configuration is performed on the secondary unit. The following messages appear on the primary unit: 709003: (Primary) Beginning configuration replication: Send to mate. 709004: (Primary) End Configuration Replication (ACT) A series of SYSLOG messages 105003 and 105004 should quickly follow on the primary unit: 105003: (Primary) Monitoring on interface 1 waiting 105004: (Primary) Monitoring on interface 1 normal Meanwhile, the secondary unit should display the following sequence as shown in Example 4-23. Example 4-23. Messages that Will Appear on the Secondary Unit After Turning FO on FWSM(config)# failover Detected an Active mate. Switching to Standby Switching to Standby. Beginning configuration replication from mate. This unit is in syncing state. 'failover' command will not be effective at this time End configuration replication from mate. 709006: (Secondary) End Configuration Replication (STB) FWSM(config)#

Note

If the write memory command is performed from the system execution space, the standby module deletes all its existing contexts and syncs the entire system execution space configuration from the active firewall. If the write memory command is performed from within a context, the standby module adds the context to its system execution space and synchronizes the context's configuration in addition.

Troubleshooting Steps

Before working through some of the common scenarios, it will be helpful for you to examine the syslog messages that are shown on the secondary FWSM after you enable logging on as shown in Example 4-24.

Example 4-24. Syslog on the Secondary FWSM

FWSM(config)# logging on FWSM(config)# logging monitor 7 FWSM(config)# logging console 7 FWSM(config)# 111008: User 'enable_15' executed the 'logging con 7' command.         Detected an Active mate. Switching to Standby         Switching to Standby. FWSM(config)# Beginning configuration replication from mate. This unit is in syncing state. 'failover' command will not be effective at this time End configuration replication from mate. 709006: (Secondary) End Configuration Replication (STB) Access Rules Download Complete: Memory Utilization < 1% 105003: (Secondary) Monitoring on interface 2 waiting 105003: (Secondary) Monitoring on interface 1 waiting 105004: (Secondary) Monitoring on interface 2 normal 105004: (Secondary) Monitoring on interface 1 normal FWSM# 

After the units are synchronized with each other, you can find the status of a unit on both the primary and secondary with the show failover command. Example 4-25 shows the output of the show failover command on the primary unit.

Example 4-25. Monitoring Failover Status on Primary FWSM

FWSM(config)# show failover Failover On Failover unit Primary Failover LAN Interface fover Reconnect timeout 0:00:00 Poll frequency 15 seconds         This host: Primary - Active                 Active time: 29925 (sec)                 Interface outside (10.30.1.1): Normal                 Interface inside (10.1.1.1): Normal         Other host: Secondary - Standby                 Active time: 285 (sec)                 Interface outside (10.30.1.2): Normal                 Interface inside (10.1.1.2): Normal Stateful Failover Logical Update Statistics         Link : Unconfigured. FWMS# 

Example 4-26 shows the same on the secondary unit.

Example 4-26. The Status of FO With the show failover Command

FWSM(config)# show failover Failover On Failover unit Secondary Failover LAN Interface fover Reconnect timeout 0:00:00 Poll frequency 15 seconds         This host: Secondary - Standby                 Active time: 285 (sec)                 Interface inside (10.1.1.2): Normal                 Interface outside (10.30.1.2): Normal         Other host: Primary - Active                 Active time: 30750 (sec)                 Interface inside (10.1.1.1): Normal                 Interface outside (10.30.1.1): Normal Stateful Failover Logical Update Statistics         Link : Unconfigured. FWSM(config)# 

Work through the following steps to troubleshoot the failover problem:

Step 1.	Level 1 syslog specifies a reason for failover. So always check the syslog to determine the root cause. For example, if the switch port failed on the inside interface of Active Firewall, you would see the following message on the Primary (Active) Firewall. 411002: Line protocol on Interface inside, changed state to down 105007: (Primary) Link status 'Down' on interface 1 104002: (Primary) Switching to STNDBYinterface check, mate is healthier Syslog from Secondary (Standby) Firewall will report the following message: 104001: (Secondary) Switching to ACTIVEmate want me Active
Step 2.	Issue the show failover history command to see the reasons for the past state changes, which will help narrow the possibilities for the failover. FWSM# show fail history ====================================================================== >From State To State Reason ====================================================================== Init Disabled Set by the CI config cmd Disabled Listen Set by the CI config cmd Listen Active No Active unit found Active Active Other unit got stuck in learn state after sync =======================================================================
Step 3.	Execute show vlan on both FWSM units and make sure you have the same VLANs downloaded on both units.
Step 4.	Execute show interface on both FWSM units to make sure they are up. Map the VLANs to the right interfaces.
Step 5.	Test the connectivity by pinging to the Failover interface IP. Be sure to allow the ICMP for the interfaces.
Step 6.	Verify that the Failover interface VLAN is configured on the switch, and is not removed from the switch accidentally.
Step 7.	Make sure all VLANS are trunked between the switches.
Step 8.	Be sure to turn on dot1Q across the board on the switch.