Section 10-3. Verifying Firewall Connectivity

10-3. Verifying Firewall Connectivity

When you install a firewall or make configuration changes to one, you might need to verify that it can communicate on all its interfaces. Users might also report problems they experience when trying to pass through the firewall. You need a logical approach to verifying the firewall's operation and troubleshooting its connectivity.

You can follow these basic steps to verify that a firewall can communicate with its neighboring networks:

Step 1.	Test with ping packets.
Step 2.	Check the ARP cache.
Step 3.	Check the routing table.
Step 4.	Use traceroute from a workstation to verify the forwarding path through the firewall.
Step 5.	Check the access lists.
Step 6.	Verify address translation operation.
Step 7.	Look for active shuns.
Step 8.	Check user authentication.
Step 9.	See what has changed.

TIP

In case you need to open a case with the Cisco TAC, you can gather all the necessary information from your firewall with one command. Configure your terminal emulator to log the session to a file, and issue the show tech-support command. The saved file can be uploaded or sent to the TAC engineer.

Step 1: Test with Ping Packets

If you know that some hosts on one interface are having trouble reaching hosts on another interface, you can begin troubleshooting with the most basic tool, the ping or ICMP echo.

You should begin by testing with ping packets from the firewall itself to hosts on each side. From a management session, you can use this command:

PIX 6.x FWSM 2.x	Firewall# ping [if_name] host
PIX 7.x	Firewall# ping [if_name] host [data pattern] [repeat count] [size bytes] [timeout seconds] [validate]

The destination address is host. If the firewall can find a routing table entry for the destination, it can figure out which interface to use as the source and source address. You can override that choice by specifying the interface name as if_name (inside, for example).

Beginning with PIX 7.x, the ping command has been expanded to allow more options. This is similar to the same command in the Cisco IOS software.

TIP

Be aware that although the firewall transmits the ping packet (ICMP echo request), it doesn't necessarily allow the reply (ICMP echo reply) to return and terminate on its interface.

By default, PIX and ASA platforms permit inbound ICMP packets to terminate on their interfaces, but FWSM doesn't. In any event, make sure that the firewall is configured to permit at least the specific ICMP packet types that are useful for troubleshooting. You can use the following icmp commands to accomplish this:

 Firewall(config)# icmp permit any echo if_name Firewall(config)# icmp permit any echo-reply if_name Firewall(config)# icmp permit any time-exceeded if_name Firewall(config)# icmp deny any if_name 

Try sending pings from the firewall to a known host in each direction. If the pings are successful, the firewall can communicate with nearby hosts on the most basic level.

Now go to a host on each side of the firewall, and try to send ping packets toward the firewall. First, you can try to ping the firewall interfaces directly.

If that is successful, you might want to try sending pings from a host on one side of the firewall to a host on the other. This tests the full round-trip path out and back through the firewall. Make sure you have configured any access lists to allow inbound ICMP echo, echo-reply, and time-exceeded packet types on the firewall interfaces. This might need to be configured on each of the firewall interfaces along the path, depending on the firewall platform and whether an access list has been applied to the interfaces. In addition, PIX 7.x offers an ICMP inspection engine that should be enabled so that ICMP traffic can be inspected and permitted.

TIP

You can also enable ICMP debugging on the firewall to get detailed information about how ICMP packets are being handled. Be careful, though, because any type of packet debugging can adversely load an already-busy firewall. Also, when you enable ICMP debugging, the firewall reports on any ICMP packet that arrives, is inspected, and is forwarded. This might result in much more information than you are expecting, especially when you are trying to test by pinging a single host.

Use the following command to start ICMP debugging:

 Firewall# debug icmp trace 

Be sure to stop the debugging session as soon as you're finished with it. You can use the no debug icmp trace command, as well as no debug all or undebug all.

For example, suppose host 192.168.199.100 on the inside of a firewall is trying to ping host 172.16.89.5 on the outside. The ICMP debug output is as follows:

 Firewall# debug icmp trace ICMP trace on Warning: this may cause problems on busy networks Firewall# 1: ICMP echo-request from inside:192.168.199.100 to   172.16.89.5 ID=0 seq=1435 length=80 2: ICMP echo-request: translating inside:192.168.199.100 to   outside:172.16.89.161 3: ICMP echo-reply from outside:172.16.89.5 to 172.16.89.161 ID=0   seq=1435 length=80 4: ICMP echo-reply: untranslating outside:172.16.89.161 to   inside:192.168.199.100 Firewall# no debug icmp trace ICMP trace off 

Step 2: Check the ARP Cache

Like any host on a network, a firewall must have the basic mechanism to relate IP addresses (Layer 3) to MAC addresses (Layer 2). This is done by building and maintaining an ARP cache. Normally, when a host knows a destination's IP address, it sends an ARP request in the hope that the destination will send an ARP reply with its MAC address. The firewall can build its ARP cache by sending its own ARP requests or by listening to other ARP replies on its interfaces.

Connectivity through the firewall can be affected if the firewall has a stale or incorrect ARP entry. For example, the next-hop router on an interface might have just changed its IP address. A host on the local network of a firewall interface could also change its IP address. If the firewall has a previous ARP entry and doesn't hear an ARP reply with the new information, it continues to use the stale entry.

For example, consider a host that starts out with IP address 192.168.199.100. Later, it changes to 192.168.199.101. Here is the firewall ARP cache before and after the address change:

 Firewall# show arp         inside 192.168.199.100 0050.e2c6.f680 Firewall# show arp         inside 192.168.199.101 0050.e2c6.f680         inside 192.168.199.100 0050.e2c6.f680 Firewall# show arp timeout arp timeout 14400 seconds 

With the default ARP timeout value, the old ARP entry remains in the cache for up to 4 hours! (In PIX 7.x, the show arp timeout command is actually show running-config arp timeout.)

Now imagine the opposite case, in which a change in MAC addresses occurs. Suppose the hosts on a network have been using an existing router as their default gateway. The router has always been 192.168.199.1 (00d0.0457.3bfc). Now a firewall is being installed in the network, and it will become the default gateway for the internal hosts. Therefore, the firewall uses 192.168.199.1, and the change in gateway platforms is transparent to the end users.

For security reasons, Cisco firewalls try to stay silent and don't announce any information about themselves. As a result, they don't send gratuitous ARP requests on their interfaces. One side effect of this appears when a firewall replaces an existing router on the network. All the internal hosts still have an ARP cache entry for 192.168.199.1 as 00d0.0457.3bfc, which belonged to the router. The firewall comes up as 192.168.199.1, but with its own MAC address of 0090.276c.3d0a. Because it doesn't announce itself with a gratuitous ARP, the hosts never notice the change in MAC addresses, and they continue to use the old router MAC address until their ARP entries expire. In other words, none of the hosts can send packets to their default gateway (the firewall) until the old ARP entries are flushed.

You should also be aware of how a firewall maintains its own ARP cache. If an ARP cache entry exists for an IP-to-MAC address mapping, a new ARP reply has the following effect:

• If the MAC address is the same, but the IP address is different, a new ARP entry is added to the cache.

• If the IP address is the same, but the MAC address is different, the existing ARP entry is overwritten with the new information.

You can follow these steps to verify ARP cache information:

1.

Display the ARP cache:

 Firewall# show arp 

Each ARP entry contains the station's MAC address, the corresponding IP address, and the firewall interface where the ARP reply was heard. A sample ARP cache is as follows:

 Firewall # show arp         stateful 192.168.199.2 0030.8587.5432         outside 172.16.11.71 0003.4725.2e23         outside 172.16.11.67 00d0.01e6.6ffc         inside 192.168.25.9 0003.4725.2e94         inside 192.168.25.10 0003.a088.5769         inside 192.168.254.2 0000.0c07.ac01 

2.

(Optional) Verify the ARP cache timeout value:

PIX 6.x FWSM 2.x	Firewall# show arp timeout
PIX 7.x	Firewall# show running-config arp timeout

Each ARP entry has an aging or persistence timer associated with it. By default, the firewall keeps an entry for 14,400 seconds (4 hours) before discarding it. If subsequent ARP replies are received for that entry before the timer expires, the timer is reset.

TIP

To adjust the ARP persistence timeout value, you can use the arp timeout seconds configuration command.

3.

(Optional) Flush the ARP cache:

 Firewall# clear arp 

If an entry exists and the host changes its IP address, an ARP reply might not be received. The stale entry, with its now-outdated IP-MAC address mapping, continues to be used for up to 4 hours.

In this case, you should flush the stale ARP entry so that a new one can be created. It isn't possible to flush just one entry; the entire ARP cache must be flushed at one time.

4.

(Optional) Define a static ARP entry:

 Firewall(config)# arp interface_name ip_address mac_address [alias] 

Sometimes you might need to define a static ARP cache entry that will never expire. This can be handy if there is a host that only listens on its network interface and never sends an ARP reply. The static ARP entry is associated with the firewall interface named interface_name and binds ip_address to mac_address (dotted-triplet format as in xxxx.yyyy.zzzz).

You can add the alias keyword to create a static ARP cache entry that the firewall uses to generate proxy ARP replies. When other hosts send an ARP request for that address, the firewall uses the alias entry to send back an ARP reply.

Step 3: Check the Routing Table

A firewall maintains a routing table, much as a router does. The routing table is consulted when a packet has been inspected and is ready to be forwarded out a firewall interface. The firewall must have a route for the destination network in its routing table before a packet can be sent on its way.

You can verify the routing table contents with the show route command, as in this example:

 Firewall# show route O IA 192.168.167.1 255.255.255.255 [110/11] via 192.168.198.4, 0:00:01, inside C    192.168.77.0 255.255.255.0 is directly connected, dmz C    192.168.198.0 255.255.255.0 is directly connected, inside C    10.1.0.0 255.255.0.0 is directly connected, outside S*   0.0.0.0 0.0.0.0 [1/0] via 10.1.1.1, outside Firewall# 

Here, routes marked with C are directly connected to the firewall. In other words, they are subnets that are configured on the firewall interfaces. Routes marked with O have been learned through OSPF. The default route shown is labeled S because it has been statically configured on the firewall.

Verify that a destination network you are trying to reach through the firewall actually appears in the firewall's routing table. Obviously, if a specific subnet doesn't appear, the default route is used.

Step 4: Use Traceroute to Verify the Forwarding Path

Traceroute is a common tool you can use to discover the path from one host to another through the network. The originating host sends special packets toward the destination. Each router that is encountered along the way returns a message to the source, indicating that it was present on the path.

The host generating a traceroute sends packets toward the destination with the IP time-to-live (TTL) field incrementing from 1. The first traceroute packet has a TTL of 1; the first router to receive the packet must decrement the TTL (as all routers are required to do). The TTL is now 0, so the router drops the packet and returns an ICMP time-exceeded message to the source. The next traceroute packet has a TTL of 2, so the second-hop router returns an ICMP message, and so on.

As the ICMP time-exceeded messages are received at the traceroute source, the router IP address is displayed, along with the round-trip time. As soon as the destination host returns its ICMP message, a record of each router hop along the path is available.

Figure 10-12 illustrates this process. Host PC-A performs a traceroute to host PC-B. Each successive traceroute packet has a higher TTL and is returned by routers progressively further along the path. Notice that the firewall participates in the traceroute process only if ICMP packets are used as traceroute probes. Otherwise, the firewall only inspects and forwards the traceroute packets without modification and does not appear as a hop.

Figure 10-12. Traceroute as a Progression of Probe Packets

If you use traceroute to troubleshoot or verify connectivity through a firewall, be aware that the firewall might be missing from the traceroute hop information. If it is missing, UDP traceroute packets have been used; if it appears, ICMP traceroute packets have been used. If the traceroute begins to time out before the destination host is reached, something along the way is preventing connectivity. This could be the firewall if it has not been configured to allow the traceroute packets in both directions.

When PCs perform a traceroute (the tracert command), they send ICMP echo-request packets with increasing TTL values. When Cisco routers and switches perform a traceroute (the trace IOS or traceroute CatOS command), they send UDP packets to port 33434 with increasing TTL values. (Some platforms use a port slightly greater than 33434, and others begin with 33434 and increase the port number with each packet sent.)

You should configure the firewall to permit the following types of packets through each of its interfaces along the traceroute path:

• ICMP echo-request Traceroute packets sent from the source.

• ICMP echo-reply When the target host is finally reached, it returns an echo-reply message.

• ICMP unreachable Messages returned by distant hosts for path MTU discovery.

• ICMP time-exceeded Messages returned by routers indicating a traceroute hop.

• UDP Traceroute packets sent from Cisco devices.

• DNS Used to look up domain names of each traceroute hop (if requested).

You can permit these packet types on the firewall by using the following template to configure an access list that will be applied to a firewall interface:

 Firewall(config)# access-list acl_name permit icmp any any eq echo Firewall(config)# access-list acl_name permit icmp any any eq echo-reply Firewall(config)# access-list acl_name permit icmp any any eq unreachable Firewall(config)# access-list acl_name permit icmp any any eq time-exceeded Firewall(config)# access-list acl_name permit udp any range 32768 65535 any range   33434 33523 Firewall(config)# access-list acl_name permit udp any dns_address eq domain 

The fifth line, permitting UDP traffic, is optional and should be used only if you require UDP traceroute response from the firewall. You should use caution if you decide to include that command, because it gives open access from any host to any host over a wide range of UDP ports. The UDP port range must be kept wide because the traceroute UDP port tends to vary across router and switch platforms.

Finally, if you use ICMP traceroute probes and you want the firewall to return an ICMP message to declare its presence, you need one final configuration change. (The firewall doesn't accept any UDP traceroute probes, so it doesn't return any ICMP error messages about them. As a result, the firewall doesn't appear as a hop when UDP probes are used.)

By default, an FWSM firewall drops any ICMP packet destined for a firewall interface address, whereas PIX and ASA platforms permit them. To get the firewall to interpret the traceroute probe and send back an ICMP time-exceeded message, you should enable specific ICMP processing on the inbound firewall interface:

 Firewall(config)# icmp permit source-address source-mask echo if_name Firewall(config)# icmp permit source-address source-mask echo-reply if_name Firewall(config)# icmp permit source-address source-mask time-exceeded if_name Firewall(config)# icmp permit source-address source-mask echo if_name 

TIP

Beginning with PIX 7.x, you should enable the ICMP inspection engine to examine any ICMP traffic passing through the firewall. You can do this with the inspect icmp command while configuring a policy map.

On any Cisco firewall platform, you should enable the fixup or inspection for ICMP error messages with the inspect icmp error command (PIX 7.x) or fixup icmp error command (PIX 6.x and FWSM 2.x). No stateful inspection is performed; instead, the firewall examines the ICMP error packets and translates the appropriate addresses in the original header portion of the error payload.

Step 5: Check the Access Lists

If you find specific hosts or types of traffic that cannot pass through the firewall, you should verify the access lists that are applied to the firewall interfaces.

In some cases, corporate security policies might dictate that the firewall should deny a certain type of traffic. In other cases, an access list might be misconfigured or might have an outdated configuration that denies traffic by mistake.

You can follow these steps to help pinpoint an ACL configuration that could be blocking traffic:

1.

Identify which ACL is applied to an interface:

PIX 6.x FWSM 2.x	Firewall# show access-group
PIX 7.x	Firewall# show running-config access-group

If your firewall has many access lists configured, you might not remember which one has been applied or bound to a particular interface or in which direction it was bound (PIX 7.x allows both in and out directions). You also might not remember the name of a certain access list. The output of this command shows which ACL is bound to which interface, as in this example:

 Firewall# show running-config access-group access-group acl_out_v19 in interface outside access-group acl_inside in interface inside access-group acl_temp_dmz in interface dmz access-group mylist_3 in interface dmz2 Firewall# 

2.

(Optional) Check the ACL commands in the configuration:

 Firewall# show run | {include | grep} matching_text 

If extensive ACLs are defined, this is a much more efficient way to find matching statements than just paging through the firewall configuration (show run or write term). You can quickly see any references to matching_text in the ACL configuration.

However, if object groups are defined and they are applied in the ACL, this method might not display intuitive results. In other words, the object groups and ACLs are configured separately, so this search displays matching text from either configuration section.

3.

Check ACL activity:

 Firewall# show access-list acl_name [| {include | grep} matching_text] 

Each condition in the access list is shown, along with a count of the number of "hits" or times any packet has matched that condition on the interface. You can add an optional filter to search the ACL for specific matching_text. If your ACL is very large, you might want to use the | include filter to look for subnet or host addresses or specific protocol or port numbers within the ACL.

If you know the specific ACL rule that you are expecting to be active in permitting or denying traffic, you can check its hit count (hitcnt). If the hit count is 0, no packets have matched that condition; if the count is positive or rising, it is matching traffic. In the following example, the first line shown permits public access to the 192.168.106.0/24 subnet using SQL, but no matching traffic has been seen. The second line allows public access to a web server, and almost 6 million matching packets have been seen.

 access-list acl_outside line 720 permit udp any 192.168.106.0 255.255.255.0   eq 1433 (hitcnt=0) access-list acl_outside line 761 permit tcp any host 192.168.106.13 eq www   (hitcnt=5997791) 

TIP

If you have configured many object groups in your firewall, you might find it difficult to sort through both object group and access list definitions to find specific conditions. The show access-list command also expands any object group references in the ACL so that the object group contents are shown with their respective hit counts.

In this way, the firewall rules can be shown as they are interpreted. In the following example, an object group called web-servers has been configured and referenced in an ACL:

 Firewall# show access-list acl_outside | begin web-servers access-list acl_outside line 740 permit tcp any object-group web-servers eq www access-list acl_outside line 740 permit tcp any host 172.17.69.20 eq www   (hitcnt=8743) access-list acl_outside line 740 permit tcp any host 172.17.69.30 eq www   (hitcnt=16432) access-list acl_outside line 740 permit tcp any host 172.17.69.40 eq www   (hitcnt=1711) access-list acl_outside line 740 permit tcp any host 172.17.69.50 eq www   (hitcnt=4913495) [output deleted] 

Notice that each line has the same ACL line number (740), indicating that the object group is referenced in that line of the ACL acl_outside. The object group has been expanded so that the ACL looks like it was written with separate rules for each web server. This makes it much easier to search for things without trudging through complex nested object groups and ACL statements.

Step 6: Verify Address Translation Operation

Before a host can send traffic through a firewall, the firewall must have the following items configured properly:

• Address translation (using static or nat/global commands)

• An access list that permits the traffic; this allows connections to be established

You can monitor the xlate and conn entries for specific hosts in several ways, as described in the following sections.

Monitoring Translations

You can see what (if any) address translations have been created for a host by using a form of the show xlate command:

 Firewall# show xlate [detail] [global | local ip1[-ip2] [netmask mask]] lport |   gport port[-port]] [interface if1[,if2][,ifn]] [state static [,dump]   [,portmap] [,norandomseq] [,identity]] [debug] [count] Firewall# show xlate [{global | local} ip1[-ip2] [netmask mask]] [{lport | gport}   port[-port]] [interface if1[,if2][,ifn]] [state {static | portmap | identity |   norandomseq}] [debug] [detail] 

Remember with xlate entries, global represents the translated address on the lower-security interface, and local is the address on the higher-security interface. Use any of the following commands according to the IP address information you already know.

You can select a global or local address or a range of addresses. You can add a local port (lport) or global port (gport) or a range of ports. In addition, you can specify one or more firewall interfaces that are involved with the translation. You can specify the translation type as static, portmap (PAT), identity (identity NAT), or noramdomseq (no randomization of the TCP initial sequence number).

The command output is shown in a brief format (local and global addresses only) by default. The output can also be requested as detail (the default format plus the firewall interfaces and translation type) or as debug (the detail format plus the idle time).

To get a feel for the current number of xlate table entries, you can use the following command:

 Firewall# show xlate count 

TIP

The show xlate commands covered in the following sections give basic information about the translated addresses, along with any port translation. Here is an example:

 Firewall# show xlate local 172.27.112.26 18356 in use, 22368 most used PAT Global 10.1.1.1(11104) Local 172.27.112.26(4685) PAT Global 10.1.1.1(24560) Local 172.27.112.26(4695) 

You can also display translation flags and timeout values by adding the debug keyword to each command:

 Firewall# show xlate local 172.27.112.26 debug 18729 in use, 22368 most used Flags: D - DNS, d - dump, I - identity, i - inside, n - no random,        o - outside, r - portmap, s - static TCP PAT from inside:172.27.112.26/4697 to outside:10.1.1.1/23228 flags   r idle 0:03:19 timeout 0:00:30 TCP PAT from inside:172.27.112.26/4695 to outside:10.1.1.1/24560 flags   r idle 48:06:38 timeout 0:00:30 

Finding Xlate Entries Based on a Global Address

To find xlate entries based on a global address, use the following command:

 Firewall# show xlate global global-ip [netmask mask] [gport global-port] 

For example, the following output shows some of the PAT entries that have been created using global address 12.163.11.72:

 Firewall# show xlate global 12.163.11.72 15624 in use, 26615 most used PAT Global 12.163.11.72(10454) Local 172.21.40.111(1033) PAT Global 12.163.11.72(10406) Local 172.21.96.91(1052) PAT Global 12.163.11.72(10416) Local 172.21.100.59(2749) [output omitted] 

Finding Xlate Entries Based on a Local Address

To find xlate entries based on a local address, use the following command:

 Firewall# show xlate local local-ip [netmask mask] [lport local-port] 

For example, the following command is used to display the xlate entries for all local addresses using local port 123 (UDP 123 is used for NTP):

 Firewall# show xlate lport 123 15630 in use, 26615 most used PAT Global 128.1.10.72(190) Local 172.21.60.179(123) PAT Global 128.1.10.72(108) Local 172.21.56.125(123) PAT Global 128.1.10.72(122) Local 172.21.156.122(123) PAT Global 128.1.10.72(116) Local 172.21.198.207(123) [output omitted] 

Finding All Static Xlate Entries

To find all static xlate entries, use the following command:

 Firewall# show xlate state static 

The following command displays some of the xlate entries that have been configured with the static command:

 Firewall# show xlate state static Global 10.16.98.118 Local 172.16.100.99 Global 10.16.98.101 Local 172.21.192.116 Global 10.16.98.105 Local 172.17.197.66 Global 10.16.98.111 Local 172.17.232.74 Global 10.16.98.107 Local 172.20.196.434. 

Finding All PAT Entries (Dynamic Xlates)

To find all PAT entries (dynamic xlates), use the following command:

 Firewall# show xlate state portmap 

For example, the following command displays all the dynamic PAT entries that are currently in the xlate table:

 Firewall# show xlate state portmap 15723 in use, 26615 most used PAT Global 68.163.1.2(10452) Local 172.16.104.89(4499) PAT Global 68.163.1.2(10454) Local 172.16.40.111(1033) PAT Global 68.163.1.2(10406) Local 172.16.96.91(1052) 

Finding All Identity Entries

To find all identity entries (global and local addresses are identical), use the following command:

 Firewall# show xlate state identity 

For example, the following command displays all identity xlate entries that are currently in use:

 Firewall# show xlate state identity 1 in use, 12 most used Global 192.168.198.17 Local 192.168.198.17 

Monitoring Connections

You can see what (if any) UDP and TCP connections have been established for a host by using a form of the show conn command. The command has the following syntax:

 Firewall# show conn [state state_type] [{foreign | local} ip1[-ip2] netmask mask]   [long] [{lport | fport} port1[-port2]] [protocol {tcp | udp}] 

With no additional keywords, show conn displays the entire conn table. In PIX 7.x, only the conn table size is shown unless show conn all is given.

For TCP connections, you can specify the connection state as state state_typse, where the state type is one of the keywords listed in Table 10-13.

Table 10-13. TCP Connection State Types

Keyword	Flag	Description
up	U	The TCP connection is up and functional
finin	r	TCP FIN requested in the inbound direction and acknowledged
finout	R	TCP FIN requested in the outbound direction and acknowledged
http_get		HTTP GET connection
smtp_data	M	SMTP data connection
nojava		A connection that has Java applets blocked by the java filter command
data_in	I	Data has moved in the inbound direction
data_out	O	Data has moved in the outbound direction
rpc	R	SunRPC
h225	h	H.225
h323	H	H.323
sqlnet_fixup_data	q	SQL*Net
conn_inbound	B	Connection with the initial SYN originating from the outside
sip	T	SIP
mgcp	g	MGCP
ctiqbe	C	Computer Telephony Interface Quick Buffer Encoding (CTIQBE)
skinny	k	Cisco Skinny SCCP

You can also specify the foreign or local IP address (or range of addresses) used in the connections, as well as the local port (lport) or foreign port (fport) or a range of ports. The IP protocol can also be given as tcp or udp.

With connections, "foreign" represents the host on the lower-security interface, and "local" is the address of the host on the higher-security interface. Table 10-14 lists the possible display formats.

Table 10-14. Display Formats Available with the show conn Command

Keyword	Information Displayed
(Default)	Local and foreign IP addresses and ports, idle time, connection byte count, connection flags
detail	Default format plus firewall interfaces
long	detail format plus address translations

You can also display the current number of conn table entries with the following command:

 Firewall# show conn count 

The output from this command shows the total number of conn table entries, regardless of protocol. It isn't possible to show a breakdown of entries by protocol (UDP and TCP, for example).

The show conn commands listed in the following sections display the protocol, foreign and local addresses and ports, connection idle time, connection data volume, and connection flags. Here is a brief example:

 Firewall# show conn foreign 10.10.39.14 10449 in use, 577536 most used TCP out 10.10.39.14:1033 in 192.168.29.37:524 idle 0:08:06 Bytes 11076 flags UIOB 

You can also display similar information along with a listing of the connection flags by adding the detail keyword:

 Firewall# show conn foreign 10.10.39.14 detail 10385 in use, 577536 most used Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,        B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,        E - outside back connection, F - outside FIN, f - inside FIN,        G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i -   incomplete,        k - Skinny media, M - SMTP data, m - SIP media, O - outbound data,        P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN,        R - UDP RPC, r - inside acknowledged FIN, S - awaiting inside SYN,        s - awaiting outside SYN, T - SIP, t - SIP transient, U - up TCP outside:10.10.39.14/1033 inside:192.168.29.37/524 flags UIOB 

Notice that the foreign and local host addresses are shown with the firewall interface names where they reside. If you are interested in which host (foreign or local) initiated the connection, you have to interpret the connection flags.

In the preceding example, you can determine the following facts from flags UIOB:

• U The connection is up (fully established).

• I Inbound data has been received from the foreign host.

• O Outbound data has been sent from the local host.

• B The initial TCP SYN flag came from the outside (foreign host). Therefore, the connection was initiated by the foreign host.

Because UDP is not connection-oriented, it isn't possible to see which host initiated a UDP session.

Finding Conn Entries Based on a Foreign Address

To find conn entries based on a foreign address, use the following command:

 Firewall# show conn foreign foreign_ip [netmask mask] 

The foreign address foreign_ip is used to find all the relevant connections in the conn table. For example, you can use the following command to find all active conn table entries involving the foreign address 198.133.219.25 (http://www.cisco.com):

 Firewall# show conn foreign 198.133.219.25 7046 in use, 57005 most used TCP out 198.133.219.25:80 in 172.21.4.5:3050 idle 0:03:32 Bytes 10762 flags UIO Firewall# 

Finding Conn Entries Based on a Local Address

To find conn entries based on a local address, use the following command:

 Firewall# show conn local local_ip [netmask mask] 

For example, local address 172.16.193.187 is found to have the following open conn table entries:

 Firewall# show conn local 172.16.193.187 17267 in use, 57005 most used TCP out 147.208.14.61:80 in 172.16.193.187:4536 idle 0:00:03 flags UIO TCP out 64.215.168.97:80 in 172.16.193.187:4538 idle 0:00:03 flags UIO UDP outside:65.98.25.54:53 inside:172.16.193.187:51455 flags d [output omitted] 

Finding Conn Entries Based on Protocol Information

To find conn entries based on protocol information, use the following command:

 Firewall# show conn protocol {tcp | udp} [fport foreign_port | lport local_port] 

The following TCP connections using foreign port 1433 are currently in this firewall's conn table:

 Firewall# show conn protocol tcp fport 1433 TCP out 64.191.149.170:443 in 172.16.1.10:1988 idle 0:00:55 Bytes 0 flags saA TCP out 160.109.67.70:443 in 172.16.1.11:1173 idle 0:03:56 Bytes 157428 flags UIO TCP out 64.73.28.116:443 in 172.16.1.12:1294 idle 0:01:06 Bytes 921 flags UIO [output omitted] 

Monitoring Specific Hosts

If you are looking for stateful inspection information for a particular host, you can spend time sifting through the xlate and conn tables. However, there is a much more direct way to get this information in a single command:

PIX 6.x	Firewall# show local-host [ip_address]
PIX 7.x	Firewall# show local-host [ip_address] [all] [detail]

You can get information about any "local" host as long as it resides on a firewall interface with a security level greater than 0 and you know its IP address. Beginning with PIX 7.x, a local host can reside on any firewall address, regardless of the security level. The firewall does all the table lookups for you and returns any information it finds. The output also includes any connection limits that might apply to the host.

If you don't specify an IP address, all local hosts that are currently in the xlate and conn tables are displayed. (PIX 7.x is an exception; if no IP address is given, only a count of connections on each interface is displayed.)

In the following example, information about the host at 172.18.10.10 is sought. This subnet has been defined as a static identity translation so that addresses 172.18.10.x on the inside also appear on the outside.

 Firewall# show local-host 172.18.10.10 Interface inside: 15721 active, 15829 maximum active, 0 denied local host: <172.18.10.10>,     TCP connection count/limit = 1/unlimited     TCP embryonic count = 0     TCP intercept watermark = unlimited     UDP connection count/limit = 0/unlimited   AAA:   Xlate(s):     Global 172.18.10.10 Local 172.18.10.10   Conn(s):     TCP out 172.16.3.14:53957 in 128. 172.18.10.10:23 idle 0:00:03 Bytes 545   flags UIOB Firewall# 

You can see the static identity xlate entry by the IP address appearing as both global and local under the Xlate(s) section. The local host had only one active connectiona Telnet session from the foreign host 172.16.3.14.

The show local-host command is especially useful for local hosts that have dynamic xlate or PAT entries for outbound connections. In the following example, you are interested in the activities of the local host at 172.21.96.22. All inside local hosts are translated using the firewall's outside interface address and dynamic port information.

 Firewall# show local-host 172.21.96.22 Interface inside: 15725 active, 15829 maximum active, 0 denied local host: <172.21.96.22>,     TCP connection count/limit = 176/unlimited     TCP embryonic count = 11     TCP intercept watermark = unlimited     UDP connection count/limit = 0/unlimited   AAA:   Xlate(s):     PAT Global 207.246.96.46(44112) Local 172.21.96.22(1168)     PAT Global 207.246.96.46 (44120) Local 172.21.96.22(1176)     PAT Global 207.246.96.46 (58859) Local 172.21.96.22(1293)     PAT Global 207.246.96.46 (29899) Local 172.21.96.22(1339)     PAT Global 207.246.96.46 (33585) Local 172.21.96.22(1469)  [otuput deleted]   Conn(s):     TCP out 217.123.215.64:6883 in 172.21.96.22:1168 idle 0:00:21 Bytes 15918696   flags UIO     TCP out 66.127.201.130:6881 in 172.21.96.22:1176 idle 0:00:22 Bytes 24003969   flags UIO     TCP out 219.111.75.71:6881 in 172.21.96.22:1293 idle 0:00:58 Bytes 730251   flags UIO     TCP out 156.34.72.34:6881 in 172.21.96.22:1339 idle 0:00:58 Bytes 4969   flags UIO     TCP out 200.104.11.50:6883 in 172.21.96.22:1469 idle 0:00:19 Bytes 4744   flags UIO [output omitted] 

Here, you find that local host 172.21.96.22 has 176 active (and established) TCP connections to foreign hosts. As well, the firewall has tracked 11 TCP connections that are embryonic or half-opened. This can be a cause of concern if many embryonic connections are opened to the local host. The firewall is using its default settings and allowing an unlimited number of embryonic connections to be attempted.

The show local-host command has also conveniently summarized all xlate entries involving host 172.21.96.22 so that you can see all the local and foreign port number pairs being used. Below that, all its active connections are summarized, as if you had used the show conn command.

By default, the show local-host command doesn't display any connections that are active to or from the firewall itself. Beginning with PIX 7.x, you can add the all keyword to display all the local host's active connectionsincluding those that terminate or originate at the firewall. The firewall interfaces can be considered local hosts too.

In the following example, a client (172.21.4.60) on the firewall's outside interface (172.16.1.1) has opened an SSH connection to the firewall. Notice that the firewall interface name NP Identity Ifc is displayed as the interface where the SSH connection terminates.

 Firewall# show local-host 172.16.1.1 all Interface dmz: 0 active, 0 maximum active, 0 denied Interface inside: 0 active, 1 maximum active, 0 denied Interface outside: 1 active, 14 maximum active, 0 denied Interface NP Identity Ifc: 1 active, 2 maximum active, 0 denied local host: <172.16.1.1>,     TCP flow count/limit = 0/unlimited     TCP embryonic count to (from) host = 0 (0)     TCP intercept watermark = unlimited     UDP flow count/limit = 0/unlimited   Conn:     TCP out 172.21.4.60:3244 in 172.16.1.1:22 idle 0:00:00 bytes 621703 flags UIOB Firewall# 

Clearing Xlate Table Entries

On several occasions, the xlate table might contain stale or incorrect entries. This can happen if you make configuration changes to the static, global, or nat commands on a firewall or to an interface access list. As soon as that happens, it is likely that the xlate table has existing entries that use previous or outdated translations.

For example, if several hosts are using Telnet connections, and a new policy is added to an access list to deny the Telnet protocol, new Telnet connections are blocked. The existing connections remain open until the application closes them or until the firewall idles them out. Therefore, the xlate entries and the corresponding conn table entries for the active Telnet sessions remain active too. Those entries must be cleared manually if the policy needs to be enforced immediately.

You can clear xlate entries manually, either one at a time or the whole table at once. Use one of the following commands:

Firewall# clear xlate global global_ip [netmask mask] [gport global_port]	Clears an xlate entry based on a global address.
Firewall# clear xlate local local_ip [netmask mask] [lport local_port]	Clears an xlate entry based on a local address.
Firewall# clear xlate interface if_name_1[,if_name_2]	Clears xlate entries bound to a pair of firewall interfaces.
Firewall# clear xlate	Clears the entire xlate table.

Although this is a quick way to flush the entire xlate table, you should be careful. Any active connections using the address translations are dropped immediately. This means that users could see applications or sessions terminate in the middle of important work.

If you must clear the xlate table, do so at a time of low usage or during a downtime window.

Adjusting Table Timeout Values

You can also adjust various idle timers that affect address translations and connections maintained by the firewall. Use the following commands if you feel a timeout adjustment is needed:

• Xlate entry timer:

   Firewall(config)# timeout xlate hh[:mm[:ss]] 

  
  

  By default, xlate entries involving TCP connections are be deleted after they have been idle (no data passed) for 3 hours. The minimum idle time is 1 minute, but the xlate idle timer can't be set to a value that is less than the uauth timer (the default is 5 minutes).

  Xlate portmap (PAT) entries created for UDP always idle out after 30 seconds. This idle timer cannot be configured.

• TCP conn entry timer:

   Firewall(config)# timeout conn hh[:mm[:ss]] 

  
  

  Conn table entries for TCP connections idle out after 1 hour by default. The minimum idle period is 5 minutes. You can use a time value of 0:0:0 or 0 to indicate that TCP conn entries should never time out. In that case, TCP connections must close themselves when both hosts exchange FIN flags.

• TCP half-closed conn entry timer:

   Firewall(config)# half-closed hh[:mm[:ss]] 

  
  

  A TCP connection is half-closed if only one of the pair of hosts signals a connection termination with its FIN flag. If the other host doesn't respond with its FIN handshake, the firewall goes ahead and closes the connection after an idle period.

  By default, half-closed connections are deleted after 10 minutes. The minimum idle time is 5 minutes.

  NOTE

  Half-open TCP connections (also called embryonic connections) result if only one of the pair sends the TCP SYN flag. The firewall automatically closes half-open connections if a fixed period of 2 minutes elapses before the three-way TCP handshake completes.

  
  

• UDP conn entry timer:

   Firewall(config)# udp hh[:mm[:ss]] 

  
  

  UDP sessions, because they are connectionless, must simply time out. No mechanism exists for one host to close or terminate a UDP session. By default, a firewall deletes a UDP conn entry after it has been idle (no data passing) for 2 minutes. The minimum idle time is 1 minute.

Step 7: Look for Active Shuns

A Cisco firewall can shun (block) traffic coming from specific source addresses on an interface. This feature is useful when a host is generating malicious traffic and needs to be stopped. If you have manually added shuns to your firewall, or if another system has added them automatically, you might forget that they are in place.

In fact, when shuns are defined, they are dynamic in nature and aren't added to the firewall configuration. If you are troubleshooting why a host isn't receiving or sending any traffic, you might not find the reason by looking through the firewall configuration.

You can display a list of active shuns with the following EXEC command:

 Firewall# show shun [src_ip] 

If you provide a specific source address src_ip, only shuns involving that address are shown.

You can also look at shun activity and duration with the following command:

 Firewall# show shun statistics 

To delete a shun, you can use the following EXEC command:

 Firewall# no shun src_ip [dst_ip sport dport [protocol]] 

You must provide the source address (src_ip). The destination address (dst_ip), source and destination ports, and protocol are all optional. If they are omitted, 0s are assumed in those fields.

For example, suppose a user has called to report that her host at 172.21.196.38 cannot reach the public network. You have trouble locating the cause of the problem until you look at a list of the active shuns on the firewall:

 Firewall# show shun Shun 172.16.44.59 0.0.0.0 0 0 Shun 172.20.36.62 0.0.0.0 0 0 Shun 172.31.69.84 0.0.0.0 0 0 Shun 172.21.196.38 0.0.0.0 0 0 Shun 172.18.77.186 0.0.0.0 0 0 Shun 172.16.103.161 0.0.0.0 0 0 Shun 172.16.100.166 0.0.0.0 0 0 Shun 172.21.196.230 0.0.0.0 0 0 Shun 172.16.73.135 0.0.0.0 0 0 Firewall# show shun statistics stateful=OFF, cnt=0 dmz2=OFF, cnt=0 outside=OFF, cnt=210499 inside=ON, cnt=2605814591 Shun 172.16.44.59 cnt=269, time=(392:44:35) Shun 172.20.36.62 cnt=125493, time=(392:44:35) Shun 172.31.69.84 cnt=49384, time=(392:44:35) Shun 172.21.196.38 cnt=80694744, time=(392:44:35) Shun 172.18.77.186 cnt=36524, time=(392:44:35) Shun 172.16.103.161 cnt=1021, time=(392:44:35) Shun 172.16.100.166 cnt=9835, time=(392:44:35) Shun 172.21.196.230 cnt=52, time=(392:44:35) Shun 172.16.73.135 cnt=321282, time=(392:44:35) Firewall# 

The shun in question (shaded in the output) has been quite active, blocking 80,694,744 packets since it was defined more than 392 hours ago. When you have determined that it is safe to delete that shun, you can use the following command:

 Firewall# no shun 172.21.196.38 

TIP

Keep in mind that shun definitions are dynamic in nature. You should keep an offline list of all the shuns you add manually. If the firewall loses power or reloads, or a failover condition occurs, all the active shuns are lost. In other words, you have to reconfigure them as soon as you realize they are no longer active.

As a long-term plan, you should consider converting the shuns into access list entries so that they become part of the firewall configuration. This makes them more permanent and leverages the firewall's capability to generate Syslog information about their activity.

Step 8: Check User Authentication

You can configure user authentication in several forms on a Cisco firewall. Refer to the information in the following sections to verify that authentication is working properly.

Authentication Proxy (Uauth)

If your firewall is configured as an authentication proxy, a loss of connectivity might be related to authentication problems. You can view the active authenticated users with the show uauth com-mand. Any users who have successfully authenticated are shown, along with their IP addresses and uauth timeout values:

 Firewall# show uauth                         Current    Most Seen Authenticated Users       1          1 Authen In Progress        0          1 user 'hucaby' at 192.168.199.33, authenticated    absolute   timeout: 0:05:00    inactivity timeout: 0:00:00 Firewall# 

If a user is listed as authenticated but is unable to connect through the firewall, there might be a problem with authorization. For example, a AAA server could be downloading an authorization filter or ACL name associated with the user, but that ACL isn't defined on the firewall. If this is the case, the Syslog server should show a message similar to this:

 Feb 26 2004 21:55:42 Firewall : %PIX-3-109016: Can't find authorization ACL   'engineering' on 'PIX' for user 'hucaby' 

If you do find a problem with a user's authentication session, you can clear that person from the uauth table with the clear uauth username command.

If no users are being successfully authenticated through the firewall, you should look on a more fundamental level. Make sure the firewall can communicate with the proper RADIUS or TACACS+ server. You can use the debug aaa authentication command to watch debug messages on the firewall console or Telnet session as a user tries to authenticate.

For example, suppose a firewall is configured to query a RADIUS server (192.168.11.49) for user authentication. A user named hucaby is getting an authentication prompt from the firewall but is being rejected. The debug aaa authentication command produces the following output:

 Firewall# debug aaa authentication Firewall# 34: Received response: , session id 201242794 35: Making authentication request for host 192.168.11.49, user , session id:   201242794 36: Processing challenge for user , session id: 201242794, challenge: Please   register for access Username: 37: sending challenge to user: , challenge: Please register for access Username: , session id: 201242794 38: Received response: hucaby, session id 201242794 39: Making authentication request for host 192.168.11.49, user hucaby, session id:   201242794 40: Processing challenge for user hucaby, session id: 201242794, challenge:   Password: 41: sending challenge to user: hucaby, challenge: Password: , session id:   201242794 42: Received response: , session id 201242794 43: Making authentication request for host 192.168.11.49, user hucaby, session id:   201242794 44: Authentication failed for user :hucaby, pass :firewallzrc00l, session id:   201242794 45: retrying Authentication for user :hucaby, pass : firewallzrc00l, session id:   201242794 46: Received response: , session id 201242794 47: Making authentication request for host 192.168.11.49, user , session id:   201242794 48: Processing challenge for user , session id: 201242794, challenge: Please   register for access Username: 49: sending challenge to user: , challenge: Please register for access Username: , session id: 201242794 

The user is being prompted for login credentials, and you can see how the firewall is communicating the userID and then the password to the RADIUS server. The highlighted message indicates that the RADIUS server has rejected the user's information because the password is incorrect. In fact, the password entered by the user is displayed in the debug output.

Content Filtering

Another area to look at is content filtering through Websense or N2H2. If one of these services denies a user's access to a URL, you should check the server's configuration for that user or user group.

At the most basic level, you should see evidence that the firewall is talking to the content-filtering server. You can use the show url-server stats command. If the counters shown are nonzero and are increasing with user traffic, the content-filtering communication is working properly, as shown in this example:

 Firewall# show url-server stats URL Server Statistics: ---------------------- Vendor                           websense URLs total/allowed/denied        420691/363332/57359 HTTPSs total/allowed/denied      0/0/0 FTPs total/allowed/denied        0/0/0 URL Server Status: ------------------ 192.168.204.17             UP URL Packets Sent and Recieved Stats: ----------------------------------- Message                 Sent    Recieved STATUS_REQUEST          310338  309198 LOOKUP_REQUEST          416011  414554 LOG_REQUEST             0       NA ----------------------------------- Firewall# 

If the firewall is having trouble reaching the content-filtering server, you might see one of the following Syslog messages:

 %PIX-2-304007: URL Server IP_address not responding, ENTERING ALLOW mode. %PIX-3-304003: URL Server IP_address timed out URL url %PIX-3-304006: URL Server IP_address not responding %PIX-6-304004: URL Server IP_address request failed URL url 

As soon as the content-filtering server again can be reached, you should see this Syslog message:

 %PIX-2-304008: LEAVING ALLOW mode, URL Server is up. 

Step 9: See What Has Changed

If you have installed, configured, and tested a firewall, trusted users should be able to pass through it according to the security policies. At the same time, the firewall should deny or drop all untrusted users and traffic. Suppose things have been working like this for some time, but one day users begin to call and complain.

One possible cause of a problem is that someone somewhere has changed something on your network. One good troubleshooting approach is to ask, "What changed during the time when problems began to occur?"

In the case of a firewall, it is entirely possible that someone has made a change to the configuration, especially if you work with a staff of other firewall administrators. How can you determine if a change was made and who made it?

1.	Implement an orderly change-management process. If an administrator needs to alter a firewall configuration, the change should be announced in advance, approved by peer review, and performed at an expected time. Records of the change-management activity should be kept as an audit trail.
2.	Keep copies of the firewall configuration archived. As each change is made, archive the new configuration. Not only will you have an additional audit trail, but you also will make a downgrade situation easier if you need to back out of a large configuration change.
3.	Use AAA authentication for firewall management access. This forces each firewall administrator to have a unique user ID and password, which makes management access more secure. In addition, the firewall generates an audit trail each time a configuration change is made or a command is executed. AAA authentication should always be used in conjunction with Syslog collection for the sole purpose of maintaining a thorough audit trail. When AAA authentication is not used, firewall administrators and users take on the generic identity enable_1 (privilege level 1) when they first log in. After using the enable command, users take on the generic identity enable_15 (privilege level 15). Naturally, when AAA authentication is used, users appear as their own user IDs. If you were trying to sift through the Syslog records to see who made changes to a firewall configuration, which of these messages would be more useful? Feb 26 2005 08:06:30 Firewall-A : %PIX-5-111008: User 'enable_15' executed the 'clear xlate' command. Feb 26 2005 08:09:13 Firewall-A : %PIX-5-111008: User 'enable_15' executed the 'no access-list acl_outside' command. or Feb 26 2005 08:06:30 Firewall-A : %PIX-5-111008: User 'hucaby' executed the 'clear xlate' command. Feb 26 2005 08:09:13 Firewall-A : %PIX-5-111008: User 'hucaby' executed the 'no access-list acl_outside' command. Clearly, you would like to track down user hucaby to see why he cleared the xlate table and deleted the ACL applied to the outside interface at the start of the business day.