Several tools and commands are available on the PIX Firewall to troubleshoot all kinds of issues with PIX Firewall. In this section, we will attempt to go through all such tools and commands, which will be used in the rest of the chapter for troubleshooting specific issues.
show commands on PIX Firewall are used to display statistics and information about the PIX firewall both current and past. show commands are used mainly for troubleshooting and the monitoring the health of the PIX firewall. Some of most useful show commands are shown in the sections that follow.
show xlate [detail]
This command shows the translation details through the PIX firewall. Example 3-5 shows both the summary and details of translation that are built up on the PIX firewall. It is recommended to look at the details of the translation which gives the interfaces involved the flow of a packet. This is useful to identify and correct any NAT related mis-configuration.
Example 3-5. Translation Through the PIX Firewall
PIX (config) # show xlate 3 in use, 3 most used PAT Global 22.214.171.124(0) Local 10.1.1.50 ICMP id 340 PAT Global 126.96.36.199 (1024) Local 10.1.1.50(1028) PAT Global 188.8.131.52 (1024) Local 10.1.1.50(516) PIX (config) # show xlate detail 3 in use, 3 most used Flags: D DNS, d dump, I identity, I inside, n no random, o outside, r portmap, s static TCP PAT from inside:10.1.1.50/1026 to outside:184.108.40.206/1024 flags ri UDP PAT from inside:10.1.1.50/1028 to outside:220.127.116.11/1024 flags ri ICMP PAT from inside:10.1.1.50/21505 to outside:18.104.22.168/0 flags ri PIX(config)#
Table 3-4 describes the Xlate flags.
Table 3-4. Xlate Flags Table
Static Translation Slot
Dump Translation Slot on Next Clearing Cycle
Port Map Translation
No Randomization of TCP Sequence Number
Outside Address Translation
Inside Address Translation
DNS A RR Rewrite
Identity Translation from NAT 0
show connection [detail]
This command shows the connection details output on the PIX firewall. Connection will not be built up without translation. So, if you do not see any connection, you need to find out if the translation is built up. Example 3-6 shows the translation that is built up on the PIX firewall.
Example 3-6. Shows the show connection [detail] Output from the PIX Firewall
PIX (config) # show connection 2 in use, 2 most used ! Idle time, bytes transferred and the flags are shown in the following connection TCP out 22.214.171.124:23 in 10.1.1.15:1026 idle 0:00:22 bytes 1774 flags UIO UDP out 126.96.36.199:31649 in 10.1.1.15:1028 idle 0:00:14 bytes 540 flags d ! Following command shows the interface details of the connection PIX (config) # show connection detail 2 in use, 2 most used Flags: A awaiting inside ACK to SYN, a awaiting outside ACK to SYN, B initial SYN from outside, D DNS, d dump, E outside back connection, F outside FIN, f inside FIN, G group, H H.323, I inbound data, M SMTP data, m SIP media, O- outbound data, P inside back connection, q SQL*Net data, R outside acknowledged FIN, R UDP RPC, r inside acknowledged FIN, S awaiting inside SYN, s awaiting outside SYN, T SIP, t SIP transient, U up TCP outside: 188.8.131.52/23 dmz:10.1.1.15/1026 flags UIO UDP outside: 184.108.40.206/31649 dmz:10.1.1.15/1028 flags d PIX#
A local-host is an entry that is created for any source IP on a higher security level interface. The show local-host command displays the translation, connection, and AAA information together. Example 3-7 shows a local-host information for local host IP address of 10.1.1.50.
Example 3-7. show local-host Command Output
PIX# show local-host 10.1.1.50 Interface inside: 822 active, 823 maximum active, 0 denied local host: <10.1.1.50>, TCP connection count/limit = 0/unlimited TCP embryonic count = 0 TCP intercept watermark = unlimited UDP connection count/limit = 63/unlimited AAA: Xlate(s): PAT Global 220.127.116.11(41166) Local 10.1.1.50(39075) Conn(s): UDP out 18.104.22.168:8943 in 10.1.1.50:63556 idle 0:01:31 flags PIX#
This command is used to see what inspection policies are applied and the packets matching them, as shown in Example 3-8.
Example 3-8. Output of the show service-policy Command
PIX# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 0, drop 0, reset-drop 0 Inspect: ftp, packet 0, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 0, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 0, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 0 PIX#
show asp drop
This command is used to identify the number of packets dropped by the PIX while processing the packet as shown in Example 3-9.
Example 3-9. show asp drop Command Output
PIX# show asp drop Frame drop: Invalid tcp length 9382 Invalid udp length 10 No route to host 1009 Reverse-path verify failed 15 Flow is denied by access rule 25247101 First TCP packet not SYN 36888 Bad option length in TCP 731 TCP MSS was too large 10942 TCP Window scale on non-SYN 2591 TCP Dual open denied 11 TCP data send after FIN 62 TCP failed 3 way handshake 328859 TCP SEQ in SYN/SYNACK invalid 142 TCP ACK in SYNACK invalid 278 TCP packet SEQ past window 46331 DNS Inspect packet too long 5 DNS Inspect id not matched 8270 ... PIX#
show cpu usage
This command is first introduced in PIX OS Version 6.0(1). Under normal conditions, the PIX CPU should stay below 30 percent, and can go as high as 60 percent. Anything above 60 percent is high. If the CPU reaches 100 percent, the PIX will start dropping packets. The show cpu usage command displays the CPU over time as a running average as shown below:
PIX# show cpu usage CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1% PIX#
The percentage usage prints as NA (Not Applicable) if the usage is unavailable for the specified time interval; this can happen if you try to find out CPU Usage before the 5-second, 1-minute, or 5-minute intervals.
The show traffic command displays the traffic transmitted and received on each interfaces of the PIX as shown in Example 3-10.
Example 3-10. show traffic Command Output
PIX# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec PIX#
The show blocks command and the show cpu usage command are useful in determining whether the PIX is being overloaded. The blocks are internal storage locations, similar to queues on a router; a packet is stored in a block until the PIX can process it and place it on the outbound interface xmit queue. Example 3-11 shows the show blocks output.
Example 3-11. show blocks Command Output
PIX# show block SIZE MAX LOW CNT 4 100 99 100 80 100 98 100 256 600 592 600 1550 1704 1362 1448 2048 100 100 100 2560 40 40 40 4096 30 30 30 8192 60 60 60 16384 104 104 104 65536 10 10 10 PIX#
In the show blocks command output, SIZE is the block size, MAX is the total number of block available, LOW is the lowest number of blocks available on PIX since the last reboot, and the CNT is the current number of blocks available for specific task. Both LOW and CNT for any block size hit to zero indicate a low memory condition, which requires further investigation. To determine which features are responsible for memory block utilization, refer to Table 3-5, which summarizes the different memory blocks and the purpose of different sizes of blocks.
Table 3-5. Showing Different Sizes of Memory Blocks and Their Usage
Created at boot up time
Duplicating existing blocks in DNS, isakmp, url-filtering, uauth, h323, tftp, and TCP modules
Used in TCP Intercept to generate an ACK packet, failover hello messages.
Stateful Failover, Syslog, TCP module
Ethernet Packets, buffering url filtered packets.
Only used for the Livengood (i82543) Gig Ethernet cards
show output filters
Sometimes, it is very important to view the show command output to specific lines for specific information. You can do this with the Output filter option. The syntax is as follows:
show command | begin | include | exclude | grep [-v] <regular_exp>
Following is a list of definitions for the arguments of this command:
begin Start displaying the output beginning at the First Match of the RegEx, and continue to display the remaining output.
include Display any line that matches the RegEx.
exclude Display any line that does not match the RegEx.
grep This is the same as include.
grep v This is the same as exclude.
For example, to display the interface stats starting with ethernet1, execute the following command:
PIX# show interface | begin ethernet1
To display only the route statements from the running-config, execute the following command:
PIX# show running-config | include route
To display the whole configuration except for the access-lists, you can execute the following command:
PIX# show running-config | exclude access-list
Displaying the access-list entries that contain address 10.1.1.50 can be achieved with the following command:
PIX# show access-list | grep 10.1.1.50
To display only access-list entries that have non-zero hit counts, execute the following command:
PIX# show access-list | grep v hitcnt=0
show tech-support collects output of a list of show commands. The command for show tech-support is as follows:
show tech-support [no-config | detail | tftp:]
Example 3-12 summarizes how to use the show tech-support command.
Example 3-12. How to Use the show tech-support Command
! Following command will collect all the information including the running-config for ! Troubleshooting PIX# show tech-support ! The following command collect the same output without the running-config PIX# show tech-support no-config ! The following allows you to redirect the output into different location PIX# show tech-support file ? flash: Write output to flash: file system ftp: Write output to ftp: file system tftp: Write output to tftp: file system PIX# ! The following command allows you to collect the detail show tech-support information PIX# show tech-support detail . . . . . . PIX#
Although show commands are very useful to identify the problem quickly, debug commands are required to see more detailed information about the problem under some circumstances for the connectivity issues. As debug commands affect the CPU of the PIX negatively, use the debug command as a last resort. Before turning on the respective debug command, it is very important to know how much traffic is flowing through the firewall of a specific type.
In this section, we discuss several debug commands available on PIX that will help you troubleshoot connectivity problems across the PIX firewall.
debug icmp trace
This debug command is used to see the debug output of the ping going across the PIX firewall. Ping is usually used to check the IP connectivity across the firewall.
While using ping for connectivity tests, remember these points:
You can ping only the local interface of the PIX. For example, if your PC is on the inside network, you can ping only to the inside interface of the PIX.
You cannot ping the remote interface of the PIX. For example, if you are on an inside network, you cannot ping to the DMZ or the outside interface of the PIX firewall. If you are on outside, you can ping only on the outside interface.
ICMP echo-replies must be permitted explicitly thru the PIX unless you have ICMP inspection enabled.
Figure 3-2 demonstrates that Bob is able to ping to the inside interface, not the DMZ or outside interface.
Figure 3-2. Inside Users Ping Ability to PIX Firewall
For a successful ping across the PIX firewall, you should see the request and reply packets on the debug icmp trace command output as shown in Example 3-13.
Example 3-13. debug icmp trace Output of a Successful IP Connection
PIX# debug icmp trace PIX# ! In the following line, the ingress interface is inside, inside untranslated IP ! address, and destination address of the packet is 192.168.1.50 ICMP echo-request from inside:10.1.1.50 to 192.168.1.50 ID=3239 seq=4369 length=80 ! Following line shows that 10.1.1.50 is translated to IP address 172.16.10.1 ICMP echo-request: translating inside:10.1.1.50 to outside:172.16.10.1 ! The following lines are reply packets ICMP echo-reply from outside:192.168.1.50 to 172.16.10.1 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:172.16.10.1 to inside:10.1.1.50 PIX#
The debug icmp trace command output shows enough details of the ingress, egress, source, destination IP/port, and the protocol information which can be used to define the packet flow.
You can turn off all debugs globally on the PIX firewall by issuing the no debug all or undebug all (u all is the short form) command.
To troubleshoot any application-specific issues, for example, Session Initiation Protocol (SIP) across the PIX firewall, you may run the debug for the corresponding protocol. For instance, to troubleshoot the issue pertaining to Media Gateway Control Protocol (MGCP) for voice over IP (VoIP) traffic between a phone and Call Manager, run the debug mgcp command.
debug pix process
To debug NAT within the payload of the packet, run debug pix process to see if the NAT is working correctly. For instance, if there is NAT in for MGCP, debug mgcp will not show the NAT details from the payload. For the NAT details, you must run debug pix process.
debug fixup tcp | udp
Use this command to find any inspection-related issue of a protocol. For example, to debug a problem with an FTP connection, you might need to run debug fixup tcp, which shows the FTP connection-related issue.
The capture command (introduced in Version 6.2) allows for sniffing the packet hits at the interface of the PIX firewall. The command debug packet is deprecated by this capture command. The capture command must be executed from the enable mode (not the configuration mode), and optionally, you can configure an access-list to define the interesting traffic. Traffic can be captured both before and after it passes through the PIX; one capture on the inside interface, one capture on the outside interface. You can copy captures via TFTP or HTTPS.
Following is the syntax to enable capture on the PIX Firewall for traffic analysis:
capture capture-name [access-list acl-name] [buffer buf-size] [circular-buffer] [ethernet-type type] [interface if-name] [packet-length bytes]
Table 3-6 summarizes the meaning of the arguments of the capture command.
Table 3-6. capture Command Arguments
This is the name of the capture that is used to view the information after the capture is completed.
Used to define the traffic that needs to be captured.
Capture buffer saved in RAM (default size 512kb).
The default is to stop capturing when the buffer is full. Overwrites the buffer from beginning when full. The default is non-circular.
Used to ca Ethernet packets of a particular type. The default is IP.
Used to c packets on a specific interface. The default is all interfaces.
Used to configure the maximum length to save from each packet. The default is 68 bytes.
To illustrate how to use the capture command on the PIX firewall, examine an example. Assume that an inside host (10.1.1.100) is unable to go through Telnet to the server with IP address 22.214.171.124 on the outside. Additionally, assume that the host address 10.1.1.100 is translated to 50.1.1.00. Work through the following steps to enable capture on the PIX firewall:
Create an ACL for both the inside and outside interfaces.
You must create two separate ACLs to apply with the capture for the inside and outside interfaces.
The inside interface ACL should use the untranslated source IP address and the destination IP address:
PIX(config)# access-list 100 permit tcp host 10.1.1.100 host 126.96.36.199 eq 23 PIX(config)# access-list 100 permit tcp host 188.8.131.52 eq 23 host 10.1.1.100
The outside interface ACL should use the translated source address and the destination IP address.
PIX(config)# access-list 101 permit tcp host 184.108.40.206 host 220.127.116.11 eq 23 PIX(config)# access-list 101 permit tcp host 18.104.22.168 eq 23 host 22.214.171.124
Create captures on both inside and outside interfaces.
PIX(config)# capture out-telnet access-list 101 interface outside packet-length 1500 PIX(config)# capture in-telnet access-list 100 interface inside packet- length 1500
Perform the test.
Initiate a Telnet session from the inside host (10.1.1.100) to access 126.96.36.199 on the outside.
Copy the captures off to a TFTP Server or use HTTP server on the PIX.
You can display the capture output on the PIX firewall with the following commands:
PIX# show capture in-telnet PIX# show capture out-telnet
To download the capture output to a TFTP server, use the following commands:
PIX# copy /pcap capture:out-telnet tftp://10.1.1.5/out.pcap PIX# copy /pcap capture:in-telnet tftp://10.1.1.5/in.pcap
If the HTTP server is enabled on the PIX firewall for the ASDM access, you can use the following command to download the pcap files from the PIX firewall using the web browser:
Analyze captures with sniffer software.
After downloading the captures from the PIX firewall, you can analyze the captures with sniffer capture software such as Ethereal.
The capture command has been enhanced to capture packets dropped by security policies.
PIX# capture mycapture type asp-drop ? acl-drop Flow is denied by access rule all All packet drop reasons bad-crypto Bad crypto return in packet bad-ipsec-natt Bad IPSEC NATT packet bad-ipsec-prot IPSEC not AH or ESP bad-ipsec-udp Bad IPSEC UDP packet bad-tcp-cksum Bad TCP checksum bad-tcp-flags Bad TCP flags conn-limit Connection limit reached . . . PIX#
The capture command on the PIX firewall is useful only if the packets are reaching to the PIX interface. So you need to rely on external sniffer capture software. Besides, the capture command output can be converted and saved in pcap format, which later can be opened and analyzed by sniffer capture software. Ethereal is very popular free downloadable sniffer software (www.ethereal.com).
Syslog is the best troubleshooting tool for the PIX firewall. It logs traffic both to and through the firewall. The level of detail provided by syslog is controlled by the level of detail at which PIX is configured for syslog. Seven syslog logging levels can be set on the PIX firewall as shown in Table 3-7.
Table 3-7. Syslog Messages
# of Messages (Sum)
Work through the steps that follow to configure logging on the PIX firewall.
Define what you want to capture.
The first step is to enable syslog on the PIX firewall to define the amount of logging you want to capture. There are two ways to define what you want to capture: first with the syslog level, and second with the event_list. The general syntax for enabling logging is as follows:
PIX(config)# [no] logging console | buffered | monitor | trap | mail | asdm event_list | level
While defining a different level of syslog, you can direct the logging to a monitor, console, buffer, ASDM, syslog server, or e-mail. For example, to enable the logging level to debug and capture the information in the buffer, configure the following:
PIX(config)# logging buffer debug
The No form of the command which follows will turn off debug level buffer logging.
PIX(config)# no logging buffer debug
You can change the default buffer size with the following command:
PIX(config)# [no] logging buffer-size <bytes>
For example, to set up the buffer size to be 8192 bytes, use the following command:
PIX(config)# logging buffer-size 8192
If you connect to the PIX with Telnet or SSH and want to display the level 6 logging on the monitor, use the following command:
PIX(config)# logging monitor 6
This can be written as follows:
PIX(config)# logging monitor informational
To send debug level logging to a syslog server, use the following command:
PIX(config)# logging trap debug
The following commands send the critical level information to e-mail recipient
PIX(config)# logging mail critical
You can configure the "Modifiable syslog" feature on the PIX to reduce the amount of syslog. For example, to determine what commands are being executed on the PIX, message 111009 records this information, but by default it is at level 7 (Debug).
%PIX-7-111009: User 'xyz' executed cmd: show run
So, to capture this syslog ID, the PIX must have the debug level enabled. With debug level logging, PIX generates a huge amount of logging. To cut this down, use the following command to bring the syslog ID down to some lower level, for example, level 1, which will reduce the number of messages substantially with the following command:
PIX(config)# logging message 111009 level 1
You also can use the following command:
PIX(config)# logging message 111009 level alerts
Now your syslog message should look like this:
%PIX-1-111009: User 'xyz' executed cmd: show run
To disable the modifiable syslog, you can use the following command:
PIX(config)# no logging message 111009 level alerts
Or, you can use the following command:
PIX(config)# logging message 111009 level 7
With a modifiable syslog, you will still get some logs in different lower levels (for example level 0, 1, 2, and so on). If you just want to see a specific syslog message, use the event class configuration.
An event list can be configured to allow only the specific syslog ID to be logged. An event_list provides you the flexibility to track events by class, severity, or syslog message ID. If you just want to capture syslog for ID 101001 only, you can use the following commands:
PIX(config)# logging list mylist message 101001 PIX(config)# logging buffered mylist
Define the syslog server.
You must define the external syslog server IP address to forward the syslog message to the external syslog server. If your syslog server resides on the inside network with an IP address of 10.1.1.5, use the following command:
PIX (config) # logging host inside 10.1.1.5
Define the mail server.
If you decide to send out syslog information to e-mail addresses, you need to configure the mail server and the e-mail addresses to forward the syslog information.
PIX(config)# logging from-address firstname.lastname@example.org PIX(config)# logging recipient-address email@example.com level critical PIX(config)# smtp server pri-smtp-host sec-smtp-host PIX(config)#
Turn on the time stamp for syslog.
You can configure the time stamp for logging with the following command:
PIX(config)# logging timestamp
Use the following command to turn off the timestamp:
PIX(config)# no logging timestamp
Redirect debug to syslog if needed.
To redirect the debug output to syslog, execute the following command:
PIX(config)# logging debug-trace
To turn off the redirection, use the following command:
PIX(config)# no logging debug-trace
The syslog message number used is 711011.
Turn logging on.
Finally, turn logging ON with the following command:
PIX(config)# logging enable
The following command turns off logging:
PIX(config)# no logging enable
Once logging is configured, you can verify the syslog configuration with the following command:
PIX# show running logging
To remove the logging configuration, use the following command:
PIX(config)# clear config logging
To display buffer logging syslog messages, use the following command:
PIX# show logging
To display only the syslog configuration settings, use the following command:
PIX# show logging setting
System syslog messages on PIX/ASA 5500 are found at the following link:
Syslog messages based on different severity levels on PIX/ASA 5500 can be found at the following link:
Get the syslog message ID from the syslog server, find the meaning, and perform the recommended action suggested by the syslog message ID in the previously listed links.
Traceback is a record of abnormal function calls that is usually shown on the console of the PIX firewall, when an abnormal situation occurs. Problems with PIX normal functionality may produce a console traceback message. Not every traceback is serious; some are cosmetic. But, every traceback should be decoded and analyzed. Because the traceback is in hexadecimal values, you will not be able to decode it. Therefore, you need to engage the Cisco Support team for decoding and analyzing it. The problematic function (routines) that causes the traceback might have severe effects, such as crashing the whole PIX and thereby requiring a reboot.
The method of traceback information collection depends on the version PIX is running. If your PIX is running a version earlier than 6.2, you need to connect the console to collect the traceback information for analysis. This is extremely inconvenient and poses security risks to the PIX, as you have to leave the console port connected for hours or even days to collect the traceback, as you do not know when the PIX will crash. Beginning with Version 6.2, the crash information is saved to Flash memoryby default. If saving the crash information to Flash is disabled manually, you can enable it with the following command:
PIX(config)# crashinfo save
Several often overlooked tools can help minimize the implementation and downtime of network availability. In this section, we will go through these tools:
Conduit to Access-list Converter Cisco's recommendation is to convert all conduits into access-lists. Access lists are more flexible and more efficient in terms of processing packets. Because conduits work globally on the PIX, if you have multiple interfaces, the packet coming through one of those interfaces has to go through the sequential search to find a match, whereas with an access-list this is more specific to the interface. In PIX Version 7.0, this command is deprecated completely, so you must convert your existing conduit into access-list before proceeding with the upgrade. You can download occ-121.gz for UNIX and occ-121.zip for Windows for a conduit to access-list conversation from the following location: http://www.cisco.com/cgi-bin/tablebuild.pl/pix. The output interpreter can be used as well for conversion.
Output Interpreter This is a great tool for finding common configuration errors very quickly. Here is the link for the Output interpreter:
Paste the write terminal or show running-config under the text box of Enter show command(s) output from your device for analysis.
Bugs Tracker Bugs Tracker allows you to look for a possible bug on a specific release. Search by using the string Bug Toolkit in the following link: http://www.cisco.com/kobayashi/support/tac/tools.shtml
Field Notices Field Notices contain information on whether you have severe hardware or software issues on any specific platform or version of the PIX firewall. The following link contains the field notices for the PIX firewall: http://www.cisco.com/en/US/customer/products/hw/vpndevc/ps2030/prod_field_notices_list.html
PSIRT Pages This security advisory contains Security Vulnerabilities and remedies for all Cisco products. The link for the PSIRT is: http://www.cisco.com/en/US/customer/products/products_security_advisories_listing.html