Troubleshooting NAT with a Packet Sniffer


To troubleshoot NAT, you should first verify that each necessary step has been performed.

  • Validate that an ARP entry exists for the translated IP (or that the translated IP is somehow being routed to the firewall).

  • Validate that a static host route exists on the firewall to route the translated IP address to either the untranslated address or the next hop address if the real system is more than one hop away from the firewall.

  • Validate that the rules are set up correctly. Set any security policy rule that applies to a NATted host to track long, and ensure that address translation is happening as you expect.

Wherever a verification of the configuration fails, a packet sniffer can be your friend. The remainder of this section shows you what you should see in a packet sniffer, what you shouldn't, and how to fix it.

Although there are plenty of external packet-sniffing devices, they can be expensive and inconvenient to use. Fortunately, some operating systems come with their own packet sniffers. Solaris comes with a tool called snoop . IPSO, Linux, and AIX come with tcpdump . Both of these tools will be discussed briefly in this chapter. Windows NT/2000 machines come with a limited packet sniffer in Network Monitor, but you can obtain a free copy of Ethereal (from http://www.ethereal.com), which works far better and reads in data files from snoop and tcpdump .

Since version 4.0, FireWall-1 has also come with its own packet-sniffing utility called fw monitor . Because it works at the same level as FireWall-1 (i.e., just after the MAC layer and before the network layer), its use in troubleshooting NAT issues is limited. fw monitor relies on INSPECT code and is discussed in Chapter 14.

Consider the network and configuration that were used in the earlier step-by-step example (see Figure 10.3). Let's assume that host 192.168.42.69 is attempting to connect to 192.168.0.13, the intranet Web server, which really resides at 10.0.10.80.

With a successful connection, a tcpdump of the external interface should show you the following output ( -i specifies an interface to listen to).

 #  tcpdump -i eth-s1p1  tcpdump: listening on eth-s1p1 18:51:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2 18:51:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55 18:51:54.135062 192.168.42.69.1777 > 192.168.0.13.80: S 1184222758:1184222758(0) win 16384 <mss 1460,nop,wscale 0, nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:51:54.135062 192.168.0.13.80 > 192.168.42.69.1777: S 1332216451:1332216451(0) ack 1184222759 win 32120 <mss 1460, nop,nop,timestamp 2739310[tcp]> (DF) 18:51:54.415021 192.168.42.69.1777 > 192.168.0.13.80: . ack 1 win 17376 <nop,nop,timestamp 11362405 2739310> (DF) [tos 0x10] 

If you were to use snoop , you would see the following output ( -d on snoop specifies an interface to listen to).

 #  snoop -d hme0  Using device /dev/hme (promiscuous mode) 192.168.0.2 -> (broadcast)  ARP C Who is 192.168.0.13 ? 192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 

Note that you may not necessarily see the ARP packets, especially if the originator of the packet already has the MAC address in its ARP cache. If you see SYN, SYN/ACK, and ACK packets, the connection should be established.

ARPs

The first part of the communication you should see is the request for MAC addresses via an ARP packet. When everything is working correctly, you will see an exchange like the following on the external interface with tcpdump:

 18:13:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2 18:13:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55 

With snoop , it looks like this:

 192.168.0.2 -> (broadcast)  ARP C Who is 192.168.0.13 ? 192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55 

If you do see only the first packet over and over again (e.g., the ARP "who is"), this means that nobody owns or is proving a proxy ARP for the translated address. Add a proxy ARP as described previously. In some cases (especially when Windows is the firewall), you may need to add a static host route on the external router.

SYN Packets with No Response

You should then see the SYN packet, which looks something like this with tcpdump:

 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 

With snoop , it looks like this:

 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 

If this packet repeats over and over again, one of four things may be wrong.

  • The security policy is dropping the packet. Check your logs for drops .

  • The packet is being sent to the wrong MAC address.

  • The packet is not being routed properly.

  • The packet isn't actually getting translated, thus it is getting ignored.

Verify that the MAC address the packet is being sent to is correct (in this case, it should be 0:11:22:33:44:55 ). In tcpdump , you do this with the e flag, which adds the MAC address to the output. In snoop , the only way to do this is with the v flag, which unfortunately is extremely verbose.

Also, you can show only packets going to or coming from host 192.168.0.13 by adding host 192.168.0.13 to the end of your tcpdump or snoop command line.

 #  tcpdump -e -i eth-s1p1 host 192.168.0.13  tcpdump: listening on eth-s1p1 18:21:49.201680 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:21:54.240965 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:22:07.209125 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] #  snoop v d hme0 host 192.168.0.13  Using device /dev/hme (promiscuous mode) ETHER:  ----- Ether Header ----- ETHER: ETHER:  Packet 27 arrived at 16:47:50.83 ETHER:  Packet size = 58 bytes ETHER:  Destination = 0:55:44:33:22:11, ETHER:  Source      = 0:aa:bb:cc:dd:ee, ETHER:  Ethertype = 0800 (IP) ETHER: IP:   ----- IP Header ----- IP: IP:   Version = 4 IP:   Header length = 20 bytes IP:   Type of service = 0x00 IP:         xxx. .... = 0 (precedence) IP:         ...0 .... = normal delay IP:         .... 0... = normal throughput IP:         .... .0.. = normal reliability IP:   Total length = 44 bytes IP:   Identification = 47535 IP:   Flags = 0x4 IP:         .1.. .... = do not fragment IP:         ..0. .... = last fragment IP:   Fragment offset = 0 bytes IP:   Time to live = 245 seconds/hops IP:   Protocol = 6 (TCP) IP:   Header checksum = f23f IP:   Source address = 192.168.42.69 IP:   Destination address = 192.168.0.13 IP:   No options IP: TCP:  ----- TCP Header ----- TCP: TCP:  Source port = 2000 TCP:  Destination port = 80 (HTTP) TCP:  Sequence number = 90360382 TCP:  Acknowledgement number = 0 TCP:  Data offset = 24 bytes TCP:  Flags = 0x02 TCP:        ..0. .... = No urgent pointer TCP:        ...0 .... = No acknowledgement TCP:        .... 0... = No push TCP:        .... .0.. = No reset TCP:        .... ..1. = Syn TCP:        .... ...0 = No Fin TCP:  Window = 8760 TCP:  Checksum = 0xda2b TCP:  Urgent pointer = 0 TCP:  Options: (4 bytes) TCP:    - Maximum segment size = 1460 bytes TCP: 

In the preceding case, MAC 0:aa:bb:cc:dd:ee (which is the MAC of the external router) is trying to send to MAC 0:55:44:33:22:11, which is not the correct MAC. You can usually resolve this problem by flushing the ARP cache on the external router and retrying the test.

If the packet is not being routed properly, you could see a reset (RST) packet (see the next subsection), or you could see an ICMP Destination Unreachable packet. Verify that the static host route for 192.168.0.13 is pointing to the next hop address (10.0.0.2, as shown in Figure 10.3).

If the packet is not actually being translated, you will see it very clearly in a tcpdump or snoop on the internal interface, as demonstrated below.

 #  tcpdump -i eth-s1p2 host 192.168.42.69  tcpdump: listening on eth-s1p2 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:18:25.040168 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:40:30.040342 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] #  snoop -d hme0  Using device /dev/hme (promiscuous mode) 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 

Normally, you should see the translated IP address on the internal interface. If you do not see translated packets, check your NAT rules.

SYN Followed by RST

Sometimes the packet that follows the SYN is an RST packet, as shown in the output below. (With snoop , you need V to see the TCP flags.)

 #  tcpdump -i eth-s1p1 host 192.168.0.13  tcpdump: listening on le0 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:13:22.040132 192.168.0.13.80 > 192.168.42.69.1777: R  0:0(0) ack 3911298020 win 0 [tos 0x10] #  snoop V -d hme0  Using device /dev/hme (promiscuous mode) ________________________________ 192.168.42.69 -> 192.168.0.13 ETHER Type=0800 (IP), size = 58 bytes 192.168.42.69 -> 192.168.0.13 IP  D=192.168.0.13 S=192.168.42.69 LEN=44, ID=47247 192.168.42.69 -> 192.168.0.13 TCP D=80 S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460> 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 ________________________________ 192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes 192.168.0.13 -> 192.168.42.69 IP  D=192.168.42.69 S=192.168.0.13 LEN=40, ID=61295 192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 Win=0 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 

If this happens, one of two things is wrong.

  • The remote server is not running the service specified (in this case, port 80, HTTP).

  • The packet is being routed incorrectly.

NOTE!

graphics/brain_icon.gif

In FireWall-1 4.1 and earlier, a SYN followed by a RST would also occur due to incorrectly configured anti-spoofing (i.e., 192.168.0.13 not being on the internal interface's Valid Address setting). In NG, anti-spoofing is not done on packets after they've been routed, thus eliminating this as a possible cause.


If the remote server isn't actually running the service, you will see the following in a tcpdump and snoop , respectively, on the internal interface:

 #  tcpdump -i eth-s1p1 host 10.0.10.80  tcpdump: listening on le0 18:13:22.040132 192.168.42.69.1777 > 10.0.10.80.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:13:22.040132 10.0.10.80.80 > 192.168.42.69.1777: R 0:0(0) ack 3911298020 win 0 [tos 0x10] #  snoop V -d hme0  Using device /dev/hme (promiscuous mode) ________________________________ 192.168.42.69 -> 10.0.10.80 ETHER Type=0800 (IP), size = 58 bytes 192.168.42.69 -> 10.0.10.80 IP  D=10.0.10.80 S=192.168.42.69 LEN=44, ID=47247 192.168.42.69 -> 10.0.10.80 TCP D=80 S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460> 192.168.42.69 -> 10.0.10.80 HTTP C port=1777 ________________________________ 192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes 192.168.0.13 -> 192.168.42.69 IP  D=192.168.42.69 S=10.0.10.80 LEN=40, ID=61295 192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 Win=0 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 

The internal interface should see the untranslated packets (10.0.10.80 is the system's real IP). If the packet is being routed to the wrong interface, you will also see the same behavior as in the preceding output. Verify that the static host route is set up correctly.

NOTE!

graphics/brain_icon.gif

In FireWall-1 4.1, you would also likely see a reject on Rule 0 in the logs, provided your anti-spoofing was configured correctly.


Useful tcpdump Flags

Table 10.4 contains a list of some useful flags for tcpdump , which takes commands in the following format:

 tcpdump i interface-name [other-flags] [expression] 
Table 10.4. tcpdump flags

Flag

Description

-e

Displays MAC addresses with each packet.

-i interface

Specifies an interface to listen on (required).

-l

Indicates to not buffer stdout , which is useful for piping tcpdump output to other programs.

-n

Disables name resolution on the packets shown.

-p

Indicates to not put the interface in promiscuous mode (i.e., show only the packets destined for the host).

-r filename

Reads tcpdump capture from the specified file.

-s N

Captures N number of bytes for each packet (useful with x and X ). The default is 68.

-S

Prints absolute TCP sequence numbers instead of relative ones.

-w filename

Writes captured packets to the specified file.

-x

Performs a hex dump of received packets.

-X

Performs a hex and ASCII dump of received packets (IPSO only).

tcpdump Expressions

All tcpdump commands can be followed by an expression that filters the displayed (or saved) packets to show only the packets that are interesting. Some useful expressions are shown in Table 10.5.

Table 10.5. tcpdump expressions

Expression

Description

port 80

Shows all packets with source or destination port 80 (TCP or UDP).

host 192.168.0.3

Shows all packets coming from or going to host 192.168.0.3.

host 192.168.0.3 and tcp port 80

Shows all packets coming from or going to host 192.168.0.3 that are TCP packets with a source or destination port of 80.

proto vrrp

Shows all VRRP packets. On non-IPSO platforms, use ip proto 112 .

icmp

Shows all ICMP packets.

\(src host 192.168.0.1 or src host 192.168.0.2\) and proto vrrp

Shows all VRRP packets that originate from 192.168.0.1 and 192.168.0.2 (the \ before the parenthesis is to escape it for the shell).

ether host aa:bb:cc:dd:ee:ff

Shows all packets that come from or go to the specified MAC address.

ip proto 50

Shows all packets of IP Proto 50, which by definition are IPSec AH packets.

ip[2:2] > 576

Shows IP packets that are longer than 576 bytes. [2:2] refers to the specific byte location in the TCP header and its length.

tcp[13] & 0x12 != 0

Shows only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the TCP header of the packet.

icmp[0]

Shows ICMP type 0 packets (i.e., echo reply). icmp[0] refers to the 0th byte in the ICMP header.

icmp[0] = 3 and icmp[1] = 4

Shows ICMP type 3, code 4 packets. These happen to be a response to receiving a packet that is too large to process and has the Don't Fragment bit set.

Useful snoop Flags

Table 10.6 contains a list of some useful flags for snoop , which takes commands in the following format:

 snoop [flags] [expression] 
Table 10.6. snoop flags

Flag

Description

-d interface

Specifies an interface to listen on.

-v

Enables verbose mode. All packet data is displayed for each packet. When using this option, it is recommended that you pipe the output of the snoop command to a file or you run the risk of not being able to easily terminate your snoop command.

-V

Enables a less-verbose verbose mode. A summary line is displayed for each layer in the ISO model.

-n

Disables name resolution on the packets shown.

-P

Indicates to not put the interface in promiscuous mode, which means to show only the packets actually destined for this host (not useful on a switched segment).

-i filename

Reads packets previously captured from the file filename .

-o filename

Writes packets to capture the file named filename .

-s numbytes

Captures numbytes bytes for each packet. Normally, all bytes in the packet are captured.

-p x,y

Shows packets numbered between x and y . The first packet captured is 1.

-t [adr]

Sets the timestamp format: a (absolute, i.e., wall clock time), d (delta, since capture was started), and r (relative time).

snoop Expressions

All snoop commands can be followed by an expression that filters the displayed (or saved) packets to show only the packets that are interesting. Table 10.7 presents some useful expressions. Note that many of the expressions are similar to those for tcpdump .

Table 10.7. snoop expressions

Expression

Description

port 80

Shows all packets with source or destination port 80 (TCP or UDP).

host 192.168.0.3

Shows all packets coming from or going to host 192.168.0.3. You can also omit the host qualifier.

host 192.168.0.3 and tcp port 80

Shows all packets coming from or going to host 192.168.0.3 that are TCP packets with a source or destination port of 80.

icmp

Shows all ICMP packets.

from 192.168.0.1 or to 192.168.0.2

Shows all packets that originate from 192.168.0.1 or are destined for 192.168.0.2.

ether aa:bb:cc:dd:ee:ff

Shows all packets that come from or go to the specified MAC address.

ip proto 50

Shows all packets of IP Proto 50, which by definition are IPSec AH packets.

greater 576

Shows packets longer than 576 bytes. You can use the word less instead of greater to show packets smaller than 576 bytes.

tcp[13] & 0x12 != 0

Shows only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the TCP header of the packet.



Essential Check Point FireWall-1 NG
Essential Check Point FireWall-1 NG: An Installation, Configuration, and Troubleshooting Guide
ISBN: 0321180615
EAN: 2147483647
Year: 2004
Pages: 143

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net