To troubleshoot NAT, you should first verify that each necessary step has been performed.
Wherever a verification of the configuration fails, a packet sniffer can be your friend. The remainder of this section shows you what you should see in a packet sniffer, what you shouldn't, and how to fix it. Although there are plenty of external packet-sniffing devices, they can be expensive and inconvenient to use. Fortunately, some operating systems come with their own packet sniffers. Solaris comes with a tool called snoop . IPSO, Linux, and AIX come with tcpdump . Both of these tools will be discussed briefly in this chapter. Windows NT/2000 machines come with a limited packet sniffer in Network Monitor, but you can obtain a free copy of Ethereal (from http://www.ethereal.com), which works far better and reads in data files from snoop and tcpdump . Since version 4.0, FireWall-1 has also come with its own packet-sniffing utility called fw monitor . Because it works at the same level as FireWall-1 (i.e., just after the MAC layer and before the network layer), its use in troubleshooting NAT issues is limited. fw monitor relies on INSPECT code and is discussed in Chapter 14. Consider the network and configuration that were used in the earlier step-by-step example (see Figure 10.3). Let's assume that host 192.168.42.69 is attempting to connect to 192.168.0.13, the intranet Web server, which really resides at 10.0.10.80. With a successful connection, a tcpdump of the external interface should show you the following output ( -i specifies an interface to listen to). # tcpdump -i eth-s1p1 tcpdump: listening on eth-s1p1 18:51:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2 18:51:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55 18:51:54.135062 192.168.42.69.1777 > 192.168.0.13.80: S 1184222758:1184222758(0) win 16384 <mss 1460,nop,wscale 0, nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:51:54.135062 192.168.0.13.80 > 192.168.42.69.1777: S 1332216451:1332216451(0) ack 1184222759 win 32120 <mss 1460, nop,nop,timestamp 2739310[tcp]> (DF) 18:51:54.415021 192.168.42.69.1777 > 192.168.0.13.80: . ack 1 win 17376 <nop,nop,timestamp 11362405 2739310> (DF) [tos 0x10] If you were to use snoop , you would see the following output ( -d on snoop specifies an interface to listen to). # snoop -d hme0 Using device /dev/hme (promiscuous mode) 192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ? 192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 Note that you may not necessarily see the ARP packets, especially if the originator of the packet already has the MAC address in its ARP cache. If you see SYN, SYN/ACK, and ACK packets, the connection should be established. ARPsThe first part of the communication you should see is the request for MAC addresses via an ARP packet. When everything is working correctly, you will see an exchange like the following on the external interface with tcpdump: 18:13:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2 18:13:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55 With snoop , it looks like this: 192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ? 192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55 If you do see only the first packet over and over again (e.g., the ARP "who is"), this means that nobody owns or is proving a proxy ARP for the translated address. Add a proxy ARP as described previously. In some cases (especially when Windows is the firewall), you may need to add a static host route on the external router. SYN Packets with No ResponseYou should then see the SYN packet, which looks something like this with tcpdump: 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] With snoop , it looks like this: 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 If this packet repeats over and over again, one of four things may be wrong.
Verify that the MAC address the packet is being sent to is correct (in this case, it should be 0:11:22:33:44:55 ). In tcpdump , you do this with the e flag, which adds the MAC address to the output. In snoop , the only way to do this is with the v flag, which unfortunately is extremely verbose. Also, you can show only packets going to or coming from host 192.168.0.13 by adding host 192.168.0.13 to the end of your tcpdump or snoop command line. # tcpdump -e -i eth-s1p1 host 192.168.0.13 tcpdump: listening on eth-s1p1 18:21:49.201680 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:21:54.240965 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:22:07.209125 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82: 192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] # snoop v d hme0 host 192.168.0.13 Using device /dev/hme (promiscuous mode) ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 27 arrived at 16:47:50.83 ETHER: Packet size = 58 bytes ETHER: Destination = 0:55:44:33:22:11, ETHER: Source = 0:aa:bb:cc:dd:ee, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: Total length = 44 bytes IP: Identification = 47535 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 245 seconds/hops IP: Protocol = 6 (TCP) IP: Header checksum = f23f IP: Source address = 192.168.42.69 IP: Destination address = 192.168.0.13 IP: No options IP: TCP: ----- TCP Header ----- TCP: TCP: Source port = 2000 TCP: Destination port = 80 (HTTP) TCP: Sequence number = 90360382 TCP: Acknowledgement number = 0 TCP: Data offset = 24 bytes TCP: Flags = 0x02 TCP: ..0. .... = No urgent pointer TCP: ...0 .... = No acknowledgement TCP: .... 0... = No push TCP: .... .0.. = No reset TCP: .... ..1. = Syn TCP: .... ...0 = No Fin TCP: Window = 8760 TCP: Checksum = 0xda2b TCP: Urgent pointer = 0 TCP: Options: (4 bytes) TCP: - Maximum segment size = 1460 bytes TCP: In the preceding case, MAC 0:aa:bb:cc:dd:ee (which is the MAC of the external router) is trying to send to MAC 0:55:44:33:22:11, which is not the correct MAC. You can usually resolve this problem by flushing the ARP cache on the external router and retrying the test. If the packet is not being routed properly, you could see a reset (RST) packet (see the next subsection), or you could see an ICMP Destination Unreachable packet. Verify that the static host route for 192.168.0.13 is pointing to the next hop address (10.0.0.2, as shown in Figure 10.3). If the packet is not actually being translated, you will see it very clearly in a tcpdump or snoop on the internal interface, as demonstrated below. # tcpdump -i eth-s1p2 host 192.168.42.69 tcpdump: listening on eth-s1p2 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:18:25.040168 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:40:30.040342 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] # snoop -d hme0 Using device /dev/hme (promiscuous mode) 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 Normally, you should see the translated IP address on the internal interface. If you do not see translated packets, check your NAT rules. SYN Followed by RSTSometimes the packet that follows the SYN is an RST packet, as shown in the output below. (With snoop , you need V to see the TCP flags.) # tcpdump -i eth-s1p1 host 192.168.0.13 tcpdump: listening on le0 18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 3911298019:3911298019(0) win 16384 <mss1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:13:22.040132 192.168.0.13.80 > 192.168.42.69.1777: R 0:0(0) ack 3911298020 win 0 [tos 0x10] # snoop V -d hme0 Using device /dev/hme (promiscuous mode) ________________________________ 192.168.42.69 -> 192.168.0.13 ETHER Type=0800 (IP), size = 58 bytes 192.168.42.69 -> 192.168.0.13 IP D=192.168.0.13 S=192.168.42.69 LEN=44, ID=47247 192.168.42.69 -> 192.168.0.13 TCP D=80 S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460> 192.168.42.69 -> 192.168.0.13 HTTP C port=1777 ________________________________ 192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes 192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=192.168.0.13 LEN=40, ID=61295 192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 Win=0 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 If this happens, one of two things is wrong.
NOTE!
If the remote server isn't actually running the service, you will see the following in a tcpdump and snoop , respectively, on the internal interface: # tcpdump -i eth-s1p1 host 10.0.10.80 tcpdump: listening on le0 18:13:22.040132 192.168.42.69.1777 > 10.0.10.80.80: S 3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[tcp]> (DF) [tos 0x10] 18:13:22.040132 10.0.10.80.80 > 192.168.42.69.1777: R 0:0(0) ack 3911298020 win 0 [tos 0x10] # snoop V -d hme0 Using device /dev/hme (promiscuous mode) ________________________________ 192.168.42.69 -> 10.0.10.80 ETHER Type=0800 (IP), size = 58 bytes 192.168.42.69 -> 10.0.10.80 IP D=10.0.10.80 S=192.168.42.69 LEN=44, ID=47247 192.168.42.69 -> 10.0.10.80 TCP D=80 S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460> 192.168.42.69 -> 10.0.10.80 HTTP C port=1777 ________________________________ 192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes 192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=10.0.10.80 LEN=40, ID=61295 192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 Win=0 192.168.0.13 -> 192.168.42.69 HTTP R port=1777 The internal interface should see the untranslated packets (10.0.10.80 is the system's real IP). If the packet is being routed to the wrong interface, you will also see the same behavior as in the preceding output. Verify that the static host route is set up correctly. NOTE!
Useful tcpdump FlagsTable 10.4 contains a list of some useful flags for tcpdump , which takes commands in the following format: tcpdump i interface-name [other-flags] [expression] Table 10.4. tcpdump flags
tcpdump ExpressionsAll tcpdump commands can be followed by an expression that filters the displayed (or saved) packets to show only the packets that are interesting. Some useful expressions are shown in Table 10.5. Table 10.5. tcpdump expressions
Useful snoop FlagsTable 10.6 contains a list of some useful flags for snoop , which takes commands in the following format: snoop [flags] [expression] Table 10.6. snoop flags
snoop ExpressionsAll snoop commands can be followed by an expression that filters the displayed (or saved) packets to show only the packets that are interesting. Table 10.7 presents some useful expressions. Note that many of the expressions are similar to those for tcpdump . Table 10.7. snoop expressions
|