Using a Sniffer to Diagnose Firewall Problems
|
| Many problems can be isolated by running a packet sniffer on your firewall. Our favorite is tetheral, a part of the ethereal package, because it will put the packets into a more readable form than tcpdump, which is another good option. tetheral is also handy for command line diagnosis work because it works without all the fuss of a GUI and all the "voodoo" of a more lower-level sniffer such as tcpdump. tetheral is slower than tcpdump, however. Here is an example of a connection from behind our firewall to one of our hosts on the Internet:
This example demonstrates how the sniffer caught the entire session and put it into an easier to read format for someone not familiar with raw packets. The three-way handshake is illustrated in this sniffer trace, and tetheral was even kind enough to translate the SSH protocol for users so you can see that the SSH connection is working correctly, what protocol it is using, and even what step in the SSH process is occurring. The observant reader might have also noticed the ECN flag, which is the explicit congestion notification flag. That system uses it because we don't have to worry about connectivity problems for that host with systems that do not understand ECN, as in the following example:
This connection starts off innocently enough, with a standard SYN request but with the Explicit Congestion Notification (ECN) Echo flag set (ECE) and the Congestion Reduction Windows flag (CWR) set. There is some debate about what the right thing is for an IP stack that does not understand what ECN doesignore the flags and carry on or for the paranoid, drop the packet? Unfortunately for ECN, some vendors to chose to do the latter. In our example, you can see that the connection was immediately reset (RST) by the destination. Without a sniffer, it would have been difficult to see what was going on here. What happens when we turn ECN off? The connection goes through without a hitch.
The moral of the story is to look at the connections with a close eye and check for odd flags in the packets, such as ECN, if a connection is not working, again, before moving up in the OSI model. Always rule out lower-level problems before hypothesizing about the root cause of your problem. The example above was not meant to single out ECN as a problematic extension to IP as we use ECN for all of our servers. What we're showing is how a simple change to the session can cause what appears to be a higher level problem, the need to truly isolate the root cause of a problem, and the tools to do so. With ECN, however, we like to use it on our internal and Internet reachable machine but not as often for our firewalls because there are still too many sites out there that do not support ECN correctly. The good news is that you can run ECN to your heart's content behind your firewall, turn it off on your firewall, and filter out ECN packets on your outbound interfaces without causing any problems, while gaining the advantages of better congestion control. If you use squid as an HTTP proxy on your firewall, either transparently or not, the packets will also pack the ECN flag. Let's take a look at another example. We have a firewall with two outbound Internet interfaces connected to two different ISPs. We want SMTP traffic to go over one of the interfaces, while the rest of the traffic goes over the other interface. To accomplish this we use the ROUTE patch-o-matic netfilter/iptables patch to give us the ability to route arbitrarily by source, destination, source or destination port, MAC address, and so on. The following is what the pertinent rules look like: iptables -A POSTROUTING -t mangle -s 192.168.10.0/24 \ -p tcp --dport 25 -j ROUTE --gw 1.2.3.4 --oif eth2 \ --continue gw is our upstream gateway for the second ISP connection on this firewall. When we try to connect out to port 25 on one of our servers, the connection does not go through. When we use a sniffer to look at the connection, we can see that the packets are not being NAT-ed:
What we need to add to our rules is a specific NAT rule for this route: iptables -t nat -A POSTROUTING -o eth1 -p tcp \ -s 192.168.10.0/24 --dport 25 -j SNAT --to-source 1.2.3.5 This then tells the firewall to rewrite the packet so that its source address is 1.2.3.5, which is an address our remote server can locate on the Internet. Again, the intent is to move on to the sniffer after you have moved up the OSI model and have ruled out lower-level problems. |
Win=32440 Len=0 MSS=16220 TSV=31410938 TSER=0 WS=0 0.128939 216.218.240.134 -> 68.100.73.75 TCP ssh > 47104 [SYN, ACK, ECN] Seq=0 Ack=1
|
