Appendix B: Troubleshooting with the tcpdump Utility


The tcpdump utility can be used for low-level diagnostics of packet communications. Normally, you run tcpdump on the system that is either receiving or sending the packets and examine the output as it scrolls by.

The tcpdump utility supports command-line arguments that allow you to control which packets are displayed. tcpdump works outside of the ipchains/iptables and routing rules you have created. This makes it very useful for examining packets to see what is really coming into the system.

If, for example, you sit at host realserver1 and have the following Netfilter rule in place:

 #iptables -A INPUT -s client10 -j DROP 

and then you run tcpdump with the argument to display only packets from client10 with the following command:

 #tcpdump host client10 

you will still see the packets dump onto the screen even though client10's packets never make it up to the listening daemons on the system.

You can also control which interface tcpdump listens on by adding a -i <interfacename> argument. For example, to only see the packets hitting the eth1 interface, you would use:

 #tcpdump -i eth1 host client10 

Note 

To view IP addresses instead of host names, use the -n argument to tcpdump.

(If you leave off the -i argument, tcpdump will listen on all of its devices and add the device name to each line it prints out.)

Here is what one packet from tcpdump looks like:

 16:36:25.667437 eth0 < 10-1-1-1.my.test.domain.1391 > 10-1-1- 2.my.test.domain.telnet: S 2644956594:2644956594(0) win 16384 <mss 

Let's examine this line in more detail.

After the timestamp, the eth0 is indicating that this packet was seen on the eth0 device. The next character is either a < to indicate an inbound packet or > to indicate an outbound packet. The next two parameters are the source and destination hosts followed by the port number or port name (or to be more accurate, the service name listed in /etc/services). This packet was sent to the telnet port on eth0 from the 10.1.1.1 host. The 10.1.1.1 host is expecting a response back on port 1391 (an unprivileged port). Next, the letter S indicates the SYN flag was set in the packet. The complete list of IPv4 flags is: S (SYN), F (FIN), P (PUSH), or R (RST) or a single . to indicate no IPv4 flag is set.

You have to know more about how network communication protocols are supposed to work to troubleshoot much further (see the tcpdump man page for some examples near the end that might help). But for most of the work you do to set up cluster, simply knowing that a packet was received on the correct port is often all the information you need. You test for packets received (or sent) on a particular port with the command:

 #tcpdump -n port 80 

This command listens to all HTTP packets on any Network Interface Card connected to the system.

You can combine tcpdump packet criteria (with and) to see if a specific client's HTTP request is received:

 #tcpdump -i eth1 host client10 and port 80 

This command says to listen on interface eth1 for packets sent to or from client10 on port 80.

One of the most useful things tcpdump will tell you is that the network communication between the client and the host is being reset. Look for packet reports from tcpdump that contain an R (instead of an S or no flag, which is usually the case). Using tcpdump, you can determine which host (the client or the real server or the Director) is attempting to reset the connection. This usually means it "sees" a packet coming in on an established connection but the packet does not contain the sequencing or acknowledgement number expected. If you see this happening, check to make sure there is a service or daemon running on the server that is sending the reset. Usually, the reset occurs when this server is not running the daemon the client computer is trying to talk to. You can check to see which ports have daemons listening by running the following command on the server:

 #netstat -apn 

Also look for tcpdump reports that indicate packets are only going in one direction. In other words, all of the packets you see between your web server and the host connecting to it contain only a < or they all contain only a >. This probably means you have created Netfilter rules that DROP (iptables) or DENY (ipchains) the packets you are interested in.

tcpdump might even show you an ICMP message you get back somewhere on the communication path between the web server and the client. If you see the message Admin Prohibited Filter, this indicates a firewall between you and the client is blocking the packets. (You'll know which one, because the ICMP source address—the address on the left—is the firewall that is blocking you.)

If you see a port unreachable message such as the following:

     16:41:19.273329 eth0 > host1 > host2: icmp: host1 udp port tftp unreachable [tos 0xc0] 

this probably indicates that the specified port (tftp, in this case) is either blocked by ipchains/iptables rules or that there is no daemon running (tftpd, in this case) to answer requests on this port.

Another useful diagnostic tool is the free Ethereal tool available at http://www.ethereal.com.



The Linux Enterprise Cluster. Build a Highly Available Cluster with Commodity Hardware and Free Software
Linux Enterprise Cluster: Build a Highly Available Cluster with Commodity Hardware and Free Software
ISBN: 1593270364
EAN: 2147483647
Year: 2003
Pages: 219
Authors: Karl Kopper

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net