8.5 Examples of Debugging with Tcpdump

The following sections provide specific examples of debugging with tcpdump.

8.5.1 Packet Flooding

Using tcpdump to find the source of a traffic flood is usually straightforward. Start by connecting a machine in a place where it will be able to monitor network traffic. If it is necessary to configure a switch so that packets will be sent to an additional port for monitoring, make sure to do so. Then run tcpdump and look for high talkers. Be sure to disable domain name lookups:

 Linux# tcpdump -n  17:36:16.265220 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.269171 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.273130 10.255.255.23.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.285228 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.302173 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.319372 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.334600 10.7.15.65.7000 > 10.18.1.140.7001: rx ack (66) (DF)  17:36:16.334975 10.7.15.65.7000 > 10.18.1.140.7001: rx data (36) (DF)  17:36:16.336606 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.336623 10.7.1.70.7000 > 10.18.1.140.7001: rx ack (66) (DF)  17:36:16.336939 10.7.1.70.7000 > 10.18.1.140.7001: rx data (36) (DF)  17:36:16.352253 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.356199 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.396921 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.398427 10.155.0.153.57195 > 239.255.255.253.427: udp 49  17:36:16.400831 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.404805 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.408749 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.412705 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF)  17:36:16.416750 10.255.255.27.1221 > 10.18.0.100.9995: udp 1168 (DF) 

From this short sample of output, we can see that there is a suspiciously large amount of traffic coming from 10.255.255.27 port 1221, directed at 10.18.0.100. Each packet is a UDP datagram with 1168 bytes of UDP payload data. The small difference in time stamps between packets helps convince us of the speed with which they are being sent.

Frequently, there is so much traffic on the network that it will not be so easy to determine who the high talker is. If that is the case, you may wish to rule out certain network traffic or include only certain network traffic in an attempt to focus in on the problem. For example, if you happen to know the flooding is directed at a particular host, use a filter to view traffic destined for that host:

 Linux# tcpdump -n dst victim.example.com 

8.5.2 A More Complicated Example

Imagine several workstations are having trouble accessing your Web server. The Web browser just hangs . Oddly, other machines on the same network have no problem reaching the server in a timely manner, and tests from your own workstation indicate there is no problem in connectivity. So you use a machine running tcpdump to examine the problem. First you may choose to look at the Web traffic originating from a workstation exhibiting the problem. You start tcpdump, instructing it to monitor port 80 (the port used for HTTP transactions) and then try to open the page in a Web browser:

 Linux# tcpdump host client.example.com and port 80 

But you see no traffic. Immediately, you can rule out the Web sever as the problem. If no traffic is sent to the Web server in the first place, the problem is probably not the fault of the server. So you decide to look at more traffic than just Web traffic by removing the port 80 restriction:

 Linux# tcpdump host client.example.com    18:06:11.162372 client.example.com.45600 > dns.example.com.doma... 

What is of interest is what you did not see. Although client.example.com makes a DNS request to dns.example.com, there is no response. If you look closer:

 Linux# tcpdump -xls 1500 host client  ./tcpdump-data-filter.pl    18:14:12.842409 brokenclient.example.com.55313 > dns.example.co...         4500 0048 058b 4000 ff11 9d80 0a12 0064      E..H..@........d         0a05 061e d811 0035 0034 8a44 e4ca 0010      .......5.4.D....         0001 0000 0000 0001 0377 7777 0765 7861      .........www.exa         6d70 6c65 0363 6f6d 0000 0f00 0100 0029      mple.com.......)         0800 0000 8000 0000                          ........ 

In the body of the request, you can see the DNS lookup is for www.example.com . Now the problem is clear: The clients are attempting to look up the IP address of www.example.com before connecting to the Web server, but the DNS server is not responding. This explains why some workstations can connect without difficulty; they already have the IP address of the Web server cached from an earlier transaction.