Recipe 9.16 Observing Network Traffic

9.16.1 Problem

You want to watch network traffic flowing by (or through) your machine.

9.16.2 Solution

Use a packet sniffer such as tcpdump.^[7]

^[7] In spite of its name, tcpdump is not restricted to TCP. It can capture entire packets, including the link-level (Ethernet) headers, IP, UDP, etc.

To sniff packets and save them in a file:

# tcpdump -w filename [-c count] [-i interface] [-s snap-length] [expression]

To read and display the saved network trace data:

$ tcpdump -r filename [expression]

To select packets related to particular TCP services to or from a host:

# tcpdump tcp port service [or service] and host server.example.com

For a convenient and powerful GUI, use Ethereal. [Recipe 9.17]

To enable an unconfigured interface, for a "stealth" packet sniffer:

# ifconfig interface-name 0.0.0.0 up

To print information about all of your network interfaces with loaded drivers: [Recipe 3.1]

$ ifconfig -a

9.16.3 Discussion

Is your system under attack? Your firewall is logging unusual activities, you see lots of half-open connections, and the performance of your web server is degrading. How can you learn what is happening so you can take defensive action? Use a packet sniffer to watch traffic on the network!

In normal operation, network interfaces are programmed to receive only the following:

• Unicast packets, addressed to a specific machine

• Multicast packets, targeted to systems that choose to subscribe to services like streaming video or sound

• Broadcast packets, for when an appropriate destination is not known, or for important information that is probably of interest to all machines on the network

The term "unicast" is not an oxymoron: all packets on networks like Ethernet are in fact sent (conceptually) to all systems on the network. Each system simply ignores unicast packets addressed to other machines, or uninteresting multicast packets.

A packet sniffer puts a network interface into promiscuous mode, causing it to receive all packets on the network, like a wiretap. Almost all network adapters support this mode nowadays. Linux restricts the use of promiscuous mode to the superuser, so always run packet-sniffing programs as root. Whenever you switch an interface to promiscuous mode, the kernel logs the change, so we advise running the logger command [Recipe 9.27] to announce your packet-sniffing activities.

If promiscuous mode doesn't seem to be working, and your kernel is sending complaints to the system logger (usually in /var/log/messages) that say:

modprobe: can't locate module net-pf-17

then your kernel was built without support for the packet socket protocol, which is required for network sniffers.

Rebuild your kernel with the option CONFIG_PACKET=y (or CONFIG_PACKET=m to build a kernel module). Red Hat and SuSE distribute kernels with support for the packet socket protocol enabled, so network sniffers should work.

Network switches complicate this picture. Unlike less intelligent hubs, switches watch network traffic, attempt to learn which systems are connected to each network segment, and then send unicast packets only to ports known to be connected to the destination systems, which defeats packet sniffing. However, many network switches support packet sniffing with a configuration option to send all traffic to designated ports. If you are running a network sniffer on a switched network, consult the documentation for your switch.

The primary purpose of network switches is to improve performance, not to enhance security. Packet sniffing is more difficult on a switched network, but not impossible: dsniff [Recipe 9.19] is distributed with a collection of tools to demonstrate such attacks. Do not be complacent about the need for secure protocols, just because your systems are connected to switches instead of hubs.

Similarly, routers and gateways pass traffic to different networks based on the destination address for each packet. If you want to watch traffic between machines on different networks, attach your packet sniffer somewhere along the route between the source and destination.

Packet sniffers tap into the network stack at a low level, and are therefore immune to restrictions imposed by firewalls. To verify the correct operation of your firewall, use a packet sniffer to watch the firewall accept or reject traffic.

Your network interface need not even be configured in order to watch traffic (it does need to be up, however). Use the ifconfig command to enable an unconfigured interface by setting the IP address to zero:

# ifconfig eth2 0.0.0.0 up

Unconfigured interfaces are useful for dedicated packet-sniffing machines, because they are hard to detect or attack. Such systems are often used on untrusted networks exposed to the outside (e.g., right next to your web servers). Use care when these "stealth" packet sniffers are also connected (by normally configured network interfaces) to trusted, internal networks: for example, disable IP forwarding. [Recipe 2.3]

Promiscuous mode can degrade network performance. Avoid running a packet sniffer for long periods on important, production machines: use a separate, dedicated machine instead.

Almost all Linux packet-sniffing programs use libpcap , a packet capture library distributed with tcpdump. As a fortunate consequence, network trace files share a common format, so you can use one tool to capture and save packets, and others to display and analyze the traffic. The file command recognizes and displays information about libpcap-format network trace files:

$ file trace.pcap trace.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture  length 96)

Kernels of Version 2.2 or higher can send warnings to the system logger like:

tcpdump uses obsolete (PF_INET,SOCK_PACKET)

These are harmless, and can be safely ignored. To avoid the warnings, upgrade to a more recent version of libpcap.

To sniff packets and save them in a file, use the tcpdump -w option:

# tcpdump -w trace.pcap [-c count] [-i interface] [-s snap-length] [expression]

Just kill tcpdump when you are done, or use the -c option to request a maximum number of packets to record.

If your system is connected to multiple networks, use the -i option to listen on a specific interface (e.g., eth2). The ifconfig command prints information about all of your network interfaces with loaded drivers: [Recipe 3.1]

$ ifconfig -a

The special interface name "any" denotes all of the interfaces by any program that uses libpcap, but these interfaces are not put into promiscuous mode automatically. Before using tcpdump -i any , use ifconfig to enable promiscuous mode for specific interfaces of interest:

# ifconfig interface promisc

Remember to disable promiscuous mode when you are done sniffing:

# ifconfig interface -promisc

Support for the "any" interface is available in kernel Versions 2.2 or later.

Normally, tcpdump saves only the first 68 bytes of each packet. This snapshot length is good for analysis of low-level protocols (e.g., TCP or UDP), but for higher-level ones (like HTTP) use the -s option to request a larger snapshot. To capture entire packets and track all transmitted data, specify a snapshot length of zero. Larger snapshots consume dramatically more disk space, and can impact network performance or even cause packet loss under heavy load.

By default, tcpdump records all packets seen on the network. Use a capture filter expression to select specific packets: the criteria can be based on any data in the protocol headers, using a simple syntax described in the tcpdump(8) manpage. For example, to record FTP transfers to or from a server:

# tcpdump -w trace.pcap tcp port ftp or ftp-data and host server.example.com

By restricting the kinds of packets you capture, you can reduce the performance implications and storage requirements of larger snapshots.

To read and display the saved network trace data, use the tcpdump -r option:

$ tcpdump -r trace.pcap [expression]

Root access is not required to analyze the collected data, since it is stored in ordinary files. You may want to protect those trace files, however, if they contain sensitive data.

Use a display filter expression to print information only about selected packets; display filters use the same syntax as capture filters.

The capture and display operations can be combined, without saving data to a file, if neither the -w nor -r options are used, but we recommend saving to a file, because:

• Protocol analysis often requires displaying the data multiple times, in different formats, and perhaps using different tools.

• You might want to analyze data captured at some earlier time.

• It is hard to predict selection criteria in advance. Use more inclusive filter expressions at capture time, then more discriminating ones at display time, when you understand more clearly which data is interesting.

• Display operations can be inefficient. Memory is consumed to track TCP sequence numbers, for example. Your packet sniffer should be lean and mean if you plan to run it for long periods.

• Display operations sometimes interfere with capture operations. Converting IP addresses to hostnames often involves DNS lookups, which can be confusing if you are watching traffic to and from your nameservers! Similarly, if you tunnel tcpdump output through an SSH connection, that generates additional SSH traffic.

Saving formatted output from tcpdump is an even worse idea. It consumes large amounts of space, is difficult for other programs to parse, and discards much of the information saved in the libpcap-format trace file. Use tcpdump -w to save network traces.

tcpdump prints information about packets in a terse, protocol-dependent format meticulously described in the manpage. Suppose a machine 10.6.6.6 is performing a port scan of another machine, 10.9.9.9, by running nmap -r. [Recipe 9.13] If you use tcpdump to observe this port scan activity, you'll see something like this:

# tcpdump -nn ... 23:08:14.980358 10.6.6.6.6180 > 10.9.9.9.20: S 5498218:5498218(0) win 4096 [tos 0x80] 23:08:14.980436 10.9.9.9.20 > 10.6.6.6.6180: R 0:0(0) ack 5498219 win 0 (DF) [tos 0x80] 23:08:14.980795 10.6.6.6.6180 > 10.9.9.9.21: S 5498218:5498218(0) win 4096 [tos 0x80] 23:08:14.980893 10.9.9.9.21 > 10.6.6.6.6180: R 0:0(0) ack 5498219 win 0 (DF) [tos 0x80] 23:08:14.983496 10.6.6.6.6180 > 10.9.9.9.22: S 5498218:5498218(0) win 4096 23:08:14.984488 10.9.9.9.22 > 10.6.6.6.6180: S 3458349:3458349(0) ack 5498219 win 5840  <mss 1460> (DF) 23:08:14.983907 10.6.6.6.6180 > 10.9.9.9.23: S 5498218:5498218(0) win 4096 [tos 0x80] 23:08:14.984577 10.9.9.9.23 > 10.6.6.6.6180: R 0:0(0) ack 5498219 win 0 (DF) [tos 0x80] 23:08:15.060218 10.6.6.6.6180 > 10.9.9.99.22: R 5498219:5498219(0) win 0 (DF) 23:08:15.067712 10.6.6.6.6180 > 10.9.9.99.24: S 5498218:5498218(0) win 4096 23:08:15.067797 10.9.9.9.24 > 10.6.6.6.6180: R 0:0(0) ack 5498219 win 0 (DF) 23:08:15.068201 10.6.6.6.6180 > 10.9.9.9.25: S 5498218:5498218(0) win 4096 [tos 0x80] 23:08:15.068282 10.9.9.9.25 > 10.6.6.6.6180: R 0:0(0) ack 5498219 win 0 (DF) [tos 0x80] ...

The nmap -r process scans the ports sequentially. For each closed port, we see an incoming TCP SYN packet, and a TCP RST reply from the target. An open SSH port (22) instead elicits a TCP SYN+ACK reply, indicating that a server is listening: the scanner responds a short time later with a TCP RST packet (sent out of order) to tear down the half-open SSH connection. Protocol analysis is especially enlightening when a victim is confronted by sneakier probes and denial of service attacks that don't adhere to the usual network protocol rules.

The previous example used -nn to print everything numerically. The -v option requests additional details; repeat it (-v -v ...) for increased verbosity. Timestamps are recorded by the kernel (and saved in libpcap-format trace files), and you can select a variety of formats by specifying the -t option one or more times. Use the -e option to print link-level (Ethernet) header information.

9.16.4 See Also

ifconfig(8), tcpdump(8), nmap(8). The tcpdump home page is http://www.tcpdump.org, and the nmap home page is http://www.insecure.org/nmap.

A good reference on Internet protocols is found at http://www.protocols.com. Also, the book Internet Core Protocols: The Definitive Guide (O'Reilly) covers similar material.