Using TCPDump to Capture Specific Protocols

In this section, I'll give some examples that show you how to capture various forms of network traffic for monitoring purposes. Included among the examples, you'll see what a DNS query looks like through TCPDump, some ICMP (ping) examples, and various TCP- and UDP-based protocols. After you see how normal traffic looks, I'll then show you some of the fun stuff. Specifically, I'll show what some types of attacks look like through TCPDump so that you might be able to quickly detect these when coming into (or out of) your network.

Throughout this section, I'll be using a few different programs to generate traffic for TCPDump to capture. My primary tool for TCP-related captures will be telnet. I'll use telnet to generate traffic and mirror what the real protocol (or close to it) does in the real world. Generation of DNS queries will be accomplished using both the dig command and the host command. The ping and TRaceroute commands will be used. Finally, the hping2 command will be used to generate ICMP traffic as well as other interesting packets, especially in the attack section. With the exception of hping2, all of these programs are installed on most major Linux distributions.

Using TCPDump in the Real World

So far in this chapter, you've seen a number of examples for using TCPDump to capture various types of traffic. These examples were given to show the usage of TCPDump in relation to expressions and other options. Now it's time to give you real-life examples of using TCPDump to capture specific types of traffic. The situations in which you might use these examples will vary, but I'll try to give some clue as to why you might use a given example, where I can. It might be helpful to see how a filter expression is built when trying to capture in the real world. I briefly touched on this topic earlier. However, before giving recipe-type solutions, I'll show you how to build a filter with the specific goal of capturing an HTTP conversation.

BUILDING A FILTER TO CAPTURE AN HTTP CONVERSATION

HTTP is the language of the Web. Usually HTTP rides over TCP, which in turn rides on IP. I'm choosing HTTP as the first real-world capture only because people are generally familiar with browsing a web page, even though they may not be familiar with the underlying protocol.

Recall that IP is a connectionless protocol whereas TCP is a connection-oriented protocol. TCP uses a three-way handshake to begin a conversation. HTTP takes advantage of the connection-oriented nature of TCP and in fact knows nothing of lower-layered (remember the OSI model) protocols. As far as HTTP is concerned, it hands its data down to the next lower layer and is done. To that end, when an HTTP conversation is initiated, the first thing you should see through TCPDump is the three-way handshake of TCP followed by protocol-specific data.

For the most part, HTTP traffic flows to a destination of port 80.

NOTE

The ports on which various services normally operate can be found by examining the file /etc/services. With that in mind, the true source for port-number assignments is IANA. You can view the most current and complete list of official port-number assignments at the URL http://www.iana.org/assignments/port-numbers. However, remember that there's nothing preventing someone from running a service on a port other than the official port numbers!

Because HTTP is usually found on port 80, it would be a good idea to start with a basic TCPDump expression that looked only for port 80 traffic, such as this one:

 tcpdump 'port 80' 

Running that command and then generating some traffic by surfing to a web page yields these results:

 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 17:15:38.934337 IP client.braingia.org.4485 > test.example.com.www: \          S 523004834:523004834(0) win 5840 \                 <mss 1460,sackOK,timestamp 249916003 0,nop,wscale 0> 17:15:38.984650 IP test.example.com.www > client.braingia.org.4485: S \          2810959978:2810959978(0) ack 523004835 win 5792 \                 <mss 1460,sackOK,timestamp 1320060704 249916003,nop,wscale 0> 17:15:38.984684 IP client.braingia.org.4485 > test.example.com.www: \          . ack 1 win 5840 <nop,nop,timestamp 249916008 1320060704> 17:15:38.985326 IP client.braingia.org.4485 > test.example.com.www: \         P 1:462(461) ack 1 win 5840 <nop,nop,timestamp 249916008 1320060704> 17:15:39.038067 IP test.example.com.www > client.braingia.org.4485: . \         ack 462 win 6432 <nop,nop,timestamp 1320060710 249916008> 17:15:39.065141 IP test.example.com.www > client.braingia.org.4485: . \         1:1449(1448) ack 462 win 6432 <nop,nop,timestamp 1320060712 249916008> 17:15:39.065183 IP client.braingia.org.4485 > test.example.com.www: . \         ack 1449 win 8688 <nop,nop,timestamp 249916016 1320060712> 

NOTE

I've separated the results for greater readability, and I'll continue to do so throughout the rest of this chapter.

Notice the first two lines from the TCPDump output. In this case, the first line tells me that I really should have more verbose output enabled in order to see anything interesting within the packet, and the second line gives a status of the interface on which TCPDump is listening, as well as the size of the capture.

The next line is the first line of the capture and is also coincidentally the first packet (SYN) sent in the TCP three-way handshake. The first thing you'll notice on the line is a timestamp, followed by the protocol (IP). Next is the hostname of the computer that initiated the packet (client.braingia.org) together with the source port for the traffic (4485). This combination, source computer and source port, are known as the Source. The greater-than sign (>) shows the direction of the flow in relation to the Destination for this traffic, which, as you can see from that line, is test.example.com.www. The www signifies the destination port that the traffic is headed for on the destination computer.

The next item of interest on the TCPDump output line is the Flags section, in this case indicated by an uppercase S. Recall from Chapter 1, "Preliminary Concepts Underlying Packet-Filtering Firewalls," that the TCP header can contain various flags to indicate certain conditions for the packet. If you guessed that S indicates a packet with the SYN flag set, you may have just won a valuable prize. Following the Flags section is the Sequence Number space for the packet, indicating the sequence numbers that will be covered within this packet. In the case of the example, the Sequence Number space (523004834:523004834(0)) is zero length. The next item on this line is the Window size, as indicated by the win 5840 in the output. Finally, enclosed within brackets are options contained in the packet. Although these options can be of interest at certain times, you'll rarely need to care much about them in the real world.

You've now seen one single packet of a TCP three-way handshake through TCPDump. Don't worry, it does get more exciting than this, really it does. The next line contained in the capture contains the response packet coming back from test.example.com. Notice that the timestamp has increased and the protocol is still IP. However, now the source computer is test.example.com on port 80 and the destination is client.braingia.org.4485. Notice also that the SYN flag is set as evidenced by the S following the source > destination area. The sequence number space is different now though, 2810959978:2810959978(0). This is because test.example.com chose its own sequence number as part of the process. The first difference of real interest because we haven't seen it before is the ack 523004835. This is the second part of the TCP three-way handshake, namely, what's commonly referred to as the SYN-ACK packet. In this packet, the original destination computer is answering or acknowledging the call to initiate a TCP connection on the specified port. Notice that the number following the ack is equal to the original sequence number (523004834) plus one. This is the protocol itself in action.

The third packet in the capture, depicted again here for reference, is the final packet in the connection setup for TCP:

 17:15:38.984684 IP client.braingia.org.4485 > test.example.com.www: . \          ack 1 win 5840 <nop,nop,timestamp 249916008 1320060704> 

In this packet, the original source acknowledges the connection setup. Notice that there is a single dot (.) where the flags would normally show up. This usually means that there are no flags set; however, some flags like ACK show up in a different place on the output line. The source side sets the ACK flag and also sets an initial sequence number for this connection. At this point, the TCP connection is said to be established. That sure seems like a lot of work, but didn't I promise that this section was about HTTP? Sure enough. The next packets with capture output indicate that an HTTP connection is progressing:

 17:15:38.985326 IP client.braingia.org.4485 > test.example.com.www: \         P 1:462(461) ack 1 win 5840 <nop,nop,timestamp 249916008 1320060704> 17:15:39.038067 IP test.example.com.www > client.braingia.org.4485: . ack \         462 win 6432 <nop,nop,timestamp 1320060710 249916008> 17:15:39.065141 IP test.example.com.www > client.braingia.org.4485: . \         1:1449(1448) ack 462 win 6432 <nop,nop,timestamp 1320060712 249916008> 17:15:39.065183 IP client.braingia.org.4485 > test.example.com.www: . \         ack 1449 win 8688 <nop,nop,timestamp 249916016 1320060712> 

The source sends the beginning of the data for this communication. Notice that the PUSH flag was set in the initial packet and that the sequence numbers increment. The two sides acknowledge sequence numbers and data is transferred. But with the TCPDump command that I ran (tcpdump 'port 80'), there isn't much else to see. Therefore, I'll improve that command to include the options I normally include to peek inside the packets. I'll leave it up to the reader to see what each of the options actually does, as explained earlier in the chapter. Here's the improved command for TCPDump:

 tcpdump -vv -x -X -s 1500 'port 80' 

With this command running, I can then generate additional web traffic. Here are two-and-a-half packets from the result, picking up from just after the three-way handshake:

 18:18:51.986230 IP (tos 0x0, ttl 64, id 10907, offset 0, flags [DF], \         length: 513) client.braingia.org.4564 > test.example.com.www: \                 P [tcp sum ok] 1:462(461) ack 1 win 5840 <nop,nop,timestamp                         250295308 1320440053>     0x0000: 0090 2741 78f0 00e0 1833 2ee8 0800 4500 ..'Ax....3....E.     0x0010: 0201 2a9b 4000 4006 044b c0a8 010a 455d ..*.@.@..K....E]     0x0020: 0302 11d4 0050 0c9b 1ea0 9627 33f8 8018 .....P.....'3...     0x0030: 16d0 4915 0000 0101 080a 0eeb 340c 4eb4 ..I.........4.N.     0x0040: 50f5 4745 5420 2f20 4854 5450 2f31 2e30 P.GET./.HTTP/1.0     0x0050: 0d0a 486f 7374 3a20 7777 772e 6272 6169 ..Host:.text.exam     0x0060: 6e67 6961 2e6f 7267 0d0a 4163 6365 7074  ple.com..Accept     0x0070: 3a20 7465 7874 2f68 746d 6c2c 2074 6578 :.text/html,.tex     0x0080: 742f 706c 6169 6e2c 2061 7070 6c69 6361 t/plain,.applica     0x0090: 7469 6f6e 2f6d 7377 6f72 642c 2061 7070 tion/msword,.app     0x00a0: 6c69 6361 7469 6f6e 2f70 6466 2c20 6170 lication/pdf,.ap     0x00b0: 706c 6963 6174 696f 6e2f 6f63 7465 742d plication/octet-     0x00c0: 7374 7265 616d 2c20 6170 706c 6963 6174 stream,.applicat     0x00d0: 696f 6e2f 782d 7472 6f66 662d 6d61 6e2c ion/x-troff-man,     0x00e0: 2061 7070 6c69 6361 7469 6f6e 2f78 2d74 .application/x-t     0x00f0: 6172 2c20 6170 706c 6963 6174 696f 6e2f ar,.application/     0x0100: 782d 6774 6172 2c20 6170 706c 6963 6174 x-gtar,.applicat     0x0110: 696f 6e2f 7274 662c 2061 7070 6c69 6361 ion/rtf,.applica     0x0120: 7469 6f6e 2f70 6f73 7473 6372 6970 742c tion/postscript,     0x0130: 2061 7070 6c69 6361 7469 6f6e 2f67 686f .application/gho     0x0140: 7374 7669 6577 2c20 7465 7874 2f2a 0d0a stview,.text/*..     0x0150: 4163 6365 7074 3a20 6170 706c 6963 6174 Accept:.applicat     0x0160: 696f 6e2f 782d 6465 6269 616e 2d70 6163 ion/x-debian-pac     0x0170: 6b61 6765 2c20 6175 6469 6f2f 6261 7369 kage,.audio/basi     0x0180: 632c 202a 2f2a 3b71 3d30 2e30 310d 0a41 c,.*/*;q=0.01..A     0x0190: 6363 6570 742d 456e 636f 6469 6e67 3a20 ccept-Encoding:.     0x01a0: 677a 6970 2c20 636f 6d70 7265 7373 0d0a gzip,.compress..     0x01b0: 4163 6365 7074 2d4c 616e 6775 6167 653a Accept-Language:     0x01c0: 2065 6e0d 0a55 7365 722d 4167 656e 743a .en..User-Agent:     0x01d0: 204c 796e 782f 322e 382e 3472 656c 2e31 .Lynx/2.8.4rel.1     0x01e0: 206c 6962 7777 772d 464d 2f32 2e31 3420 .libwww-FM/2.14.     0x01f0: 5353 4c2d 4d4d 2f31 2e34 2e31 204f 7065 SSL-MM/1.4.1.Ope     0x0200: 6e53 534c 2f30 2e39 2e36 630d 0a0d 0a  nSSL/0.9.6c.... 18:18:52.039595 IP (tos 0x0, ttl 48, id 25346, offset 0, flags [DF], \         length: 52) test.example.com.www > client.braingia.org.4564: .                 [tcp sum ok] 1:1(0) ack 462 win 6432 <nop,nop,timestamp                         1320440059 250295308>     0x0000: 00e0 1833 2ee8 0090 2741 78f0 0800 4500 ...3....'Ax...E.     0x0010: 0034 6302 4000 3006 ddb0 455d 0302 c0a8 .4c.@.0...E]....     0x0020: 010a 0050 11d4 9627 33f8 0c9b 206d 8010 ...P...'3....m..     0x0030: 1920 6799 0000 0101 080a 4eb4 50fb 0eeb ..g.......N.P...     0x0040: 340c                   4. 18:18:52.047021 IP (tos 0x0, ttl 48, id 25347, offset 0, flags [DF], \         length: 1500) test.example.com.www > client.braingia.org.4564:\                  . 1:1449(1448) ack 462 win 6432 <nop,nop,timestamp \                         1320440059 250295308>     0x0000: 00e0 1833 2ee8 0090 2741 78f0 0800 4500 ...3....'Ax...E.     0x0010: 05dc 6303 4000 3006 d807 455d 0302 c0a8 ..c.@.0...E]....     0x0020: 010a 0050 11d4 9627 33f8 0c9b 206d 8010 ...P...'3....m..     0x0030: 1920 b9f7 0000 0101 080a 4eb4 50fb 0eeb ..........N.P...     0x0040: 340c 4854 5450 2f31 2e31 2032 3030 204f 4.HTTP/1.1.200.O     0x0050: 4b0d 0a44 6174 653a 2054 7565 2c20 3237 K..Date:.Tue,.27     0x0060: 204a 756c 2032 3030 3420 3233 3a31 393a .Jul.2004.23:19:     0x0070: 3030 2047 4d54 0d0a 5365 7276 6572 3a20 00.GMT..Server:.     0x0080: 4170 6163 6865 2f31 2e33 2e32 3620 2855 Apache/1.3.26.(U     0x0090: 6e69 7829 2044 6562 6961 6e20 474e 552f nix).Debian.GNU/     0x00a0: 4c69 6e75 7820 6d6f 645f 6d6f 6e6f 2f30 Linux.mod_mono/0     0x00b0: 2e31 3120 6d6f 645f 7065 726c 2f31 2e32 .11.mod_perl/1.2     0x00c0: 360d 0a43 6f6e 6e65 6374 696f 6e3a 2063 6..Connection:.c     0x00d0: 6c6f 7365 0d0a 436f 6e74 656e 742d 5479 lose..Content-Ty     0x00e0: 7065 3a20 7465 7874 2f68 746d 6c3b 2063 pe:.text/html;.c     0x00f0: 6861 7273 6574 3d69 736f 2d38 3835 392d harset=iso-8859- <output truncated> 

Notice that this output contains an actual request (see the first packet, near GET./.HTTP/1.0) and also contains a portion of the response from the web server. All of this traffic is in plain text because HTTP is not encrypted. This output contains both hex and ASCII. To obtain output with just ASCII, remove the -x and -X from the command and replace them with a single -A. I personally find the dual hex and ASCII output to be helpful at times.

That's all there is to capturing HTTP traffic with TCPDump. Obvious improvements for the command would be to expand the expression to look for a specific source or destination. It's important to understand that only traffic on port 80 will be found with the command as given. If you're running HTTP traffic on another port, substitute that port instead of (or in addition to) the port found in the sample command.

CAPTURING AN SMTP CONVERSATION

Capturing an SMTP conversation is not unlike capturing an HTTP session. Begin with the basic TCPDump options that you'd like to use and then build an expression to grab the appropriate type of data, including protocol, port, and source or destination hosts. For example, here's a simple capture of port 25 traffic along with my normal TCPDump choice of options:

 tcpdump -vv -x -X -s 1500 'port 25' 

The TCP three-way handshake is again present, as you might expect:

 20:40:08.638690 murphy.debian.org.45772 > test.example.com.smtp: \         S [tcp sum ok] 1485971964:1485971964(0) win 5840 <mss 1460,                 sackOK,timestamp 795074473 0,nop,ws cale 0> (DF) \                          (ttl 57, id 65109, len 60) 0x0000  4500 003c fe55 4000 3906 deae 9252 8a06    E..<.U@.9....R.. 0x0010  455d 0302 b2cc 0019 5892 21fc 0000 0000    E]......X.!..... 0x0020  a002 16d0 8ffe 0000 0204 05b4 0402 080a    ................ 0x0030  2f63 dfa9 0000 0000 0103 0300         /c.......... 20:40:08.638769 test.example.com.smtp > murphy.debian.org.45772: S \          [tcp sum ok] 2853594323:2853594323(0) ack 1485971965 win 5792 \                 <mss 1460,sackOK,timestamp 132 1286843 795074473,nop,wscale 0> \                          (DF) (ttl 64, id 0, len 60) 0x0000  4500 003c 0000 4000 4006 d604 455d 0302    E..<..@.@...E].. 0x0010  9252 8a06 0019 b2cc aa16 64d3 5892 21fd    .R........d.X.!. 0x0020  a012 16a0 f5b6 0000 0204 05b4 0402 080a    ................ 0x0030  4ec1 3cbb 2f63 dfa9 0103 0300         N.<./c...... 20:40:08.640600 murphy.debian.org.45772 > test.example.com.smtp: . \          [tcp sum ok] 1:1(0) ack 1 win 5840 <nop,nop,timestamp \                 795074473 1321286843> (DF) (ttl 57, id 65110, len 52) 0x0000  4500 0034 fe56 4000 3906 deb5 9252 8a06    E..4.V@.9....R.. 0x0010  455d 0302 b2cc 0019 5892 21fd aa16 64d4    E]......X.!...d. 0x0020  8010 16d0 244c 0000 0101 080a 2f63 dfa9    ....$L....../c.. 0x0030  4ec1 3cbb                   N.<. 

There's nothing really new of interest during the three-way handshake process. Notice that the ASCII output isn't of much use during the three-way handshake though.

As with HTTP, after the initial TCP handshake is done, the SMTP conversation gets underway:

 20:40:08.683352 test.example.com.smtp > murphy.debian.org.45772: P \          [tcp sum ok] 1:51(50) ack 1 win 5792 <nop,nop,timestamp \                 1321286848 795074473> (DF) (ttl 64,id 22639, len 102) 0x0000  4500 0066 586f 4000 4006 7d6b 455d 0302    E..fXo@.@.}kE].. 0x0010  9252 8a06 0019 b2cc aa16 64d4 5892 21fd    .R........d.X.!. 0x0020  8018 16a0 bd07 0000 0101 080a 4ec1 3cc0    ............N.<. 0x0030  2f63 dfa9 3232 3020 6466 7730 2e69 6367    /c..220.test.exa 0x0040  6d65 6469 612e 636f 6d20 4553 4d54 5020    mple.com.ESMTP. 0x0050  506f 7374 6669 7820 2844 6562 6961 6e2f    Postfix.(Debian/ 0x0060  474e 5529 0d0a                 GNU).. 20:40:08.684581 murphy.debian.org.45772 > test.example.com.smtp: . [tcp sum ok]  1:1(0) ack 51 win 5840 <nop,nop,timestamp 795074478 1321286848> (DF) (ttl 57, i d 65111, len 52) 0x0000  4500 0034 fe57 4000 3906 deb4 9252 8a06    E..4.W@.9....R.. 0x0010  455d 0302 b2cc 0019 5892 21fd aa16 6506    E]......X.!...e. 0x0020  8010 16d0 2410 0000 0101 080a 2f63 dfae    ....$......./c.. 0x0030  4ec1 3cc0                   N.<. 20:40:08.685428 murphy.debian.org.45772 > test.example.com.smtp: P [tcp sum ok]  1:25(24) ack 51 win 5840 <nop,nop,timestamp 795074478 1321286848> (DF) (ttl 57,  id 65112, len 76) 0x0000  4500 004c fe58 4000 3906 de9b 9252 8a06    E..L.X@.9....R.. 0x0010  455d 0302 b2cc 0019 5892 21fd aa16 6506    E]......X.!...e. 0x0020  8018 16d0 3cc4 0000 0101 080a 2f63 dfae    ....<......./c.. 0x0030  4ec1 3cc0 4548 4c4f 206d 7572 7068 792e    N.<.EHLO.murphy. 0x0040  6465 6269 616e 2e6f 7267 0d0a         debian.org.. 

CAPTURING AN SSH CONVERSATION

Although it's not possible to actually capture an SSH conversation, you can look at some of the connection setup portions of the protocol. Because SSH is encrypted, though, none of the credentials or other data during the actual session is available for you to view. It should be noted, however, that if you can gain access to the private key of the server, you could theoretically decrypt the contents of the SSH connection. Doing this is well beyond the scope of this text.

I'll leave it as an exercise for the reader to capture an SSH connection, including setup, should you want to view what a normal connection looks like for SSH.

CAPTURING OTHER TCP-BASED PROTOCOLS

Capturing other TCP-based protocols follows much the same process as that in the examples shown. For example, capturing POP3 connections can be accomplished and the entire stream can be captured because POP3, like SMTP, is not encrypted during transit. One protocol is of particular interest because it has confounded network administrators for a long time. That protocol is FTP.

FTP utilizes two TCP ports, 20 and 21. Port 21 is normally used for commands and is sometimes referred to as the control channel. Port 20 in FTP is used for data and is sometimes aptly titled the data channel. Therefore, if you want to capture FTP traffic with TCPDump, you need to grab both ports 20 and 21 to see everything.

A trend over the past few years has been protocols that use nonstandard ports to circumvent firewalls and packet capturing and filtering. Such programs, including much of the peer-to-peer software, can be somewhat difficult to find during packet captures because most of the data during the conversation is binary and is thus not human-readable.

CAPTURING A DNS QUERY

TCPDump handles DNS queries a little differently than a packet that's simply TCP. More information can be gleaned from just the initial packet result line as opposed to making it necessary to increase the snaplen with the -s option. For example, consider the following trace of a simple DNS query that was looking for the IP address of a host named www.braingia.org:

 21:18:39.289121 192.168.1.10.1514 > 192.168.1.1.53: 60792+ A? www.braingia.org. (34) (DF) 21:18:39.289568 192.168.1.1.53 > 192.168.1.10.1514: 60792*- 1/2/2 A 192.168.1.50 (118) (DF) 

In the packet trace we see host 192.168.1.10 on an ephemeral port communicating with a destination of 192.168.1.1 on port 53. A query ID number is given; in this case it's 60792. You see the query ID number is followed by a +. This symbol indicates that the querier asked for recursion on this query. The A? on the trace line indicates that this was an address query. The query was for www.braingia.org, as is shown, and the size of the query is 34 bytes, which does not include IP or UDP overhead.

The answer comes quickly, as we see in the next line of the trace, which shows that the source is now 192.168.1.1 talking to the destination of 192.168.1.10. As you can see, the answer was contained in the same query ID, 60792; however, this time there are two extra characters, the * and the -. The * in a response indicates that this is an authoritative answer, and the - indicates that recursion is available and not set. The next portion of the response, 1/2/2, indicates the number of answer records (1), the number of name server records (2), and the number of additional records (2). The first answer given in this case is of type A and is 192.168.1.50. Finally, the size of the response is given to be 118 bytes.

CAPTURING PINGS

Although it may seem innocent enough, ICMP (the protocol behind ping) has been used fairly often as a means by which to attack hosts and otherwise wreak havoc. Therefore, as a security analyst, an administrator, or a curious bystander, you should know that it's in your best interest to see some of the normal activity for ICMP through TCPDump so that you might be able to spot an anomaly later.

I'll go out onto a limb and say that ICMP is most frequently used for the simple Echo Request and Echo Reply provided by ping. However, ICMP can be and is used for much more than that including informing a fast sender when to slow down (Source Quench), redirecting to other hosts (Redirect), and many other areas. Refer to Chapter 1 for more information on ICMP, or, as always, refer to the original RFC on ICMP for the authoritative information on the protocol.

Attacks Through the Eyes of TCPDump

You've seen what normal TCP and UDP packet traces look like through TCPDump, but how will you know whether someone or something is acting abnormally? Unfortunately, finding nefarious activity is not that easy. Buried in normal packet traces may be signs that someone is attempting an attack on your server. An attacker will obviously attempt to disguise his activity, making detection even more difficult. Not only do you have to wade through all the normal traffic within a packet trace, but you then have to search for the proverbial needle in a haystack to find what may be an attack attempt or even one in progress.

Recall from Chapter 10, "Intrusion Detection Tools," that not everything that falls outside of normal activity can be termed an attack. Some of it is the result of malfunctioning or misconfigured equipment. More often than not, abnormal activity spotted in a packet trace is due to reconnaissance of one form or another. And the large majority of reconnaissance work is done through automation. Rather than spending many fruitless hours searching for a vulnerable host, attackers will automate the process and have the program alert them when it finds an interesting host.

There are naturally exceptions to the rule of automation. Attacks may be the result of directed activity against your server or network. Before the attack there is usually some manual reconnaissance that takes place. This may include the would-be attacker manually crafting or creating packets to attempt to exploit possible holes in your server or network. More often than not, however, the attacker will have had some automated recon data that directed his or her efforts in your direction. If an automated scan alerted the attacker that one of your servers may be vulnerable to a particular type of attack, you may find the hosts or your entire subnet within the sights of the attacker.

Leaving a host vulnerable or making a host appear vulnerable and then observing the attacks is the premise behind a honeypot. A honeypot is a host or device that shows up as vulnerable to an attacker and thus looks like a target for attack. The idea is that by watching the methods that attackers use to exploit a hole or watching what they do when they compromise the host, the observer can learn from it and defend against such activity.

As if all the possible reasons already given for seeing abnormal activity aren't enough, here's one more: You'll also encounter accidental connections that may appear to be attacks. In other words, at times someone simply mistypes an IP address when attempting to connect to his or her server. Anyone who has ever answered the telephone only to find out that the caller dialed incorrectly can relate to this situation.

In summary, there are some basic categories within which abnormal activity might fall:

• Automated or semiautomated recon scan

• Directed attack

• Misconfigured equipment

• Wrong number

• Malfunctioning equipment

With those categories in mind, this section examines some abnormal packet traces or traces that you shouldn't see under normal conditions. By no means does this section include all the possible crafted and abnormal packets. The hope is to give you an understanding of some of the types of things to look for when performing an investigation.

NORMAL SCAN (NMAP)

Sometimes an attacker will scan your subnet or individual IP address for open ports. This scan can be anything from an innocent attempt to look for a service to reconnaissance for an attack. Many times, these scans are completely automated, with an attacker setting up one or more robots (bots) to automatically scan for vulnerable versions of software to exploit.

This simulation was created with the nmap program with the following command line:

 nmap -sT 192.168.1.2 

The TCPDump capture of the port scan is shown in the following text; note that I've truncated the output because nmap scanned for more than 1,650 ports. I've divided the capture to make explaining it easier as well.

Nmap scans begin with an ICMP Echo Request to the target host, as shown here. Note, however, that this ICMP exchange can be disabled by the person running the nmap scan, so it might not always show up:

 12:31:21.834284 IP 192.168.1.10 > 192.168.1.2: icmp 8: echo request seq 27074 12:31:21.834508 IP 192.168.1.2 > 192.168.1.10: icmp 8: echo reply seq 27074 

Next nmap looks for port 80, the default HTTP port. Notice that the scan comes from an ephemeral port on the scanner's side aimed for port 80 on the recipient. In this case, the recipient host 192.168.1.2 is listening on port 80 and a TCP response is sent back to the scanning host with the TCP RST flag set and the sequence number set:

 12:31:21.834318 IP 192.168.1.10.60034 > 192.168.1.2.80: . ack 2624625246 win 4096 12:31:21.834363 IP 192.168.1.2.80 > 192.168.1.10.60034: R \         2624625246:2624625246(0) win 0 

Next, nmap looks for the telnet port (tcp/23). Notice the difference between this and the preceding scan. Aside from the ports being different, the response packet is also different. In this case, the recipient host is not listening on TCP port 23, so it responds with a packet with the TCP RST flag set but with the TCP sequence number set to 0:

 12:31:21.935005 IP 192.168.1.10.3171 > 192.168.1.2.23: S 752173650:752173650(0) \         win 5840 <mss 1460,sackOK,timestamp 1421906912 0,nop,wscale 0> 12:31:21.935046 IP 192.168.1.2.23 > 192.168.1.10.3171: R 0:0(0) ack 752173651 win 0 

The following packets are essentially the same as the preceding telnet port scan insofar as the recipient host is not listening on the ports being scanned:

 12:31:21.935129 IP 192.168.1.10.3172 > 192.168.1.2.554: S 758180552:758180552(0) \         win 5840 <mss 1460,sackOK,timestamp 1421906912 0,nop,wscale 0> 12:31:21.935186 IP 192.168.1.2.554 > 192.168.1.10.3172: R 0:0(0) ack 758180553 win 0 12:31:21.935149 IP 192.168.1.10.3174 > 192.168.1.2.21: S 751983738:751983738(0) \         win 5840 <mss 1460,sackOK,timestamp 1421906912 0,nop,wscale 0> 12:31:21.935289 IP 192.168.1.2.21 > 192.168.1.10.3174: R 0:0(0) ack 751983739 win 0 12:31:21.935255 IP 192.168.1.10.3175 > 192.168.1.2.1723: S 757954867:757954867(0) \         win 5840 <mss 1460,sackOK,timestamp 1421906912 0,nop,wscale 0> 12:31:21.935320 IP 192.168.1.2.1723 > 192.168.1.10.3175: R 0:0(0) \         ack 757954868 win 0 

Finally, another open port is found. This time the open port is tcp/25, the well-known SMTP port:

 12:31:21.935381 IP 192.168.1.10.3176 > 192.168.1.2.25: S 762467904:762467904(0) \         win 5840 <mss 1460,sackOK,timestamp 1421906912 0,nop,wscale 0> 12:31:21.935448 IP 192.168.1.2.25 > 192.168.1.10.3176: S 2645882457:2645882457(0) \         ack 762467905 win 5792 <mss 1460,sackOK,timestamp \                 921140115 1421906912,nop,wscale 7> 

As stated previously, the nmap scan continues for another 1,650 or so ports. I chose to save a tree by not showing the remainder of the port scans here. Rest assured that they look largely the same as the ones already shown.

The response you take when port scanned depends on your security policy. If I notice a wide port scan, one in which a large number of ports are scanned, I'll usually take steps to block the host. However, because I'm not at the computer 24 hours a day (close, but not quite), I use a tool called PortSentry to monitor for this type of activity. PortSentry has apparently fallen out of active development since being purchased by Cisco. You can still download the last version of PortSentry at http://sourceforge.net/projects/sentrytools/. However, I won't cover the tool in this chapter because it's quite unclear whether it will exist by the time you're reading this book. Other antiport scan software exists, including a plug-in for Snort.

SMURF ATTACK

A Smurf attack is a DoS attack whereby the attacker sends ICMP Echo Requests with a forged source address to one or more broadcast addresses. The forged source address is the recipient of the attack, and it will be inundated with echo replies from the broadcast addresses of other networks. Imagine echo replies coming from 254 hosts directed at a machine with a small or slow Internet connection. Now imagine those replies coming from 100 networks of 254 hosts each. It doesn't take long for an entire network to become bogged down receiving ICMP replies.

The following trace was created with the hping2 command:

 hping2 -1 -a 192.168.1.2 192.168.1.255 

On the network under attack, only one host responded to the broadcasted ping; however, there's no way to guarantee that other networks wouldn't have a large number of hosts that respond:

 12:57:06.871156 IP 192.168.1.2 > 192.168.1.255: icmp 8: echo request seq 0 12:57:06.871637 IP 192.168.1.8 > 192.168.1.2: icmp 8: echo reply seq 0 12:57:07.870259 IP 192.168.1.2 > 192.168.1.255: icmp 8: echo request seq 256 12:57:07.871008 IP 192.168.1.8 > 192.168.1.2: icmp 8: echo reply seq 256 12:57:08.870132 IP 192.168.1.2 > 192.168.1.255: icmp 8: echo request seq 512 12:57:08.870880 IP 192.168.1.8 > 192.168.1.2: icmp 8: echo reply seq 512 

There is no good host-based defense for a Smurf attack. Even if ICMP replies are disabled on the individual host, the bandwidth is still being consumed by all the replies coming into the network.

To effectively counter a Smurf attack, ICMP Echo Requests directed to broadcast addresses mustn't cross router boundaries. This means that you're relying on others to be good netizens. In addition, ICMP echo replies must be filtered as far upstream from your location as possible. However, I'm not an advocate of filtering ICMP echo replies. A better solution is to rate-limit the echo replies as far upstream as possible while still allowing the replies for all the good that they do in problem diagnosis.

XMAS TREE AND TCP HEADER FLAGS

The Xmas Tree attack is so named because all the bit flags are set on within the TCP header. The idea is to cause the recipient host to respond, thus causing a DoS. Recall the TCP flag bits SYN, RST, ACK, URG, and others from Chapter 1. These bits should never all appear at the same time, and when they do it's an indication of a crafted packet.

Xmas Tree attacks are quite uncommon. However, it's important to consider the TCP flags when examining packets. Setting these flags with invalid combinations is almost always an indication of a crafted packet (though it also could in a few instances indicate broken or misconfigured software). The goal of the crafted packet might be anything from reconnaissance to an active attack such as one to get through a firewall.

The following capture sets the TCP flags SYN, FIN, RST, and PUSH, which should never show up in a real packet. It was created with the hping2 command:

 hping2 -SFRP 192.168.1.2 

There are three packets in this capture. Notice the source port increments and that the destination port is 0. The TCP flags are also shown, SFRP in this case. Seeing this in the wild should cause the intrusion analyst to immediately begin investigating the packets according to the security policy.

 13:20:03.989780 IP (tos 0x0, ttl 64, id 2270, offset 0, flags [none], length: \          40) 192.168.1.10.2687 > 192.168.1.2.0: SFRP [tcp sum ok] \                 925164686:925164686(0) win 512 13:20:04.989734 IP (tos 0x0, ttl 64, id 9285, offset 0, flags [none], \         length: 40) 192.168.1.10.2688 > 192.168.1.2.0: SFRP [tcp sum ok] \                  1113258177:1113258177(0) win 512 13:20:05.989731 IP (tos 0x0, ttl 64, id 26951, offset 0, flags [none], \         length: 40) 192.168.1.10.2689 > 192.168.1.2.0: SFRP [tcp sum ok] \                  2097818687:2097818687(0) win 512 

LAND ATTACK

The LAND attack is a DoS attack against computers running Microsoft Windows. The attack was originally reported to affect Windows 95 and Windows NT back in 1997. Microsoft eventually patched the vulnerability for the operating systems. However, the vulnerability resurfaced in Microsoft's newer operating systems, including Windows XP Service Pack 2 and even Windows Server 2003. Obviously, a problem of this type resurfacing was quite embarrassing for Microsoft, especially because it has undertaken countless initiatives to secure its operating systems and software.

The LAND attack is quite trivial and occurs when the source and destination addresses and ports are set to the recipient host and the SYN flag is set.

hping2 again provides an easy way to re-create this for testing:

 hping2 -k -S -s 25 -p 25 -a 192.168.1.2 192.168.1.2 

The capture through TCPDump is shown next. Notice that the source and destination addresses and ports are the same and that the source port is not incrementing and that it's also below the ephemeral ports:

 13:42:28.079339 IP 192.168.1.2.25 > 192.168.1.2.25: S 764505725:764505725(0) win 512 13:42:29.079462 IP 192.168.1.2.25 > 192.168.1.2.25: S 2081780101:2081780101(0) \         win 512 13:42:30.079461 IP 192.168.1.2.25 > 192.168.1.2.25: S 390202112:390202112(0) win 512 

Recording Traffic with TCPDump

While consulting for a small Internet provider, I noticed that there was a routine and significant spike in network traffic at about 3 a.m. every morning and lasting anywhere from 15 minutes to an hour. My goal was to determine the cause of this traffic spike. Because the traffic was routine and at an odd hour, my initial thought was that the traffic was the result of an automatic update process for the servers on the network.

Most of the servers in the network were running Debian Linux and using apt-proxy. This meant that only one local server would contact the off-site Debian update servers and obtain any updates necessary. All the other servers in the local network would then contact that local master server. This setup cut the Internet utilization immensely.

Although the master server could certainly be a contributing factor, I didn't feel that there would be enough update traffic on a nightly basis to warrant such a significant spike in traffic. My assumption was confirmed when I looked at the update schedule on the master server and found that it was actually looking for updates at a different time anyway, and thus it wasn't contributing to the 3 a.m. spike at all.

With that cause eliminated, I needed to look at the traffic itself at 3 a.m. However, I wasn't really looking forward to staying awake until that hour, and if I was awake, I might not be in shape to read a packet trace. Enter cron. By using cron to fire TCPDump, I could capture the packet trace to a file for later analysis. No great surprise here and I wasn't breaking any new ground, but it seemed like a fair solution to the problem. I configured the switch to copy packets to the port on which the monitoring machine was connected and got to work on the TCPDump portion.

TCPDump offers a couple features that come in handy for this type of trace. The first feature is the capability to write the output to a file (record, if you will) and then read that file in later (playback). The second helpful feature is the capability to exit after capturing a certain number of packets. Granted, I could've used another means to stop the packet trace, such as another cron job to kill the TCPDump capture, but I thought that using TCPDump's native capability was the quickest and easiest solution.

All the TCPDump commands I've shown so far in this chapter have used expressions such as port 80 or host <n>.<n>.<n>.<n>. Expressions are helpful when you're looking for specific and known traffic. However, expressions aren't of much help when you're unsure of what exactly you're searching for, as was the case here. The best option was to capture everything and then work on a filter during playback.

The two TCPDump options of interest that haven't been covered yet in the chapter are -w and -c. The -w option causes TCPDump to place the raw output into the specified file so that it can later be fed back through TCPDump. The resulting file is in TCPDump's native format and is thus not readable by a plain-text pager such as cat, less, or more. The -c option informs TCPDump that it should exit after it captures <N> packets. Getting <N> correct seemed to be the most difficult part of the capture.

The capture results needed to show me only basic information about the packet, including source and destination, as well as a few bits of the packet. To that end, the TCPDump command was rather easy to craft:

 /usr/sbin/tcpdump -c 25000 -w dumpfile -n 

In this command I have TCPDump exiting after capturing 25,000 packets, writing the capture to a file called dumpfile, and not performing DNS queries for the source and destination. Getting to this command did require some level of testing to see just how long it took to capture 25,000 packets and what information was included in the capture. After it was tested, I entered the command into cron with this schedule:

 5 3 * * * /usr/sbin/tcpdump -c 25000 -w dumpfile -n 

That is, every morning at 3:05 a.m. the capture would take place. Then at a saner hour, like 11 a.m. when I get out of bed, I'd look to see whether indeed there was traffic the prior evening that required me to look at the dumpfile. If there was traffic, I'd log in to the server and run a command to read the dumpfile:

 tcpdump -r dumpfile -X -vv 

Running this command gave me an idea of what traffic was out there. Intermingled with the normal traffic was an FTP conversation between one of the ISP's larger customers and another host on the Internet. The FTP traffic was easily the most frequent packet I was seeing within the trace. Another night's packet trace confirmed it. I had the ISP contact its customer to find out whether this was known activity and, if so, to let the customer know that it might be going over its bandwidth allocation for the month.

This example is somewhat typical of a security analyst's job. Spot an anomaly, investigate the anomaly while ruling out possibilities, and take action based on the investigation. Although the ultimate cause of this particular anomaly turned out not to be any type of unauthorized attack, the result of the investigation was a happier customer because that customer could take corrective action before exceeding its bandwidth for the month.