Sniffers


Sniffers are arguably one the most important tool sets in any network engineer's toolkit. Network sniffers are some of the first tools we use when we're debugging firewalls or just networks in general. They give visibility into the network itself in ways that host-based tools on the firewall or end nodes can't. You might read that your firewall rules are supposed to do one thing, but until you get down and look with a sniffer, there's no way to independently validate what your packets are doing. We can't tell you how many times by using a sniffer we've discovered some PEBKAC (Problem Exists Between Keyboard and Chair) that was being blamed on the firewall (that is, pointing to a nonexistent DNS server). We regularly use sniffers such as tcpdump and ethereal on both the firewall and client systems to watch sessions from source to destination and independently verify that our rules are, in fact, doing what we expect them to do. This is especially handy when attempting to debug bizarre VPN issues through NAT firewalls.

tcpdump is a command line network sniffer based on the libpcap library and is available for nearly every operating system under the sun. This is the defacto standard sniffer and comes with nearly every Linux distribution these days. Assuming, by some freak chance, that your distribution does not come with tcpdump, you can find it at http://www.tcpdump.org/.

Whereas tcpdump is nice, powerful, and comes with nearly every Linux distribution, ethereal (http://www.ethereal.com) is still our favorite network sniffer. It includes both a graphical and command line sniffer (tethereal), supports numerous protocol decoders, is available for both Windows and Linux, and can use data captured from tcpdump. One especially great feature with the command line version of ethereal (tethereal), is the ability to format the output like other command line sniffers, including tcpdump and Sun Microsystem's default sniffer, snoop.

For protocol level debugging, we highly recommend ngrep (http://ngrep.sourceforge.net/), written by a good friend of ours, Jordan Ritter. ngrep is a network version of the Unix command, grep, which makes it very useful for debugging either protocol-level problems like you'd find in an instant messaging or our personal favorite, string matching firewall rules. We discuss string matching later in the book, but in essence, think about implementing Anti-Virus or Intrusion Detection rules in your firewall. ngrep is a good way to have an independent mechanism for debugging when those rules fire.

Figure 8.1. A ngrep capture of a weather-reporting applet querying an external server for updates.


SSLdump (http://www.rtfm.com/ssldump/) is a network protocol analyzer that can decode SSLv3 and TLS connections and will display them in text form via stdout. This is handy because it allows you to pipe the rules to something else. We use SSLdump for breaking down SSL connections, whether for standard web traffic or other protocols we're encapsulating in an SSL session (SSL POP, SSL IMAP, etc.). Assuming you have the correct keys, you can also use SSLdump to look inside of an SSL encrypted session.

This is especially useful for debugging MTU issues that will pop up from time to time on DSL or multi-encapsulated sessions. SSLdump gives you visibility into the SSL session itself, to pick up potential error conditions that would not be visible through a regular sniffer.

Figure 8.2. An ssldump capture of an HTTPS connection to a website.


Analyzing Traffic Utilization

Sometimes the problems in the network are not that you cannot get into or out of a network; rather, it's the performance. One of the most common issues we've found that cause this problem are systems that are consuming vast amounts of network resources, such as numerous streaming audio or video users, distributed denial of service attack zombies, or just a large download in progress. Finding the source of bandwidth consumption can be time-consuming, and if other firewalls or NAT devices are involvedlike a great big game of network hide-and-go-seek.

etherape (http://etherape.sourceforge.net/) is a visualization tool for network data. It's a highly useful tool for looking at just how much and what kind of traffic systems are attempting to send information through your firewall. Unlike some of the other command line tools, etherape is entirely graphic, so piping anything from etherape to something else is out of the question. We're big CLI fans, but we like etherape because it's a good visualization toolit's good for smaller networks or low traffic segments. If you are familiar with etherman, you will recognize etherape right away.

Figure 8.3. etherape in visualizing the traffic for the system 192.168.32.190.


We use etherape for a quick-and-dirty look at what kind of traffic is pulsing through the firewall and where. It's very easy to get an idea of what is going through the firewall in realtime. However, it's not very good for long-term trending. If you want to spot who's slowing down your network quickly, this is the first tool we'd try.

Like the process monitoring command, top, iftop (http://www.ex-parrot.com/~pdw/iftop/) monitors what systems are using the bandwidth on your system. To use iftop, you designate an interface for it to listen on, and iftop will display the current bandwidth usage by a pair of hosts in a top-like format. This can really help isolate network utilization problems.

iftop is the second line tool if you don't have an X11R6 Windowing system to run etherape, and you can handle finding a system by wading through a lot of output (don't worryit does sort). iftop will help you find out just who is using the most traffic through your firewall.

tcprack (http://www.rhythm.cx/~steve/devel/tcptrack/) is also a sniffer, much like iftop. Its chief difference from iftop is that it displays the connection state information in a more "technically" friendly manner. iftop leans more towards the graphical. Each has its place, depending on what you are trying to accomplish. For instance, we like to use iftop to determine how much bandwidth a particular connection is using when doing "off the cuff" checks when we can't use etherape. An example might be the age old question, "Why is my connection so slow?" iftop can be really handy for this sort of quick visualization of a real-time connection; whereas, we tend to use tcptrack when we are interested in the state of a connection.

Figure 8.4. iftop displaying the traffic for the system 192.168.32.190 without host port information displayed. This is an aggregate display of the bandwidth used by the host pairs. If you want more detailed information, such as the ports in use, iftop can display that information as well.


Network Traffic Analyzers

When running a firewall(s) over time, good trending information will give you the visibility into identifying long-term problems. A really good example of this is an ISP where we had collocated some servers. Their network was misconfigured in such a way that other systems were dumping traffic onto the external segment on which our firewall was running, in turn charging us for traffic we weren't using. With accurate data from vnstat, we were able to demonstrate just how much traffic was really being consumed and were able to reap a considerable refund on our bill.

vnstat (http://humdi.net/vnstat/) is a trending tool. It will, when invoked, maintain a daily log of all the traffic it sees on a given interface. vnstat can, of course, look at more than one interface, but you have to designate the interface(s) to monitor. Unlike some of the sniffer tools that accomplish the same, vnstat works complete from /proc, meaning that it can be run as a nonpriviliged user, which is why we prefer this tool over other network traffic analyzers. Anytime you can get the same results without needing privileged access is a very good thing.

Here is an example of vnstat output from one of our firewalls. Run a 2.4 kernel with openswan installed and IPSEC in use:

 $ vnstat                      rx      /      tx      /   total    / estimated eth0:     yesterday      26.76 MB  /    56.04 MB  /  82.80 MB         today      47.40 MB  /    21.87 MB  /  69.28 MB  /   140  MB eth1:     yesterday      1,044 MB  /    96.68 MB  /  1,141 MB         today     543.30 MB  /    79.83 MB  / 623.13 MB  /  1,295 MB ipsec0:     yesterday          0 MB  /        0 MB  /      0 MB         today          0 MB  /        0 MB  /      0 MB  /      0 MB ipsec1:     yesterday       8.29 MB  /    34.99 MB  /  43.29 MB         today       4.19 MB  /    10.48 MB  /  14.68 MB  /     28 MB 

Useful Control Tools

cutter (http://www.lowth.com/cutter/) allows you to abort connections routed over your firewall by using the FIN-ACK-RST technique to close connections. Let's say you want to cut off a specific SSH connection to a specific hostbut not all traffic to the host. This tool will allow you to just shutdown the connection(s) that you want without affecting any other traffic to the host. cutter is purely an action-oriented tool. It will not tell you anything about a connection or your network, except to let you know that it sent the RST or if the connection does not exist.

Let's say we want to cut off all connections to the host 205.241.45.82. With cutter, this process is very simple and straightforward:

 cutter 205.241.45.82 For connection 10.10.10.8:39010 -> 205.241.45.82:22         sending RST from 205.241.45.82:22 to 10.10.10.8:39010         sending RST from 68.100.73.75:39010 to 205.241.45.82:22 For connection 10.10.10.8:56690 -> 205.241.45.82:22         sending RST from 205.241.45.82:22 to 10.10.10.8:56690         sending RST from 68.100.73.75:56690 to 205.241.45.82:22 

You also can be more specific, such as

 cutter 68.100.73.75 205.241.45.82 

which would only RST the connections between these two hosts. You also can specify the exact ports you want to target, further narrowing the scope of the connections you want to close down.

 cutter 68.100.73.75 22 205.241.45.82 58099 

In the previous example, only a very specific connection, that initiated from 205.241.45.82 to the SSH service running on 68.100.73.75, would be closed.

Network Probes

tcptraceroute (http://michael.toren.net/code/tcptraceroute/) is what its name sounds like, a traceroute that can use tcp. As a brief refresher, recall that regular traceroute sends out ICMP ECHO packets or UDP packets to determine the route a packet takes to get to a destination. Because many firewalls block ICMP and certain UDP ports, such as traceroute ports, there is a need for something that can probe the route of a packet despite these limitations...enter tcptraceroute.

Here is an example output:

 tcptraceroute www.gotroot.com Selected device ppp0, address 166.180.32.200 for outgoing packets Tracing the path to www.gotroot.com (205.241.45.98) on TCP port 80, 30 hops max  1  * * *  2  * * *  3  * * *  4  * * *  5  65.sub-66-174-104.myvzw.com (66.174.104.65)  148.481 ms  142.835 ms  137.663 ms  6  sl-gw23-pen-1-0-1.sprintlink.net (144.228.179.73)  142.666 ms  130.920 ms  141.407 ms  7  sl-bb25-pen-0-5.sprintlink.net (144.232.5.29)  105.704 ms  129.811 ms  132.654 ms  8  sl-gw1-pen-10-0.sprintlink.net (144.232.5.6)  151.633 ms  137.984 ms 125.672  ms  9  sl-twora-3-1.sprintlink.net (144.228.116.146) 137.638 ms     sl-twora-3-0.sprintlink.net (144.223.22.62) 206.195 ms     sl-twora-3-1.sprintlink.net (144.228.116.146) 343.609 ms 10  plesk.shinn.net (205.241.45.98) [open]  135.099 ms 183.420 ms 165.599 ms 

The astute reader will note that in the previous example, a destination port was not set. The default is port 80, which, being one of the more common services running on the Internet, seems sane enough. You also can designate a source and/or destination port for tcptraceroute to further get around any firewall rules that may be preventing a successful trace of the packet's route. Here is a simple example:

 tcptraceroute -p 1000 www.gotroot.com 22 Warning: --track-id implied by specifying the local source port Selected device ppp0, address 166.180.32.200, port 1000 for outgoing packets Tracing the path to www.gotroot.com (205.241.45.98) on TCP port 22, 30 hops max  1  * * *  2  * * *  3  * * *  4  * * *  5  65.sub-66-174-104.myvzw.com (66.174.104.65)  100.426 ms  152.877 ms  274.671 ms  6  sl-gw23-pen-1-0-1.sprintlink.net (144.228.179.73)  224.906 ms  238.076 ms  147.663 ms  7  sl-bb25-pen-0-5.sprintlink.net (144.232.5.29)  123.684 ms  131.316 ms  129.185 ms  8  sl-gw1-pen-10-0.sprintlink.net (144.232.5.6)  219.549 ms  132.361 ms  99.711 ms  9  sl-twora-3-1.sprintlink.net (144.228.116.146)  156.652 ms     sl-twora-3-0.sprintlink.net (144.223.22.62)  116.261 ms  132.005 ms 10  plesk.shinn.net (205.241.45.98) [closed]  107.728 ms  131.536 ms  158.638 ms 

Probing Tools

nmap (http://www.insecure.org) is a highly advanced port scanner with OS detection capabilities, banner grabbing, IDS evasion, and network discovery capabilitiesamong all the other wonderful features this tool provides. Besides being a nifty security tool, nmap is also a good network diagnostics tool. It gives the operator the ability to test remote connections via a number of different means, such as opening connections without normal three-way handshakes. We cover using nmap in more detail later in the book. Suffice it to say, a port scanner is the ideal tool to quickly verify what kinds of traffic can pass through firewall filter rules (in either direction!).

Following is a brief teaser on nmap. To quickly scan a system to determine if any ports are open, you can use this simple command:

 nmap -sS <host name, IP, or network address> 

This would launch what is referred to as a "SYN" scan against the host or network. This is basically a "half open" scan, if you recall from Chapter 5, "The OSI Model: Start from the Beginning." This would start the first part of the TCP connection by sending the SYN request to the host or hosts. If there were anything listening on that port, it would reply with an ACK. If nothing were there, the host could reply with an RST (this is what happens when you use a -j REJECT action with your rules), or it could simply not reply at all (such as what happens when you use a -j DROP action with your firewall rules). nmap then pulls all this information together and then tells you which ports are open.

Another simple and brief foray into nmap would involve how to test a specific port. You can do this with the following command:

 nmap -sS -p 22 <host name, IP, or network address> 

This would cause nmap to test only one port, 22, or the SSH port, to see if the service was open to the host doing the testing. You also can perform other types of tests with nmap, such as tcp connect scans, with the -sT switch instead of the -sS switch. This would carry out a full three-way handshake before reporting any port as being openand other types of scans as well. We will explore the use of nmap with firewalls in Chapter 10, "Testing Your Firewall Rules (for Security!)."



    Troubleshooting Linux Firewalls
    Troubleshooting Linux Firewalls
    ISBN: 321227239
    EAN: N/A
    Year: 2004
    Pages: 169

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net