Diagnostic Logging

This first trick to figuring out what's causing a strange problem with your firewall is to log everything netfilter is doing. Logging will help you to keep a record of how your rules are being executed, and a sniffer can give you visibility into the packets themselves to determine why a connection is behaving in a certain manner. As opposed to normal logging, which consists of logging behavior you want to know about, such as suspicious behavior or providing audit trails for certain connections, diagnostic logging is all about verbosity. You want to see things that you normally would not want to log, such as the execution of all your ruleseven packets that you normally allow and do not care about. Naturally, this form of logging is not something you normally want to do with your logs, as it can fill them up with tons of information that you don't care about. Therefore, these techniques are only for diagnostic purposes and are not recommended for ongoing use with production firewalls. They can fill up your logs quickly and might also reduce performance on the firewall.

The goal with diagnostic logging is to see everything. You don't want to assume that something is working or that a packet is taking one route through one chain, INPUT for instance, when it's really being handled by the FORWARD chain. By logging everything, you can empirically ascertain the actual sequence of events.

Scripts To Do This for You

In classic engineer form, we prefer to use good scripts to add in and remove logging rules for us on-the-fly. You can always add in rules manually, but we prefer to have tools at our disposal to diagnose problems quickly so that we can start ruling out false root causes.

 #!/bin/bash # $Id: iptables-trace,v 1.14 2004/11/13 00:31:15 apc Exp $ # Tony Clayton <t ny-netfilter@clayt n.ca> # You may use and edit this code freely. If you make changes to # it that are generally useful, please email them to me and/or # post them on the netfilter mailing list. LOGPREFIX='${table:0:1}:${chain:0:14}:$rulenum:${target:0:14}' log_entry() {   local action=$1   local table=$3 chain=$5   shift 5   if [ "$last_chain" != "$chain" ]; then     rulenum=1   fi   case $action in   (skip) ;;   (add)     local rulespec     while [ "$1" != "-j" ]; do       rulespec="$rulespec $1"       shift;     done     shift;     target=$*     eval prefix="${LOGPREFIX}"     iptables -t $table -I $chain $rulenum $rulespec -j LOG \            --log-level debug --log-prefix "*${prefix:0:27}:"     let rulenum=$rulenum+1     ;;   (delete)     iptables -t $table -D $chain $rulenum     let rulenum=$rulenum-1     ;;   esac   last_chain=$chain } start() {   for table in $(cat /proc/net/ip_tables_names); do     rulenum=1     iptables-save -t $table | grep '^-' | \       while read cmd; do         log_entry add -t $table $cmd         let rulenum=$rulenum+1       done   done } stop() {   for table in $(cat /proc/net/ip_tables_names); do     iptables-save -t $table | grep '^-' | \       while read cmd; do         echo $cmd | grep -q -e '--log-prefix "*'         if [ $? -eq 0 ]; then           log_entry delete -t $table $cmd         else           log_entry skip -t $table $cmd         fi         let rulenum=$rulenum+1       done   done } case "$1" in   start) start         ;;   stop) stop         ;;   *) echo $"Usage: $0 {start|stop}"      exit 1 esac exit 0 

When activated, the system will generate a log entry for each chain in the ruleset, prefacing these debugging rules with an asterisk "*" to help tell them apart from normal logging rules and also to make it easier to remove these rules via the iptables-trace script. The net field denotes the type of chain being logged, "m" for mangle, "f" for filter, and "n" for nat. Following that is the name of the chain, OUTPUT for instance, then the index number for the rule to tell you in what order this rule is triggered, and finally the target of the rule.

Like an init script, this one is invoked with a stop or start command. It then parses all the iptables rules and adds in a LOG target for that rule so that you can track the progress of a packet as it moves through the system. Tools like fwbuilder and others also can add in these sorts of diagnostic/debugging rules on-the-fly through their interfaces. The intent is to add in a logging rule for everything and to assume nothing is working correctly.

The catch all Logging Rule

Another useful logging trick is to put in place at the end of all your firewall rules, before your final DROP or REJECT rule, a catch all logging rule. This will help you to diagnose packets that are dropped by the firewall because of a missing rule, typo, or other mistake. This is also a useful security measure because the firewall will now log all the packets you will reject as part of your "unless allow, deny" security philosophy.

 # catch all dropped packets $IPTABLES -A INPUT -p all -m limit --limit 1/second \ -j LOG --log-level info --log-prefix "FINAL -- DROP " \ --log-tcp-sequence --log-tcp-options log-ip-options 

Keep in mind that in our example we are limiting the number of packets that will be logged to one per second. This is to prevent the firewall logging system from being overwhelmed by either too much traffic or by a deliberate attempt by an attacker to overwhelm the system. This is just a suggestion, though. If you want to log every packet, your iptables command would look like this:

 # catch all dropped packets $IPTABLES -A INPUT -p all -j LOG --log-level info \ log -prefix "FINAL -- DROP " --log-tcp-sequence \ --log-tcp-options log-ip-options 

The iptables Trace Patch

There is also a nifty little patch for iptables called trACE. It is not presently part of the stock iptables binary or any stock kernels, so you will have to patch both to get this option. We provide instructions for patching on our website at http://www.gotroot.com. Where possible, we also provide links to and copies of some iptables binaries that include this option, and you will find it as a module in the kernels we provide on our website. The patch is also available from the netfilter website, http://www.netfilter.org. It's included in the patch-o-matic archives.

TRACE is a target, much in the same way that ACCEPT or DROP are. You mark a packet with trACE, and the system will log its "flow" through the netfilter system via its table or chain name.

We think the -j TRACE target should be included in every iptables binary and kernel distribution. Although it can be slow, it's a useful tool designed specifically to help with diagnosing firewall rule problems.

Unfortunately, as TRACE is not included in any Linux distributions, we're not going to spend any time discussing it in this book, as it's not really available for most users. Our goal is to focus on tools that are in widespread use. If you are interested in using -j trACE, please check our website (www.gotroot.com) for examples.