Tools Discussed in this Chapter | Troubleshooting Linux Firewalls

Sniffers:

Probably the most useful tool for us when diagnosing any type of tunneling or NAT-ing problems has been a really good sniffer. Personally we prefer ethereal (www.ethereal.com), which comes with most major Linux distributions and is available for just about every operating system under the sun (including Linux, Solaris, Windows, OSX, even BeOS). Another advantage to ethereal is that it has both a graphical and command line interface, and what is especially useful is that you can change the format of the command line interface to emulate the output from other sniffer software (tcpdump and snoop, among others).

In the case of VPN diagnostics, as long as the VPN creates a device, as freeswan creates the ipsec0 device, you can attach to that device and look at the kinds of traffic going through the VPN. This is imperative if you are attempting to NAT traffic into a VPN.

openswan, http://www.openswan.org, is an IPSEC implementation for Linux that includes support for numerous authentication mechanisms and is in general very compatible with the myriad IPSEC implementations from other open source and commercial vendors.

IPSEC: Internal SystemsBehind a NAT/MASQ Firewall Cannot Connect to an External IPSEC Server

Extended Description: You have an internal system (Host-A) that creates an IPSEC VPN to another system, for example a corporate VPN server (Host-B), through your firewall. Non-VPN traffic functions properly, however the VPN connection cannot be established. In this "road warrior" configuration, the internal system (laptop in diagram here) only connects to the remote system as an end-node; it does not perform any routing.

Figure 19.1. Internal systems cannot connect to external IPSEC server.

This is a pretty standard configuration with very common problems. In my experience with five separate VPN implementations, I've never had it be the same problem twice. Some clients are more sensitive to NAT scenarios than others, and some are just painfully slow when it comes to authentication. The first step is to ensure that you can reach the remote VPN server in the first place, as well as other sites on the Internet. Verify that you can access that system with ping, traceroute, and so on. I spent a whole afternoon tearing my hair out in frustration about a VPN not working, only to find out that they had changed IPs.

The next step is to get a sniffer running on the firewall itself and to look at the inbound and outbound traffic in each direction to verify that the traffic is traversing the firewall correctly (or at all!). One component to look into with VPN software is that you enable it to use NAT Traversal Mode. If traffic is not passing through the firewall, this is definitely the first place to start.

Assuming traffic is not passing through the firewall, this could be due to any number of issues, starting with the MTU settings mentioned earlier in this chapter. The following is a quick checklist of the more obvious problems that will cause a breakdown in this VPN configuration:

Are you running a DSL or PPPoE connection between Host-A and the VPN server, Host-B? If so, check the MTU settings on the firewall and Host-A. The default configurations can be as high as 1500, which will absolutely break your IPSEC VPN. Recommended configurations can back this off to as low as 1300 before you can get a successful connection working. We also have found that shutting down the interfaces and reloading the network card modules on the firewall are required to get this to go into effect 100% of the timeeither that or a reboot.
Does your IPSEC client have a NAT traversal setting? Most of them do, as well as other encapsulation options. Enable these options first. If the IPSEC client requires these options to traverse NAT safely, no amount of tinkering on the firewall itself is going to help you.
Is another IPSEC VPN running to Host-B successfully? We have had the situation pop up on a very basic iptables/netfilter configuration where multiple systems behind the firewall were connecting to the same remote VPN server. Connection tracking was not enabled. Instead, it was using a very basic output rule, so only the first system to connect would work. The right answer here is to enable connection tracking! We also discovered on a 2.4 kernel that just adding the rules after the VPN connection was up didn't resolve the problem; even flushing the rules and clearing the connection tracking table didn't help. In the end, a reboot solved the problem. Later it was determined that this was an issue with the IPSEC VPN client itself.

Example Rules:

 $IPTABLES -A INPUT  -m state --state ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A FORWARD -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

IPSEC: Firewall Cannot Establish IPSEC VPNs

Extended Description: You have a NAT/MASQUERADING firewall that is IPSEC capable (free/openswan). This firewall acts as an end-node, NAT/MASQUERADING traffic into the VPN itself so all connections inside the VPN, as well as to the external network, appear to be coming from the firewall itself. The VPN will either not connect or when connected won't route correctly.

Generally you find this kind of configuration where you do not or cannot allow the remote network access to your network. I use this configuration personally because I do not trust the hosts on the other end of the VPN to be allowed to directly communicate with my network. This is also an optimal Extranet configuration.

As mentioned earlier in the chapterand I cannot stress this enough with IPSECthe MTU settings will cause issues here. This is generally the case when the VPN itself cannot be established with the remote system. In cases where the VPNs themselves are established and communicating successfully, routing to the remote network is the trouble. The next thing to check are the NAT rules.

Figure 19.2. Firewall cannot establish IPSEC VPNs.

The great way to diagnose this type of problem is with a sniffer like ethereal. My method is to look at the ipsec interface itself, in the case of free/openswan this is ipsec0, ipsec1, ipsec2, and so on. The following output is from my home network through the firewall, minimoose, to a remote network showing a successful VPN tunnel:

 [root@minimoose root]# tethereal -i ipsec0 107.113060 10.10.12.1 -> 10.10.11.102 TCP 58844 > ssh [SYN] Seq=2062573597 Ack=0 Win=5840 Len=0 MSS=1460 TSV=565962409 TSER=0 WS=0 107.175181 10.10.11.102 -> 10.10.12.1 TCP ssh > 58844 [SYN, ACK] Seq=2910414569 Ack=2062573598 Win=5792 Len=0 MSS=1460 TSV=90414655 TSER=565962409 WS=0 107.175746 10.10.12.1 -> 10.10.11.102 TCP 58844 > ssh [ACK] Seq=2062573598 Ack=2910414570 Win=5840 Len=0 TSV=565962416 TSER=90414655

This configuration consists of an internal host called winona (IP 10.10.12.100), SSH-ing through the firewall minimoose (10.10.12.1 internal IP, dynamic IP on the external interface), to the remote system 10.10.11.102. Connections from the 10.10.12.0/24 network have been NAT-ed to appear to come from the host minimoose using the following rules:

 $IPTABLES -t nat -A POSTROUTING -o ipsec0  -s 10.10.12.0/24 -d 10.10.11.0/24 -j SNAT --to-source 10.10.12.1

We would use this configuration where the remote network is only expecting to see a single source IP address for accessing hosts on its internal network. Otherwise the remote firewall would discard, or not know how to route the 10.10.12.0/24 traffic back to your network. Here is an example of what that would look like:

 [root@minimoose root]# tethereal -i ipsec0 Capturing on ipsec0   0.000000 10.10.12.198 -> 10.10.11.102 TCP 58845 > ssh [SYN] Seq=2724808935 Ack=0 Win=5840 Len=0 MSS=1460 TSV=566028529 TSER=0 WS=0   0.000000 10.10.12.198 -> 10.10.11.102 TCP 58845 > ssh [SYN] Seq=2724808936 Ack=0 Win=5840 Len=0 MSS=1460 TSV=566028529 TSER=0 WS=0

Note that the host 10.10.11.102 isn't responding. This is due to the fact that 10.10.11.102 doesn't have a route back to the 10.10.12.0/24 network. Most likely it is responding out its default route. Another important element in this configuration to pay attention to is the -o flag, which in iptables designates the interface this rule is bound to. In multi-IPSEC configurations, this is an important rule to pay attention to, given that in general the default behavior is going to be to bind this to assume the external interface on the system. This would make the source for the packets be the external IP address for all your VPN traffic, thereby breaking your VPN connection.

IPSEC: Firewall Can Establish Connections to a Remote VPN Server, but Traffic Does not Route Correctly Inside the VPN

Extended Description: Two firewalls are configured to connect RFC 1918 networks to each other over a VPN. In the following example, the left side network 10.10.10.0/24 is attempting to be connected to the right side network 10.10.11.0/24 from the firewall, minimoose, to the firewall, jenner, over an IPSEC connection such as free/openswan.

This configuration is a true NAT to NAT VPN configuration that you would most likely be using in an environment where you are linking two corporate networks together, perhaps in the case of eliminating a legacy WAN connection.

As mentioned earlier in the chapter, not every IPSEC implementation is going to be compatible with every other. This is especially true when you get into the various authentication mechanisms used. We favor openswan (www.openswan.org), which for the most part handles all the standard authentication mechanisms. Assuming that the issue is with the initial connections themselves not working correctly, the MTU settings or an ISP policy that blocks IPSEC traffic could be the culprit. The best way to verify this is with a sniffer. If you can see your traffic outbound from one firewall but never see it arrive on the other, then the problem is most likely an upstream filter blocking your traffic. If the traffic arrives but is discarded, then the problem is likely MTU related.

Figure 19.3. Traffic does not route through VPN.

Assuming that the VPN itself can be established, but the traffic is not routing between networks, set up sniffers on both sides to capture traffic between systems. Typically the issues in this area are related to the source and/or destination of the packets traversing the VPN. Like the section here, the systems on each end will get confused as to where to send the return traffic. The following is a basic working example of the preceding configuration using freeswan/openswan to link both sides. Keep in mind that this is a basic configuration and that you would not want to run this as it is in a production environment. Specifically, the FORWARD NEW rule should be more specific, covering traffic only from your trusted network to whatever destinations you want the firewall to forward traffic for:

 #!/bin/sh IPTABLES=/sbin/iptables #Connection tracking rules $IPTABLES -A INPUT  -m state state ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A OUTPUT -m state state NEW,ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A FORWARD -m state state NEW,ESTABLISHED,RELATED -j ACCEPT # eth0 is our external dynamic interface, eth1 is our internal interface $IPTABLES -t nat -A POSTROUTING -o eth0 -j MASQUERADE # enable IP Forwarding echo 1 > /proc/sys/net/ipv4/ip_forward

This is a very simplistic example of the previous scenario. A major mistake here is where you bind the nat rulethe correct place to put it is on your external interface only. This way the address translation is limited to only the external interface and does not affect your other interfaces, ipsec0 or etH1. This allows the basic IP forwarding to handle routing between the two networks. Here's how the traffic looks in ethereal:

 [root@minimoose root]# tethereal -i ipsec0 Capturing on ipsec0   0.000000 10.10.10.198 -> 10.10.11.102 TCP 58845 > ssh [SYN] Seq=2724808935 Ack=0 Win=5840 Len=0 MSS=1460 TSV=566028529 TSER=0 WS=0   0.068717 10.10.11.102 -> 10.10.10.198 TCP ssh > 58845 [SYN, ACK] Seq=3602031957 Ack=2724808936 Win=5792 Len=0 MSS=1460 TSV=90480758 TSER=566028529 WS=0   0.069324 10.10.10.198 -> 10.10.11.102 TCP 58845 > ssh [ACK] Seq=2724808936 Ack=3602031958 Win=5840 Len=0 TSV=566028536 TSER=90480758   0.128028 10.10.11.102 -> 10.10.10.198 SSH Server Protocol: SSH-2.0-NOYB

Here is the same configuration but with the nat rule bound to all the interfaces using the following rule:

 $IPTABLES -t nat -A POSTROUTING -j MASQUERADE [root@minimoose root]# tethereal -i ipsec0 Capturing on ipsec0   0.000000 123.4.5.6 -> 10.10.11.102 TCP 58845 > ssh [SYN] Seq=2724808935 Ack=0 Win=5840 Len=0 MSS=1460 TSV=566028529 TSER=0 WS=0     0.069324 123.4.5.6 -> 10.10.11.102 TCP 58845 > ssh [ACK] Seq=2724808936 Ack=3602031958 Win=5840 Len=0 TSV=566028536 TSER=90480758

In this case, the 10.10.11.0/24 network sees traffic coming over the VPN that has the wrong source address, specifically the masqueraded external IP from the firewall. Assuming an "unless allow, deny" model on the other side, a properly configured network is going to DROP or REJECT those packets because they are not matching the allowed rules on the other side.

This leads to the next scenario, assuming that the connections are being established, the routing is correct, and traffic still is not traversing the VPN successfullyit could be a DROP/REJECT rule blocking the traffic in or outbound. The best way to rule out that type of rule is to first test the configuration with no filtering. If things work, then you know it's a filter rule. You'll have to go through rule by rule to rule out which ones are being problematic, either by commenting out each filter or adding in diagnostic logging so you can see exactly which rules are being triggered and where.

PPTP: Cannot Establish PPTP Connections Through the Firewall

Extended Description: You have an internal system that creates a Point to Point Tunneling Protocol (PPTP) VPN to another system, for example a corporate VPN server, through your firewall. Non-VPN traffic functions properly; however, the PPTP VPN connection cannot be established. In this "road warrior" configuration, the internal system (laptop in diagram here) only connects to the remote system as an end-node; it does not perform any routing.

Figure 19.4. Internal systems cannot establish PPTP VPNs through firewall.

VPNs, like PPTP, suffer from problems seen in both Layer 3 and Layer 4 -based VPNs, given that it lives in both worldsby using both a Layer 4 TCP control port, as well as a Layer 3 data protocol. This means that not only is a PPTP connection sensitive to MTU settings, it is also going to require a special connection tracking module as well as paying attention to filter rules on the Layer 4 traffic.

PPTP traffic uses protocol 47 (GRE), TCP port 1723, and requires some additional patches to be added into the Linux kernel to traverse NAT firewalls. Specifically, the kernel has to be patched with the pptp-conntrak-nat.patch from patch-o-matic and have the following options enabled:

 Networking options <*>   IP: GRE tunnels over IP   [*]     IP: broadcast GRE over IP Networking options ->   -> IP: Netfilter Configuration   -> Connection tracking     <*>   GRE protocol support       <*>    PPTP protocol support

Without these modules and settings built into the kernel, PPTP connections will not work through the firewall. Furthermore, these are not set in default Redhat kernels. So if you are running a Redhat box and have not specifically rebuilt your system to support these options, then no amount of tinkering on the firewall is going to get this to work!

Just like IPSEC, PPTP is very sensitive to the MTU settings between endpoints. This is due to the fact that PPTP really uses PPP (Point to Point Protocol) inside the PPTP encapsulated session to handle the real network traffic. Because of the overhead involved in the encapsulation, you'll need to keep an eye on both the firewall and your client's MTU. In general, if you're having connection or performance issues, try setting both systems to the same MTU.

The next step is to verify that you can actually connect to the remote PPTP system. This is such a well-known service, ISPs with a no-VPN policy will likely block this PPTP connection. Verify with your ISPs that they allow this type of traffic in advance. If you have the capability to set up sniffers on both ends of the PPTP connection, verify that both endpoints are communicating both the TCP and protocol 47 traffic correctly. PPTP needs both!

Finally, ensure that your firewall is allowing protocol 47 (GRE) traffic outbound. The following example rules are from the pptpclient website (http://pptpclient.sourceforge.net/howto-diagnosis.phtml). Document the proper firewall rules to work with PPTP:

 # eth0 is our external interface $IPTABLES --insert OUTPUT 1 \ --source 0.0.0.0/0.0.0.0 \ --destination 0.0.0.0/0.0.0.0 \ --jump ACCEPT --protocol gre \ --out-interface eth0 $IPTABLES --insert INPUT 1 \ --source 0.0.0.0/0.0.0.0 \ --destination 0.0.0.0/0.0.0.0 \ --jump ACCEPT --protocol gre \ --in-interface eth0