Things to Consider with IPSEC

Having spent far too much time attempting to get the overly elaborate IPSEC VPNs working between our houses, businesses, and business partners (one of us has four separate VPNsthe other has five!), we can safely say...we hate IPSEC. Setting aside the firewall issues for a moment, IPSEC is less a standard and more a compilation of "shoulds." "Should" might even be too strong of a word...it's more like a lot of "yeah, kindas." Needless to say, before blaming your firewall for your VPN woes, it's important to dive into some of the very common issues with getting IPSEC to work.

First, IPSEC implementations are not always compatible. Even though IPSEC is supposed to be a standard, that does not mean that the implementations out there have to play nice with one another. Because of this, we tend to favor openswan (for the time being) for IPSEC VPNs. Even though 2.6 Linux kernels have native IPSEC support, we still prefer openswan because, for the most part, it just "works" with most of the IPSEC implementations out there. It's also relatively easy to set up on both 2.4 and 2.6 Linux systems, and standardizing it is a good thing. Further, openswan does use the native IPSEC support in 2.6 kernels; it just abstracts it away so you can use openswan configurations from 2.4 systems without any changes.

Nevertheless, you will still find that even something as old and robust as openswan might not work with the VPNs you need to support. For instance, one of our laptops has openswan, the Cisco IPSEC package, and (given that it doesn't work with Redhat 9) Checkpoints VPN-1 IPSEC client running in a Redhat 7.3 Vmware (www.vmware.com) image. The main issue we've run into on the compatibility front is that they use different authentication mechanismsso always check your documentation first! Also some VPN software packages need to be configured to be NAT-aware, so to speak. They work perfectly outside of a firewall and not at all behind it. By default, IPSEC wasn't designed for NAT-ing, so more often than not, it won't work behind a firewall unless the implementation has been extended to work in NAT environments.

Network hardware matters! Specifically with IPSEC, you're going to have to worry about the Maximum Transmission Unit (MTU) size with DSL connections, some wireless cellular connections, or if you're doing any other tunneling such as Point to Point Protocol over Ethernet (PPPoE). We've found that you have to play with the MTU settings starting around 1470, dropping as low as 1300. We've even run into some cellular wireless services that required us to go even lower. It depends on your configuration, IPSEC implementation, and networking environment upstream. This is a critical issue with IPSECMTUs really do matter. If the VPN isn't working, check the MTU and try lowering. We recommend 1300 as a good starting point.

Another important step is to bind it to the right interface. This is more of a reminder of an embarrassing mistake made on another system and a warning on not to over-think the problem. Check the simple things firstlike the network card...