8.6. Handling Failure

Firewalls, like any other computer application, can fail. The cost of failure can vary depending on the role the firewall is serving. The failure of a firewall used in a SOHO environment can mean lost productivity for a handful of users. The failure of a firewall that protects an e-commerce web site can mean thousands of dollars in lost revenue.

For a number of years, commercial routers and firewalls have had the capability to provide high availability in the face of device failure. Various vendors support various architectures as described earlier in the Section 8.1.6 section. For those who chose to use open source firewalls, however, the options have been extremely limited. Most of the solutions we authors have seen were based on homemade scripts and cron jobs. Thankfully options are emerging in the FreeBSD and OpenBSD arenas that begin to give network engineers high availability options using the firewalls they know and love.

In order to have a highly available firewall cluster, two services must be provided. First, the firewalls need to have the ability to share an IP and MAC address between multiple hosts. This link and network layer redundancy ensures that the failure of a firewall is handled transparently on the local network. The other service required is to synchronize state information between firewalls to keep active sessions alive even when one of the firewalls fails. This makes firewall failures transparent to the hosts that are communicating end-to-end through the firewalls. Without synchronization, all active sessions through the firewall would need to be reestablished.

8.6.1. CARP

Sharing IP and MAC addresses is key to providing seamless access to hosts directly connected to the firewall. Sharing the IP address is relatively easy. However, sharing the MAC address is the hard part. If each firewall in the cluster used its own MAC address for a shared IP address, failure of the primary firewall would cause problems for directly connected hosts. These hosts would have to expire their ARP entry for the old IP/MAC address pair and acquire the new pair via a new ARP request. This expiration and renewal process can take an extended period of time (on the order of minutes), which is generally not acceptable in a highly available architecture.

The solution is to create a virtual MAC address that is shared between multiple firewalls. Cisco originally developed and patented the Hot Standby Router Protocol (HSRP) to accomplish this feat on their routers. After some time, the IETF worked on creating the Virtual Routing Redundancy Protocol (VRRP) to do basically the same thing. In fact, VRRP was so similar that Cisco and the IETF became embroiled in a patent dispute that eventually led to developers having to pay a license fee to Cisco if they wished to use VRRP. For the open source development community, this meant that the current standard for sharing an IP/MAC address pair was unusable.

The Common Address Redundancy Protocol (CARP) is a patent-free mechanism for sharing an IP/MAC address pair and handling failure between devices in a cluster. It originally was released in OpenBSD 3.5 and has since been ported to FreeBSD. CARP is substantially different from VRRP and should prevent it from being the target of patent disputes. CARP allows the open source community to finally begin to create real clustering capabilities, not just for firewalls but any clustered service such as web or mail servers.

8.6.2. CARP Configuration

CARP is included in the default OpenBSD distribution. However, as of this writing, there is only a placeholder in the FreeBSD source tree for CARP. To obtain CARP functionality in FreeBSD, you must apply the patch at http://people.freebsd.org/~mlaier/CARP/ and follow the directions in the patch for installation.

The configuration of CARP is basically the same for OpenBSD and FreeBSD. In this example, we will provide configuration guidance for a hot-standby architecture on OpenBSD. However, these commands and configurations are similar in intent in FreeBSD. This example is based on the example network topology in Figure 8-9.

Figure 8-9. Example CARP network

First ensure CARP is enabled. By default, net.inet.carp.allow should be enabled, however it doesn't hurt to check to prevent troubleshooting problems later. Also, you will need to enable net.inet.carp.preempt. This variable allows you to force one host, if it is online, to always be primary in the Hot-Standby architecture. Without preemption, the first CARP host that is put on the network is the master. From an administration standpoint, this is not ideal as you are never sure why one host is master and not the other. With preemption, if the secondary host is the CARP master, than you know the primary firewall has failed.

CARP interfaces are treated similarly to any other network interface, meaning they can be controlled via hostname files or ifconfig(8). The IP address that is shared between two hosts must be in the same IP subnet as the IP address on that interface. For instance, the 192.168.0.1 address is shared on the 192.168.0.0/24 network in this example. Two CARP interfaces need to be created; one for the front and one for the back of the cluster. /etc/hostname.carp0 on the primary firewall should have the following:

inet 192.168.0.1  255.255.255.0 192.168.0.255 vhid 1 pass password

While /etc/hostname.carp1 should have:

inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 2 pass password

The vhid variable is the virtual host ID. It must be the same on both CARP hosts for a given shared IP address. The pass variable is a password used to validate the CARP traffic between the hosts. Unlike HSRP where passwords are sent in the clear, CARP uses SHA-1 to hash the update with the password to ensure the integrity of the updates.

On the secondary firewall, /etc/hostname.carp0 should look like this:

inet 192.168.0.1  255.255.255.0 192.168.0.255 vhid 1 advskew 100 pass password

The advskew variable modifies the CARP announcements from the secondary host to ensure that if the primary host is online, it preempts the secondary. /ect/hostname.carp1 should also have an advskew value of 100.

The firewalls on both hosts must be configured to allow CARP traffic between them. CARP uses a unique IP protocol identifier, 112, which pf knows by the carp keyword. CARP is also state based so you can configure the firewall to keep state for greater efficiency.

pass on { interface1 interface 2 } proto carp keep state

Upon rebooting, the systems will share 192.168.0.1 and 10.1.1.1. At this point a host will only take over when the primary host stops sending CARP traffic on all interfaces. Hopefully in the future, CARP will allow failover to occur if any interfaces go down.

8.6.3. pfsync

Once a firewall cluster is sharing IP addresses, there is still the issue of transferring state information between the two machines. If one firewall fails over to the other without sharing state information, any existing sessions transiting the firewall cluster will be dropped. This is not ideal.

OpenBSD has the capability through pfsync to exchange state information between firewalls. While there is no native pfsync-like capability in IPFW, pfsync is available if you use the FreeBSD PF port rather than IPFW.

pfsync passes state information between multiple firewalls. When a state is created, updated, or removed on one firewall, pfsync will notify the other firewall. If one firewall fails, the other will be able to seamlessly handle traffic that was in state on the primary firewall.

pfsync traffic sent between firewalls is not authenticated. Therefore, it is advisable to create a private network, such as a cross-over cable, between the two hosts to prevent attackers from injecting pfsync traffic into the firewalls. This has the added advantage of not burdening an existing network interface with the extra load of pfsync traffic, which can be quite large on busy firewalls.

On both hosts, create an /etc/hostname.pfsync0 file with the following:

up syncif interface

The interface should be whatever interface you want the pfsync data to go across. pfsync uses its own IP identifier (240) for traffic between the firewalls. pf knows this IP ID as the keyword pfsync. Add the following to your firewall rules.

pass quick on { interface } proto pfsync

Reboot the machine to get all the changes to take effect. Now, with CARP and pfsync, your firewall cluster is running in a Hot-Standby architecture. You should test the firewall cluster by shutting down the primary host and verifying that the secondary host takes over.