Failover Detection

The PIX can detect several types of failovers. One mechanism it uses is the hello message. This message is sent every 3 “15 seconds out every interface to test communication. The default is 15 seconds, but it can be changed with the firewall poll command.

If a firewall unit doesn't see a hello message in two updates (30 seconds), both firewalls start to initiate failover tests to determine and confirm which of the firewalls has failed. If the primary is confirmed down, the standby moves into the active role; if the secondary firewall has failed, the primary continues to operate with no failover.

Make sure you understand that Hello messages are sent across all interfaces, including the serial or LAN-based cable. By default, hello messages are sent every 15 seconds, and if two messages are missed, the failover process begins.

Causes for Failovers

Failovers occur for many reasons. When a failover does occur, both firewalls work together to promote the standby firewall to the active state if possible. If the primary firewall detects an interface going down, it tells the secondary to move into the active state. On the other hand, the secondary promotes itself if it notices that the primary is offline. The following events cause failovers:

• The primary firewall is turned off or the power supply fails.

• The primary firewall is rebooted.

• An interface on the active firewall goes down or the serial interface cable fails.

• The primary firewall experiences a block memory exhaustion condition.

When using the serial cable as the failover link between the firewalls, power off detection can take place. If the primary firewall's power is turned off, the secondary firewall starts to promote itself to active state within 15 seconds. If a LAN-based cable is used, the power failure cannot be detected .

The Four Interface Tests

The PIX firewall issues four tests to determine whether the active firewall is truly faulty before promoting the secondary to active. As stated previously, hello messages are sent to detect interfaces on the opposite firewall. If two messages are missed, a series of tests is initiated to probe more deeply and help justify a failover. Table 11.3 explains the failover tests.

Table 11.3. Four Failover Tests

Test	Function
NIC status test	This tests the up/down status of the interface. If the link is down or unplugged, or the intermediate switch is turned off or plugged in to a switch-performing spanning tree, the interface is detected as a failure and the active firewall becomes the standby firewall. However, if the link is determined to be up, the test succeeds and the PIX searches more deeply during the second test to test for other possible problems that caused the missing hello messages.
Network activity	The PIX monitors the activity of the link for 5 seconds; if valid frames are detected, the failover testing is aborted. If no valid frames are detected, meaning the test failed, the PIX moves on to the third test.
ARP test	This test sends ARP requests to the last 10 IP addresses queued in the ARP table. If any response comes back, the testing is aborted and the firewall is considered operational. However, if no responses come back, the PIX moves to the fourth test.
Ping test	This is the last-chance test. A broadcast ping of 255.255.255.255 is sent, and if any device (host, router, and so on) replies, the test is considered a success and failover is aborted. However, if no requests come back, the failover to the standby takes place.

During the testing, if any valid frames are received from the other PIX, the testing is aborted and the systems are deemed operational. The results of each test are passed back and forth between the primary and secondary firewalls to determine which firewall is operational. For example, the primary might determine that the secondary interfaces are down and thus not promote the secondary firewall to the active state.

The network activity test monitors for traffic for 5 seconds. If no traffic is found, the PIX moves to the next test, instead of to standby mode.

Failed State

When a firewall is deemed as failed, it disables all its network interfaces. However, every 15 seconds the failed PIX tries to test all the interfaces and automatically moves into the standby state. If problems still exist, it fails again.

To manually move the failed firewall back into the standby state, the failover reset command can be issued. For example, if Jack unplugs an interface, the PIX moves into the failed state. After Jack plugs the interface back in, the PIX automatically moves into the standby state in 15 seconds, as long as everything else is functioning correctly. Or Jack could issue the failover reset command if he doesn't want to wait 15 seconds. If a problem still exists after the command has been issued, the PIX again moves into the failed state.