Section 4-5. Firewall Reloads and Crashes

4-5. Firewall Reloads and Crashes

During a firewall's normal operation, you might find an occasion to manually reboot or reload it. As well, the firewall might crash on its own, because of some unexpected software or hardware problem, and possibly reload itself. The following sections discuss the methods you can use to make a firewall reload and diagnose the cause of a crash.

Reloading a Firewall

To manually trigger a firewall reload, choose one of the options discussed in the following sections. You can initiate a firewall reload only from privileged EXEC (enable) mode. On a PIX 7.x or FWSM firewall platform running in multiple-context security mode, you can initiate a reload only from the system execution space.

Reloading a Firewall Immediately

You can use the following command to initiate an immediate reload. Be aware that as soon as the reload begins, all existing connections through the firewall are dropped, and traffic cannot pass through for a minute or more. Reloads should be initiated only during a network maintenance window.

FWSM 2.x	Firewall# reload [noconfirm]
PIX 6.x	Firewall# reload [noconfirm]
PIX 7.x	Firewall# reload [noconfirm] [max-hold-time {minutes | hhh:mm}] [quick] [save-config] [reason text]

With FWSM 2.x and PIX 6.x, a manual reload occurs immediately. The firewall prompts you for a confirmation of the reload and to save the running configuration if it differs from the startup configuration. To answer the confirmations, you can enter y or just press the Enter key. You can avoid all the confirmation prompts by using the noconfirm keyword.

PIX 7.x adds more flexibility to the reload process. By default, the firewall informs all the administrative users with active sessions (console, Telnet, and SSH) of the impending reload. It also attempts to shut down each of its software modules (IPSec VPN remote access, for example) before the reload. This allows every reload to happen gracefully, without catching users and functions off-guard.

You can use one or more of these keywords with the reload command to change the reload behavior:

• max-hold-time You can limit how long the firewall waits to inform users and shut down security functions. Specify the number of minutes to wait (1 to 33960) or as hours and minutes hhh:mm (up to 566 hours 0 minutes). After that maximum hold time expires, the firewall is free to abruptly perform the reload.

• quick Perform an abrupt reload without informing users or shutting down processes.

• save-config Force the firewall to save the running configuration to the nonvolatile startup configurationeven if it has already been previously saved.

• reason Provides a descriptive reason for the reload as text (an arbitrary string of up to 255 characters). The reason text is displayed to any active administrative users connected with console, Telnet, or SSH sessions.

Reloading a Firewall at a Specific Time and Date

Beginning with PIX 7.0, you can schedule a firewall reload at some specific time in the future. This might be handy if reloads are permitted only during a predetermined maintenance window. The command for reloading a firewall at a specific time and date is as follows:

FWSM 2.x
PIX 6.x
PIX 7.x	Firewall# reload at hh:mm [month day | day month] [max-hold-time {minutes | hhh:mm}] [noconfirm] [quick] [save-config] [reason text]

A firewall can schedule a reload at a specific hour and minute, given as hh:mm. You can also specify a month and day (1 to 31). The month can be abbreviated as long as it denotes a unique month name. If no date is given, the reload time is assumed to be within the next 24 hours. Also, the date and time must be within 576 hours 0 minutes (or 24 days) from the time the command is issued.

The remaining reload keywords are optional and are discussed in the preceding section.

Before using this command, be certain that the firewall clock is set and is accurate. You should also confirm that the reload time is based on the same time zone information that the firewall is using. Otherwise, your concept of the reload schedule might be quite different from the firewall's concept! You can view the current firewall date and time with the show clock command.

If needed, you can set the date and time with the following command:

 Firewall# clock set hh:mm:ss {month day | day month} year 

Refer to section 9-1, "Managing the Firewall Clock," in Chapter 9 for complete information about setting and maintaining the clock.

Reloading a Firewall After a Time Interval

You can schedule a firewall reload after a specified amount of time has elapsed. This might be useful if the reload should be delayed from the present time, to allow users a "grace period" to complete their work. The command for reloading a firewall after a time interval is as follows:

FWSM 2.x
PIX 6.x
PIX 7.x	Firewall# reload in {minutes | hh:mm} [max-hold-time {minutes | hhh:mm}] [noconfirm] [quick] [save-config] [reason text]

You can schedule a reload at some specific delay after the current time. The delay is given as minutes (1 to 34560) or as hours and minutes as hhh:mm. In either case, the delay limit is 576 hours 0 minutes from the current time.

The remaining reload keywords are optional and are discussed in the section "Reloading a Firewall Immediately."

TIP

In PIX 7.x, you can display a scheduled reload that is pending with the show reload command. If you need to cancel the reload ahead of time, use the reload cancel command. Obviously, as soon as the actual reload has begun, there is no way to cancel it.

The following example shows how a reload is scheduled at a certain date and time and how that reload is later canceled.

 Firewall# show clock 16:10:01.743 EST Wed Dec 15 2004 Firewall# reload at 23:50 dec 18 save-config reason Firewall is being   reloaded for an OS Image upgrade Cryptochecksum: f96855f6 676cf300 d7f31ae8 bd4cbdcf 3250 bytes copied in 0.530 secs Proceed with reload? [confirm] Firewall# *** *** --- SHUTDOWN in 79:38:41 --- *** *** Message to all terminals: *** ***   Firewall is being reloaded for an OS Image upgrade Firewall# Firewall# show reload Reload scheduled for 23:50:00 EST Sat Dec 18 2004 (in 79 hours and 38   minutes) by enable_15 from ssh (remote 172.21.6.1) Reload reason: Firewall is being reloaded for an OS Image upgrade Firewall# Firewall# reload cancel Firewall# *** *** --- SHUTDOWN ABORTED --- Firewall# Firewall# show reload No reload is scheduled. Firewall# 

When a reload has been successfully scheduled, the firewall also generates Syslog message ID 199007. If a reload is canceled, a Syslog message 199008 is generated. Both messages are default severity level 5.

Obtaining Crash Information

If a firewall crashes for some reason, you might be left with few clues other than it reloaded or was left at the monitor prompt. You can configure the firewall to save a crashinfo image of its RAM memory as a file in Flash memory. The image also includes the output of many commands that show the status of the firewall and its resources. A firewall stores only one crashinfo file; subsequent crashes overwrite it. Therefore, only information from the most recent crash is available.

After a crash, you can examine the crashinfo file to determine the cause of the crasheven after a power cycle or reload. This information is useful to the engineers at the Cisco Technical Assistance Center (TAC) as they track down the source of a firewall problem.

The following sections discuss preparing for and examining a crashinfo image.

Enabling Crashinfo Creation

You can use the following configuration command to automatically save a crashinfo file during a firewall crash:

FWSM 2.x
PIX 6.x	Firewall(config)# crashinfo save enable
PIX 7.x	Firewall(config)# no crashinfo save disable

By default, crashinfo collection is enabled on all firewall and security appliance platforms. If you decide to disable crashinfo collection, you can use the following command:

FWSM 2.x
PIX 6.x	Firewall(config)# crashinfo save disable
PIX 7.x	Firewall(config)# crashinfo save disable

In PIX releases, you can see the current status of crashinfo creation by using the show crashinfo save command. On an FWSM, use the show crashdump save keyword.

When a PIX 6.x platform crashes, an image of the firewall RAM is saved in Flash memory as file number 4. The size of the image equals the amount of RAM in the firewall.

In PIX 7.x, the crashinfo file is created and saved in a hidden Flash file system. The actual file can't be seen in a directory listing, nor can its location be specified or changed.

Generating a Test Crashinfo Image

To create a test version of a crashinfo file and store it in Flash memory, enter the following command:

FWSM 2.x
PIX 6.x	Firewall(config)# crashinfo test
PIX 7.x	Firewall# crashinfo test

The firewall operation is not hindered or interrupted by entering this command. This command can be used if you just want to get a feel for the crashinfo process and the crashinfo file contents. (Notice that this is a configuration command for PIX 6.x and a privileged EXEC command for PIX 7.x.)

Forcing an Actual Firewall Crash

You can make the firewall produce an error that actually causes itself to crash and to save a legitimate crashinfo image file by entering the following command:

FWSM 2.x	Firewall(config)# crashdump force {page-fault| watchdog}
PIX 6.x	Firewall(config)# crashinfo force {page-fault| watchdog}
PIX 7.x	Firewall# crashinfo force {page-fault| watchdog}

Either a page fault or a watchdog fatal error can be used to cause the crash. A page-fault crash occurs when a firewall process tries to access memory that is outside its own use. A watchdog crash occurs because a firewall process has become unresponsive or hung. In practice, the type of crash doesn't really matter. The goal is to confirm the process of saving a crashinfo file when the firewall actually crashes.

Notice that this is a configuration command for FWSM and PIX 6.x and is a privileged EXEC command for PIX 7.x.

CAUTION

This command is useful only for testing the crashinfo procedure. Because it actually crashes the firewall when it is used, you should use this command only under controlled circumstances. As soon as the firewall crashes, no traffic is inspected or passes through it.

In the following example, a watchdog timeout error is used to trigger an actual firewall crash:

 Firewall# config terminal Firewall# crashinfo force watchdog WARNING: This command will force the PIX to crash and reboot. Do you wish to   proceed? [confirm]: Watchdog timeout failure! Please contact customer support.     PC       SP       STATE       Runtime    SBASE     Stack Process Hrd 001eaa09 0090edc4 00555848          0 0090de3c 3628/4096 arp_timer Lsi 001effad 009d1fec 00555860          0 009d1074 3928/4096 FragDBGC Lwe 00119abf 00a5555c 00558fc0          0 00a546f4 3688/4096 dbgtrace Lwe 003e3f55 00a576ec 0054e188          0 00a557a4 7788/8192 Logger Hwe 003e80d0 00a5a7e4 0054e438          0 00a5886c 8024/8192 tcp_fast Hwe 003e8049 00a5c894 0054e438          0 00a5a91c 8024/8192 tcp_slow Lsi 003006f9 0293c984 00555860          0 0293b9fc 3944/4096 xlate clean Lsi 00300607 0293da24 00555860          0 0293caac 3888/4096 uxlate clean [output omitted] Rebooting.... 

Viewing the Crashinfo Information

After the crashinfo file is written, it can't be uploaded from the firewall. Instead, you can view its contents when you enter the following command:

FWSM 2.x	Firewall# show crashdump
PIX 6.x	Firewall# show crashinfo
PIX 7.x	Firewall# show crashinfo

The show crashinfo command (or show crashdump for FWSM 2.x) displays all the relevant process and stack trace information that was active during the crash. You should capture all this text output with a terminal emulator so that it can be saved to a file and sent to Cisco TAC.

The first line of output also tells what triggered the crashinfo image. If the line is ":Saved Test Crash," the image was saved by the crashinfo test command. Otherwise, ":Saved Crash" indicates that the file is the result of an actual firewall crash, as in the following example:

 Firewall# show crashinfo : Saved_Crash Thread Name: ci/console (Old pc 0x0011f217 ebp 0x02fb2d50) Traceback: 0: 00104bf8 1: 00102155 2: 0010014a 3: 00338b55 4: 00338c1b [output omitted] 

Deleting the Previous Crashinfo File Contents

Each time a crashinfo file is created, its contents replace the previous file contents. Therefore, only one crashinfo file is stored at any given time, and Flash memory is not used up after multiple crashes.

You can, however, clear the contents of the current crashinfo file with the following command:

FWSM 2.x	Firewall(config)# clear crashdump
PIX 6.x	Firewall(config)# clear crashinfo
PIX 7.x	Firewall# clear crashinfo

This might be useful if your firewall has experienced a crash in the past and you have already resolved the cause of that crash. In that case, you might want an empty crashinfo file to signify that no further crashes have occurred.