16.3 Core Dumps, Assertions, and Stack Traces

16.3 Core Dumps, Assertions, and Stack Traces

If you are unlucky, Squid may experience a fatal error while running. These sorts of errors come in three flavors: assertions, bus errors, and segmentation violations.

An assertion is a sanity check in the source code. It is a tool, used by developers, to make sure that some condition is always true before proceeding. If the condition is false, the program exits and creates a core file so that the developer can analyze the situation. Here is a typical example:

 int some_array[100]; void some_func(int idx) {         ...         assert(idx < 100);         some_array[idx]++;         ... } 

Here, the assertion makes sure that the value of the array index is within the bounds of the array. It would be an error to access array elements greater than (or equal to) 100. If, somehow, the value of idx isn't less than 100, the program prints a message like this when it runs:

 assertion failed: filename.c:123: "idx < 100" 

If this happens with Squid, you'll see an "assertion failed" message in cache.log . In addition, your operating system should create a core file, which is helpful in the post-mortem analysis. I'll explain what to do with a core file at the end of this section.

A bus error is "a fatal failure in the execution of a machine language instruction resulting from the processor detecting an anomalous condition on its bus." ^[1] They typically occur when the processor attempts an operation on a nonaligned memory address. You are, perhaps, more likely to see a bus error on a 64-bit processor system, such as the Alpha and some SPARC CPUs. Fortunately, they are easy to fix.

^[1] From the Free On-line Dictionary of Computing (FOLDOC), http:// wombat .doc.ic.ac.uk/foldoc/.

Segmentation violation errors are, unfortunately , more common and sometimes harder to fix. A "SEGV" usually occurs when the process tries to access an invalid memory area. It might be a NULL pointer or a memory address outside the scope of the process. They are particularly difficult to track down when the cause (the bug) and effect (the SEGV) are separated in time.

By default, Squid traps bus errors and segmentation violations, and attempts a clean shutdown when they occur. You'll see something like this in cache.log :

 FATAL: Received Bus Error...dying. 2003/09/29 23:18:01 storeDirWriteCleanLogs: Starting... 

In most cases, Squid is able to write clean versions of the swap.state files. Just before exiting, Squid calls abort( ) to create a core file. The core file may help you, or other developers, track down and fix the bug.

A core file is generally more useful when it is created immediately following the error, rather than calling the clean shutdown procedure first. You can tell Squid not to trap bus errors and segmentation violations with the -C command line option:

 % squid -C ... 

Note that some operating systems use the filename core , while others prepend the process name (i.e., squid.core ). Once you have the core file, use a debugger to get a stack trace. gdb is the GNU debugger ”a companion to the GNU C compiler. If you don't have gdb , try running dbx or adb instead. Here's how you can use gdb to get a stack trace:

 % gdb /usr/local/squid/sbin/squid /path/to/squid.core ... Core was generated by 'squid'. Program terminated with signal 6, Abort trap. ... 

Then, type where to print the stack trace:

 (gdb) where #0  0x28168b54 in kill ( ) from /usr/lib/libc.so.4 #1  0x281aa0ce in abort ( ) from /usr/lib/libc.so.4 #2  0x80a2316 in death (sig=10) at tools.c:301 #3  0xbfbfffac in ?? ( ) #4  0x80abe0a in storeDiskdSend (mtype=4, sd=0x82101e0, id=1214000,     sio=0x9e90a10, size=4096, offset=-1, shm_offset=0)     at diskd/store_io_diskd.c:485 #5  0x80ab726 in storeDiskdWrite (SD=0x82101e0, sio=0x9e90a10,     buf=0x13e94000 "...", size=4096, offset=-1, free_func=0)     at diskd/store_io_diskd.c:251 #6  0x809d2fb in storeWrite (sio=0x9e90a10, buf=0x13e94000 "...",     size=4096, offset=-1, free_func=0) at store_io.c:89 #7  0x80a1c2d in storeSwapOut (e=0xc5a7800) at store_swapout.c:259 #8  0x809b667 in storeAppend (e=0xc5a7800, buf=0x810f9a0 "...", len=57344)     at store.c:533 #9  0x807873b in httpReadReply (fd=134, data=0xc343590) at http.c:642 #10 0x806492f in comm_poll (msec=10) at comm_select.c:445 #11 0x8084404 in main (argc=2, argv=0xbfbffa8c) at main.c:742 #12 0x804a465 in _start ( ) 

As you can see, the stack trace prints the name of each function, its arguments, and the source code filenames and line numbers . This information is extremely useful when tracking down bugs . In some cases, however, it isn't sufficient. You might be asked to execute additional commands in the debugger, such as printing the value of a variable from within a certain function:

 (gdb) frame 4 #4  0x80abe0a in storeDiskdSend (mtype=4, sd=0x82101e0, id=1214000,     sio=0x9e90a10, size=4096, offset=-1, shm_offset=0)     at diskd/store_io_diskd.c:485 485         x = msgsnd(diskdinfo->smsgid, &M,  msg_snd_rcv_sz, IPC_NOWAIT); (gdb) set print pretty (gdb) print M  = {   mtype = 4,   id = 1214000,   seq_no = 7203103,   callback_data = 0x9e90a10,   size = 4096,   offset = -1,   status = -1,   shm_offset = 0 } 

After you've reported a bug, try to keep the core file around for a few days, in case you need additional information from it.

16.3.1 Can't Find the Core File?

core files are written in the process' current directory. By default, Squid doesn't change its current directory at startup. Thus, your core file, if any, should be written in the directory in which Squid was started. You won't find a core file if the filesystem doesn't have enough free space or if the process owner doesn't have write permission in the directory. You can use the coredump_dir directive to make Squid use a specific location ”somewhere with plenty of space and sufficient permissions.

Process resource limits may also prevent the creation of a core file. One of the process limit parameters is the size of the core dump file. Usually, most systems set this to "unlimited" by default. You can check the current limit from your shell with the limits or ulimit commands. Note, however, that your shell's limit might be different than the Squid process limit, especially when Squid is started automatically at boot time. If you suspect process limits prevent generation of a core file, try this:

 csh% limit coredumpsize unlimited csh% squid -NCd1 

On FreeBSD, a sysctl parameter controls whether or not the operating system generates a core file for processes that call setuid( ) and/or setgid( ) . Squid uses those functions if you start it as root . To get a core dump, then, you must tell the kernel to create the core file with this command:

 # sysctl kern.sugid_coredump=1 

See the sysctl.conf manpage for information on how to set the variable automatically when your system boots.