Crash 11: A closer look

We will start with the stack traceback and print out the inode involved. Using the owner field of the inode structure (the process currently handling the inode, not the user who owns the inode), we will find out what process is at work. We will confirm the process by looking at the variable masterprocp , which is a pointer to the current running process.

 Hiya on zen...  adb -k vmunix.11 vmcore.11  physmem 1ff8  $c  _panic(0xf8174abb,0xf817e784,0xf817e798,0x283,0xa000,0xf84c6b98) + 6c  _assfail(0xf817e784,0xf817e798,0x283,0xfcfdbdb0,0x1,0xf83817a8) + 3c  _pvn_vplist_dirty(0xfcfdbdb0,0x0,0x0,0xf83817a8,0xf8359758,0xfcfdbdb0) + 250  _ufs_putpage(0xfcfdbdb0,0xfcfdbda8,0x0,0x0,0x0,0xf82f43d8) + 33c  _ufs_l_putpage(0xfcfdbdb0,0x0,0x0,0x0,0x0,0x0) + 30  _syncip(  0xfcfdbda8  ,0x0,0x0,0x7,0x1101,0x0) + 124  _update(0xf81a7900,0x104e,0xfcfdbda8,0xfcf1e808,0x0,0xfcfdbda8) + 2f0  _ufs_sync(0x0,0xf8175ac8,0xf817d820,0xf81a5c48,0xf8175ac8,0xfce792c4) + 4  _sync(0xf84c6fe0,0x120,0xf8173468,0xf8173588,0xf84c7000,0xf8175a88) + 3c  _syscall(0xf84c7000) + 3b4  0xfcfdbda8$<inode  0xfcfdbda8:     forw            back                  fcfdbbe8        f81a7900  0xfcfdbdb0:     flag    refct   shlockc exlockc                  0       8       0       0  0xfcfdbdb8:     vfsmnt          vop             pages           vfsp                  0               f817d8b8        f83723c0        fce4f658  0xfcfdbdc8:     type            rdev            data                  2               01      0250    fcfdbda8  0xfcfdbdd8:     devvp           flag    maj     min     ino                  fce51b94        111  026     0204    28682  0xfcfdbde8:     fs              dquot           owner           count                  fde66000        0  f82f43d8  2  0xfcfdbdf8:     nextr           freef           freeb                  24576           0               0  0xfcfdbe04:     delayoff        delaylen        nextrio         writes                  0               0               c000            0  0xfcfdbe04:     mode    links   uid     gid     size                  040700  2       1889    17      162816  0xfcfdbe14:  atime 1994 Jul 23 03:31:08  mtime 1994 May 26 12:03:15                  ctime 1994 May 26 12:03:15  0xfcfdbe28:     addresses  0xfcfdbe2c:     101a8           10300           10458           10638                  10708           10860           109b8           10b10                  10ca0           10dc0           10f18           110f0                  1e4f8           0               0  0xfcfdbe68:     flags           blocks          gen                  0               336             1967826125  *time=Y   1994 Jul 23 03:31:14   f82f43d8$<proc  0xf82f43d8:     link            rlink           nxt             prev                  f8198b90        0               f82f4260        f82f5aa4  0xf82f43e8:     as              segu            stack           uarea                  fce6c6e8        f84c4000        f84c6f58  f84c7000  0xf82f43f8:     upri    pri     cpu     stat    time    nice                  071     01      037     03      06      024  0xf82f43fe:     slp     cursig  sig                  0       0       0  0xf82f4404:     mask            ignore          catch                  2000            300001          2000  0xf82f4410:     flag            uid     suid    pgrp                  80001           0       0       52  0xf82f441c:     pid     ppid    xstat   ticks  186  1       0       10  0xf82f4424:     cred            ru              tsize                  fce12780        0               0  (Output trimmed)   !ps axk vmunix.11 vmcore.11  grep 186  186 ? R   586:57 update  !  f84c7000$<u  0xf84c7000:     pc              sp              psr             uwm                  f80532d0        f84c6af0        11900ae2        0  0xf84c7010:  0xf84c7358:     flags                  80000000  0xf84c7364:     procp           ar0             comm                  f82f43d8        f84c6fb4  update^@^@^@^@  0xf84c7380:     arg0            arg1            arg2                  f7fffff3        f7fffff0        0  0xf84c73a0:     uap             qsave                           error                  f84c6fe0        f80e918c        f84c6ec0        0  0xf84c73ac:     rv1             rv2             eosys                  0               0               0  0xf84c73c0:     signal                  0               1               0               0                  0               0               0               0                  0               0               0               0  (Output trimmed)   masterprocp/  X   _masterprocp:   _masterprocp:   f82f43d8 

The update program was trying to update (sync) inode 28682 on the device with major and minor numbers of 026 and 204.

Checking quickly, let's make sure that the macro we used, /usr/lib/adb/inode , isn't reporting the major and minor numbers as octal values.

 Hiya on zen...  cd /usr/lib/adb  Hiya on zen...  more inode  ./"forw"16t"back"n2X  +$<<vnode  +/"devvp"16t"flag"8t"maj"8t"min"8t"ino"nXx  2b  D4+  +/"fs"16t"dquot"16t"owner"16t"count"n4X  +/"nextr"16t"freef"16t"freeb"nD2X  +$<<dino  .,<9-1$<inode  Hiya on zen... 

Good thing we checked! So the reported major/minor of 26/204 is really, in decimal, major 22 and minor 132. What device is that on Zen?

 Hiya on zen...  cd /dev  Hiya on zen...  ls -l  grep " 132 "  brw-r-----  1 root      22, 132 Dec  9  1992 id010e  crw-r-----  1 root      72, 132 Jan 12 12:15 rid010e  crw-r-----  1 root      17, 132 Dec  9  1992 rsd16e  brw-r-----  1 root       7, 132 Dec  9  1992 sd16e  Hiya on zen... 

What caused a change to inode 28682 which required it to be updated to the disk at nearly 3 a.m.? The find command! find looks at each inode. By nature of UFS, referencing an inode for any reason results in updating the inode's "last access time" field. Thus, the inode is "dirty." In other words, the inode is modified and needs to be updated on disk. Compare the last access time in the inode structure and the crash time. They are within seconds of each other.

As you can see, running a find command on a UNIX system actually results in a lot of disk activity.

Let's whip through the other crashes and see if the same inode and process are at work.