Walking the stack by hand | PANIC! UNIX System Crash Dump Analysis Handbook (Bk/CD-ROM)

We also see that three of the four CPUs were idle. The fourth was handling the panic. Let's confirm this via adb . This is a particularly interesting example, because the first thing we notice is that the $c output below is short. Occasionally, the $c output of adb will come up a bit short. If it doesn't look as if a system call or interrupt is at the bottom of the stack traceback, be suspicious! To make sure we aren't missing anything exciting, we walked the stack by hand, just to be on the safe side.
  $c  complete_panic(0x0,0xe18cfd1c,0xe18b1ec0,0xe18b1ec0,0xe18b1ee4,0xe00fdd28) + d0  do_panic(?) + 20  vcmn_err(0xe00dc000,0xe18cfd1c,0xe18cfd1c,0x73,0x73,0x3)  cmn_err(0x3,0xe00dc000,0x0,0x0,0x0,0x0) + 1c  e18d8ec0$<thread   <---   Using the thread address from the non-idle CPU structure seen earlier  adb  0xe18d8ec0:                  link            stk             stksize                  0               e18d8e60        1ec0  0xe18d8ecc:                  bound           affcnt          bind_cpu                  e00fd240        0               -1  0xe18d8ed4:                  flag            procflag        schedflag       state                  9               0               11              4  0xe18d8ee0:     pri             epri            pc              sp                  169             0               e000ffe8  e18d8cc8  0xe18d8eec:     wchan0          wchan           cid             clfuncs  .... remainder of thread output omitted ....   e18d8cc8/16X  0xe18d8cc8:     0               f               0               d                  1               e00d1800        e00fd240        1                  e00e7400        1               e00d1c00        e00fd240                  0               e00e7400  e18d8d30       e000fbf0   e000fbf0/i  do_panic+0x20:  call    complete_panic  e18d8d30/16X  0xe18d8d30:     f               37              f               0                  f5659000        e00e7710        4               e010166c                  e00e0054        e18d8e38        e00e7400        f                  c               0  e18d8d90       e000fbc0   e000fbc0/i  panic+0x1c:     call    do_panic  e18d8d90/16X  0xe18d8d90:     f               1               f               4                   1               e00e7710        4               e010166c                   e00e0054        e00dc000        12534f0         62                   1               0  e18d8df0        e003f3f4   e003f3f4/i  clock+0x58:     call    panic  e18d8df0/16X  0xe18d8df0:     e00fa024        1               e00d1c00        4                   a               5               2               4                   0               4               fff01080        ffefefc8                   ffffffff        64  e18d8e60        e000666c   e000666c/i  L14_front_end+0x2c8:            call    clock  e18d8e60/16X  0xe18d8e60:     41401fc2        f54c93f8        f54c93fc        e00fd240                   a               a9              e18d8ec0        e18cf7e0                   8               f54b4400        fff01080        ffefefc8                   2               8  e18cf888        f54c93e4   f54c93e4/i  md_daemon+0x14: call    md_get_status  e18cf888/16X  0xe18cf888:     414019c3        e0009944        e0009948        e00fd240                   a               e18b1ec0        e18d8ec0        f54b4518                   1               f54b4518        c58f6ff         f54b4538                   f54b4528        0  e18cf8e8        f54debfc   f54debfc/i  md_stripe_strategy+0x23c:       call    md_daemon  e18cf8e8/16X  0xe18cf8e8:     f68127e8        1               f56f4e00        f5a9cadc                   f5a9cb04        f5a9caf8        0               f6832c00                   4d746           0               0               0                   8               f68127e8  e18cf960        e003ba6c   e003ba6c/i  bwrite+0x8c:    call    bdev_strategy  e18cf960/16X  0xe18cf960:     11eb39          71b8c           f5499f84        3                   a               e18b1ec0        e18d8ec0        e18cf918                   f68127e8        0               c0b             ffefefc8                   80a             e00fd240  e18cf9c0        e003c3d8   e003c3d8/i  bflush+0x19c:   call    bwrite  e18cf9c0/16X  0xe18cf9c0:     f5704b30        f5705148        f5705148        f5704b30                   c0b             28f             e00fd0ac        2000                   ffffffff        0               0               f6812848                   f6105f68        f68127e8  e18cfa20        f5499e94   f5499e94/i  ufs_update+0x20c:               call    bflush  e18cfa20/16X  0xe18cfa20:     414019c7        d8              f681ca00        0                   f54a5c6c        1               0               f5490308                   12              f5aca600        f681cad8        f5aca62c                   f5b10000        0  e18cfa88        f549cb20   f549cb20/i  ufs_sync+0x1c:  call    ufs_update  e18cfa88/16X  0xe18cfa88:     0               7007ee          e00e9d90        2                   0               8               e00fdd84        e00fdd84                   0               0               f54ad500        f5490308                   f549cb04        e00fdd86  e18cfae8        e0093630   e0093630/i  sync+0x54:      jmpl    %o4, %o7  e18cfae8/16X  0xe18cfae8:     e00fa024        e00e6c00        e000ff98        f5408000                   10              80a             200             e00e7518                   e00e4fa4        0               e00fdd84        e00d6d00                   2               20  e18cfb48        e009421c   e009421c/i  vfs_syncall+0x58:               call    sync  e18cfb48/16X  0xe18cfb48:     410019c2        e002bd94        e001009c        e00fd240                   a               e00e4c00        e00e4c00        4                   e00e4fc4        e00e4fbc        ffffffec        410010e3                   41001ce3        0  e18cfba8        e00100a8   e00100a8/i  complete_panic+0x190:           call    vfs_syncall  e18cfba8/16X  0xe18cfba8:     c               1               e00d2000        9                   1               e00d2000        e00fd240        0                   0               e18cfd1c        e18b1ec0        e18b1ec0                   e18b1ee4        e00fdd28  e18cfc10        e000fbf0   e000fbf0/i  do_panic+0x20:  call    complete_panic  e18cfc10/16X  0xe18cfc10:     41401cc4        e00401cc        7a              7a                   7a              ffef0000        7               418010e6                   e00dc000        e18cfd1c        e18cfd1c        73                   73              3  e18cfc70        e00401bc   e00401bc/i  cmn_err+0x1c:   call    vcmn_err  e18cfc70/16X  0xe18cfc70:     73              0               63              63                   63              63              63              63                   3               e00dc000        0               0                   0               0  e18cfcd0        e0011220   e0011220/i  vx_handler+0xa4:                call    cmn_err  e18cfcd0/16X  0xe18cfcd0:     41801dc6        ffd8d898        1               e00d1c00                   ffef0000        0               f0120000        0                   ffed4440        7               e00041e0        0                   e00d2010        e00d2008  e18cfd50        ffd56178   ffd56178/i  data address not found  e18cfd50/16X  0xe18cfd50:     ffd5d0e0        0               ffd48d8c        ffef0000                   e001117c        ffd7ecf0        ffefefc8        ffefebdc                   2               ffd418a0        b               e00d1fb6                   0               1  e18cfdb0        e0037878   e0037878/i  prom_enter_mon+0x60:            jmpl    %o1, %o7  e18cfdb0/16X  0xe18cfdb0:     60              e0006758        f               10                   1               e00e7710        1               e010166c                   2e0000          6b              2e0000          4                   1               6b  e18cfe10        e0010860   e0010860/i  debug_enter+0x100:              call    prom_enter_mon  e18cfe10/16X  0xe18cfe10:     10              e0029dd8        e00d30c8        e00fa084                   e00fd240        0               4               0                   0               0               44              0                   740000          740002  e18cfe80        e0010744   e0010744/i  abort_sequence_enter+0x18:      call    debug_enter  e18cfe80/16X  0xe18cfe80:     740000          44              0               10                   e0000000        1               0               80                   0               1               44              1                   0               f576205c  e18cfee0        f5760a94   f5760a94/i  zs_high_intr+0xc4:              jmpl    %g1, %o7  e18cfee0/16X  0xe18cfee0:     f5762460        e00b1800        0               e00f7000                   0               2               1               e00d1993                   f543861c        2               3               c                   e00fd240        f5762000  e18cff40        e000cdd8   e000cdd8/i  slow_intr+0x9c: jmpl    %i1, %o7  e18cff40/16X  0xe18cff40:     410001c4        e00e6c00        e00458b4        e00fd240                   a               e18b1ec0        e18d8ec0        e00e7518                   c               f57609d0        e00e6f14        0                   0               0  e18cffa0        e000637c   e000637c/i  _level1+0xa0:   call    slow_intr  e18cffa0/16X  0xe18cffa0:     41400cc5        e0044258        e004425c        e000637c                   c               414000c5        1               e18b1d58                   1b              0               0               ffffffff                   0               e00fd240  e18b1e00        e0044294   e0044294/i  idle+0xcc:      call    disp_getwork  e18b1e00/16X  0xe18b1e00:     41000ac2        e0006754        e00fd240        e00ee110                   e00fd280        0               e00fd28c        e00fd28d                   e00fd240        0               e00fd244        e00fd35c                   0               0  e18b1e60        e000a0c8   e000a0c8/i  thread_start+4: jmpl    %i7, %o7  e18b1e60/16X  0xe18b1e60:     0               0               0               0                   0               0               0               0                   0               0               0               0                   0               0               0  e00441c8   e00441c8/i  idle:   idle:           save    %sp, -0x60, %sp
What did we learn by doing this manual stack traceback? Something quite important actually! The first CPU was also idle when the panic was forced. Basically, this Dragon was quietly snoozing until someone walked up to its console and woke it up.