Using the threadlist macro

To find out what the Dragon was dreaming about, we need to look at all of the threads. We do this via the adb macro threadlist on Solaris 2 systems. On Solaris 1 systems, we use the traceall macro.

Note

Threadlist and traceall output can be thousands of lines long on heavily used servers. You might want to consider sending the adb output to a file. Also, you'll discover that it actually takes adb quite a while to do all of this thread tracing for you. This threadlist took over six minutes to run. So, yes, this is a good time to get coffee!

Here is the command we used to collect the thread information.

 Hiya on p4d-2000a...  time adb -k unix.0 vmcore.0 > allthreads   $<threadlist   $q  real     6:23.0  user     1:31.8  sys      4:46.6  Hiya on p4d-2000a...

While some of you might be curious to see the whole threadlist output, we think it best to keep the output shown here a bit limited. First, we'll show you a few of what appear to be boring threads, examples of what we are initially ignoring .

 thread_id e18dbec0  <---   Thread quietly awaiting a certain condition  ?(?) + dffff7a8  cv_wait(0xe00fd1c8,0xe00fd1c8,0x0,0x10000,0x0,0x0)  background(0x0,0xfffffffe,0x10000,0xf5f3e488,0x0,0xe00fd1c8) + e8                  thread_id e18e1ec0  ?(?) + dffff7a8  cv_wait(0xe01023f0,0xe01023f0,0xc70f6ff,0x1,0x3,0xf5437100)  qwriter_outer_thread(0x0,0xf5437100,0xe00fdbc4,0x0,0xe01023f0,0x0) + 74                  thread_id e18eaec0  <---   Problem with $c output  (We'll get back to this)  ?() + dffff7a8  data address not found                  thread_id e18f1ec0  <---   A CPU being awakened for the panic  ?(?) + dffff7a8  poll_obp_mbox(0xffeef001,0xfc,0x1,0x4,0xe00ee098,0xfc)  level14_handler(0xe18f1d54) + 4  L14_front_end(?) + 68  flush_windows(0x414000e0)  resume(?) + 4  disp_getwork(0xf571d800,0x0,0x0,0xffffffff,0x0,0xf571c000)  idle(0xf571d800,0x0,0xf571d804,0xf571d91c,0x1,0x0) + cc                  thread_id f572ee00  ?(?) + dffff7a8  biowait(0xf60950f0,0x0,0x0,0xf64604d0,0xf68ed3ec,0xf60950f0)                  thread_id f63f0e00  <---   Thread quietly awaiting a signal  ?(?) + dffff7a8  cv_wait_sig(0x0,0xf628a558,0x1,0xf623f000,0xe1d5b9e0,0xf63f0e00)  strwaitq(0xf628a500,0xf5fad746,0x400,0xf5fad700,0xe1d5b7a4,0x2) + 234  strread(0x0,0xf628a558,0xf628a510,0xe1d5b828,0x0,0x600000) + 14c  rw(0x400,0xe1d5b920,0x1,0x0,0xe00d626c,0xf5d9c804) + 1e4  syscall(0xe00de9f0) + 3d8

Here are the more interesting threads. The underlined data shows what the trained eye takes notice of.

 thread_id e18c9ec0  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,f5a98fe0)  polltime(0xf6263000,0x0,0xc64f6ff,0xb36940ff,0x124c054,0xf571d800) + 8  callout_execute(0xe00fe5d0,e00fe9e0,0x124c054,80000000,f59da8fc,a0f67454) + 8c  realtime_timein(0xe00fe5d0,0x0,0x100,0x410000e0,0x41000de0,0x0) + 14  softint(0xe003e2d0,0xe00eaea0,0x0,0x0,0x0,0xe00eb1c4) + 8c  softlevel1(0x0,0xc6c76ff,0xe00d7088,0x1,0x41000de2,0xe00eb1c4) + 4  soft_pseudo_intr(0xe00d709c,0x0,0xe00a337c,0x0,0x0,0x1) + 44  cpu_intr(0x1,0x1f00606,0xe1d1c9e0,0xe000ce30,0xe00ae708,0x0) + 70  _level1(0xe00fd240,0x0,0xe0000000,0x0,0x4b,0x100)                  thread_id e18d8ec0  ?(0xe00e7400,0x1,0xe00d1c00,0xe00fd240,0x0,0xe00e7400) + dffff7a8  do_panic(0xe00e0054,0xe18d8e38,0xe00e7400,0xf,0xc,0x0) + 20  panic(0xe00e0054,0xe00dc000,0x12534f0,0x62,0x1,0x0) + 1c  clock(0x0) + 58                  thread_id e1bd8ec0  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  pollwakeup(0xf5c3ad40,0x41,0xe00ec800,0xe00ecb34,0x0,0xe0102248) + 3c  strrput(0xf6acd700,0x0,0xdec76ff,0xf5c3ad00,0x1,0xf5c3ad58) + 320  putnext(0xf5c3aa00,0xf6acd700,0xf5c3ad00,0x1ff00,0xe007acd8,0xf5c3aaa8) + 134                  thread_id f5ab7400  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  pollwakeup(0xf6acea40,0x41,0xe00ec800,0xe00ecb34,0x0,0xe0102248) + 3c  putback(0xf6acea58,0xf6acea58,0xf69e6600,0x0,0xe1ef8808,0x0) + 98                  thread_id f5a3ce00  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  cv_wait_sig_swap(0x0,0xe00ecb34,0x1,0xf5d2f800,0xe1b9a9e0,0xf5a3ce00) + 170  poll(0x1,0xf6b7738c,0xf6a9edf8,0x0,0x0,0xffffffff) + 54c  syscall(0xe00de9f0) + 3d8                  thread_id f6756000  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  poll(0xf66b21b0,0xe1c47920,0x0,0x0,0x1,0x0) + 208  syscall(0xe00de9f0) + 3d8                  thread_id f57a1c00  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  pollwakeup(0xf63e1d40,0x41,0xe00ec800,0xe00ecb34,0x0,0xe0102248) + 3c  strrput(0xf6b7ba80,0x0,0xabd0e0ff,0xf63e1d00,0xe1b88808,0xf63e1d58) + 320  putnext(0xf63e1800,0xf6b7ba80,0xf63e1d00,0x1ff00,0xe007acd8,0xf63e18a8) + 134                  thread_id e1b6bec0  ?(?) + dffff7a8  mutex_enter(0xe00ecb34  ,0xe00ecb3a,0xe00ecb34,0xb36940ff,0xf5a98ffb,0xf5a98fe0)  pollwakeup(0xf6098b40,0x41,0xe00ec800,0xe00ecb34,0x0,0xe0102248) + 3c  strrput(0xf66df700,0x0,0xdb5f6ff,0xf6098b00,0x0,0xf6098b58) + 320

Looking at this trimmed output, we see seven calls to mutex_enter() . This routine is called with one parameter, the pointer to a mutual exclusive lock. What is surprising is that each of the seven threads is calling mutex_enter() with the same mutex pointer, 0xe00ecb34.