Section 14.2. Examining User Process Stacks within a Kernel Image

14.2. Examining User Process Stacks within a Kernel Image

A kernel crash dump can save memory pages of user processes in Solaris. We explain how to save process memory pages and how to examine user processes by using the kernel crash dump.

14.2.1. Enabling Process Pages in a Dump

We must modify the dump configuration to save process pages. We confirm the dump configuration by running dumpadm with no option.

# /usr/sbin/dumpadm          Dump content: all pages           Dump device: /dev/dsk/c0t0d0s1 (swap)    Savecore directory: /var/crash/example      Savecore enabled: yes 

If Dump content is not all pages or curproc, no process memory page will be dumped. In that case, we run dumpadm -c all or dumpadm -c curproc.

14.2.2. Invoking MDB to Examine the Kernel Image

We gather a crash dump and confirm that user pages are contained.

# /usr/bin/mdb unix.0 vmcore.0    Loading modules: [ unix krtld genunix ufs_log ip nfs random ptm    logindmux ] > ::status debugging crash dump vmcore.0 (64-bit) from rmcferrari operating system: 5.11 snv_31 (i86pc) panic message: forced crash dump initiated at user request dump content: all kernel and user pages 

The dump content line shows that this dump includes user pages.

14.2.3. Locating the Target Process

Next, we search for process information with which we are concerned. We use nscd as the target of this test case. The first thing to find is the address of the process.

> ::pgrep nscd S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME R    575      1    575    575      0 0x42000000 ffffffff866f1878 nscd 

The address of the process is ffffffff866f1878. As a sanity check, we can look at the kernel thread stacks for each processwe'll use these later to double-check that the user stack matches the kernel stack, for those threads blocked in a system call.

> 0t575::pid2proc |::print proc_t p_tlist |::list kthread_t t_forw stack pointer for thread ffffffff866cb060: fffffe8000c7fdd0 [ fffffe8000c7fdd0 _resume_from_idle+0xde() ]   fffffe8000c7fe10 swtch+0x185()   fffffe8000c7fe80 cv_wait_sig_swap_core+0x17a()   fffffe8000c7fea0 cv_wait_sig_swap+0x1a()   fffffe8000c7fec0 pause+0x59()   fffffe8000c7ff10 sys_syscall32+0x101() stack pointer for thread ffffffff866cc140: fffffe8000c61d70 [ fffffe8000c61d70 _resume_from_idle+0xde() ]   fffffe8000c61db0 swtch+0x185()   fffffe8000c61e10 cv_wait_sig+0x150()   fffffe8000c61e50 door_unref+0x94()   fffffe8000c61ec0 doorfs32+0x90()   fffffe8000c61f10 sys_syscall32+0x101() stack pointer for thread ffffffff866cba80: fffffe8000c6dd10 [ fffffe8000c6dd10 _resume_from_idle+0xde() ]   fffffe8000c6dd50 swtch_to+0xc9()   fffffe8000c6ddb0 shuttle_resume+0x376()   fffffe8000c6de50 door_return+0x228()   fffffe8000c6dec0 doorfs32+0x157()   fffffe8000c6df10 sys_syscall32+0x101() stack pointer for thread ffffffff866cb720: fffffe8000c73cf0 [ fffffe8000c73cf0 _resume_from_idle+0xde() ]   fffffe8000c73d30 swtch+0x185()   fffffe8000c73db0 cv_timedwait_sig+0x1a3()   fffffe8000c73e30 cv_waituntil_sig+0xab()   fffffe8000c73ec0 nanosleep+0x141()   fffffe8000c73f10 sys_syscall32+0x101() ... 

It appears that the first few threads on the process are blocked in the pause(), door(), and nanosleep() system calls. We'll double-check against these later when we traverse the user stacks.

14.2.4. Extracting the User-Mode Stack Frame Pointers

The next things to find are the stack pointers for the user threads, which are stored in each thread's lwp.

[View full width]
> ffffffff866f1878::walk thread |::print kthread_t t_lwp->lwp_regs|::print "struct regs" r_rsp |=X                 8047d54         fecc9f80        febbac08        fea9df78        fe99df78          fe89df78           fe79df78                 fe69df78        fe59df78        fe49df78        fe39df58        fe29df58          fe19df58           fe09df58                 fdf9df58        fde9df58        fdd9df58        fdc9df58        fdb9df58          fda9df58           fd99df58                    fd89d538         fd79bc08 

Each entry is a thread's stack pointer in the user process's address space. We can use these to traverse the stack in the user process's context.