Network status: netstat

On both Solaris 1 and Solaris 2 systems, the UNIX netstat command can be used with your postmortem files. No special option is needed. For example:

netstat -d vmunix.5 vmcore.5

Unfortunately, some of the options don't work on savecore files even if you request it, so when using netstat , compare the output to the output you get when netstat is run on the system being used for analysis work. If the output is the same, you can assume that the command wasn't using the crash files.

NFS status: nfsstat

Just as with the netstat command, you can use the nfsstat command on your system crash dump files. No special option is needed, just the names of the unix and core file.

nfsstat -n vmunix.2 vmcore.2

As with the netstat command, always compare the output of the nfsstat command run on the live system with the output for the postmortem files, just to make sure you are really getting system crash data.

Address resolution protocol status: arp

The UNIX arp program can be used with postmortem files to extract information from the address resolution protocol table. Like many UNIX commands, you need only append the name of the unix and core files to the command line.

arp -a vmunix.1 vmcore.1

As always, compare the output to that from a live system to make sure it's reporting statistics for the savecore files.

Interprocess communication status: ipcs

The ipcs command is used to report status on the interprocess communication facilities: shared memory, message queues and semaphores. i pcs exists on both Solaris 1 and Solaris 2 systems. To use ipcs against your crash dump files, use option - C followed by the corefile name and -N followed by the namelist name. For example:

ipcs -a -C vmcore.4 -N vmunix.4

Again, compare the output to that from a live system to make sure you're viewing the proper data.

If you aren't familiar with ps , pstat , netstat , nfsstat , arp , or ipcs , please refer to the manual pages.

The crash program

Solaris systems provide a program called crash that was specifically written for use in examining memory on both live systems and in postmortem files; crash can be used to print out various structures and tables of the kernel in easy-to-read formats.

On Solaris 1 systems, the crash program can be found in the /etc directory. On Solaris 2 systems, it lives in the /usr/sbin directory.

While it is tempting for us to include a whole chapter or two on how to use crash , we would rather just refer you to the manual pages. However, since Solaris 2's process status, ps , command doesn't run on postmortem files, we will show you how to get the process status by using the crash program instead.

Figure 6-3 Using the crash utility to get process status information

Hiya...

crash -d vmcore.0 -n unix.0

dumpfile = vmcore.0, namelist = unix.0, outfile = stdout 
>

help p

p [-e] [-f] [-l] [-w filename] [([-p] [-a] tbl_entry  #procid)...  -r] 
        tbl_entry = slot number  address  symbol  expression  range 
process table 
alias: proc 
        acceptable aliases are uniquely identifiable initial substrings 
>

p -e

PROC TABLE SIZE = 490 
SLOT ST  PID  PPID  PGID   SID   UID PRI CPU   NAME        FLAGS 
   0 r     0     0     0     0     0  98  69 sched          load sys lock 
   1 r     1     0     0     0     0  98  32 init           load 
   2 r     2     0     0     0     0  98  13 pageout        load sys lock nowait 
   3 r     3     0     0     0     0  98  80 fsflush        load sys lock nowait 
   4 r   369     1   369   369     0  98  14 sac            load jctl 
   5 r   370     1   370   370 11628  98  69 sh             load 
   6 r   331   323   323   323     0  98  13 lpNet          load nowait jctl 
   7 r   261     1   261   261     0  98   2 keyserv        load 
   8 r   259     1   259   259     0  98  80 rpcbind        load 
   9 r   267     1   267   267     0  98  64 ypbind         load 
  10 r   269     1   269   269     0  98  14 kerbd          load 
  11 r   279     1   279   279     0  98  80 inetd          load 
  12 r   286     1   286   286     0  98  80 automountd     load 
  13 r   290     1   290   290     0  98  18 statd          load 
  14 r   292     1   292   292     0  98  45 lockd          load 
  15 r   303     1   303   303     0  98  28 syslogd        load nowait 
  16 r   323     1   323   323     0  98  28 lpsched        load nowait 
  17 r   313     1   313   313     0  98  50 cron           load 
  18 r   332     1   332   332     0  98  37 sendmail       load 
  19 r   373   369   369   369     0  98  20 ttymon         load jctl 
  20 r   528   523   520   520     0  98  61 adb            load 
  21 r   496     1   495   495 11628  98  11 fm_flb         load 
  22 r   407   370   370   370 11628  98  80 openwin        load 
  23 z   399   370   399   370 11628  98  80 zombie         load 
  24 r   372   369   369   369     0  98  21 listen         load nowait jctl 
  25 r   492     1   492   477 11628  98  80 maker4X.exe    load 
  26 r   411   407   370   370 11628  98  12 xinit          load 
  27 r   412   411   412   370 11628  98  80 Xsun           load 
  28 r   413   411   413   370 11628  98 107 sh             load 
  29 r   432   279   279   279     0  98  23 rpc.ttdbserver load 
  30 r   421     1   421   370 11628  98  17 fbconsole      load 
  31 r   520   517   520   520 11628  98  80 sh             load 
  32 r   428     1   428   370 11628  98  35 vkbd           load 
  33 r   431     1   431   370 11628  98  52 ttsession      load 
  34 r   435   413   435   370 11628  98   4 olwm           load 
  35 r   436   435   435   370 11628  98  24 olwmslave      load 
  37 r   444     1   413   370 11628  98  41 clock          load 
  39 r   523   520   520   520     0  98  51 sh             load 
  40 r   451     1   413   370 11628  98  77 cmdtool        load jctl 
  41 r   454   451   454   454 11628  98  61 sh             load 
  42 r   471   435   471   471 11628  98  80 x_cdplayer     load 
  43 r   517   435   517   517 11628  98  80 cmdtool        load jctl 
>

q

Hiya...

Use of the p command with the -l option in crash is interesting, giving you the user credentials information. Basically, this is who owns the process. The output of the -l option is very long, so we'll just show a snippet of it here.

Figure 6-4 Process credentials as displayed by the crash utility

SLOT ST  PID  PPID  PGID   SID   UID PRI CPU   NAME        FLAGS 
  20 r   528   523   520   520     0  98  61 adb            load 

        Session: sid: 520, ctty: vnode(fc5acd04) maj(  24) min  1) 
        Process Credentials: uid: 0, gid: 1, real uid: 0, real gid: 1 
        as: fc1ef550 
        wait code: 0, wait data: 0 
        sig: effff178   link 0 
        parent: fc451800        child: 0 
        sibling: 0 threadp: fc4dc600 
        utime: 41       stime: 20       cutime: 0       cstime: 0 
        trace: 0        sigmask: effff178       class: 0 
        lwptotal: 1     lwpcnt: 1       lwprcnt: 1 
        lwpblocked: 0

This is the section of the output for the adb process that was running at the time of the crash. The crash dump in question was caused by the adb session in Chapter 5, when we set rootdir to zero and caused the system to panic.

Again, if you want to experiment with the crash command, go for it! However, please note that some early versions of crash were a bit buggy .