Section 3.5. Process Introspection Commands

3.5. Process Introspection Commands

Solaris provides a set of utilities for inspecting the state of processes. Most of the introspection tools can be used either on a running process or postmortem on a core file resulting from a process dump. The general syntax is as follows:

$ ptool pid $ ptool pid/lwpid $ ptool core 

See the man pages for each of these tools for additional details.

3.5.1. Process Stack: pstack

The stacks of all or specific threads within a process can be displayed with the pstack command.

$ pstack 23154 23154:  shadow -a shadow -i 193 -s ffffffff10000000 -m /var/tmp/fbench9Cai2S -----------------  lwp# 1 / thread# 1  --------------------  ffffffff7e7ce0f4 lwp_wait (2, ffffffff7fffe9cc)  ffffffff7e7c9528 _thrp_join (2, 0, 0, 1, 100000000, ffffffff7fffe9cc) + 38  0000000100018300 threadflow_init (ffffffff3722f1b0, ffffffff10000000, 10006a658, 0, 0, 1000888b0) + 184  00000001000172f8 procflow_exec (6a000, 10006a000, 0, 6a000, 5, ffffffff3722f1b0) + 15c  0000000100026558 main (a3400, ffffffff7ffff948, ffffffff7fffeff8, a4000, 0, 1) + 414  000000010001585c _start (0, 0, 0, 0, 0, 0) + 17c -----------------  lwp# 2 / thread# 2  --------------------  000000010001ae90 flowoplib_hog (30d40, ffffffff651f3650, 30d40, ffffffff373aa3b8, 1, 2e906) + 68  00000001000194a4 flowop_start (ffffffff373aa3b8, 0, 1, 0, 1, 1000888b0) + 408  ffffffff7e7ccea0 _lwp_start (0, 0, 0, 0, 0, 0) 

The pstack command can be very useful for diagnosing process hangs or the status of core dumps. By default it shows a stack backtrace for all the threads within a process. It can also be used as a crude performance analysis technique; by taking a few samples of the process stack, you can often determine where the process is spending most of its time.

You can also dump a specific thread's stacks by supplying the lwpid on the command line.

sol8$ pstack 26258/2 26258:  shadow -a shadow -i 62 -s ffffffff10000000 -m /var/tmp/fbenchI4aGkZ -----------------  lwp# 2 / thread# 2  --------------------  ffffffff7e7ce138 lwp_mutex_timedlock (ffffffff10000060, 0)  ffffffff7e7c4e8c mutex_lock_internal (ffffffff10000060, 0, 0, 1000, ffffffff7e8eef80, ffffffff7f402400) + 248  000000010001da3c ipc_mutex_lock (ffffffff10000060, 1000888b0, 100088800, 88800, 100000000, 1) + 4  0000000100019d94 flowop_find (ffffffff651e2278, 100088800, ffffffff651e2180, 88800, 100000000, 1) + 34  000000010001b990 flowoplib_sempost (ffffffff3739a768, ffffffff651e2180, 0, 6ac00, 1, 1) + 4c  00000001000194a4 flowop_start (ffffffff3739a768, 0, 1, 0, 1, 1000888b0) + 408  ffffffff7e7ccea0 _lwp_start (0, 0, 0, 0, 0, 0) 

3.5.2. Process Memory Map: pmap -x

The pmap command inspects a process, displaying every mapping within the process's address space. The amount of resident, nonshared anonymous, and locked memory is shown for each mapping. This allows you to estimate shared and private memory usage.

sol9$ pmap -x 102908  102908:   sh  Address   Kbytes Resident   Anon  Locked Mode    Mapped File  00010000      88      88       -       - r-x--   sh  00036000       8       8       8       - rwx--   sh  00038000      16      16      16       - rwx--     [ heap ]  FF260000      16      16       -       - r-x--   en_.so.2  FF272000      16      16       -       - rwx--   en_US.so.2  FF280000     664     624       -       - r-x--   libc.so.1  FF336000      32      32       8       - rwx--   libc.so.1  FF360000      16      16       -       - r-x--   libc_psr.so.1  FF380000      24      24       -       - r-x--   libgen.so.1  FF396000       8       8       -       - rwx--   libgen.so.1  FF3A0000       8       8       -       - r-x--   libdl.so.1  FF3B0000       8       8       8       - rwx--     [ anon ]  FF3C0000     152     152       -       - r-x--   ld.so.1  FF3F6000       8       8       8       - rwx--   ld.so.1  FFBFE000       8       8       8       - rw---     [ stack ]  --------   -----   -----   -----   ------  total Kb    1072    1032      56       - 

This example shows the address space of a Bourne shell, with the executable at the top and the stack at the bottom. The total Resident memory is 1032 Kbytes, which is an approximation of physical memory usage. Much of this memory will be shared by other processes mapping the same files. The total Anon memory is 56 Kbytes, which is an indication of the private memory for this process instance.

You can find more information on interpreting pmap -x output in Section 6.8.

3.5.3. Process File Table: pfiles

A list of files open within a process can be obtained with the pfiles command.

sol10# pfiles 21571 21571:  /usr/lib/ssh/sshd   Current rlimit: 256 file descriptors    0: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2       O_RDWR|O_LARGEFILE       /devices/pseudo/mm@0:null    1: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2       O_RDWR|O_LARGEFILE       /devices/pseudo/mm@0:null    2: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2       O_RDWR|O_LARGEFILE       /devices/pseudo/mm@0:null    3: S_IFCHR mode:0000 dev:286,0 ino:38639 uid:0 gid:0 rdev:215,2       O_RDWR FD_CLOEXEC       /devices/pseudo/crypto@0:crypto    4: S_IFIFO mode:0000 dev:294,0 ino:13099 uid:0 gid:0 size:0       O_RDWR|O_NONBLOCK FD_CLOEXEC    5: S_IFDOOR mode:0444 dev:295,0 ino:62 uid:0 gid:0 size:0       O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[89]       /var/run/name_service_door    6: S_IFIFO mode:0000 dev:294,0 ino:13098 uid:0 gid:0 size:0       O_RDWR|O_NONBLOCK FD_CLOEXEC    7: S_IFDOOR mode:0644 dev:295,0 ino:55 uid:0 gid:0 size:0       O_RDONLY FD_CLOEXEC  door to keyserv[169]       /var/run/rpc_door/rpc_100029.1    8: S_IFCHR mode:0000 dev:286,0 ino:26793 uid:0 gid:0 rdev:41,134       O_RDWR FD_CLOEXEC       /devices/pseudo/udp@0:udp    9: S_IFSOCK mode:0666 dev:292,0 ino:31268 uid:0 gid:0 size:0       O_RDWR|O_NONBLOCK         SOCK_STREAM         SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49640)         sockname: AF_INET6 ::ffff:129.146.238.66  port: 22         peername: AF_INET6 ::ffff:129.146.206.91  port: 63374   10: S_IFIFO mode:0000 dev:294,0 ino:13098 uid:0 gid:0 size:0       O_RDWR|O_NONBLOCK   11: S_IFIFO mode:0000 dev:294,0 ino:13099 uid:0 gid:0 size:0       O_RDWR|O_NONBLOCK FD_CLOEXEC 

The Solaris 10 version of pfiles prints path names if possible.

3.5.4. Execution Time Statistics for a Process: ptime

A process can be timed with the ptime command for accurate microstate accounting instrumentation.^[1]

^[1] Most other time commands now source the same microstate-accounting-based times.

$ ptime sleep 1 real        1.203 user        0.022 sys         0.140 

3.5.5. Process Signal Disposition: psig

A list of the signals and their current disposition can be displayed with psig.

sol8$ psig $$ 15481:          -zsh HUP             caught   0 INT             blocked,caught   0 QUIT            blocked,ignored ILL             blocked,default TRAP            blocked,default ABRT            blocked,default EMT             blocked,default FPE             blocked,default KILL            default BUS             blocked,default SEGV            blocked,default SYS             blocked,default PIPE            blocked,default ALRM            blocked,caught    0 TERM            blocked,ignored USR1            blocked,default USR2            blocked,default CLD             caught   0 PWR             blocked,default WINCH           blocked,caught    0 URG             blocked,default POLL            blocked,default STOP            default 

3.5.6. Process Libraries: pldd

A list of the libraries currently mapped into a process can be displayed with pldd. This is useful for verifying which version or path of a library is being dynamically linked into a process.

sol8$ pldd $$ 482764: -ksh /usr/lib/libsocket.so.1 /usr/lib/libnsl.so.1 /usr/lib/libc.so.1 /usr/lib/libdl.so.1 /usr/lib/libmp.so.2 

3.5.7. Process Flags: pflags

The pflags command shows a variety of status information for a process. Information includes the mode32-bit or 64-bitin which the process is running and the current state for each thread within the process (see Section 3.1 in Solaris^™ Internals for information on thread state). In addition, the top-level function on each thread's stack is displayed.

sol8$ pflags $$ 482764: -ksh               data model = _ILP32  flags = PR_ORPHAN   /1:         flags = PR_PCINVAL|PR_ASLEEP [ waitid(0x7,0x0,0xffbff938,0x7) ] 

3.5.8. Process Credentials: pcred

The credentials for a process can be displayed with pcred.

sol8$ pcred $$ 482764: e/r/suid=36413  e/r/sgid=10         groups: 10 10512 570 

3.5.9. Process Arguments: pargs

The full process arguments and optionally a list of the current environment settings can be displayed for a process with the pargs command.

$ pargs -ae 22961 22961:  /opt/filebench/bin/filebench argv[0]: /opt/filebench/bin/filebench envp[0]: _=/opt/filebench/bin/filebench envp[1]: MANPATH=/usr/man:/usr/dt/man:/usr/local/man:/opt/SUNWspro/man:/ws/on998- tools/teamware/man:/home/rmc/local/man envp[2]: VISUAL=/bin/vi ... 

3.5.10. Process Working Directory: pwdx

The current working directory of a process can be displayed with the pwdx command.

$ pwdx  22961 22961:  /tmp/filebench