Section 1.1. Observability Tools

1.1. Observability Tools

The commands, tools, and utilities used for observing system performance and behavior can be categorized in terms of the information they provide and the source of the data. They include the following.

• Kernel-statistics-gathering tools. Report kstats, or kernel statistics, collected by means of counters. Examples are vmstat, mpstat, and netstat.

• Process tools. Provide system process listings and statistics for individual processes and threads. Examples are prstat, ptree, and pfiles.

• Forensic tools. Track system calls and perform in-depth analysis of targets such as applications, kernels, and core files. Examples are truss and MDB.

• Dynamic tools. Fully instrument-running applications and kernels. DTrace is an example.

In combination, these utilities constitute a rich set of tools that provide much of the information required to find bottlenecks in system performance, debug troublesome applications, and even help determine what caused a system to crashafter the fact! But which tool is right for the task at hand? The answer lies in determining the information needed and matching it to the tools available. Sometimes a single tool provides this information. Other times you may need to turn detective, using one set of tools, say, DTrace, to dig out the information you need in order to zero in on specific areas where other tools like MDB can perform in-depth analysis.

Determining which tool to use to find the relevant information about the system at hand can sometimes be as confusing to the novice as the results the tool produces. Which particular command or utility to use depends both on the nature of the problem you are investigating and on your goal. Typically, a systemwide view is the first place to start (the "stat" commands), along with a full process view (prstat(1)). Drilling-down on a specific process or set of processes typically involves the use of several of the commands, along with dtrace and/or MDB.

1.1.1. Kstat Tools

The system kernel statistics utilities (kstats) extract information continuously maintained in the kernel Kstats framework as counters that are incremented upon the occurrence of specific events, such as the execution of a system call or a disk I/O. The individual commands and utilities built on kstats can be summarized as follows. (Consult the individual man pages and the following chapters for information on the use of these commands and the data they provide.)

• mpstat(1M). Per-processor statistics and utilization.

• vmstat(1M). Memory, run queue, and summarized processor utilization.

• iostat(1M). Disk I/O subsystem operations, bandwidth, and utilization.

• netstat(1M). Network interface packet rates, errors, and collisions.

• kstat(1M). Name-based output of kstat counter values.

• sar(1). Catch-all reporting of a broad range of system statistics; often regularly scheduled to collect statistics that assist in producing reports on such vital signs as daily CPU utilization.

The utilities listed above extract data values from the underlying kstats and report per-second counts for a variety of system events. Note that the exception is netstat(1), which does not normalize values to per-second rates but rather to the per-interval rates specified by the sampling interval used on the command line. With these tools, you can observe the utilization level of the system's hardware resources (processors, memory, disk storage, network interfaces) and can track specific events systemwide, to aid your understanding of the load and application behavior.

1.1.2. Process Tools

Information and data on running processes are available with two tools and their options.

• ps(1). Process status. List the processes on the system, optionally displaying extended per-process information.

• prstat(1M). Process status. Monitor processes on the system, optionally displaying process and thread-level microstate accounting and per-project statistics for resource management.

Per-process information is available through a set of tools collectively known as the ptools, or process tools. These utilities are built on the process file system, procfs, located under /proc.

• pargs(1). Display process argument list.

• pflags(1). Display process flags.

• pcred(1). Display process credentials.

• pldd(1). Display process shared object library dependencies.

• psig(1). Display process signal dispositions.

• pstack(1). Display process stack.

• pmap(1). Display process address space mappings.

• pfiles(1). Display process opened files with names and flags.

• ptree(1). Display process family tree.

• ptime(1). Time process execution.

• pwdx(1). Display process working directory.

Process control is available with various ptools.

• pgrep(1). Search for a process name string, and return the PID.

• pkill(1). Send a kill signal or specified signal to a process or process list.

• pstop(1). Stop a process.

• prun(1). Start a process that has been stopped.

• pwait(1). Wait for a process to terminate.

• preap(1). Reap a zombie (defunct) process.

1.1.3. Forensic Tools

Powerful process- and thread-level tracing and debugging facilities included in Solaris 10 and OpenSolaris provide another level of visibility into process- or thread-execution flow and behavior.

• truss(1). Trace functions and system calls.

• mdb(1). Debug or control processes.

• dtrace(1M). Trace, analyze, control, and debug processes.

• plockstat(1M). Track user-defined locks in processes and threads.

Several tools enable you to trace, observe, and analyze the kernel and its interaction with applications.

• dtrace(1M). Trace, monitor, and observe the kernel.

• lockstat(1M). Track kernel locks and profile the kernel.

• mdb(1) and kmdb(1). Analyze and debug the running kernel, applications, and core files.

Last, specific utilities track hardware-specific counters and provide visibility into low-level processor and system utilization and behavior.

• cputrack(1). Track per-processor hardware counters for a process.

• cpustat(1M). Track per-processor hardware counters.

• busstat(1M). Track interconnect bus hardware counters.