Background on Linux and Performance Analysis

Linux entered the marketplace primarily through grassroots efforts that recognized the price/performance benefits of Linux on small servers. Linux has been very effective in that market space and has become an increasingly popular operating system for use in the wider enterprise market. This popularity has encouraged hardware vendors to port Linux to platforms with more processors, more processor types, more physical memory, and larger I/O configurations. Software vendors have seen the benefits of porting more complex software to Linux, and end users are putting together more complex configurations. As these workloads increase in complexity, end users are beginning to expect the same or higher levels of performance as they had with proprietary systems. These increased expectations of performance translate into performance analysis and improvements in scalability of the operating system's scheduler, memory management, disk, and network I/O subsystems.

Historically, systems vendors provided a proprietary operating system and performed the analysis of core subsystems internally and with a relatively small set of critical applications that were key to the core of their businesses. Those vendors were able to use advanced and often proprietary performance analysis tools, including sophisticated bus monitors, proprietary chip counters, simulations tools, native performance monitoring packages, and tools to analyze the performance of complex software and hardware configurations. Many of these complex tools were made available to selected customers or used through service contracts to allow customers to analyze their workload's performance characteristics and to assess the results of complex tuning changes. The intent of part of this book is to explore some of the tools and the best practices that can be more universally applied today to improve the overall performance of Linux and the applications that run on Linux.

Linux provides some new challenges in performance analysis but also offers some unique advantages. Specifically, Linux runs on a much wider range of hardware platforms than most proprietary operating systems, including RISC and CISC architectures, 32-bit and 64-bit systems, uniprocessor systems, embedded systems, small (two to four CPUs) and large (eight to 128) symmetrical multiprocessing systems. The workloads on these Linux-based systems and the underlying processor combinations tend to be more varied than any other single operating system has addressed, including desktops, laptops, embedded systems, mail servers, file servers, application servers, database servers, and content servers. Although many workloads have been used at various times in tuning Linux, Linux has not been tuned for a specific, single workload. Moreover, the same core Linux kernel and base libraries are expected to run reasonably well on all of these workloads. Because of these trade-offs and the widespread expectations that Linux works reasonably well on all workloads, there is a good chance that your unique workload will not run optimally on Linux without the benefit of system analysis, performance monitoring, and (potentially) system and application tuning.

Performance in these increasingly complex workloads is influenced by many factors. Obviously, processor speed, memory size, number of network or disk I/O controllers, and size and speed of disks are significant components of all workload performance characteristics. Other major components include the application's own performance characteristics, the nature of the workload, the communications between applications, the access patterns for data on disk or on the network, and, of course, the usage models of the end users using the applications.

Tuning a particular workload often starts with the assumption that many of the physical characteristics of the environment will remain unchanged, such as the number of processors, the processor type, number of disks, and so on. The primary goals in performance analysis are to identify what components of the workload are the current bottlenecks to increased performance and overall throughput and, when the workload has saturated the capacity of the available hardware, to understand what hardware configuration improvements would increase the workload's throughput and performance.

Many Linux performance tools are available today in the open-source community in addition to a few commercial tools. Many of the examples in this chapter are drawn from a common distribution running a Linux 2.4 kernel, although in general the same tools should exist and have the same capabilities in any distribution based on the Linux 2.6 kernel. In some cases, the data collected by the kernel is provided through different interfaces or based on different in-kernel accounting mechanisms. In any case, the information provided by the tool should be similar or identical. If your distribution does not have a particular tool installed or you believe that a particular tool is not sufficiently current, you may want to consult the distribution media for the tool or pick up the latest version from the tool's maintainer. See, for instance, http://linuxperf.nl.linux.org/general/tools/tools.html for a pointer to some of the currently available tools.