Preface

Welcome to the second edition of Solaris™ Internals and its companion volume, Solaris™ Performance and Tools. It has been almost five years since the release of the first edition, during which time we have had the opportunity to communicate with a great many Solaris users, software developers, system administrators, database administrators, performance analysts, and even the occasional kernel hacker. We are grateful for all the feedback, and we have made specific changes to the format and content of this edition based on reader input. Read on to learn what is different. We look forward to continued communication with the Solaris community.

About These Books

These books are about the internals of Sun's Solaris Operating Systemspecifically, the SunOS kernel. Other components of Solaris, such as windowing systems for desktops, are not covered. The first edition of Solaris™ Internals covered Solaris releases 2.5.1, 2.6, and Solaris 7. These volumes focus on Solaris 10, with updated information for Solaris 8 and 9.

In the first edition, we wanted not only to describe the internal components that make the Solaris kernel tick, but also to provide guidance on putting the information to practical use. These same goals apply to this work, with further emphasis on the use of bundled (and in some cases unbundled) tools and utilities that can be used to examine and probe a running system. Our ability to illustrate more of the kernel's inner workings with observability tools is facilitated in no small part by the inclusion of some revolutionary and innovative technology in Solaris 10DTrace, a dynamic kernel tracing framework. DTrace is one of many new technologies in Solaris 10, and is used extensively throughout this text.

In working on the second edition, we enlisted the help of several friends and colleagues, many of whom are part of Solaris kernel engineering. Their expertise and guidance contributed significantly to the quality and content of these books. We also found ourselves expanding topics along the way, demonstrating the use of dtrace(1), mdb(1), kstat(1), and other bundled tools. So much so that we decided early on that some specific coverage of these tools was necessary, and chapters were written to provide readers with the required background information on the tools and utilities. From this, an entire chapter on using the tools for performance and behavior analysis evolved.

As we neared completion of the work, and began building the entire manuscript, we ran into a bit of a problemthe size. The book had grown to over 1,500 pages. This, we discovered, presented some problems in the publishing and production of the book. After some discussion with the publisher, it was decided we should break the work up into two volumes.

Solaris™ Internals. This represents an update to the first edition, including a significant amount of new material. All major kernel subsystems are included: the virtual memory (VM) system, processes and threads, the kernel dispatcher and scheduling classes, file systems and the virtual file system (VFS) framework, and core kernel facilities. New Solaris facilities for resource management are covered as well, along with a new chapter on networking. New features in Solaris 8 and Solaris 9 are called out as appropriate throughout the text. Examples of Solaris utilities and tools for performance and analysis work, described in the companion volume, are used throughout the text.

Solaris™ Performance and Tools. This book contains chapters on the tools and utilities bundled with Solaris 10: dtrace(1), mdb(1), kstat(1), etc. There are also extensive chapters on using the tools to analyze the performance and behavior of a Solaris system.

The two texts are designed as companion volumes, and can be used in conjunction with access to the Solaris source code on

http://www.opensolaris.org

Readers interested in specific releases before Solaris 8 should continue to use the first edition as a reference.

Intended Audience

We believe that these books will serve as a useful reference for a variety of technical staff members working with the Solaris Operating System.

Application developers can find information in these books about how Solaris OS implements functions behind the application programming interfaces. This information helps developers understand performance, scalability, and implementation specifics of each interface when they develop Solaris applications. The system overview section and sections on scheduling, interprocess communication, and file system behavior should be the most useful sections.

Device driver and kernel module developers of drivers, STREAMS modules, loadable system calls, etc., can find herein the general architecture and implementation theory of the Solaris OS. The Solaris kernel framework and facilities portions of the books (especially the locking and synchronization primitives chapters) are particularly relevant.

Systems administrators, systems analysts, database administrators, and Enterprise Resource Planning (ERP) managers responsible for performance tuning and capacity planning can learn about the behavioral characteristics of the major Solaris subsystems. The file system caching and memory management chapters provide a great deal of information about how Solaris behaves in real-world environments. The algorithms behind Solaris tunable parameters are covered in depth throughout the books.

Technical support staff responsible for the diagnosis, debugging, and support of Solaris will find a wealth of information about implementation details of Solaris. Major data structures and data flow diagrams are provided in each chapter to aid debugging and navigation of Solaris systems.

System users who just want to know more about how the Solaris kernel works will find high-level overviews at the start of each chapter.

Beyond the technical user community, those in academia studying operating systems will find that this text will work well as a reference. Solaris OS is a robust, feature-rich, volume production operating system, well suited to a variety of workloads, ranging from uniprocessor desktops to very large multiprocessor systems with large memory and input/output (I/O) configurations. The robustness and scalability of Solaris OS for commercial data processing, Web services, network applications, and scientific workloads is without peer in the industry. Much can be learned from studying such an operating system.

OpenSolaris

In June 2005, Sun Microsystems introduced OpenSolaris, a fully functional Solaris operating system release built from open source. As part of the OpenSolaris initiative, the Solaris kernel source was made generally available through an open license offering. This has some obvious benefits to this text. We can now include Solaris source directly in the text where appropriate, as well as refer to full source listings made available through the OpenSolaris initiative.

With OpenSolaris, a worldwide community of developers now has access to Solaris source code, and developers can contribute to whatever component of the operating system they find interesting. Source code accessibility allows us to structure the books such that we can cross-reference specific source files, right down to line numbers in the source tree.

OpenSolaris represents a significant milestone for technologists worldwide; a world-class, mature, robust, and feature-rich operating system is now easily accessible to anyone wishing to use Solaris, explore it, and contribute to its development.

Visit the Open Solaris Website to learn more about OpenSolaris:

http://www.opensolaris.org

The OpenSolaris source code is available at:

http://cvs.opensolaris.org/source

Source code references used throughout this text are relative to that starting location.

How the Books Are Organized

We organized the Solaris™ Internals volumes into several logical parts, each part grouping several chapters containing related information. Our goal was to provide a building block approach to the material by which later sections could build on information provided in earlier chapters. However, for readers familiar with particular aspects of operating systems design and implementation, the individual parts and chapters can stand on their own in terms of the subject matter they cover.

Volume 1: Solaris™ Internals

Part One: Introduction to Solaris Internals

Chapter 1 Introduction

Part Two: The Process Model

Chapter 2 The Solaris Process Model

Chapter 3 Scheduling Classes and the Dispatcher

Chapter 4 Interprocess Communication

Chapter 5 Process Rights Management

Part Three: Resource Management

Chapter 6 Zones

Chapter 7 Projects, Tasks, and Resource Controls

Part Four: Memory

Chapter 8 Introduction to Solaris Memory

Chapter 9 Virtual Memory

Chapter 10 Physical Memory

Chapter 11 Kernel Memory

Chapter 12 Hardware Address Translation

Chapter 13 Working with Multiple Page Sizes in Solaris

Part Five: File Systems

Chapter 14 File System Framework

Chapter 15 The UFS File System

Part Six: Platform Specifics

Chapter 16 Support for NUMA and CMT Hardware

Chapter 17 Locking and Synchronization

Part Seven: Networking

Chapter 18 The Solaris Network Stack

Part Eight: Kernel Services

Chapter 19 Clocks and Timers

Chapter 20 Task Queues

Chapter 21 kmdb Implementation

Volume 2: Solaris™ Performance and Tools

Part One: Observability Methods

Chapter 1 Introduction to Observability Tools

Chapter 2 CPUs

Chapter 3 Processes

Chapter 4 Disk Behavior and Analysis

Chapter 5 File Systems

Chapter 6 Memory

Chapter 7 Networks

Chapter 8 Performance Counters

Chapter 9 Kernel Monitoring

Part Two: Observability Infrastructure

Chapter 10 Dynamic Tracing

Chapter 11 Kernel Statistics

Part Three: Debugging

Chapter 12 The Modular Debugger

Chapter 13 An MDB Tutorial

Chapter 14 Debugging Kernels

Updates and Related Material

To complement these books, we created a Web site at which we will place updated material, tools we refer to, and links to related material on the topics covered. We will regularly update the Web site (http://www.solarisinternals.com) with information about this text and future work on Solaris™ Internals. The Web site will be enhanced to provide a forum for Frequently Asked Questions (FAQs) related to the text, as well as general questions about Solaris internals, performance, and behavior. If bugs are discovered in the text, we will post errata on the Web site as well.

Notational Conventions

Table P.1 describes the typographic conventions used throughout these books, and Table P.2 shows the default system prompt for the utilities we describe.

Table P.1. Typographic Conventions

Typeface or Symbol	Meaning	Example
AaBbCc123	Command names, file names, and data structures.	The vmstat command. The <sys/proc.h> header file. The proc structure.
AaBbCc123()	Function names.	page_create_va()
AaBbCc123(2)	Manual pages.	Please see vmstat(1M).
AaBbCc123	Commands you type within an example.	$ vmstat r b w swap free re mf 0 0 0 464440 18920 1 13
AaBbCc123	New terms as they are introduced.	A major page fault occurs when...
MDB	The modular debuggers, including the user-mode debugger (mdb) and the kernel in-situ debugger (kmdb).	Examples that are applicable to both the user-mode and the in-situ kernel debugger.
mdb	The user-mode modular debugger.	Examples that are applicable the user-mode debugger.
kmdb	The in-situ debugger	Examples that are applicable to the in-situ kernel debugger.

Table P.2. Command Prompts

Shell	Prompt
Shell prompt	minimum-osversion$
Shell superuser prompt	minimum-osversion#
The mdb debugger prompt	>
The kmdb debugger prompt	[cpu]>

A Note from the Authors

Once again, a large investment in time and energy proved enormously rewarding for the authors. The support from Sun's Solaris kernel development group, the Solaris user community, and readers of the first edition has been extremely gratifying. We believe we have been able to achieve more with the second edition in terms of providing Solaris users with a valuable reference text. We certainly extended our knowledge in writing it, and we look forward to hearing from readers.