Section 1.7. Memory Management

1.7. Memory Management

Every object in the system is managed as a memory object in some form; data structures, kernel text, process address space segments, processes, threads, etc., all exist and are managed as objects in memory. Thus, the Solaris virtual memory (VM) system can be considered the core of the operating systemit manages the system's memory on behalf of the kernel and processes. The main task of the VM system is to manage efficient allocation of the system's physical memory to the processes and kernel subsystems running within the operating system. The VM system uses slower storage media (usually disk) to store data that does not fit within the physical memory of the system, thus accommodating programs larger than the size of physical memory. The VM system is what keeps the most frequently used portions within physical memory and the lesser-used portions on the slower secondary storage.

For processes, the VM system presents a simple linear range of memory, known as an address space. Each address space is broken into several segments that represent mappings of the executable; heap space (general-purpose, process-allocated memory), shared libraries, and a program stack. Each segment is divided into equal-sized pieces of virtual memory, known as pages, and a hardware memory management unit (MMU) manages the mapping of page-sized pieces of virtual memory to physical memory. Figure 1.4 shows the relationship between an address space, segments, the memory management unit, and physical memory.

Figure 1.4. Address Spaces, Segments, and Pages

The virtual memory system is implemented in a layered, modular fashion. The components that deal with physical memory management are mostly hardware-platform specific. The platform-dependent portions are implemented in the hardware address translation (HAT) layer.

1.7.1. Global Memory Allocation

The VM system implements demand paging. Pages of memory are allocated on demand as they are referenced, and hence portions of an executable or shared library are allocated on demand. Loading pages of memory on demand dramatically lowers the memory footprint and the startup time of a process. When an area of virtual memory is accessed, the hardware MMU raises an event to tell the kernel that an access has occurred to an area of memory that does not have physical memory mapped to it. This event is a page fault. The heap of a process is similarly allocated. Initially, only virtual memory space is allocated to the process; when memory is first referenced, a page fault occurs and memory is allocated one page at a time.

The virtual memory system uses a global paging model that implements a single global policy to manage the allocation of memory between processes. A scanning algorithm calculates the least used portion of the physical memory. A kernel thread (the page scanner) scans memory in physical page order when the amount of free memory falls below a preconfigured threshold. Pages that have not been used recently are stolen and placed onto a free list for use by other processes.

1.7.2. The Cyclic Page Cache

All modern operating systems implement some form of file system caching, such that frequently referenced files have their contents in physical memory, providing significantly faster access. The virtual memory system and file system page cache have been tightly integrated in Solaris from the very beginning, originating in SunOS 4.0. In Solaris OS, all of free physical memory can be used to cache file system data.

The original page cache design implemented a special kernel address space segment, called segmap, to manage the mapping of files to physical memory. Since segmap is fixed in size (the actual size will vary based on Solaris release, hardware architecture, and physical memory size), a mechanism is required for handling workloads that exceed the size of segmap. In such situations, a page replacement algorithm is implemented, where least recently used pages are moved out of the segmap, making room for new pages to be cached. The moved pages may still be resident in physical memory, but a file I/O reference will require that the page be moved back into the segmap segment.

Before Solaris 8, pages moved out of segmap needed intervention by the page scanner in order to be available for use by other memory consumers. In Solaris 8, a new cyclic page cache was implemented; pages moved out of segmap are now placed on a cache list. Cache list pages appear to the VM system as free memory, available to other memory consumers. The page scanner is no longer required to reclaim memory in the form of pages that have been pushed out of segmap.

1.7.3. Kernel Memory Management

The Solaris kernel requires memory for kernel instructions, data structures, and caches. Most of the kernel's memory is not pageable; that is, it is allocated from physical memory that cannot be stolen by the page scanner. This characteristic avoids deadlocks that could occur within the kernel if a kernel memory management function caused a page fault while holding a lock for another critical resource. The kernel cannot rely on the global paging used by processes, so it implements its own memory allocation systems.

A core kernel memory allocatorthe slab allocatorallocates memory for kernel data structures. As the name suggests, the allocator subdivides large contiguous areas of memory (slabs) into smaller chunks for data structures. Allocation pools are organized so that like-sized objects are allocated from the same continuous segments, thereby dramatically reducing fragmentation that could result from continuous allocation and deallocation.

The original slab allocator, integrated into Solaris 2.4, was significantly enhanced over time to keep pace with larger multiprocessor systems and to extend its use as a general-purpose kernel memory allocator. A new, per-processor caching scheme was introduced to provide scalable performance. A general-purpose resource allocator was added in the form of a virtual memory allocator, called vmem. The original slab allocator was designed to manage kernel heap memory allocations; the addition of the vmem layer extends the kernel memory allocator to much broader use as a general-purpose allocator of arbitrary-sized resources. Finally, the design, architecture, and algorithms used by the new kernel memory allocator were applied to a user-level memory allocator and implemented as a plug-in replacement for the malloc(3C) interface through a new library, libumem.

The new kernel memory allocator was implemented in Solaris 8. The user-level allocator, libumem, was introduced in Solaris 10.