Section 16.5. Memory Allocation

16.5. Memory Allocation

In Solaris, memory allocation is a two-step process. The first step assigns virtual memory. This step occurs when an application calls brk() to extend its heap or when the application maps in a file. The second step assigns physical memory to back the virtual memory. The assignment of physical memory does not occur until the application first tries to read or write to the new virtual address. At that point, the kernel will select a physical memory page and create a mapping from the application's virtual address to this physical page.

The key to delivering the best performance on systems with sizable memory locality differences is to ensure that physical memory is allocated close to the threads that are expected to access it. This allows for both lower latency and higher bandwidth. Obviously, when a process allocates memory, the kernel cannot predict with certainty how that memory will be used. However, the kernel can make several different assumptions that hold in many cases.

The simplest policy that one can adopt when allocating memory for locality awareness is "first touch." This simply means that memory is allocated from the home lgroup of the thread that first tries to access that memory. This approach assumes that whichever thread first accesses the memory is likely to be the thread that will access that memory most frequently in the future. This assumption obviously holds for single-threaded applications, but it also holds for many multi-threaded applications. First touch is the default memory allocation policy used for private memory. Note that memory is allocated from the thread's home lgroup, even if the thread is running remote from its home at the time the memory is allocated. This behavior reflects an assumption that the thread will primarily be scheduled to run on its home lgroup.

Shared memory (for example, Intimate Shared Memory (ISM), MAP_SHARED memory-mapped files, etc.) is, by definition, likely to be accessed by multiple threads. Assuming that some significant number of those threads will be running in different lgroups, the kernel allocates shared memory by using the default random memory placement policy. This policy optimizes for bandwidth while trying to minimize average latency for the threads accessing it throughout the server. It spreads the memory across as many memory banks as possible, distributing the load across many memory controllers and bus interfaces, thereby preventing any single component from becoming a performance-limiting hot spot. In addition, random placement improves the reproducibility of performance measurements by ensuring that the relative locality of threads and memory remains roughly constant across multiple runs of an application.