Section 6.3. Types of Paging

6.3. Types of Paging

In this section, we quickly review the two major types of "paging": file I/O paging and anonymous memory paging. Understanding them will help you interpret the system metrics and health. Figure 6.1 puts paging in the context of physical memory's life cycle.

Figure 6.1. Life Cycle of Physical Memory

6.3.1. File I/O Paging: "Good" Paging

Traditional Solaris file systems (including UFS, VxFS, NFS, etc.) use the virtual memory system as the primary file cache (ZFS is an exception). We cover file systems caching in more detail in Section 14.8 in Solaris^™ Internals.

File system I/O paging is the term we use for paging reads and writes files through file systems in their default cached mode. Files are read and written in multiples of page-size units to the I/O or to the network device backing the file system. Once a file page is read into memory, the virtual memory system caches that page so that subsequent file-level accesses don't have to reread pages from the device. It's normal to see a substantial amount of paging activity as a result of file I/O. Beginning with Solaris 8, a cyclic file system cache was introduced. The cyclic file system cache recirculates pages from the file system through a central pool known as the cache list, preventing the file system from putting excessive paging pressure on other users of memory within the system. This feature superseded the priority paging algorithms used in Solaris 7 and earlier to minimize these effects.

Paging can be divided into the following categories:

• Reading files. File system reads that miss in the file cache are performed as virtual memory page-ins. A new page is taken off the free list, and an I/O is scheduled to fill the page from its backing store. Files read with the system call read(2) are mapped into the segmap cache and are eventually placed back onto the tail of the cache list. The cache list becomes an ordered list of file pages; the oldest cached pages (head of the cache list) are eventually recycled as file system I/O consumes new pages from the free list.

  Smaller I/Os typically exhibit a one-to-one ratio between file system cache misses and page-ins. In some cases, however, the file system will group reads or issue prefetch, resulting in larger or differing relationships between file I/O and paging.

• Writing files. The process of writing a file also involves virtual memory operationsupdated files are paged out to the backing I/O in multiples of page-size chunks. However, the reporting mechanism exhibits some oddities; for example, only page-outs that hint at discarding the page from cache show as file system page-outs in the kstat and vmstat statistics.

• Reading executables. The virtual memory system reads executables (program binaries) into memory upon exec and reads shared libraries into a process's address space. These read operations are basically the same as regular file system reads; however, the virtual memory system marks and tracks them separately to make it easy to isolate program paging from file I/O paging.

Paging of executables is visible through vmstat statistics; executable page-ins, page-outs, and frees are shown in the epi, epo, and epf columns. File page-ins, page-outs, and frees are shown in the fpi, fpo, and fpf columns.

$ vmstat -p 3      memory          page          executable      anonymous      filesystem    swap  free re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf  411696 12720 38 35473 15738 0 217112 20 0  848   13 14146 14331 23  377  559  409356 35344 11 1823 9717 0 141771 104  0   22   96 9376 9389   62  295  306  345264 26724 53 5234 2329 0 8727  28    0    0  263 2200 2200  103  217  129  301104 48032 36 7313 8451 0 102271 76   0   75  167 8199 8241   15  157  135 

6.3.2. Anonymous Memory Paging: "Bad" Paging

Anonymous memory paging is the term we use when the virtual memory system migrates anonymous pages to the swap device because of a shortage of physical memory. Most often, this occurs when the sum of the process heaps, shared memory, and stacks exceeds the available physical memory, causing the page scanner to begin shifting out to the swap device those pages that haven't recently been used. The next time the owning process references these pages, it incurs a data fault and must go to sleep while waiting for the pages to be brought back in from the swap device.

Anonymous paging is visible through the vmstat statistics; page-ins and page-outs are shown in the api and apo columns.

$ vmstat -p 3      memory           page          executable      anonymous      filesystem    swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf  411696 12720  38 35473 15738 0 217112 20 0  848   13 14146 14331 23  377  559  409356 35344  11 1823 9717 0 141771 104  0   22   96 9376 9389   62  295  306  345264 26724  53 5234 2329 0 8727  28    0    0  263 2200 2200  103  217  129  301104 48032  36 7313 8451 0 102271 76   0   75  167 8199 8241   15  157  135 

Although swap I/O is just another form of file system I/O, it is most often much slower than regular file I/O because of the random movement of memory to and from the swap device. Pages are collected and queued to the swap device in physical page order by the page scanner and are efficiently issued to the swap device (clustering allows up to 1-Mbyte I/Os). However, the owning process typically references the pages semi-sequentially in virtual memory order, resulting in random page-size I/O from the swap device. We know from simple I/O metrics that random 8-Kbyte I/O is likely to yield service times of around 5 milliseconds, significantly affecting performance.

6.3.3. Per-Zone Paging Statistics

The DTraceToolkit includes a command to display the vmstat statistics per-zone. It uses the zonename DTrace variable to summarize by zone.

# zvmstat 1       ZONE   re    mf  fr  sr  epi  epo  epf  api  apo  apf   fpi  fpo  fpf     global   54   316   1   0    0    0    0    0    0    0     0    1    1  workzone1    0     0   0   0    0    0    0    0    0    0     0    0    0       ZONE   re    mf  fr  sr  epi  epo  epf  api  apo  apf   fpi  fpo  fpf     global  157   659   1   0   10    0    0    0    0    0     3    2    1  workzone1  770  1085   0   0   48    0    0    0    0    0   928    0    0       ZONE   re    mf  fr  sr  epi  epo  epf  api  apo  apf   fpi  fpo  fpf     global   56   317   0   0    6    0    0    0    0    0     2    0    0  workzone1 1478    21   0   0    0    0    0    0    0    0  1635    0    0                                                                     See DTraceToolkit