Section 6.6. Scan Rate as a Memory Health Indicator

6.6. Scan Rate as a Memory Health Indicator

Solaris uses a central physical memory manager to reclaim memory from various subsystems when there is a shortage. A single daemon performs serves this purpose: the page scanner. The page scanner returns memory to the free list when the amount of free memory falls below a preset level, represented by a preconfigured tunable parameter, lotsfree. Knowing the basics about the page scanner will help you understand and interpret the memory health and performance statistics.

The scanner starts scanning when free memory is lower than lotsfree number of pages free plus a small buffer factor, deficit. The scanner starts scanning at a rate of slowscan pages per second at this point and gets faster as the amount of free memory approaches zero. The system parameter lotsfree is calculated at startup as 1/64th of memory, and the parameter deficit is either zero or a small number of pagesset by the page allocator at times of large memory allocation to let the scanner free a few more pages above lotsfree in anticipation of more memory requests.

Figure 6.3 shows that the rate at which the scanner scans increases linearly as free memory ranges between lotsfree and zero. The scanner starts scanning at the minimum rate set by slowscan when memory falls below lotsfree and then increases to fastscan if memory falls low enough.

Figure 6.3. Page Scanner Rate, Interpolated by Number of Free Pages

The page scanner and its metrics are an important indicator of memory health. If the page scanner is running, there is likely a memory shortage. This is an interesting departure from the behavior you might have been accustomed to on Solaris 7 and earlier, where the page scanner was always running. Since Solaris 8, the file system cache resides on the cache list, which is part of the global free memory count. Thus, if a significant amount of memory is available, even if it's being used as a file system cache, the page scanner won't be running.

The most important metric is the scan rate, which indicates whether the page scanner is running. The scanner starts scanning at an initial rate (slowscan) when freemem falls down to the configured watermarklotsfreeand then runs faster as free memory gets lower, up to a maximum (fastscan).

You can perform a quick and simple health check by determining whether there is a significant memory shortage. To do this, use vmstat to look at scanning activity and check to see if there is sufficient free memory on the system.

Let's first look at a healthy system. This system is showing 970 Mbytes of free memory in the free column and a scan rate (sr) of zero.

$ vmstat -p 3      memory           page          executable      anonymous      filesystem    swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf  1512488 837792 160 20 12   0   0    0    0    0    0    0    0   12   12   12  1715812 985116 7  82   0   0   0    0    0    0    0    0    0   45    0    0  1715784 983984 0   2   0   0   0    0    0    0    0    0    0   53    0    0  1715780 987644 0   0   0   0   0    0    0    0    0    0    0   33    0    0 

Looking at a second case, we can see two of the key indicators showing a memory shortageboth high scan rates (sr > 50000 in this case) and very low free memory (free < 10 Mbytes).

$ vmstat -p 3      memory           page          executable      anonymous      filesystem    swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf  2276000 1589424 2128 19969 1 0 0    0    0    0    0    0    0    0    1    1  1087652 388768 12 129675 13879 0 85590 0 0   12    0 3238 3238   10 9391 10630  608036 51464  20 8853 37303 0 65871 38   0  781   12 19934 19930 95 16548 16591   94448  8000  17 23674 30169 0 238522 16 0  810   23 28739 28804 56  547  556 

Given that the page scanner runs only when the free list and cache list are effectively depleted, then any scanning activity is our first sign of memory shortage. Drilling down further with ::memstat (see Section 6.4) shows us where the major allocations are. It's useful to check that the kernel hasn't grown unnecessarily large.

6.6.1. Using prstat to Estimate Memory Slowdowns

Using the microstate measurement option in prstat, you can observe the percentage of execution time spent in data faults. The microstates show 100% of the execution time of a thread broken down into eight categories; the DFL column shows the percentage of time spent waiting for data faults to be serviced. The following example shows a severe memory shortage. The system was running short of memory, and each thread in filebench is waiting for memory approximately 90% of the time.

$ prstat -mL    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID  15625 rmc      0.1 0.7 0.0 0.0  95 0.0 0.9 3.2  1K 726  88   0 filebench/2  15652 rmc      0.1 0.7 0.0 0.0  94 0.0 1.8 3.6  1K  1K  10   0 filebench/2  15635 rmc      0.1 0.7 0.0 0.0  96 0.0 0.5 3.2  1K  1K   8   0 filebench/2  15626 rmc      0.1 0.6 0.0 0.0  95 0.0 1.4 2.6  1K 813  10   0 filebench/2  15712 rmc      0.1 0.5 0.0 0.0  47 0.0  49 3.8  1K 831 104   0 filebench/2  15628 rmc      0.1 0.5 0.0 0.0  96 0.0 0.0 3.1  1K 735   4   0 filebench/2  15725 rmc      0.0 0.4 0.0 0.0  92 0.0 1.7 5.7 996 736   8   0 filebench/2  15719 rmc      0.0 0.4 0.0 0.0  40  40  17 2.9  1K 708 107   0 filebench/2  15614 rmc      0.0 0.3 0.0 0.0  92 0.0 4.7 2.4 874 576  40   0 filebench/2  15748 rmc      0.0 0.3 0.0 0.0  94 0.0 0.0 5.5 868 646   8   0 filebench/2  15674 rmc      0.0 0.3 0.0 0.0  86 0.0 9.7 3.2 888 571  62   0 filebench/2  15666 rmc      0.0 0.3 0.0 0.0  29  46  23 2.1 689 502 107   0 filebench/2  15682 rmc      0.0 0.2 0.0 0.0  24  43  31 1.9 660 450 107   0 filebench/2