| Process Resource Manager (PRM) allows us to prioritize CPU, memory, and IO workloads based on the requirements of users and applications. We do this using a PRM concept known as shares that represent the amount of available resource in the system as a whole. We can quantify a minimum level of performance when the system is being heavily utilized by guaranteeing a proportion of the total available number of shares of system resources to individual PRM groups of users and applications. PRM can be configured to allow users/applications to use unused shares when no one else needs them; this is the default behavior for CPU shares, but it needs to be configured for memory. At its heart, PRM has a simple configuration file ( /etc/prmconf ) that specifies the share entitlement for each user , group of users, and applications are assigned for CPU, memory, and disk IO. PRM effectively introduces a new scheduling policy into the system. This scheduling policy can operate in stark contrast to the default HP-UX Timesharing scheduling policy where particularly busy processes are penalized by having their priority decayed quicker in favor of less-busy processes. PRM will guarantee that processes are allocated their full share of resources regardless of how busy individual processes are. PRM is sometimes referred to as the Fair Share Scheduler. WorkLoad Manager (WLM) goes one step further than PRM and allows us to build into our performance profile goal-based performance metrics. The metrics will be measured against data that we provide on a continual basis from our applications. For example, if we specify that a minimum level of performance equates to 10,000 transactions in our database, we can extract this figure from our application and pass it to WLM. If the current metric falls below the specified goal, WLM can dynamically retune available system resources to meet the specified goal. These goal-based metrics are known as Service Level Objectives. At its heart, WLM has its own configuration file, but ultimately WLM interfaces directly with PRM to manage the allocation of system resources. 11.10.1 A simple PRM configuration to manage CPU shares PRM comes standard when the 11i Enterprise Operating Environment is purchased. Otherwise, it is a separately ordered product. Installing the product requires a reboot of the system. Once installed, the default PRM configuration file ( /etc/prmconf ) contains numerous examples and explanations of how to set up CPU, memory, and disk IO shares. We will look at setting up CPU shares to begin with. A fundamental concept in PRM is the idea of a share . A share entitlement is a proportion of the total available to the system. For CPU shares, this means a proportion of all available CPU time. Initially, this may sound as if we specify a share entitlement as a percentage. We can do this if we want to, if it makes the configuration easier to understand. However, if at some later date we add other users/group/applications to the configuration, we might want to rework our percentage figures so they all add up to 100 percent. This is no longer technically necessary with PRM, although some administrators find it easier to manage (since PRM version C.01.08 shares do not need to add to 100 percent). Here's an example of not using percentages to represent shares entitlements : -
Fred, Barney, and Wilma all belong to separate PRM groups. -
Fred's group is allocated six CPU shares . -
Barney's group is allocated four CPU shares . -
Wilma's group is allocated two CPU shares . -
The total number of shares is effectively 12. -
Fred has effectively been allocated 50 percent of the shares . -
Barney has effectively been allocated 33.3 percent of the shares . -
Wilma has effectively been allocated 16.7 percent of the shares . -
We add a new user called Betty in her own PRM group. -
We want Betty to be assigned the same proportion of CPU shares as Barney. We simply set Betty's CPU shares to equal 4. The fact that Barney now has 4/16 = 25 percent of the CPU shares is not important; his proportion of the overall shares in relation to everyone else is still the same. Let's start with this example as a specific configuration. The configuration process is relatively straightforward. It can be summarized as follows : -
Consider retuning the shares to the default OTHERS PRM group. By default, all users currently in the /etc/passwd file will be assigned membership of the OTHERS PRM group. This group is allocated 100 CPU shares. The idea here is that non-root processes owned by other system users will be guaranteed a certain level of performance. This is a good idea. It may be appropriate to either retune the number shares the OTHERS group is allocated or create specific groups for specific system users. Root is automatically assigned to the PRM_SYS group, which effectively has no limit to the shares it can consume. Non-root users can be added to the PRM_SYS group, although this may be dangerous and may allow a non-root process to consume all of an available resource. -
Create PRM groups. These are nothing to do with groups in /etc/group , although it is common to use the same group names . -
Assign share entitlements to PRM groups. -
Assign real UNIX users listed in /etc/passwd to PRM groups. It is this link between a user and a PRM group that effectively gives shares to particular users. Users are not directly given their own share entitlement, but will use a proportion of the share entitlement allocated for the group. All other things being equal, all members of the group will accrue time equally. The CPU they use is charged against their respective groups. Let's look at a simple example using the users and shares we mentioned in the example above. We have Fred, Barney, and Wilma all executing processes on behalf of their application. Using the HP-UX Timesharing scheduling policy, all processes will be allocated the same amount of CPU, all other things being equal. Let's take a snapshot of what Fred's, Barney's, and Wilma's processes are doing right now. I will take an excerpt from the output of top and glance to illustrate this example. Here are the main processes for each user:
TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND pts/3 3983 wilma 247 24 28K 128K run 1:13 39.39 39.32 itDB pts/2 3981 barney 248 24 28K 128K run 1:01 30.06 30.01 adminDB pts/0 3982 fred 248 24 28K 128K run 0:58 29.95 29.90 quarryDB
As we can see, these processes appear to be quite CPU- intensive . These along with other processes are causing this system to exhibit some signs of having a CPU bottleneck. Look at the CPU Load averages:
CPU BY PROCESSOR Users= 7 CPU State Util LoadAvg(1/5/15 min) CSwitch Last Pid ------------------------------------------------------------------------- 0 Enable 100.0 4.2/ 4.0/ 2.9 818 3633
With 100 percent Utilization and a Load Average of at least 4, these are good indicators that the CPU is the single resource stopping these processes from running more efficiently . Looking at each individual process, they are all blocked on priority indicating even more conclusively that this is a typical CPU bottleneck. There are no other obvious bottlenecks because the system is not paging and there is no significant IO to talk about. Without PRM, we could use nice , rtprio , and rtsched techniques to try to manage these processes. This can be time-consuming , non-deterministic , and not very accurate. Let's put together our PRM configuration file. Here are my PRM group definitions with their associated share entitlements. (I decided to leave the OTHERS group with the original share allocation. I don't want system processes being blocked from running.)
root@hpeos003[] vi /etc/prmconf ... # OTHERS:1:100:: QUARRY:2:6:: ADMIN:3:4:: IT:4:2:: # ... root@hpeos003[]
I will probably assign my users to their respective groups at this time as well. Users are listed toward the bottom of the file:
root@hpeos003[] vi /etc/prmconf ... fred::::QUARRY barney::::ADMIN wilma::::IT ... root@hpeos003[]
This is as simple as it can get. Users can belong to alternate groups if you want them to utilize the share entitlements for more than one group. Alternate groups are listed as a comma-separated list after the primary group name . All that's left to do is to install this configuration and enable the PRM scheduler:
root@hpeos003[] prmconfig -ie PRM configured from file: /etc/prmconf File last modified: Mon Nov 24 08:05:51 2003 PRM CPU scheduler state: Enabled PRM Group PRMID CPU Entitlement ------------------------------------------------------- ADMIN 3 3.57% IT 4 1.79% OTHERS 1 89.29% QUARRY 2 5.36% PRM memory manager state: Not Running PRM User Initial Group Alternate Group(s) ----------------------------------------------------------------------------- adm OTHERS barney ADMIN bin OTHERS charlesk OTHERS daemon OTHERS fred QUARRY hpdb OTHERS ids OTHERS lp OTHERS mysql OTHERS nuucp OTHERS oracle OTHERS root (PRM_SYS) smbnull OTHERS sshd OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled root@hpeos003[]
We can see that the proportions of CPU share entitlements are calculated relative to the total number of shares available for the entire system. By default, if there are any unused shares, i.e., from the OTHERS group, other groups that are consuming shares will be allocated spare shares evenly among all active groups. To reverse this behavior is to introduce a concept known as capping, whereby a group is not allowed to exceed its share entitlement regardless of whether there are spare shares available. Let's look at what the three main application processes are doing:
TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND pts/0 3982 fred 249 24 28K 128K run 9:06 49.77 49.68 quarryDB pts/2 3981 barney 249 24 28K 128K run 8:40 33.15 33.09 adminDB pts/3 3983 wilma 234 24 28K 128K run 9:11 16.66 16.63 itDB
As soon as we enabled the PRM scheduler, the performance characteristics of our three applications changed dramatically. We can see that CPU performance is closely matching the share entitlements . There is a PRM command called prmmonitor that will show us the difference between a share entitlement and current usage:
root@hpeos003[] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Mon Nov 24 08:05:51 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/24/03 Mon Nov 24 08:15:22 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 89.29% 0.00% QUARRY 2 5.36% 49.50% ADMIN 3 3.57% 33.66% IT 4 1.79% 16.83% PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[]
Again, we can see that CPU capping has not been enforced and the spare shares from the OTHERS group are being used by the three other active groups. If we want to turn capping on, it's simple and takes effect immediately:
root@hpeos003[] prmconfig -M CPUCAPON PRM configured from file: /etc/prmconf File last modified: Mon Nov 24 08:05:51 2003 PRM CPU scheduler state: Enabled, CPU cap ON PRM Group PRMID CPU Entitlement ------------------------------------------------------- ADMIN 3 3.57% IT 4 1.79% OTHERS 1 89.29% QUARRY 2 5.36% PRM memory manager state: Not Running PRM User Initial Group Alternate Group(s) -------------------------------------------------------------------------------- adm OTHERS barney ADMIN bin OTHERS charlesk OTHERS daemon OTHERS fred QUARRY hpdb OTHERS ids OTHERS lp OTHERS mysql OTHERS nuucp OTHERS oracle OTHERS root (PRM_SYS) smbnull OTHERS sshd OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled root@hpeos003[] root@hpeos003[] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Mon Nov 24 08:05:51 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/24/03 Mon Nov 24 08:19:53 2003 Sample: 1 second CPU scheduler state: Enabled, CPU cap ON CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 89.29% 0.00% QUARRY 2 5.36% 4.98% ADMIN 3 3.57% 2.99% IT 4 1.79% 1.00% PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[]
Some administrators I know have psychological problems enabling CPU capping; they believe that if there are spare CPU cycles, then we should share them out among the users currently using the system. I tend to agree with this philosophy, but I do accept that in some situations (where we get involved with billing CPU time to different parts of an organization) it is appropriate to only allow a maximum share entitlement . The only thing left to do is to ensure that we edit the startup configuration file /etc/rc.config.d/prm to CONFIGURE and ENABLE the PRM scheduler after every reboot. 11.10.1.1 PRM APPLICATION RECORDS While it is convenient to apportion CPU cycles on a PRM group basis, the real assignment of CPU cycles occurs when users execute programs. At that time, PRM can identify who executed the program and assign CPU cycles to the appropriate PRM group. There are many applications that run under a single user ID (e.g., Oracle) regardless of how many users are using the application. In these situations, we need to configure PRM Application Records. These will identify a single program or multiple programs, initially by identifying individual scripts/program files. While this might seem like a good idea initially, it is common for application startup routines to spawn child processes based on command line arguments and/or environment variables . In these situations, we can specify what are called alternate names of the processes that PRM can identify (from a ps listing) in order to identify processes for individual applications. In this example, we have an application with a startup script called /home/haydes/launcher .
root@hpeos003[haydes] pwd /home/haydes root@hpeos003[haydes] ll launcher -rwxrwxr-x 1 haydes haydes 20480 Nov 25 23:26 launcher root@hpeos003[haydes]
This is the program that will launch the various components of an application depending on input from various sources. Currently, we have a wide selection of processes running for various applications:
root@hpeos003[haydes] ps -fu haydes UID PID PPID C STIME TTY TIME COMMAND haydes 6004 1 24 23:51:28 pts/1 0:00 quarry_LOG haydes 6002 1 24 23:51:27 pts/1 0:00 itDB haydes 5998 1 31 23:51:27 pts/1 0:00 adminRPT haydes 5999 1 26 23:51:27 pts/1 0:00 comp-it1 haydes 6001 1 24 23:51:27 pts/1 0:00 comp-it3 haydes 5995 1 31 23:51:26 pts/1 0:00 RECV_adm.ship haydes 6003 1 24 23:51:28 pts/1 0:00 quarryDB haydes 5996 1 31 23:51:26 pts/1 0:00 TRANS_adm.ship haydes 5997 1 31 23:51:27 pts/1 0:00 adminDB haydes 6000 1 25 23:51:27 pts/1 0:00 comp-it2 haydes 6005 1 24 23:51:28 pts/1 0:00 rdr_quarry haydes 5994 1 28 23:51:26 pts/1 0:00 FLUSH_admin root@hpeos003[haydes]
The problem we have is that all the processes are owned by the haydes user. If we try to assign different shares to different parts of the application, we will need to do it via application records. If we were to run this application under the current configuration, the user haydes would accrue all the CPU cycles ( haydes belongs to the OTHERS group).
root@hpeos003[] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 00:05:35 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/26/03 Wed Nov 26 00:07:43 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 89.29% 100.00% QUARRY 2 5.36% 0.00% ADMIN 3 3.57% 0.00% IT 4 1.79% 0.00% PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[]
Application records can be used in addition to user records to capture processes like the ones we see above. The important thing here is to understand how our application creates child processes. In our example above, the launcher program simply spawns a number of child processes and dies. The problem is that we don't know whether the child process is simply a function of the parent or whether the child executes a separate program. If it executes a separate program, we will need to identify each individual program with a unique application record. Where we have a single launcher process that doesn't execute separate programs, we will identify individual processes using the alternate name specification in the prmconf file.
<program name>::::<PRM GROUP>,<alternate name>,...,<alternate name>
The syntax of the alternate name allows the use of wildcards to identify process names. We need to spend some considerable time ensuring that we can uniquely identify processes in order to properly assign them to the relevant PRM groups. Once we have identified the various application components, we can assign them to PRM groups. I have checked with my haydes application administrators, and they don't know whether launcher executes programs or is a single program that launches simple child processes. I am going to have to try a configuration which assumes that the child processes are effectively functions of the main program, i.e., part of the same program. Here's the resulting prmconf file:
root@hpeos003[] vi /etc/prmconf ... OTHERS:1:100:: QUARRY:2:6:: ADMIN:3:4:: IT:4:2:: ... fred::::QUARRY barney::::ADMIN wilma::::IT /home/haydes/launcher::::QUARRY,quarry* /home/haydes/launcher::::ADMIN,admin*,*_adm[.]* /home/haydes/launcher::::IT,comp-it*,itDB ... root@hpeos003[]
You can see that I am identifying a single large program file ( launcher ) and using the alternate name syntax to identify individual process names. You may need to spend some time ensuring that you uniquely identifying individual applications; you don't want to be picking up the CPU usage of processes not in your PRM group. Once in place, we can check the configuration and then install and enable it.
root@hpeos003[haydes] prmconfig -s Configuration file check complete. No errors found. root@hpeos003[haydes] prmconfig -ie PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 00:17:48 2003 PRM CPU scheduler state: Enabled PRM Group PRMID CPU Entitlement ------------------------------------------------------- ADMIN 3 3.57% IT 4 1.79% OTHERS 1 89.29% QUARRY 2 5.36% PRM memory manager state: Not Running PRM User Initial Group Alternate Group(s) -------------------------------------------------------------------------------- adm OTHERS barney ADMIN betty OTHERS bin OTHERS charlesk OTHERS daemon OTHERS fred QUARRY haydes OTHERS hpdb OTHERS ids OTHERS lp OTHERS mysql OTHERS nuucp OTHERS oracle OTHERS root (PRM_SYS) smbnull OTHERS sshd OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled root@hpeos003[haydes]
The crucial thing now is to check whether PRM is assigning CPU to the appropriate applications:
root@hpeos003[haydes] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 00:17:48 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/26/03 Wed Nov 26 01:20:33 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 89.29% 100.00% QUARRY 2 5.36% 0.00% ADMIN 3 3.57% 0.00% IT 4 1.79% 0.00% PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[haydes]
You can see that the OTHERS ( haydes and his friends ) are consuming all of the CPU group. This says to me that the launcher program simply spawns a child process that immediately executes a separate program and hence is not the same file ID as the launcher . I need to be able to find the programs/scripts that constitute these processes and rework my prmconf file. Here is the result:
root@hpeos003[] vi /etc/prmconf ... OTHERS:1:100:: QUARRY:2:6:: ADMIN:3:4:: IT:4:2:: ... fred::::QUARRY barney::::ADMIN wilma::::IT /home/haydes/bin/*adm*::::ADMIN /home/haydes/bin/comp-it*::::IT /home/haydes/bin/itDB::::IT,itDB* /home/haydes/bin/quarry*::::QUARRY /home/haydes/bin/rdr_quarry::::QUARRY,rdr_quarry* ... root@hpeos003[]
You may notice that you can use wildcards in identifying script/program names. If you use wildcards in the filenames, you cannot use the alternate name syntax as well. Now let's try this configuration:
root@hpeos003[haydes] prmconfig -s Configuration file check complete. No errors found. root@hpeos003[haydes] prmconfig -ie PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 01:32:08 2003 PRM CPU scheduler state: Enabled PRM Group PRMID CPU Entitlement ------------------------------------------------------- ADMIN 3 3.57% IT 4 1.79% OTHERS 1 89.29% QUARRY 2 5.36% PRM memory manager state: Not Running PRM User Initial Group Alternate Group(s) ------------------------------------------------------------------------- adm OTHERS barney ADMIN betty OTHERS bin OTHERS charlesk OTHERS daemon OTHERS fred QUARRY haydes OTHERS hpdb OTHERS ids OTHERS lp OTHERS mysql OTHERS nuucp OTHERS oracle OTHERS root (PRM_SYS) smbnull OTHERS sshd OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled root@hpeos003[haydes]
The configuration appears to be okay. We need to see whether the applications are running under the appropriate entitlements:
root@hpeos003[haydes] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 01:32:08 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/26/03 Wed Nov 26 01:38:28 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 89.29% 0.00% QUARRY 2 5.36% 50.00% ADMIN 3 3.57% 33.33% IT 4 1.79% 16.67% PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[haydes]
This is much better. This example illustrates how important it is to work with your application suppliers to establish how to identify individual applications in your PRM Application Records. 11.10.1.2 THREAD SCHEDULING AND PRM PRM uses a novel idea when allocating CPU shares. The mental picture to try to construct is a picture of a carousel you would find at the fairground. There are a fixed number of horses (clock ticks) in the carousel . Your share entitlement will determine how many horses you are allocated and therefore how many clock ticks your threads can consume with every spin of the carousel. Very clever. One reason I mention this is in relation to other scheduling policies we talked about previously, namely real-time priorities. PRM will not restrict real-time processes from obtaining the process; the reason a process is real-time is because it is deemed more important than just about any other process. We need to be aware that a real-time process will accumulate CPU time on behalf of its PRM group. If this process is CPU-bound, it could result in no other process from that PRM group gaining access to the CPU because the real-time process has eaten all the horses . Another aspect of PRM scheduling is how it deals with multiprocessor systems. PRM will normally assign successive processes/threads for a single PRM group on successive processors. The idea here is to try to keep overall CPU loads averaged over all processors in the system (a job the statdaemon gets involved in every 1 second normally). PRM will honor any forced processor affinity directives for individual processes/threads, but will take this into account the next time it schedules a thread from a particular PRM group. For some applications, processor affinity can make significant differences to the performance of applications. While PRM will not override forced processor affinity directives, PRM's desire to spread threads over should be considered . In such situations, it is worth considering Processor Sets. We discussed Processor Sets in our discussion of a Process Life Cycle. Processor Sets can be established and managed from within PRM without using the psrset commands. We take a quick look at setting up Processor Sets from within PRM in the next short section. 11.10.1.3 PRM PROCESSOR SETS We have discussed Processor Sets previously in this chapter, so I won't go over the concepts. I have moved my haydes application to a machine with four processors:
# ioscan -fnkC processor Class I H/W Path Driver S/W State H/W Type Description =================================================================== processor 0 2/10 processor CLAIMED PROCESSOR Processor processor 1 2/11 processor CLAIMED PROCESSOR Processor processor 2 2/12 processor CLAIMED PROCESSOR Processor processor 3 2/13 processor CLAIMED PROCESSOR Processor #
The purpose of this demonstration is to show you how to set up Processor Sets from within PRM. I am going to assign two processors to my busiest application, QUARRY, and one processor to the ADMIN group, and then allow everyone else, including the IT people, to use the default Processor Set. Processor Sets are an alternative to CPU shares. Use one scheduling policy or the other. If you use both, you run this risk of giving, for example, a 50 percent share entitlement of a one-CPU Processor Set, which may be less overall CPU than 50 percent of all the processors. You have been warned . The format of a Processor Set entry is relatively simple:
<GROUP NAME>:PSET:::<No. of CPUS>:<CPU List>
If you don't specify a CPU List , PRM will assign CPUs to your group, and if you don't specify a No. of CPUS , PRM will calculate it for you. You don't need to assign all CPUs to Processor Sets. Any processors left will come under the control of the normal PRM scheduler. Here is the prmconf file for my Processor Set configuration:
# cat /etc/prmconf ... OTHERS:1:100:: QUARRY:PSET:::2:1,3 ADMIN:PSET:::1: IT:4:100:: ... fred::::QUARRY barney::::ADMIN wilma::::IT /home/haydes/bin/*adm*::::ADMIN /home/haydes/bin/comp-it*::::IT /home/haydes/bin/itDB::::IT,itDB* /home/haydes/bin/quarry*::::QUARRY /home/haydes/bin/rdr_quarry::::QUARRY,rdr_quarry* ... # #
As you can see, the only major difference is that the QUARRY and ADMIN groups are now in their own Processor Set. I have changed the share entitlement for the IT group to be the same as the OTHERS group just to ensure that everyone else on the system has an equal share of the remaining CPU. Now let's test and start the configuration:
# prmconfig -s Configuration file check complete. No errors found. # prmconfig -iep PRM configured from file: /etc/prmconf File last modified: Tue Nov 25 19:09:52 2003 PRM CPU scheduler state: Enabled PRM Group PRMID CPU Entitlement ----------------------------------------------------------------------------- ADMIN 65 25.00% (1 CPUs - 2) IT 4 12.50% OTHERS 1 12.50% QUARRY 64 50.00% (2 CPUs - 1, 3) PRM memory manager state: Not Running PRM User Initial Group Alternate Group(s) ----------------------------------------------------------------------------- adm OTHERS barney ADMIN bin OTHERS daemon OTHERS fred QUARRY haydes OTHERS hpdb OTHERS lp OTHERS mysql OTHERS nobody OTHERS nuucp OTHERS root (PRM_SYS) smbnull OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled #
Notice that the CPU entitlement printed by prmconfig is effectively the percentage of CPUs allocated to each Processor Set. Now with the applications running, we can view the Processor Set information with ps -z :
# ps -fzu haydes UID PSET PID PPID C STIME TTY TIME COMMAND haydes 0 20320 1 220 19:28:33 ? 6:23 itDB haydes 1 20321 1 255 19:28:33 ? 6:45 quarryDB haydes 0 20319 1 0 19:28:33 ? 0:05 comp-it3 haydes 1 20323 1 255 19:28:33 ? 6:43 rdr_quarry haydes 2 20314 1 232 19:28:32 ? 2:15 TRANS_adm.ship haydes 2 20313 1 233 19:28:32 ? 2:14 RECV_adm.ship haydes 2 20316 1 0 19:28:33 ? 0:00 adminRPT haydes 2 20315 1 239 19:28:33 ? 2:14 adminDB haydes 2 20312 1 0 19:28:32 ? 0:00 FLUSH_admin haydes 0 20318 1 0 19:28:33 ? 0:05 comp-it2 haydes 0 20317 1 0 19:28:33 ? 0:05 comp-it1 haydes 1 20322 1 0 19:28:33 ? 0:02 quarry_LOG #
We can see how that transposes into percentage CPU with prmmonitor :
# prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Tue Nov 25 19:09:52 2003 HP-UX uksd3120 B.11.11 U 9000/800 11/25/03 Tue Nov 25 19:30:41 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ____________________________________________________________ OTHERS 1 12.50% 0.00% IT 4 12.50% 25.00% QUARRY 64 50.00% 50.00% ADMIN 65 25.00% 25.00% PRM application manager state: Enabled (polling interval: 30 seconds) #
As you can see, each application is effectively isolated from others. In this instance, I suspect the overall application performance will improve with processes/threads staying in close CPU-proximity to where they ran before and reducing the overhead of reloading the CPU cache whenever a process/thread is moved to a new processor. Notice that IT has consumed all the available entitlement in the remaining CPUs. I am not going to apply CPU-capping because I prefer to let applications use whatever CPU cycles are available. 11.10.2 Using PRM to prioritize memory shares Main memory is a scarce resource on most systems. It is always a struggle to accommodate every user process in main memory all the time. Thankfully, HP-UX is relatively successful at managing memory in that we no longer swap entire processes out to disk; we page them out. We discussed the virtual memory system in Chapter 9, Swap and Dump Space. While we can rely on HP-UX to manage memory when the amount of free memory starts to run out, we should consider why we get to that situation in the first place. We run out of memory because processes/threads use it ”plain and simple. The question is, "Do we need to allow every group of users the same access to memory?" Normally, we limit access to memory by the various kernel parameters that control the data segment, stack, shared memory, and so on. The problem is that these limits apply to everyone on the system. On some rare occasions, I have used the POSIX-shell ulimit built-in command that can limit memory use on a per-process basis. The problem with this is that if users realize it is simply a feature of the shell, they will reset their own limits if they can get to a shell-prompt (restricted shells are worth thinking about in this instance). In reality, a single process could simply come along and grab a huge portion of main memory, up to the limits imposed by the kernel. That's why we have the kernel parameters. However, there are situations where for one group of users these limits are not enough. What do we do then? Ideally, we could give them their own server (an nPar or a vPar could be a good idea), whereby if they fill up main memory, they have no one to blame but themselves . Where this is not possible, we can turn to PRM to impose its ideas of share entitlements in respect of memory use. The memory share entitlements are calculated in the same way as CPU share entitlements . As well as basic share entitlement, there are a few other characteristics of memory shares we need to consider. These options can be expressed alongside basic memory shares in the /etc/prmconf file: -
CAP : Will we cap the amount of memory to which a PRM group has access? If we specify a CAP, it is expressed as an absolute percentage of main memory. -
SUPPRESS : This option determines which processes within a PRM group are to be suppressed if the PRM group reaches its entitlement. The default is to suppress all processes. Processes that lock memory are not suppressed. The idea is that PRM is effectively going to STOP processes until such times as the PRM group utilization falls below its entitlement (this explains why sending a process the STOP signal while the same process has locked it data/instructions in memory, makes no sense; it's locked and will not give up its memory). The other option is to suppress only the largest process. This needs to be considered carefully because sometimes the largest process is the process that has locked its memory. -
IMPORT : This, along with EXPORT, is used to implement memory isolation (when both are set to = 0). When left blank, proportional overachievement is possible. -
EXPORT : This, along with IMPORT, is used to implement memory isolation (set both to = 0). When left blank, proportional over-achievement is possible. -
LOCKABLE : This is the portion of a PRM group's entitlement that is available to be locked. Individual processes within the group will need the MLOCK privilege (see setprivgrp ) in order to lock themselves in memory. The idea behind memory records is to match the PRM group ID with the PRM group ID for CPU share records. Let's look at our previous example using CPU shares:
root@hpeos003[] more /etc/prmconf ... OTHERS:1:100:: QUARRY:2:6:: ADMIN:3:4:: IT:4:2:: ... fred::::QUARRY barney::::ADMIN wilma::::IT root@hpeos003[]
After analyzing the applications for these groups of users, we realize that the configuration will look something like this: -
ADMIN group need a large proportion of the memory on this system and is not willing to share any of its spare memory with anyone : SHARE = 10, IMPORT = 0, EXPORT = 0. -
QUARRY needs a reasonable amount of memory, but its main database process is prone to grabbing large chunks of memory if allowed, a fact that upsets a large part of the user community: SHARE = 5, CAP = 75. -
IT generally doesn't need lots of memory, but occasionally they run large program compilations. At this time, it would be nice to have some additional memory: SHARE = 3. -
OTHER processes don't really know how much they need, but we don't want them to swamp the memory system unnecessarily. SHARE = 1. A memory record takes the following form:
#!PRM_MEM:<GROUP>:<SHARE>:<CAP>:<SUPPRESS>::<IMPORT>:<EXPORT>:<LOCKABLE>
The memory records to implement this configuration would look something like this:
root@hpeos003[] vi /etc/prmconf ... OTHERS:1:100:: #!PRM_MEM:1:1::::::: QUARRY:2:6:: #!PRM_MEM:2:5:75:LARGEST::::: ADMIN:3:4:: #!PRM_MEM:3:10::::0:0:: IT:4:2:: #!PRM_MEM:4:3::::::: ... fred::::QUARRY barney::::ADMIN wilma::::IT /home/haydes/bin/*adm*::::ADMIN /home/haydes/bin/comp-it*::::IT /home/haydes/bin/itDB::::IT,itDB* /home/haydes/bin/quarry*::::QUARRY /home/haydes/bin/rdr_quarry::::QUARRY,rdr_quarry* ... root@hpeos003[]
I have grouped the memory records with the group records in an attempt to make it easier to associate the PRM group ID with the group name:
root@hpeos003[] prmconfig -s Configuration file check complete. No errors found. root@hpeos003[]
Now let's get the configuration up and running:
root@hpeos003[] prmconfig -ie PRM configured from file: /etc/prmconf File last modified: Tue Nov 25 22:15:07 2003 PRM CPU scheduler state: Enabled PRM Group PRMID CPU Entitlement ------------------------------------------------------- ADMIN 3 3.57% IT 4 1.79% OTHERS 1 89.29% QUARRY 2 5.36% PRM memory manager state: Enabled (polling interval: 10 seconds) PRM memory manager logging state: Disabled Memory Upper PRM Group PRMID Entitlement Bound Suppress ------------------------------------------------------------------ ADMIN 3 52.63% ALL IT 4 15.79% ALL OTHERS 1 5.26% ALL QUARRY 2 26.32% 75% LARGEST PRM User Initial Group Alternate Group(s) ----------------------------------------------------------------------------- adm OTHERS barney ADMIN bin OTHERS charlesk OTHERS daemon OTHERS fred QUARRY hpdb OTHERS ids OTHERS lp OTHERS mysql OTHERS nuucp OTHERS oracle OTHERS root (PRM_SYS) smbnull OTHERS sshd OTHERS sys OTHERS uucp OTHERS webadmin OTHERS wilma IT www OTHERS PRM application manager state: Enabled (polling interval: 30 seconds) PRM application manager logging state: Disabled Disk manager state: Disabled root@hpeos003[]
With the application up and running, here is the output from prmmonitor :
root@hpeos003[] prmmonitor 1 1 PRM configured from file: /etc/prmconf File last modified: Wed Nov 26 04:30:06 2003 HP-UX hpeos003 B.11.11 U 9000/800 11/26/03 Wed Nov 26 04:47:13 2003 Sample: 1 second CPU scheduler state: Enabled CPU CPU PRM Group PRMID Entitlement Used ------------------------------------------------------------ OTHERS 1 89.29% 0.97% QUARRY 2 5.36% 49.51% ADMIN 3 3.57% 33.01% IT 4 1.79% 16.50% Wed Nov 26 04:47:13 2003 Sample: 1 second Memory manager state: Enabled Paging: No Memory Upper PRM Group PRMID Entitlement Bound Usage Procs Stops ----------------------------------------------------------------------------- OTHERS 1 5.26% 0.69% 1 QUARRY 2 26.32% 75% 0.03% 2 ADMIN 3 52.63% 41.70% 5 IT 4 15.79% 12.52% 2 PRM application manager state: Enabled (polling interval: 30 seconds) root@hpeos003[]
As you can see, processes are currently not using their entitlement, but my memory shares are in place in case they do. | 