11.7 Memory Limitations for 32-bit Operating Systems, magic Numbers , and Memory Windows Processes running under the limitation of a 32-bit address space have serious limitations regarding the total amount of data they can address. Each quadrant in the address space limits each of the major data structures, i.e., user text/code and user data to 1GB in size . For these private data areas, this limitation is probably not too much of a concern. When we consider the Shared Objects in the address space, we must remember that these objects are accessible to all processes on the system. Quadrants 3 and 4 of every process will reference the same address space. This means that all processes are sharing an address space of effectively 1.75GB (the last 256MB is reserved for IO routines). I will say that again because it is important: Every process in the system is sharing a data area that is 1.75GB in size . What we are effectively saying is that all of the shared memory segments created by every process in the system is limited to 1.75GB in total. This is a serious limitation. If we had four instances of a database running simultaneously on one server, the total amount of shared memory to all of the instances would be 1.75GB in total. I believe that this was a driving factor in having a separate server for each instance of an application, i.e., a server to support the finance database, a separate server to support the sales database, and so on. Now that we have larger 64-bit servers, surely this limitation has gone. The limitation is effectively gone as long as we are running a 64-bit application in a 64-bit operating system. With a large number of 32-bit applications still in operation, this limitation is still an issue. It is conceivable to run a 32-bit application in a 64-bit operating system (the operating system uses a concept known as address swizzling to transform a 64-bit address into a 32-bit address). With backward compatibility a major issue for major vendors , it is common that programs are simply copied from an older 32-bit server to a new 64-bit server without being recompiled. What we have is a 32-bit application running in a 64-bit operating system and the corresponding limitations to the address space for the 32-bit applications. All of the 32-bit applications are still limited to 1.75GB of shared objects. We need a solution to this limitation. The best solution is to recompile the application to be 64-bit. This is not always possible. Some software vendors don't support their applications running in 64-bit mode. They insist that the application remain 32-bit. Such applications will experience the limitations we are speaking of here. What we need to do is consult with our application administrators and discuss the fact that by default all of the 32-bit instances of the application will have access to only 1.75GB of shared objects in total. If this is not an issue, then we can halt this discussion here. If this is going to be a problem, we need to find a solution. The solutions involve either changing something called the magic number of a program or using memory windows . 11.7.1 Program magic numbers When a program is compiled, a flag is set in the header of the program to instruct the kernel about any special attributes pertaining to how the kernel should treat this program when it is being executed. This flag is known as the magic number and is set by the compiler, the linker, or a command called chatr . When a 32-bit program is compiled, by default the magic number is set to SHARE_MAGIC (known as a shared executable ). This means that the kernel will locate objects within the address space as described so far. We can view the magic number using the chatr command:
root@hpeos003[charlesk] cc -o bigcpu bigcpu.c root@hpeos003[charlesk] chatr bigcpu bigcpu: shared executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space disabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled root@hpeos003[charlesk]
The problem we have is that a compiled 32-bit binary by default is limited to 1.75 GB of space for share objects. If we are able to have the program recompiled with a special flag, we can change this behavior. What we need is a program known as a normal executable (magic number = EXEC_MAGIC ). This requires the application developer to use the N option to compile the program:
root@hpeos003[charlesk] cc -N -o bigcpu bigcpu.c root@hpeos003[charlesk] chatr bigcpu bigcpu: normal executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space disabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled root@hpeos003[charlesk]
This allows the kernel to use the space immediately after the private code finishes as the start of the private data area, as depicted in Figure 11-11. Figure 11-11. EXEC_MAGIC executable. 
In itself, this has given us a program that can use +1GB of space for private data. If the application uses large private Memory Mapped Files, it might be an ideal solution to have a 1GB private data area. For our applications that use shared objects, this has not eliminated our problem of being limited to 1.75GB for all shared objects for the entire system. This first step in making an executable an EXEC_MAGIC executable is a crucial step. Now that we have an EXEC_MAGIC executable, we can further change the magic number in order to allow the shared objects to use more space in the address range. What we will do is transform our EXEC_MAGIC executable into a SHMEM_MAGIC executable by using the chatr M command:
root@hpeos003[charlesk] chatr -M bigcpu bigcpu: current values: normal executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space disabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled new values : normal SHMEM_MAGIC executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space disabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled root@hpeos003[charlesk]
This has made a significant impact on how the kernel will position objects in the address space of this process. If we think back to our original discussion regarding application programs, we said that it is common for programs to have small quantities of private data and instructions. The bulk of the memory used by an application attributed to shared objects. With this new magic number in place, the shared objects are allowed to utilize the vast majority of the address space. As we can see from Figure 11-12, we have dramatically increased the size of the space for shared objects. The total space available to shared objects is now  2.75GB. Remember, this 2.75GB is the space shared among all applications running in the system because we are discussing shared objects . If this is still not enough space, we need to discuss a concept known as memory windows . Figure 11-12. SHMEM_MAGIC executable. 
Before we leave magic numbers, there's one more possibility we should discuss. When we complied our EXEC_MAGIC executable, this allowed our program to utilize +1GB of space for user/private data. If this application makes extensive use of user/private data, e.g., a video-editing program, that doesn't use shared objects extensively, we can further modify the magic number in such a way as to allow the kernel to map user/private data into Quadrants 3 and 4. NOTE: This is applicable only for 32-bit programs running under 64-bit HP-UX that do not make extensive use of shared objects such as shared memory segments. We use special options to the chatr command to allow either just Quadrant 3 to map user/private data (+q3p - quadrant 3 private ) or to allow both Quadrants 3 and 4 to map user/private data (+q4p - quadrant 3 and 4 private ). Figure 11-13. EXEC_MAGIC + private Quadrant 4 and/or 3 data. 
Here's an example using the chatr command:
root@hpeos003[charlesk] chatr +q4p enable bigcpu bigcpu: current values: normal SHMEM_MAGIC executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space disabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled new values: normal SHMEM_MAGIC executable shared library dynamic path search: SHLIB_PATH disabled second embedded path disabled first Not Defined shared library list: dynamic /usr/lib/libc.2 shared library binding: deferred global hash table disabled plabel caching disabled global hash array size:1103 global hash array nbuckets:3 shared vtable support disabled static branch prediction disabled executable from stack: D (default) kernel assisted branch prediction enabled lazy swap allocation disabled text segment locking disabled data segment locking disabled third quadrant private data space disabled fourth quadrant private data space enabled third quadrant global data space disabled data page size: D (default) instruction page size: D (default) nulptr references disabled shared library private mapping disabled shared library text merging disabled root@hpeos003[charlesk]
This gives an effective private data area of up to approximately 2.85GB with +q3p enabled and approximately 3.8GB with +q4p enabled. To make use of this additional space, the only other change we will probably have to do is to reconfigure the kernel parameter maxdsiz to allow for a larger user data segment. 11.7.2 Memory windows We are continuing our discussion regarding 32-bit applications running in 64-bit HP-UX that are experiencing problems with the limited address space that 32-bit brings . We are specifically interested in applications that have tried to utilize the SHMEM_MAGIC magic number, but still come up against problems because this only increased the total pool of shared objects to 2.75GB in size. The problem could be that we have multiple instances of a database running on the same server. This last point is a crucial point if we are to utilize memory windows . Where we have consolidated multiple servers onto one more powerful server, it is not uncommon to have multiple applications running simultaneously. If these applications are all 32-bit applications experiencing the address space problems mentioned above, we might be able to utilize memory windows depending on the answer to an important question: " Do these applications ever access shared objects from another application? " What we mean by this, using an example, is: "Does the finance application ever attach to and read from a shared memory segment belonging to the sales application?" If the answer to this question is a definite no, then each application can make use of memory windows . A memory window allows a process/application to utilize a unique definition for Quadrants 2 and 3 of its address space. This alleviates the problem of the system-wide limitation of 1.75GB of shared objects for SHARE_MAGIC executables and 2.75GB for SHMEM_EXEC executables. Quadrant 4 is still shared because we need somewhere to locate shared libraries, and it is in Quadrant 4 that we have important kernel routines including IO libraries. Effectively, we get an address space that behaves like the diagram we see in Figure 11-14. Figure 11-14. SHMEM_MAGIC + memory windows. 
Two steps to implement memory windows are relatively straightforward: - Tune the max_mem_window kernel parameter to be the maximum number of memory windows you require. To size this, we should estimate how many 32-bit applications will need their own memory windows .
root@hpeos003[] kmtune -q max_mem_window Parameter Current Dyn Planned Module Version ============================================================================= max_mem_window 10 - 10 root@hpeos003[]
The default for this parameter is actually 0. I have modified this parameter ahead of time. Unfortunately, to modify this parameter will require a reboot of this system.
- Set up a /etc/services.window configuration file to define each memory window .
root@hpeos003[] vi /etc/services.window finance 10 sales 20 root@hpeos003[]
This file doesn't exist by default. As you can see I am simply giving each memory window a name and an associated numerical value. This step is technically not necessary but it makes managing memory windows a little easier.
- Modify the relevant application startup routines to launch the application in its own memory window using the getmemwindow and setmemwindow commands. The getmemwindow command will simply extract the memory window number from the /etc/services.window file:
root@hpeos003[] getmemwindow finance 10 root@hpeos003[]
We can use this in a startup routine for the finance application to ensure that the application always starts up on the proper memory window.
root@hpeos003[] cat /finance/scripts/fin.start #!/sbin/sh PATH=/sbin:/usr/sbin:/usr/bin DEPARTMENT=finance WINDOW=$(getmemwindow $DEPARTMENT) setmemwindow b i $WINDOW /finance/bin/finDB root@hpeos003[] root@hpeos003[] /finance/scripts/fin.start root@hpeos003[]
The b option is necessary when we have a SHMEM_MAGIC to allow Quadrants 2 and 3 to be treated as a contiguous block of space.
- Monitor the use of memory windows with the free contributed utility shminfo available from ftp:// contrib :9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo.
root@hpeos003[] shminfo -w 10 libp4 (7.0): Opening /stand/vmunix /dev/kmem Loading symbols from /stand/vmunix shminfo (3.6) Shared space from Window id 10 : Space Start End Kbytes Usage Q2 0x04c3bc00.0x40000000-0x7fffffff 1048576 FREE Q3 0x0422a800.0x80000000-0x800fffff 1024 SHMEM id=7007 Q3 0x0422a800.0x80100000-0xbfffffff 1047552 FREE root@hpeos003[]
I can still use the ipcs command, but I will not get any information relating to memory windows:
root@hpeos003[] ipcs -mbop IPC status from /dev/kmem as of Sat Nov 22 03:47:16 2003 T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID Shared Memory: m 0 0x411c28bb --rw-rw-rw- root root 0 348 667 667 m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 667 667 m 2 0x412001e0 --rw-rw-rw- root root 1 8192 667 679 m 3 0x301c5666 --rw-rw-rw- root root 1 1048576 1739 1739 m 2004 0x5e10001b --rw------- root root 1 512 1978 1978 m 205 0x00000000 D-rw------- root root 6 1052672 2060 2060 m 1006 0x00000000 D-rw------- www other 6 184324 2068 2068 m 7007 0x00001ed2 --rw-rw-rw- root sys 1 1048576 6352 6379 root@hpeos003[]
This application will now operate in its own memory window . We need to ensure that all relevant commands associated with this application have a wrapper script that will use the setmemwindow command to launch the program in the correct memory window .
| 