3.8. Large Page Support Large page usage is primarily intended to provide performance improvements to applications that allocate a lot of memory and frequently access that memory. The performance improvement is mainly due to the reduced number of misses in the Translation Lookaside Buffer (TLB), a cache of virtual-to-physical translation. The number of misses can be reduced when the TLB is able to map a larger virtual memory range. This is possible due to the multiple page size support provided by most modern architectures. For example, the 32-bit Intel architecture supports 4K and 4MB (2MB in PAE mode) page sizes; Itanium supports multiple page sizes 4K, 8K, 64K, 256K, 1MB, 4MB, 16MB, and 256MB; SUN UltraSPARC supports 8K, 64K, 512K, and 4MB; and the 64-bit PowerPC (ppc64) supports 4K, 64K, 16MB, and 64GB. In this section, we give information for application developers on how to use large page support provided by the Linux kernel. The 2.6 kernel has built-in support for hugetlbpage (a term used by the Linux community for large page). With the kernel configured with hugetlbpage support, the output of the command cat /proc/meminfo should display an output having the information about hugetlbpage, as follows: HugePages_Total: 20 HugePages_Free: 20 Hugepagesize: 16384 KB
A filesystem of type hugetlbfs should also appear in /proc/filesystems. Before the user-space application can use hugetlbpage support, huge pages have to be allocated in the kernel by the administrator. /proc/sys/vm/nr_hugepages indicates the current number of configured huge pages in the kernel. To configure ten huge pages in the system, use this command: echo 10 > /proc/sys/vm/nr_hugepages
The allocation will be successful only if there is enough physically contiguous memory in the system at that moment. The deallocation of huge pages is possible only if there are enough huge pages free that can be transferred back to the regular memory pool. Pages that are used as hugetlb pages are reserved inside the kernel and cannot be used for other purposes. There are two ways for application developers to use the hugetlbpage support: The system V shared memory system call (shmget, shmat) The mmap system call
It is possible for the same application to use a combination of mmap and shm*. In the following programming examples, we show how to use these system calls for the hugetlbpage support. These programs originally appeared in /usr/src/linux/Documentation/vm/hugetlbpage.txt. In Example 3-2, the application uses the System V shared memory system calls to request 256MB of memory backed by huge pages. The SHM_HUGETLB flag is used in the shmget system call to inform the kernel that huge pages are requested. Example 3-2. Using Huge Page Memory via System V Shared Memory Interfaces /* * Example of using huge page memory in a user application using Sys V * shared memory system calls. In this example, the app is requesting * 256MB of memory that is backed by huge pages. The application * uses the flag SHM_HUGETLB in the shmget system call to inform the * kernel that it is requesting huge pages. * * For the ia64 architecture, the Linux kernel reserves Region number 4 * for huge pages. That means the addresses starting with 0x800000... * will need to be specified. Specifying a fixed address is not * required on ppc64, i386 or x86_64. * * Note: The default shared memory limit is quite low on many kernels; * you may need to increase it via: * * echo 268435456 > /proc/sys/kernel/shmmax * * This will increase the maximum size per shared memory segment to * 256MB. The other limit that you will hit eventually is shmall which * is the total amount of shared memory in pages. To set it to 16GB on * a system with a 4KB pagesize do: * * echo 4194304 > /proc/sys/kernel/shmall */ #include <stdlib.h> #include <stdio.h> #include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h> #include <sys/mman.h> #ifndef SHM_HUGETLB #define SHM_HUGETLB 04000 #endif #define LENGTH (256UL*1024*1024) #define dprintf(x) printf(x) /* Only ia64 requires this */ #ifdef __ia64__ #define ADDR (void *)(0x8000000000000000UL) #define SHMAT_FLAGS (SHM_RND) #else #define ADDR (void *)(0x0UL) #define SHMAT_FLAGS (0) #endif int main(void) { int shmid; unsigned long i; char *shmaddr; if ((shmid = shmget(2, LENGTH, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W)) < 0) { perror("shmget"); exit(1); } printf("shmid: 0x%x\n", shmid); shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS); if (shmaddr == (char *)-1) { perror("Shared memory attach failure"); shmctl(shmid, IPC_RMID, NULL); exit(2); } printf("shmaddr: %p\n", shmaddr); dprintf("Starting the writes:\n"); for (i = 0; i < LENGTH; i++) { shmaddr[i] = (char)(i); if (!(i % (1024 * 1024))) dprintf("."); } dprintf("\n"); dprintf("Starting the Check..."); for (i = 0; i < LENGTH; i++) if (shmaddr[i] != (char)i) printf("\nIndex %lu mismatched\n", i); dprintf("Done.\n"); if (shmdt((const void *)shmaddr) != 0) { perror("Detach failure"); shmctl(shmid, IPC_RMID, NULL); exit(3); } shmctl(shmid, IPC_RMID, NULL); return 0; }
|
For ia86 architecture, the kernel reserves a specific region for huge pages. That means a fixed address in the calling process must be specified. A fixed address is not required for i386, x86_64, or ppc64. You may need to increase the maximum size per shared memory segment to 256MB. This can be done by issuing this command: echo 268435456 > /proc/sys/kernel/shmmax
The other limit that you need to be concerned about is /proc/sys/kernel/shmall, which is the total amount of shared memory in pages. mmap system call
In this case, it is required that the system administrator mount a filesystem of type hugetlbfs first. Any files created on this mount point will be backed by huge pages. mount none /mnt/huge -t hugetlbfs -o uid=1000,gid=100
The preceding command mounts a filesystem of type hugetlbfs on the directory /mnt/huge with the owner and group of the root of the filesystem set to 1000 and 100, respectively. Example 3-3 provides an application using the mmap system call to request a memory of size 256MB that is backed by huge pages. Example 3-3. Using Huge Page Memory via the mmap Routine /* * Example of using huge page memory in a user application using the * mmap system call. Before running this application, make sure that * the administrator has mounted the hugetlbfs filesystem (on some * directory like /mnt) using the command mount -t hugetlbfs nodev * /mnt. In this example, the app is requesting memory of size 256MB * that is backed by huge pages. * * For ia64 architecture, Linux kernel reserves Region number 4 for * huge pages. That means the addresses starting with 0x800000... will * need to be specified. Specifying a fixed address is not required * on ppc64, i386 or x86_64. */ #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <sys/mman.h> #include <fcntl.h> #define FILE_NAME "/mnt/hugepagefile" #define LENGTH (256UL*1024*1024) #define PROTECTION (PROT_READ | PROT_WRITE) /* Only ia64 requires this */ #ifdef __ia64__ #define ADDR (void *)(0x8000000000000000UL) #define FLAGS (MAP_SHARED | MAP_FIXED) #else #define ADDR (void *)(0x0UL) #define FLAGS (MAP_SHARED) #endif void check_bytes(char *addr) { printf("First hex is %x\n", *((unsigned int *)addr)); } void write_bytes(char *addr) { unsigned long i; for (i = 0; i < LENGTH; i++) *(addr + i) = (char)i; } void read_bytes(char *addr) { unsigned long i; check_bytes(addr); for (i = 0; i < LENGTH; i++) if (*(addr + i) != (char)i) { printf("Mismatch at %lu\n", i); break; } } int main(void) { void *addr; int fd; fd = open(FILE_NAME, O_CREAT | O_RDWR, 0755); if (fd < 0) { perror("Open failed"); exit(1); } addr = mmap(ADDR, LENGTH, PROTECTION, FLAGS, fd, 0); if (addr == MAP_FAILED) { perror("mmap"); unlink(FILE_NAME); exit(1); } printf("Returned address is %p\n", addr); check_bytes(addr); write_bytes(addr); read_bytes(addr); munmap(addr, LENGTH); close(fd); unlink(FILE_NAME); return 0; }
Note that read and write system calls are not supported on files residing on a hugetlbfs filesystem. A regular chown, chgrp, and chmod (with the right permissions) could be used to change the file attributes on hugetlbfs. |
|
