Non-Uniform Memory Access (NUMA)

In Non-Uniform Memory Access (NUMA) architecture, a machine consists of multiple nodes, with each node consisting of one or more processors and local memory. A single system image of the operating system is typically used by NUMA machines, and all of the memory is accessible to tasks executing on any of the processors on any node. Local memory access times are faster than remote memory access times, and it is desirable for tasks executing on a processor on a given node to have its associated working set on the local memory of that node.

Migration of a task to another processor on the same node might be necessary, but migration of a task to another node should be avoided because of the cost of remote fetches. Some architectures have an onboard cache for remote memory to try to mitigate the cost of remote fetches, which further complicates task placement methodology.

A variant on this situation occurs when memory that is closest or fastest to a processor is dependent on some other conditionfor example, electrically by location on the bus, or logically by other components' management of the memory. In all cases, however, the same issue remains: some memory is "better" than others.

Successful scheduling on a NUMA machine is more complex than on a single processor. A policy that frequently migrates tasks from their original processor might still perform well if it can keep those tasks close to the memory they use. Processor speeds have increased over the years, and it can be well worth the extra instruction cycles to determine how best to keep memory references local rather than remote.

As if successful scheduling on a NUMA machine weren't challenging enough, Linux has also provided answers to scheduling on machines with multithread support. This gets even more interesting as you work around the virtual CPU.