2.3 Multiple CPU Systems

2.3 Multiple CPU Systems

Adding a second CPU to the computer increases the complexity of synchronizing the system's activities. Each CPU has its own set of registers including the PS register containing the SPL and mode information. Therefore, any attempt by software to set the SPL to a non-zero value with the goal of synchronizing with an ISR will be faulty. It will provide synchronization but only with ISRs that happen to run on that particular CPU. The ISRs and other kernel mode code running on another CPU will not be aware of the SPL setting on this CPU.

Figure 2-3 depicts a two CPU system. Note that either CPU can access any of the physical memory or I/O devices on the system.

Figure 2-3: Dual-CPU Hardware Components

2.3.1 Asynchronous Multiprocessing (ASMP)

One solution to the synchronization issue would be to force all code that runs in kernel mode (including ISRs) to run on a particular CPU. This is a form of ‘funneling’ that forces activities to run on a particular CPU (usually the primary CPU). Forcing all of the kernel mode activities to run on a single CPU in a multiprocessor system (more than one CPU in the box) is a solution referred to as Asymmetric Multiprocessing (ASMP). ASMP is relatively simple to implement and allows the SPL synchronization for kernel mode activities to continue working even though there are multiple CPUs involved. However, it causes high scheduling overhead as each process that needs to get into kernel mode (for a system call, for example) must be rescheduled to run on the primary CPU.

ASMP worked, but didn't work well enough, and it scaled poorly. As more CPUs were added, fewer of their CPU cycles could actually be utilized. There had to be another way to synchronize between multiple CPUs. Remember, we are still talking about a single computer. We have not yet reached a point where we are adding more computers or clustering computers, but we will.

Figure 2-4 shows a two-CPU system in an asymmetric arrangement. The top CPU has been designated as the primary CPU and will handle all interrupts and run all kernel mode code.

Figure 2-4: Asynchronous Multiprocessing

2.3.2 Symmetric Multiprocessing (SMP) – Tightly Coupled

In order for a system to be deemed Symmetric, all of the CPUs in the system must be able to execute all modes of processing. In UNIX, the two modes are user mode and kernel mode. Software running on any CPU in a multiprocessor system needs to be coordinated with respect to access to kernel data structures that may be referenced (perhaps simultaneously) by other kernel mode software (including ISRs) running on other CPUs. The CPU-centric SPL synchronization method is not sophisticated enough to handle multiple CPUs.

Tru64 UNIX uses a memory interlocking mechanism and special ‘load-locked’, ‘store-conditional’ instructions to achieve the necessary extra level of synchronization. Memory is accessible to all CPUs in the system. The idea is to protect kernel structures or arrays with a spinlock bit. The spinlock is nothing but a bit in memory that is designated to coordinate access to a particular structure, field, array, device, etc. The term ‘spinlock’ is generic. Tru64 UNIX refers to these locks as ‘simple locks’. Let's face it, on the surface, they are about as simple as they can get. If you can set the bit, you have access to the resource represented by the bit; if you cannot (because it is already set), you do not have access to the resource represented by the bit. When access is denied, the requestor usually will ‘spin’ and repetitively check to see if the bit is clear. Once it is clear, the requestor can set it and access the protected resource. Do not get this confused with application level file locks and related activities. File locks are available to an application program, while spinlocks (‘simple locks’) are available to the operating system kernel. User mode application code will never directly request a simple lock.

Now we have a mechanism that provides synchronization between multiple CPUs within a system. The next step is to figure out how to coordinate access to resources when the accessing code is running on multiple systems. For the record, we have just discussed synchronization between ‘tightly coupled’ CPUs that use the same memory copy of the operating system software running on more than one CPU.

Figure 2-5 depicts a ‘tightly coupled’ system with two CPUs in an SMP arrangement. Note that both CPUs can run user mode and kernel mode code.

Figure 2-5: Symmetric Multiprocessing