Parallel Computing on Heterogeneous Networks, by Alexey Lastovetsky
ISBN 0-471-22982-2 Copyright 2003 by John Wiley & Sons, Inc.
Local networks of computers (NoCs) are the most common and available parallel architecture. Nowadays not only big businesses and organizations but practically any medium or small enterprize has several computers interconnected in a local network.
Generally, a local network of computers consists of PCs, workstations, SMP servers, and even MPP supercomputers and clusters interconnected via mixed network equipment (see Figure 5.1). At a first glance this architecture is very similar to the MPP architecture. Like the latter it provides a number of processors not sharing global main memory and interconnected via a communication network. Therefore the most natural model of program for NoCs is also a set of parallel processes, each running on a separate processor and using message passing to communicate with the others. That is, message passing is the basic programming model for this architecture.
Figure 5.1: Local network of computers.
Because of this similarity between MPPs and NoCs, it might be expected that NoCs would be as widely used as MPPs for high-performance parallel computing. In reality, NoCs are hardly used for parallel computing. The reason is why the huge performance potential of millions NoCs around the world is so underutilized is that parallel programming for NoCs is a lot more difficult than parallel programming for MPPs.
Unlike MPPs, which are designed and manufactured specifically for high-performance parallel computing, a typical NoC is a naturally developed computer system. A NoC is a general-purpose computer system that is developed incrementally over a relatively long period of time. As a result NoCs are not nicely regular nor balanced for high-performance computing. On the contrary, irregularity, heterogeneity, and instability are their inherent features unlike the architecture of MPPs. These very features make parallel programming for NoCs particularly challenging.
There are three sources of difficulties. The first one is the heterogeneity of processors. Generally speaking, in a NoC, different processors are of the different architecture.
The second source is the communication network. As we noted above, it is not designed for high-performance parallel computing.
The third source is the multiple-user capacity of NoCs. A NoC is not a strongly centralized computer system. It consists of relatively autonomous computers, each of which may be used and administered independently by its users. In NoCs different components are not as strongly integrated and controlled as in MPPs.
In the chapter we discuss the sources and analyze the programming challenges coming from each of these sources.