7.3 Gigabit Ethernet

Though with larger overhead to computing nodes and lower bandwidth for inter-node communication compared with Myrinet, Gigabit Ethernet (GigE), with its advantage in price, can sometimes be the choice for an HPC environment. This is especially true if applications to be run on such a system are:

• Embarrassingly Parallel. In this case, there is no communication, or very little communication, between processes. Examples are sequence searching programs in life sciences (BLAST, FASTA), low level image processing, Mandelbrot set, and Monte Carlo simulations.

• Hybrid message passing and multi-threading . In this case, one can adjust the number of processes and the number of threads to achieve a balanced computing and communication performance. For example, for a cluster of two SMP nodes each with 32 processors, you can run a hybrid MPI/OpenMP job with two MPI processes and 32 OpenMP threads per MPI process, making full use of the total 64 processors. You can also start four MPI processes each with 16 threads, and so on.

In our case study, discussed in the following section, we have a configuration of two p655 nodes interconnected by Gigabit Ethernet.

In this case, the GM driver and MPICH_GM are not needed. You can use ANL's MPICH to compile and run MPI programs. The installation is more or less the same as MPICH_GM. It is worth noting that in order to use the SMP feature, it is better to have two kinds of installations:

• One with the following configuration option:

  ./configure --with-device=ch_p4 --with-comm=shares

• And the other with:

  ./configure --with-device=ch_shared

The first one will work with a cluster of SMP nodes in which the intra-node communication will be done through a shared memory mechanism, while the inter-node communications go to the network adapters.

The second one will only work on a single SMP node using shared memory for communications. However, somehow it performs better than the first one.

So, it's useful to have both installed.