Chapter 6: Setting up Clusters

Overview

Philip Papadopoulos

Your day is starting off well—many boxes of the fastest-ever cluster nodes have arrived on your shipping dock and you are ready to tear them open, set them up, and start running your favorite code just as soon as you can. The space is planned out, you know what kinds of nodes you need in terms of login, storage, compute and other types of devices, the electricians finished putting in the plugs yesterday, and the cooling has been upgraded to keep your cluster nicely chilled. It then dawns up you, you are looking at literally hundreds of components—computers, power distribution units, networking switches, racks (or tables, or shelves), disks, and a monitor or two. You need to have a plan for how to layout and organize your cluster for the physical buildout. The harder part is really deciding on how you are going to get the operating system onto each one of the nodes so that you can get started computing in the shortest time possible. There isn't a "right way" or a "wrong way" to do this. And, in reality, how you provision your nodes has quite a bit to do with personal taste and administrative style. There are, however, are two broad issues to keep in mind—scalability and reproducibility.

Initial setup is closely aligned with management of the cluster (see Chapter 13). Setup is not a one-time task for a few reasons. First, nodes do break and replacement nodes need to be turned on and provisioned with the latest software. Second, clusters often incrementally expand requiring you to provision new (and sometimes physically different) nodes over the lifetime of a the machine. Third, Linux moves quite rapidly—with 3 package updates every two days on the current release of Redhat—at some point, patching simply won't work and a wholesale re-installation is needed to make the cluster stable and consistent. A real plan is needed for both management and setup. These two aspects support each other.

This chapter is organized as follows: the challenges of cluster setup are introduced first, some simple but effective tips for physically organizing your system, an overview of general approaches to cluster setup is given, and finally some detail is given about two popular approaches to cluster setup: NPACI Rocks's description-based installation, and OSCAR's image-based setup. This chapter is not meant to be a step-by-step instruction manual for any particular method, but has the primary purpose of giving the reader some comparisons and motivation for why different teams choose very different approaches.

Before launching into the deep details it is worth noting that this chapter covers traditional Beowulfs where each node has a disk and contains a local copy of the operating system. Single system image toolkits like Scyld and SCore have their own custom installers. In particular, Scyld's setup and management are given in Chapter 18. Diskless systems are not covered in this section.

After reading this chapter, the reader should have a good overview of how different provisioning methodologies work. Setup and ongoing management are intimately connected, and an administrator's style often dictates how a system is provisioned. In the end, instantiated clusters, whether built with Rocks, OSCAR, SCE, or even other lesser-known systems toolkits, have very similar functionality. Afterall, each has a basic Linux OS, a queueing system, monitoring, message passing, and I/O subsystems. The key to evaluating each of these systems for your own use is how does a particular approach reduce your time for administrative issues and increase your time for actually using the cluster.