10.4 Distributed make

     

GNU make supports a little known (and only slightly tested ) build option for managing builds that uses multiple systems over a network. The feature relies upon the Customs library distributed with Pmake . Pmake is an alternate version of make written in about 1989 by Adam de Boor (and maintained ever since by Andreas Stolcke) for the Sprite operating system. The Customs library helps to distribute a make execution across many machines in parallel. As of version 3.77, GNU make has included support for the Customs library for distributing make .

To enable Customs library support, you must rebuild make from sources. The instructions for this process are in the README.customs file in the make distribution. First, you must download and build the pmake distribution (the URL is in the README), then build make with the ”with-customs option.

The heart of the Customs library is the customs daemon that runs on each host participating in the distributed make network. These hosts must all share a common view of the filesystem, such as NFS provides. One instance of the customs daemon is designated the master. The master monitors hosts in the participating hosts list and allocates jobs to each member. When make is run with the ”jobs flag greater than 1, make contacts the master and together they spawn jobs on available hosts in the network.

The Customs library supports a wide range of features. Hosts can be grouped by architecture and rated for performance. Arbitrary attributes can be assigned to hosts and jobs can be allocated to hosts based on combinations of attributes and boolean operators. Additionally, host status such as idle time, free disk space, free swap space, and current load average can also be accounted for when processing jobs.

If your project is implemented in C, C++, or Objective-C you should also consider distcc (http://distcc.samba.org) for distributing compiles across several hosts. distcc was written by Martin Pool and others to speedup Samba builds. It is a robust and complete solution for projects written in C, C++, or Objective-C. The tool is used by simply replacing the C compiler with the distcc program:

 $  make --jobs=8 CC=distcc  

For each compilation, distcc uses the local compiler to preprocess the output, then ships the expanded source to an available remote machine for compilation. Finally, the remote host returns the resulting object file to the master. This approach removes the necessity for having a shared filesystem, greatly simplifying installation and configuration.

The set of worker or volunteer hosts can be specified in several ways. The simplest is to list the volunteer hosts in an environment variable before starting distcc :

 $  export DISTCC_HOSTS='localhost wasatch oops'  

distcc is very configurable with options for handling host lists, integrating with the native compiler, managing compression, search paths, and handling failure and recovery.

ccache is another tool for improving compilation performance, written by Samba project leader Andrew Tridgell. The idea is simple, cache the results of previous compiles. Before performing a compile, check if the cache already contains the resulting object files. This does not require multiple hosts, or even a network. The author reports a 5 to 10 times speed up in common compilations. The easiest way to use ccache is to prefix your compiler command with ccache :

 $  make CC='ccache gcc'  

ccache can be used together with distcc for even greater performance improvements. In addition, both tools are available in the Cygwin tool set.



Managing Projects with GNU make
Managing Projects with GNU Make (Nutshell Handbooks)
ISBN: 0596006101
EAN: 2147483647
Year: 2003
Pages: 131

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net