| The motivation behind recursive make is simple: make works very well within a single directory (or small set of directories) but becomes more complex when the number of directories grows. So, we can use make to build a large project by writing a simple, self-contained makefile for each directory, then executing them all individually. We could use a scripting tool to perform this execution, but it is more effective to use make itself since there are also dependencies involved at the higher level. For example, suppose I have an mp3 player application. It can logically be divided into several components : the user interface, codecs, and database management. These might be represented by three libraries: libui.a , libcodec.a , and libdb.a . The application itself consists of glue holding these pieces together. A straightforward mapping of these components onto a file structure might look like Figure 6-1. Figure 6-1. File layout for an MP3 player 
A more traditional layout would place the application's main function and glue in the top directory rather than in the subdirectory app/player . I prefer to put application code in its own directory to create a cleaner layout at the top level and allow for growth of the system with additional modules. For instance, if we choose to add a separate cataloging application later it can neatly fit under app/catalog . If each of the directories lib/db , lib/codec , lib/ui , and app/player contains a makefile , then it is the job of the top-level makefile to invoke them. lib_codec := lib/codec lib_db := lib/db lib_ui := lib/ui libraries := $(lib_ui) $(lib_db) $(lib_codec) player := app/player .PHONY: all $(player) $(libraries) all: $(player) $(player) $(libraries): $(MAKE) --directory=$@ $(player): $(libraries) $(lib_ui): $(lib_db) $(lib_codec)
The top-level makefile invokes make on each subdirectory through a rule that lists the subdirectories as targets and whose action is to invoke make : $(player) $(libraries): $(MAKE) --directory=$@
The variable MAKE should always be used to invoke make within a makefile . The MAKE variable is recognized by make and is set to the actual path of make so recursive invocations all use the same executable. Also, lines containing the variable MAKE are handled specially when the command-line options ”touch ( -t ), ”just-print ( -n ), and ”question ( -q ) are used. We'll discuss this in detail in Section 6.1.1 later in this chapter. The target directories are marked with .PHONY so the rule fires even though the target may be up to date. The ”directory ( -C ) option is used to cause make to change to the target directory before reading a makefile . This rule, although a bit subtle, overcomes several problems associated with a more straightforward command script: all: for d in $(player) $(libraries); \ do \ $(MAKE) --directory=$$d; \ done
This command script fails to properly transmit errors to the parent make . It also does not allow make to execute any subdirectory builds in parallel. We'll discuss this feature of make in Chapter 10. As make is planning the execution of the dependency graph, the prerequisites of a target are independent of one another. In addition, separate targets with no dependency relationships to one another are also independent. For example, the libraries have no inherent relationship to the app/player target or to each other. This means make is free to execute the app/player makefile before building any of the libraries. Clearly, this would cause the build to fail since linking the application requires the libraries. To solve this problem, we provide additional dependency information. $(player): $(libraries) $(lib_ui): $(lib_db) $(lib_codec)
Here we state that the makefile s in the library subdirectories must be executed before the makefile in the player directory. Similarly, the lib/ui code requires the lib/db and lib/codec libraries to be compiled. This ensures that any generated code (such as yacc / lex files) have been generated before the ui code is compiled. There is a further subtle ordering issue when updating prerequisites. As with all dependencies, the order of updating is determined by the analysis of the dependency graph, but when the prerequisites of a target are listed on a single line, GNU make happens to update them from left to right. For example: all: a b c all: d e f
If there are no other dependency relationships to be considered , the six prerequisites can be updated in any order (e.g., "d b a c e f"), but GNU make uses left to right within a single target line, yielding the update order: "a b c d e f" or "d e f a b c." Although this ordering is an accident of the implementation, the order of execution appears correct. It is easy to forget that the correct order is a happy accident and fail to provide full dependency information. Eventually, the dependency analysis will yield a different order and cause problems. So, if a set of targets must be updated in a specific order, enforce the proper order with appropriate prerequisites. When the top-level makefile is run, we see: $ make make --directory=lib/db make[1]: Entering directory `/test/book/out/ch06-simple/lib/db' Update db library... make[1]: Leaving directory `/test/book/out/ch06-simple/lib/db' make --directory=lib/codec make[1]: Entering directory `/test/book/out/ch06-simple/lib/codec' Update codec library... make[1]: Leaving directory `/test/book/out/ch06-simple/lib/codec' make --directory=lib/ui make[1]: Entering directory `/test/book/out/ch06-simple/lib/ui' Update ui library... make[1]: Leaving directory `/test/book/out/ch06-simple/lib/ui' make --directory=app/player make[1]: Entering directory `/test/book/out/ch06-simple/app/player' Update player application... make[1]: Leaving directory `/test/book/out/ch06-simple/app/player'
When make detects that it is invoking another make recursively, it enables the ”print-directory ( -w ) option, which causes make to print the Entering directory and Leaving directory messages. This option is also enabled when the ”directory ( -C ) option is used. The value of the make variable MAKELEVEL is printed in square brackets in each line as well. In this simple example, each component makefile prints a simple message about updating the component. 6.1.1 Command-Line Options Recursive make is a simple idea that quickly becomes complicated. The perfect recursive make implementation would behave as if the many makefile s in the system are a single makefile . Achieving this level of coordination is virtually impossible , so compromises must be made. The subtle issues become more clear when we look at how command-line options must be handled. Suppose we have added comments to a header file in our mp3 player. Rather than recompiling all the source that depends on the modified header, we realize we can instead perform a make ”touch to bring the timestamps of the files up to date. By executing the make ”touch with the top-level makefile, we would like make to touch all the appropriate files managed by sub- make s. Let's see how this works. Usually, when ”touch is provided on the command line, the normal processing of rules is suspended . Instead, the dependency graph is traversed and the selected targets and those prerequisites that are not marked .PHONY are brought up to date by executing touch on the target. Since our subdirectories are marked .PHONY , they would normally be ignored (touching them like normal files would be pointless). But we don't want those targets ignored, we want their command script executed. To do the right thing, make automatically labels any line containing MAKE with the + modifier, meaning make runs the sub- make regardless of the ”touch option. When make runs the sub- make it must also arrange for the ”touch flag to be passed to the sub-process. It does this through the MAKEFLAGS variable. When make starts, it automatically appends most command-line options to MAKEFLAGS . The only exceptions are the options ”directory ( -C ), ”file ( -f ), ”old-file ( -o ), and ”new-file ( -W ). The MAKEFLAGS variable is then exported to the environment and read by the sub- make as it starts. With this special support, sub- make s behave mostly the way you want. The recursive execution of $(MAKE) and the special handling of MAKEFLAGS that is applied to ”touch ( -t ) is also applied to the options ”just-print ( -n ) and ”question ( -q ). 6.1.2 Passing Variables As we have already mentioned, variables are passed to sub- make s through the environment and controlled using the export and unexport directives. Variables passed through the environment are taken as default values, but are overridden by any assignment to the variable. Use the ”environment- overrides ( -e ) option to allow environment variables to override the local assignment. You can explicitly override the environment for a specific assignment (even when the ”environment-overrides option is used) with the override directive: override TMPDIR = ~/tmp
Variables defined on the command line are automatically exported to the environment if they use legal shell syntax. A variable is considered legal if it uses only letters , numbers , and underscores. Variable assignments from the command line are stored in the MAKEFLAGS variable along with command-line options. 6.1.3 Error Handling What happens when a recursive make gets an error? Nothing very unusual, actually. The make receiving the error status terminates its processing with an exit status of 2. The parent make then exits, propagating the error status up the recursive make process tree. If the ”keep-going ( -k ) option is used on the top-level make , it is passed to sub- make s as usual. The sub- make does what it normally does, skips the current target and proceeds to the next goal that does not use the erroneous target as a prerequisite. For example, if our mp3 player program encountered a compilation error in the lib/db component, the lib/db make would exit, returning a status of 2 to the top-level makefile . If we used the ”keep-going ( -k ) option, the top-level makefile would proceed to the next unrelated target, lib/codec . When it had completed that target, regardless of its exit status, the make would exit with a status of 2 since there are no further targets that can be processed due to the failure of lib/db . The ”question ( -q ) option behaves very similarly. This option causes make to return an exit status of 1 if some target is not up to date, 0 otherwise . When applied to a tree of makefile s, make begins recursively executing makefile s until it can determine if the project is up to date. As soon as an out-of-date file is found, make terminates the currently active make and unwinds the recursion. 6.1.4 Building Other Targets The basic build target is essential for any build system, but we also need the other support targets we've come to depend upon, such as clean , install , print , etc. Because these are .PHONY targets, the technique described earlier doesn't work very well. For instance, there are several broken approaches, such as: clean: $(player) $(libraries) $(MAKE) --directory=$@ clean
or: $(player) $(libraries): $(MAKE) --directory=$@ clean
The first is broken because the prerequisites would trigger a build of the default target in the $(player) and $(libraries) makefile s, not a build of the clean target. The second is illegal because these targets already exist with a different command script. One approach that works relies on a shell for loop: clean: for d in $(player) $(libraries); \ do \ $(MAKE) --directory=$$f clean; \ done
A for loop is not very satisfying for all the reasons described earlier, but it (and the preceding illegal example) points us to this solution: $(player) $(libraries): $(MAKE) --directory=$@ $(TARGET)
By adding the variable $(TARGET) to the recursive make line and setting the TARGET variable on the make command line, we can add arbitrary goals to the sub- make : $ make TARGET=clean
Unfortunately, this does not invoke the $(TARGET) on the top-level makefile . Often this is not necessary because the top-level makefile has nothing to do, but, if necessary, we can add another invocation of make protected by an if : $(player) $(libraries): $(MAKE) --directory=$@ $(TARGET) $(if $(TARGET), $(MAKE) $(TARGET))
Now we can invoke the clean target (or any other target) by simply setting TARGET on the command line. 6.1.5 Cross-Makefile Dependencies The special support in make for command-line options and communication through environment variables suggests that recursive make has been tuned to work well. So what are the serious complications alluded to earlier? Separate makefile s linked by recursive $(MAKE) commands record only the most superficial top-level links. Unfortunately, there are often subtle dependencies buried in some directories. For example, suppose a db module includes a yacc -based parser for importing and exporting music data. If the ui module, ui.c , includes the generated yacc header, we have a dependency between these two modules. If the dependencies are properly modeled , make should know to recompile our ui module whenever the grammar header is updated. This is not difficult to arrange using the automatic dependency generation technique described earlier. But what if the yacc file itself is modified? In this case, when the ui makefile is run, a correct makefile would recognize that yacc must first be run to generate the parser and header before compiling ui.c . In our recursive make decomposition, this does not occur, because the rule and dependencies for running yacc are in the db makefile , not the ui makefile . In this case, the best we can do is to ensure that the db makefile is always executed before executing the ui makefile . This higher-level dependency must be encoded by hand. We were astute enough in the first version of our makefile to recognize this, but, in general, this is a very difficult maintenance problem. As code is written and modified, the top-level makefile will fail to properly record the intermodule dependencies. To continue the example, if the yacc grammar in db is updated and the ui makefile is run before the db makefile (by executing it directly instead of through the top-level makefile ), the ui makefile does not know there is an unsatisfied dependency in the db makefile and that yacc must be run to update the header file. Instead, the ui makefile compiles its program with the old yacc header. If new symbols have been defined and are now being referenced, then a compilation error is reported . Thus, the recursive make approach is inherently more fragile than a single makefile . The problem worsens when code generators are used more extensively. Suppose that the use of an RPC stub generator is added to ui and the headers are referenced in db . Now we have mutual reference to contend with. To resolve this, it may be required to visit db to generate the yacc header, then visit ui to generate the RPC stubs, then visit db to compile the files, and finally visit ui to complete the compilation process. The number of passes required to create and compile the source for a project is dependent on the structure of the code and the tools used to create it. This kind of mutual reference is common in complex systems. The standard solution in real-world makefile s is usually a hack. To ensure that all files are up to date, every makefile is executed when a command is given to the top-level makefile . Notice that this is precisely what our mp3 player makefile does. When the top-level makefile is run, each of the four sub- makefile s is unconditionally run. In complex cases, makefile s are run repeatedly to ensure that all code is first generated then compiled. Often this iterative execution is a complete waste of time, but occasionally it is required. 6.1.6 Avoiding Duplicate Code The directory layout of our application includes three libraries. The makefile s for these libraries are very similar. This makes sense because the three libraries serve different purposes in the final application but are all built with similar commands. This kind of decomposition is typical of large projects and leads to many similar makefile s and lots of ( makefile ) code duplication. Code duplication is bad, even makefile code duplication. It increases the maintenance costs of the software and leads to more bugs . It also makes it more difficult to understand algorithms and identify minor variations in them. So we would like to avoid code duplication in our makefile s as much as possible. This is most easily accomplished by moving the common pieces of a makefile into a common include file. For example, the codec makefile contains: lib_codec := libcodec.a sources := codec.c objects := $(subst .c,.o,$(sources)) dependencies := $(subst .c,.d,$(sources)) include_dirs := .. ../../include CPPFLAGS += $(addprefix -I ,$(include_dirs)) vpath %.h $(include_dirs) all: $(lib_codec) $(lib_codec): $(objects) $(AR) $(ARFLAGS) $@ $^ .PHONY: clean clean: $(RM) $(lib_codec) $(objects) $(dependencies) ifneq "$(MAKECMDGOALS)" "clean" include $(dependencies) endif %.d: %.c $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< \ sed 's,\($*\.o\) *:, $@: ,' > $@.tmp mv $@.tmp $@
Almost all of this code is duplicated in the db and ui makefile s. The only lines that change for each library are the name of the library itself and the source files the library contains. When duplicate code is moved into common.mk , we can pare this makefile down to: library := libcodec.a sources := codec.c include ../../common.mk
See what we have moved into the single, shared include file: MV := mv -f RM := rm -f SED := sed objects := $(subst .c,.o,$(sources)) dependencies := $(subst .c,.d,$(sources)) include_dirs := .. ../../include CPPFLAGS += $(addprefix -I ,$(include_dirs)) vpath %.h $(include_dirs) .PHONY: library library: $(library) $(library): $(objects) $(AR) $(ARFLAGS) $@ $^ .PHONY: clean clean: $(RM) $(objects) $(program) $(library) $(dependencies) $(extra_clean) ifneq "$(MAKECMDGOALS)" "clean" -include $(dependencies) endif %.c %.h: %.y $(YACC.y) --defines $< $(MV) y.tab.c $*.c $(MV) y.tab.h $*.h %.d: %.c $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< \ $(SED) 's,\($*\.o\) *:, $@: ,' > $@.tmp $(MV) $@.tmp $@
The variable include_dirs , which was different for each makefile , is now identical in all makefile s because we reworked the path source files use for included headers to make all libraries use the same include path. The common.mk file even includes the default goal for the library include files. The original makefile s used the default target all . That would cause problems with nonlibrary makefile s that need to specify a different set of prerequisites for their default goal. So the shared code version uses a default target of library . Notice that because this common file contains targets it must be included after the default target for nonlibrary makefile s. Also notice that the clean command script references the variables program , library , and extra_clean . For library makefile s, the program variable is empty; for program makefile s, the library variable is empty. The extra_clean variable was added specifically for the db makefile . This makefile uses the variable to denote code generated by yacc . The makefile is: library := libdb.a sources := scanner.c playlist.c extra_clean := $(sources) playlist.h .SECONDARY: playlist.c playlist.h scanner.c include ../../common.mk
Using these techniques, code duplication can be kept to a minimum. As more makefile code is moved into the common makefile , it evolves into a generic makefile for the entire project. make variables and user-defined functions are used as customization points, allowing the generic makefile to be modified for each directory. | 