| If we want to support a single source tree with multiple platforms and multiple builds per platform, separating the source and binary trees is necessary, so how do we do it? The make program was originally written to work well for files in a single directory. Although it has changed dramatically since then, it hasn't forgotten its roots. make works with multiple directories best when the files it is updating live in the current directory (or its subdirectories). 8.1.1 The Easy Way The easiest way to get make to place binaries in a separate directory from sources is to start the make program from the binary directory. The output files are accessed using relative paths, as shown in the previous chapter, while the input files must be found either through explicit paths or through searching through vpath . In either case, we'll need to refer to the source directory in several places, so we start with a variable to hold it: SOURCE_DIR := ../mp3_player
Building on our previous makefile , the source-to-object function is unchanged, but the subdirectory function now needs to take into account the relative path to the source. # $(call source-to-object, source-file-list) source-to-object = $(subst .c,.o,$(filter %.c,)) \ $(subst .y,.o,$(filter %.y,)) \ $(subst .l,.o,$(filter %.l,)) # $(subdirectory) subdirectory = $(patsubst $(SOURCE_DIR)/%/module.mk,%, \ $(word \ $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))
In our new makefile , the files listed in the MAKEFILE_LIST will include the relative path to the source. So to extract the relative path to the module's directory, we must strip off the prefix as well as the module.mk suffix. Next, to help make find the sources, we use the vpath feature: vpath %.y $(SOURCE_DIR) vpath %.l $(SOURCE_DIR) vpath %.c $(SOURCE_DIR)
This allows us to use simple relative paths for our source files as well as our output files. When make needs a source file, it will search SOURCE_DIR if it cannot find the file in the current directory of the output tree. Next, we must update the include_dirs variable: include_dirs := lib $(SOURCE_DIR)/lib $(SOURCE_DIR)/include
In addition to the source directories, this variable now includes the lib directory from the binary tree because the generated yacc and lex header files will be placed there. The make include directive must be updated to access the module.mk files from their source directories since make does not use the vpath to find include files: include $(patsubst %,$(SOURCE_DIR)/%/module.mk,$(modules))
Finally, we create the output directories themselves : create-output-directories := \ $(shell for f in $(modules); \ do \ $(TEST) -d $$f $(MKDIR) $$f; \ done)
This assignment creates a dummy variable whose value is never used, but because of the simple variable assignment we are guaranteed that the directories will be created before make performs any other work. We must create the directories "by hand" because yacc , lex , and the dependency file generation will not create the output directories themselves. Another way to ensure these directories are created is to add the directories as prerequisites to the dependency files (the .d files). This is a bad idea because the directory is not really a prerequisite. The yacc , lex , or dependency files do not depend on the contents of the directory, nor should they be regenerated just because the directory timestamp is updated. In fact, this would be a source of great inefficiency if the project were remade when a file was added or removed from an output directory. The modifications to the module.mk file are even simpler: local_src := $(addprefix $(subdirectory)/,playlist.y scanner.l) $(eval $(call make-library, $(subdirectory)/libdb.a, $(local_src))) .SECONDARY: $(call generated-source, $(local_src)) $(subdirectory)/scanner.d: $(subdirectory)/playlist.d
This version omits the wildcard to find the source. It is a straightforward matter to restore this feature and is left as an exercise for the reader. There is one glitch that appears to be a bug in the original makefile . When this example was run, I discovered that the scanner.d dependency file was being generated before playlist.h , which it depends upon. This dependency was missing from the original makefile , but it worked anyway purely by accident . Getting all the dependencies right is a difficult task, even in small projects. Assuming the source is in the subdirectory mp3_player , here is how we build our project with the new makefile : $ mkdir mp3_player_out $ cd mp3_player_out $ make --file=../mp3_player/makefile
The makefile is correct and works well, but it is rather annoying to be forced to change directories to the output directory and then be forced to add the ”file ( -f ) option. This can be cured with a simple shell script: #! /bin/bash if [[ ! -d $OUTPUT_DIR ]] then if ! mkdir -p $OUTPUT_DIR then echo "Cannot create output directory" > /dev/stderr exit 1 fi fi cd $OUTPUT_DIR make --file=$SOURCE_DIR/makefile "$@"
This script assumes the source and output directories are stored in the environment variables SOURCE_DIR and OUTPUT_DIR , respectively. This is a standard practice that allows developers to switch trees easily but still avoid typing paths too frequently. One last caution. There is nothing in make or our makefile to prevent a developer from executing the makefile from the source tree, even though it should be executed from the binary tree. This is a common mistake and some command scripts might behave badly . For instance, the clean target: .PHONY: clean clean: $(RM) -r *
would delete the user 's entire source tree! Oops. It seems prudent to add a check for this eventuality in the makefile at the highest level. Here is a reasonable check: $(if $(filter $(notdir $(SOURCE_DIR)),$(notdir $(CURDIR))),\ $(error Please run the makefile from the binary tree.))
This code tests if the name of the current working directory ( $(notdir $(CURDIR)) ) is the same as the source directory ( $(notdir $(SOURCE_DIR)) ). If so, print the error and exit. Since the if and error functions expand to nothing, we can place these two lines immediately after the definition of SOURCE_DIR . 8.1.2 The Hard Way Some developers find having to cd into the binary tree so annoying that they will go to great lengths to avoid it, or maybe the makefile maintainer is working in an environment where shell script wrappers or aliases are unsuitable. In any case, the makefile can be modified to allow running make from the source tree and placing binary files in a separate output tree by prefixing all the output filenames with a path. At this point I usually go with absolute paths since this provides more flexibility, although it does exacerbate problems with command-line length limits. The input files continue to use simple relative paths from the makefile directory. Example 8-1 shows the makefile modified to allow executing make from the source tree and writing binary files to a binary tree. Example 8-1. A makefile separating source and binary that can be executed from the source tree SOURCE_DIR := /test/book/examples/ch07-separate-binaries-1 BINARY_DIR := /test/book/out/mp3_player_out # $(call source-dir-to-binary-dir, directory-list) source-dir-to-binary-dir = $(addprefix $(BINARY_DIR)/, ) # $(call source-to-object, source-file-list) source-to-object = $(call source-dir-to-binary-dir, \ $(subst .c,.o,$(filter %.c,)) \ $(subst .y,.o,$(filter %.y,)) \ $(subst .l,.o,$(filter %.l,))) # $(subdirectory) subdirectory = $(patsubst %/module.mk,%, \ $(word \ $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))) # $(call make-library, library-name, source-file-list) define make-library libraries += $(BINARY_DIR)/ sources += $(BINARY_DIR)/: $(call source-dir-to-binary-dir, \ $(subst .c,.o,$(filter %.c,)) \ $(subst .y,.o,$(filter %.y,)) \ $(subst .l,.o,$(filter %.l,))) $(AR) $(ARFLAGS) $$@ $$^ endef # $(call generated-source, source-file-list) generated-source = $(call source-dir-to-binary-dir, \ $(subst .y,.c,$(filter %.y,)) \ $(subst .y,.h,$(filter %.y,)) \ $(subst .l,.c,$(filter %.l,))) \ $(filter %.c,) # $(compile-rules) define compile-rules $(foreach f, $(local_src),\ $(call one-compile-rule,$(call source-to-object,$f),$f)) endef # $(call one-compile-rule, binary-file, source-files) define one-compile-rule : $(call generated-source,) $(COMPILE.c) -o $$@ $$< $(subst .o,.d,): $(call generated-source,) $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $$< \ $(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$@) $$@: ,' > $$@.tmp $(MV) $$@.tmp $$@ endef modules := lib/codec lib/db lib/ui app/player programs := libraries := sources := objects = $(call source-to-object,$(sources)) dependencies = $(subst .o,.d,$(objects)) include_dirs := $(BINARY_DIR)/lib lib include CPPFLAGS += $(addprefix -I ,$(include_dirs)) vpath %.h $(include_dirs) MKDIR := mkdir -p MV := mv -f RM := rm -f SED := sed TEST := test create-output-directories := \ $(shell for f in $(call source-dir-to-binary-dir,$(modules)); \ do \ $(TEST) -d $$f $(MKDIR) $$f; \ done) all: include $(addsuffix /module.mk,$(modules)) .PHONY: all all: $(programs) .PHONY: libraries libraries: $(libraries) .PHONY: clean clean: $(RM) -r $(BINARY_DIR) ifneq "$(MAKECMDGOALS)" "clean" include $(dependencies) endif
In this version the source-to-object function is modified to prepend the path to the binary tree. This prefixing operation is performed several times, so write it as a function: SOURCE_DIR := /test/book/examples/ch07-separate-binaries-1 BINARY_DIR := /test/book/out/mp3_player_out # $(call source-dir-to-binary-dir, directory-list) source-dir-to-binary-dir = $(addprefix $(BINARY_DIR)/, ) # $(call source-to-object, source-file-list) source-to-object = $(call source-dir-to-binary-dir, \ $(subst .c,.o,$(filter %.c,)) \ $(subst .y,.o,$(filter %.y,)) \ $(subst .l,.o,$(filter %.l,)))
The make-library function is similarly altered to prefix the output file with BINARY_DIR . The subdirectory function is restored to its previous version since the include path is again a simple relative path. One small snag; a bug in make 3.80 prevents calling source-to-object within the new version of make-library . This bug has been fixed in 3.81. We can work around the bug by hand expanding the source-to-object function. Now we get to the truly ugly part. When the output file is not directly accessible from a path relative to the makefile , the implicit rules no longer fire. For instance, the basic compile rule %.o : %.c works well when the two files live in the same directory, or even if the C file is in a subdirectory, say lib/codec/codec.c . When the source file lives in a remote directory, we can instruct make to search for the source with the vpath feature. But when the object file lives in a remote directory, make has no way of determining where the object file resides and the target/prerequisite chain is broken. The only way to inform make of the location of the output file is to provide an explicit rule linking the source and object files: $(BINARY_DIR)/lib/codec/codec.o: lib/codec/codec.c
This must be done for every single object file. Worse , this target/prerequisite pair is not matched against the implicit rule, %.o: %.c . That means we must also provide the command script, duplicating whatever is in the implicit database and possibly repeating this script many times. The problem also applies to the automatic dependency generation rule we've been using. Adding two explicit rules for every object file in a makefile is a maintenance nightmare, if done by hand. However, we can minimize the code duplication and maintenance by writing a function to generate these rules: # $(call one-compile-rule, binary-file, source-files) define one-compile-rule : $(call generated-source,) $(COMPILE.c) $$@ $$< $(subst .o,.d,): $(call generated-source,) $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $$< \ $(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$@) $$@: ,' > $$@.tmp $(MV) $$@.tmp $$@ endef
The first two lines of the function are the explicit rule for the object-to-source dependency. The prerequisites for the rule must be computed using the generated-source function we wrote in Chapter 6 because some of the source files are yacc and lex files that will cause compilation failures when they appear in the command script (expanded with $^ , for instance). The automatic variables are quoted so they are expanded later when the command script is executed rather than when the user-defined function is evaluated by eval . The generated-source function has been modified to return C files unaltered as well as the generated source for yacc and lex : # $(call generated-source, source-file-list) generated-source = $(call source-dir-to-binary-dir, \ $(subst .y,.c,$(filter %.y,)) \ $(subst .y,.h,$(filter %.y,)) \ $(subst .l,.c,$(filter %.l,))) \ $(filter %.c,)
With this change, the function now produces this output: Argument Result lib/db/playlist.y /c/mp3_player_out/lib/db/playlist.c /c/mp3_player_out/lib/db/playlist.h lib/db/scanner.l /c/mp3_player_out/lib/db/scanner.c app/player/play_mp3.c app/player/play_mp3.c
The explicit rule for dependency generation is similar. Again, note the extra quoting (double dollar signs) required by the dependency script. Our new function must now be expanded for each source file in a module: # $(compile-rules) define compile-rules $(foreach f, $(local_src),\ $(call one-compile-rule,$(call source-to-object,$f),$f)) endef
This function relies on the global variable local_src used by the module.mk files. A more general approach would pass this file list as an argument, but in this project it seems unnecessary. These functions are easily added to our module.mk files: local_src := $(subdirectory)/codec.c $(eval $(call make-library,$(subdirectory)/libcodec.a,$(local_src))) $(eval $(compile-rules))
We must use eval because the compile-rules function expands to more than one line of make code. There is one last complication. If the standard C compilation pattern rule fails to match with binary output paths, the implicit rule for lex and our pattern rule for yacc will also fail. We can update these by hand easily. Since they are no longer applicable to other lex or yacc files, we can move them into lib/db/module.mk : local_dir := $(BINARY_DIR)/$(subdirectory) local_src := $(addprefix $(subdirectory)/,playlist.y scanner.l) $(eval $(call make-library,$(subdirectory)/libdb.a,$(local_src))) $(eval $(compile-rules)) .SECONDARY: $(call generated-source, $(local_src)) $(local_dir)/scanner.d: $(local_dir)/playlist.d $(local_dir)/%.c $(local_dir)/%.h: $(subdirectory)/%.y $(YACC.y) --defines $< $(MV) y.tab.c $(dir $@)$*.c $(MV) y.tab.h $(dir $@)$*.h $(local_dir)/scanner.c: $(subdirectory)/scanner.l @$(RM) $@ $(LEX.l) $< > $@
The lex rule has been implemented as a normal explicit rule, but the yacc rule is a pattern rule. Why? Because the yacc rule is used to build two targets, a C file and a header file. If we used a normal explicit rule, make would execute the command script twice, once for the C file to be created and once for the header. But make assumes that a pattern rule with multiple targets updates both targets with a single execution. If possible, instead of the makefile s shown in this section, I would use the simpler approach of compiling from the binary tree. As you can see, complications arise immediately (and seem to get worse and worse) when trying to compile from the source tree. | 