| Here we measure the performance of some basic operations in make . Table 10-1 shows the results of these measurements. We'll explain each test and suggest how they might affect makefile s you write. Table 10-1. Cost of operations | Operation | Executions | Seconds per execution (Windows) | Executions per second (Windows) | Seconds per execution (Linux) | Executions per second (Linux) | | make (bash) | 1000 | 0.0436 | 22 | 0.0162 | 61 | | make (ash) | 1000 | 0.0413 | 24 | 0.0151 | 66 | | make (sh) | 1000 | 0.0452 | 22 | 0.0159 | 62 | | assignment | 10,000 | 0.0001 | 8130 | 0.0001 | 10,989 | | subst (short) | 10,000 | 0.0003 | 3891 | 0.0003 | 3846 | | subst (long) | 10,000 | 0.0018 | 547 | 0.0014 | 704 | | sed (bash) | , 1000 | 0.0910 | 10 | 0.0342 | 29 | | sed (ash) | 1000 | 0.0699 | 14 | 0.0069 | 144 | | sed (sh) | 1000 | 0.0911 | 10 | 0.0139 | 71 | | shell (bash) | 1000 | 0.0398 | 25 | 0.0261 | 38 | | shell (ash) | 1000 | 0.0253 | 39 | 0.0018 | 555 | | shell (sh) | 1000 | 0.0399 | 25 | 0.0050 | 198 |
The Windows tests were run on a 1.9-GHz Pentium 4 (approximately 3578 BogoMips) [1] with 512 MB RAM running Windows XP. The Cygwin version of make 3.80 was used, started from an rxvt window. The Linux tests were run on a 450-MHz Pentium 2 (891 BogoMips) with 256 MB of RAM running Linux RedHat 9. [1] See http://www.clifton.nl/bogomips.html for an explanation of BogoMips. The subshell used by make can have a significant effect on the overall performance of the makefile . The bash shell is a complex, fully featured shell, and therefore large. The ash shell is a much smaller, with fewer features but adequate for most tasks . To complicate matters, if bash is invoked from the filename /bin/sh , it alters its behavior significantly to conform more closely to the standard shell. On most Linux systems the file /bin/sh is a symbolic link to bash , while in Cygwin /bin/sh is really ash . To account for these differences, some of the tests were run three times, each time using a different shell. The shell used is indicated in parentheses. When "(sh)" appears, it means that bash was linked to the file named /bin/sh . The first three tests, labeled make, give an indication of how expensive it is to run make if there is nothing to do. The makefile contains: SHELL := /bin/bash .PHONY: x x: $(MAKE) --no-print-directory --silent --question make-bash.mk; \ ... this command repeated 99 more times ...
The word "bash" is replaced with the appropriate shell name as required. We use the ”no-print-directory and ”silent commands to eliminate unnecessary computation that might skew the timing test and to avoid cluttering the timing output values with irrelevant text. The ”question option tells make to simply check the dependencies without executing any commands and return an exit status of zero if the files are up to date. This allows make to do as little work as possible. No commands will be executed by this makefile and dependencies exist for only one .PHONY target. The command script executes make 100 times. This makefile , called make-bash.mk , is executed 10 times by a parent makefile with this code: define ten-times TESTS += .PHONY: : @echo $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent ; \ time $(MAKE) --no-print-directory --silent endef .PHONY: all all: $(eval $(call ten-times, make-bash, -f make-bash.mk)) all: $(TESTS)
The time for these 1,000 executions is then averaged. As you can see from the table, the Cygwin make ran at roughly 22 executions per second or 0.044 seconds per run, while the Linux version (even on a drastically slower CPU) performed roughly 61 executions per second or 0.016 seconds per run. To verify these results, the native Windows version of make was also tested and did not yield any dramatic speed up. Conclusion: while process creation in Cygwin make is slightly slower than a native Windows make , both are dramatically slower than Linux. It also suggests that use of recursive make on a Windows platform may perform significantly slower than the same build run on Linux. As you would expect, the shell used in this test had no effect on execution time. Because the command script contained no shell special characters , the shell was not invoked at all. Rather, make executed the commands directly. This can be verified by setting the SHELL variable to a completely bogus value and noting that the test still runs correctly. The difference in performance between the three shells must be attributed to normal system variance. The next benchmark measures the speed of variable assignment. This calibrates the most elementary make operation. The makefile , called assign.mk , contains: # 10000 assignments z := 10 ...repeated 10000 times... .PHONY: x x: ;
This makefile is then run using our ten-times function in the parent makefile . The assignment is obviously very fast. Cygwin make will execute 8130 assignments per second while the Linux system can do 10,989. I believe the performance of Windows for most of these operations is actually better than the benchmark indicates because the cost of creating the make process 10 times cannot be reliably factored out of the time. Conclusion: because it is unlikely that the average makefile would perform 10,000 assignments, the cost of variable assignment in an average makefile is negligible. The next two benchmarks measure the cost of a subst function call. The first uses a short 10-character string with three substitutions: # 10000 subst on a 10 char string dir := ab/cd/ef/g x := $(subst /, ,$(dir)) ...repeated 10000 times... .PHONY: x x: ;
This operation takes roughly twice as long as a simple assignment, or 3891 operations per second on Windows. Again, the Linux system appears to outperform the Windows system by a wide margin. (Remember, the Linux system is running at less than one quarter the clock speed of the Windows system.) The longer substitution operates on a 1000-character string with roughly 100 substitutions: # Ten character file dir := ab/cd/ef/g # 1000 character path p100 := $(dir);$(dir);$(dir);$(dir);$(dir);... p1000 := $(p100)$(p100)$(p100)$(p100)$(p100)... # 10000 subst on a 1000 char string x := $(subst ;, ,$(p1000)) ...repeated 10000 times... .PHONY: x x: ;
The next three benchmarks measure the speed of the same substitution using sed . The benchmark contains: # 100 sed using bash SHELL := /bin/bash .PHONY: sed-bash sed-bash: echo '$(p1000)' sed 's/;/ /g' > /dev/null ...repeated 100 times...
As usual, this makefile is executed using the ten-times function. On Windows, sed execution takes about 50 times longer than the subst function. On our Linux system, sed is only 24 times slower. When we factor in the cost of the shell, we see that ash on Windows does provide a useful speed-up. With ash , the sed is only 39 times slower than subst! (wink) On Linux, the shell used has a much more profound effect. Using ash , the sed is only five times slower than subst . Here we also notice the curious effect of renaming bash to sh . On Cygwin, there is no difference between a bash named /bin/bash and one named /bin/sh , but on Linux, a bash linked to /bin/sh performs significantly better. The final benchmark simply invokes the make shell command to evaluate the cost of running a subshell. The makefile contains: # 100 $(shell ) using bash SHELL := /bin/bash x := $(shell :) ...repeated 100 times... .PHONY: x x: ;
There are no surprises here. The Windows system is slower than Linux, with ash having an edge over bash . The performance gain of ash is more pronounced ”about 50 % faster. The Linux system performs best with ash and slowest with bash (when named "bash"). Benchmarking is a never-ending task, however, the measurements we've made can provide some useful insight. Create as many variables as you like if they help clarify the structure of the makefile because they are essentially free. Built-in make functions are preferred over running commands even if you are required by the structure of your code to reexecute the make function repeatedly. Avoid recursive make or unnecessary process creation on Windows. While on Linux, use ash if you are creating many processes. Finally, remember that in most makefile s, the time a makefile takes to run is due almost entirely to the cost of the programs run, not make or the structure of the makefile . Usually, reducing the number of programs run will be most helpful in reducing the execution time of a makefile . | 