(The previous article is available here.)
Almost like text editors, build tools form a large population. You can find many of them cataloged at the Free Software Foundation site and at DMZ. You can also find many build tools by digging in well-known Open Source repositories like SourceForge or Tigirs, but it is much more difficult to spot what you are looking for on those sites. I will discuss here a small selection. This selection is highly subjective and is is biased towards Open Source tools. I have grouped the tools into several categories to help you get a better overview.
The tools in this category are not really alternatives. They are what people currently understand as Make (virtually nobody is using today the original BSD Unix Make). I mention three tools in this category: GNU Make, Opus Make, and NMake. They all share some compatibility with the format of the original Makefiles.
This tool is not widely used today but, in my opinion, Opus make is the best Make clone ever made. It has a few outstanding features that set it apart from the crowd. It has the richest set of directives allowed in the Makefile (including its own "cd", "echo", "copy", "delete", and other frequent shell commands). Even more important, these directives can take effect at parsing time or at rule execution time. This makes for more portable execution parts in rules, greatly reducing the dependence on the shell underneath. Opus make has always had logical operators in conditional expressions, regular expression substitutions, the ability to trace the parsing and to stop on lines not understood, and much more. With its "inference restart" command, it has support for one-pass building, a feature rarely seen. Along with its comprehensive set of native features, Opus make also has a fair set of emulations for other Make tools. Unfortunately, according to the Opus Web site, its development seems to have stopped back in 1998. A version of it is still distributed today with the IBM Rational ClearCase SCM tool.
GNU Make is very likely the most widely-used build tool. Being available as Open Source and being an integral part of the GNU tool set made it a popular choice on many Unix-like platforms. It is actively maintained and is a vast improvement over the original Make, especially with its large set of new macros. Over the last ten years, it has gained some important features (that should have been available from the beginning), to print at parsing time, stop parsing and exit, force the build, define new function-like macros, make case-insensitive filename comparisons on some platforms, etc. Unfortunately, GNU Make also kept most of the problems of the original Make, and much of the criticism that you'll hear of Make is actually directed to GNU Make.
NMake originates at and is maintained by AT&T laboratories. The large set of Open Source tools from AT&T (Korn shell, graphviz, etc.) is built with it. NMake is Open Source itself. There is a commercial version from Lucent as well. The AT&T-style source distribution packages rely on NMake and their own configuration tool, iffe. This build system from AT&T is very Unix-centric, exactly like the GNU build system. Note that Microsoft also has a Make clone called "nmake" bundled with their development environments. Take care; they are incompatible. Don't expect a Makefile written for one to work with the other.
The tools in this category move away from the old syntax of Makefiles but don't break with the Make tool spirit. That is, they are still developed in C or C++ and they still use some kind of text file located close to the sources to describe what has to be built. I will mention two tools in this category: Jam and Cook.
Jam is maintained and promoted by the people behind the perforce SCM tool. Perforce is commercial software, but Jam is fully Open Source. Despite some small issues with its syntax, Jam files are expressive enough for the Jam tool to come standard with a decent database of rules.
Jam was so influent that it generated a set of clones (FTJam, Boost.Jam, Boost.Jam.v2). The most interesting is Boost.Jam from the maintainers of the Boost C++ library. It introduces some good syntax extensions to the original tool. You can tell by those extensions that the authors were C++ programmers (for example, the scoping of variables looks like C++ namespaces). Boost.Jam is not the only nor the first attempt to raise the level of the build description, but more than other Make evolutions, Boost.Jam focus on real build issues. For example, it provides canned variant builds, it provides dynamic loading library abstraction for Unix and MS Windows target platforms, it cares about testing the result of the build, etc. If you ever used Make for a real-life-sized project, you had to provide such things by your own effort, and you will immediately understand what savings this approach of Boost.Jam brings. Unfortunately, Jam doesn't make the jump to content signatures and to commandline signatures. The problems related to the weak timestamp heuristic are still present.
Cook is a build tool designed by Peter Miller with quite a long history. Like Jam, it is Open Source and supports parallel builds. Like Jam, it avoids recursion. Like GNU Make and unlike Jam, it relies on a separate tool, c_incl, to get implicit dependencies in C code. The build description syntax used by Cook takes some features from Makefiles and some from LISP. (Many build tools have a LISP-like syntax. There is a good reason for that: Manipulating lists (i.e., lists of dependencies) is a frequent task when describing a build. Cook can use content signatures, or "fingerprints" in its parlance.)
I cannot express a clear preference in this category. Boost.Jam and Cook come close. But I can say that I prefer any of them over GNU Make. The reason is that they allow you to focus on build design and forget some of the low-level build implementation details. Unfortunately, they haven't achieved the widespread use that they deserve, perhaps because they are not part of a more comprehensive build system, as GNU Make is.
You may argue that my preference does not make a lot of sense. Indeed, both Cook and Jam (the original) rely on the GNU build system to get themselves built, which in turn relies on GNU Make. So it is not a matter of preference; you'll have to have GNU Make. This gives me the opportunity to introduce a thorny issue, bootstrapping. The question is: How do you build when you don't have a build tool? In particular, how do you build the build tool itself? One solution is to cross compile and then distribute binaries for the new platform. Another solution is to go back to shell scripts for the build description of the build tool itself (like Boost.Jam does). Or you can just rely on "good old GNU Make", like many self-proclaimed modern build tools do.
Some build tools are able to generate shell scripts automatically from their native build descriptions. Of course, the generated scripts don't have the full functionality of the build tool. They are usually able to do a build from scratch, and nothing more. And they usually require manual setup of the shell environment. Nevertheless, those generated shell build scripts help a lot with the bootstrapping issue.
The build tool is always just a piece in the larger software development system. Let us define more precisely the terms used below. The build tool reads a build description and then directly starts other tools to actually produce the result of the build (a report, a compiled file, etc.). A build system is a set of tools, including a build tool (by the way, a good build system will support several competing build tools). A build system may produce or adapt the build descriptions before they are used by the build tool. A build system may in some way manage the build result (for example, it may post the summary of an automatic build on a Web site). Next to the build tool, the other important tool in a build system is the configuration tool, the one that adapts or automatically generates the build descriptions used by the build tool.
To understand the paradigm shift introduced by build systems, it is important to first know the new requirements addressed by build systems: software distribution as a package of sources. The typical usage scenario for a build tool is C sourcecode development. Here, the genuine input changes frequently and locally. Avoidance is a crucial feature in this use case. How building works elsewhere is less important. Also, the addition of new components to a build has to be easy. By contrast, for software distribution, the crucial need is the ability to customize the software. It has to be adapted to new build platforms, to new runtime platforms, to sitewide conventions, etc. The builds are far less frequent. (Avoidance is not really a requirement, for example.) The difference in requirements has had an influence on the evolutions of build tools (some being used mostly from within build systems, and some being used mostly stand-alone).
The key advance in build systems is the fact that they put some executable code in the sourcecode distribution of the software to be built. We name the executable part in the source distribution the configuration tool. This changes the nature of the sourcecode distribution; now it is more like an installer package. "Installer" is a term stolen from binary distributions. (For software distributed in binary form, it is a long-time established practice that the package has to "execute" on the system that receives the software distribution). Because you are now supposed to execute some program delivered with the sources before you start the build, the source distribution has a chance to adapt itself to the current build machine. This is what makes build systems so helpful with sourcecode portability. The configuration tool may change build descriptions or sourcecode or both. Sometimes, the configuration tool has evolved into a complex interactive tool (see the several tools available to configure Linux kernel compilation).
The configuration tool inspects the build platform in small steps, with individual checks for one feature. Hereafter, we will call these steps probes. How results of probes are represented differs from one tool to another, but most of them have a caching mechanism in place so that you don't need to run a probe each time you need the result of that probe. Probes can have various granularities, can be independent or can be dependent on the result of other probes. Roughly speaking, the available set of probes is the way a configuration tool represents its knowledge about the platform to which it needs to adapt. The result of the entire set of probes defines de facto a model for a given platform. This hurts on the plane of operating systems, with more or less support from those systems' maintainers. Given the diversity of platforms to cover, configuration tools in general take on a tremendous job. This unavoidably generates frustration in different places and at different levels. Try to be positive and not judge a configuration tool only by its failures in some cases.
Before we look to some more Make alternatives, let us first compare a few build systems.
This is the system of choice in the Open Source community. It is based on GNU Make as its build tool. GBS has a few remarkable features that greatly help with software portability. The configuration tool of GBS is a shell script named "configure". That script inspects the build platform and then produces build descriptions for GNU Make as well as C sourcecode (actually one central header file, to be included by the C source files).
What makes GBS so successful? Certainly not the fact that it was the first attempt or the only attempt. Long before it, the IMake tool introduced higher-level build descriptions. IMake comes from the C code base of the X Windowing System. IMake higher-level Makefiles made the build description both more portable and easier to write. Yet IMake was never as successful as GBS, probably due to the way the knowledge about the platform was implemented for the IMake tool.
The configuration tool of GBS, the "configure" shell script, is automatically generated by a tool named autoconf. autoconf is not a build tool, but a script compiler. In my opinion, autoconf is the most valuable part of GBS, and it is largely responsible for GBS's success. The script produced by autoconf is made very portable (it doesn't have heavy requirements on the underlying system). This means that, unlike with IMake, with GBS, you have a fair chance to successfully build on a platform that was never seen by the author of the software you build.
Equally importantly, the authors of GBS were the first to seriously consider the bootstrapping issue. Indeed, it is possible to use GBS to build parts of GBS. After a few iterations, you'll get a completely new version of GBS. It is not an easy process, but at least it is possible. You may wonder, "What's the big deal? I can already use Make to build Make!" The big difference is that, in order to build Make with Make, a version of Make has to be installed on your build machine first. By contrast, you can build autoconf with GBS on a build machine where autoconf was never installed before.
If GBS is so smart, why isn't everybody using it? One reason is that GBS is very Unix-centric and C/C++ dedicated. And even if you build C programs on Unix, GBS only works effectively if your sourcecode complies with the GNU coding guidelines. This is a showstopper if you have a large body of code with its own coding guidelines (for example, not using the central header file config.h but some other code, maybe automatically generated as well).
Another reason why GBS is not used everywhere is the price it pays for its portability. For example, the probes of autoconf are encoded as M4 macros. M4 is a powerful text processor, but its syntax is pretty low level and difficult. Difficult enough to make this a major obstacle for extending the reach of GBS. While a godsend for less capable platforms, GBS has a much harder time winning the favor of developers on mainstream platforms. People simply don't want to give up their convenience (when writing probes) for the sake of some obscure platform no one's heard of. I know this is not a technical argument, but the world is such that non-technical arguments are sometimes more important (see user reactions at Freshmeat).
Another thorny issue is weak support for embedded software development and associated issues. In the embedded world, everybody is cross-compiling; the build platform and the runtime platform are separate and very different in nature. In that case, automatic inspection of the build platform before the build will not get you very far. In the embedded world, some kind of database holding the characteristics of the different platforms turns out better. The database is painful to maintain, but is the only solution that works. Autoconf has lately added some features to support cross-compilation.
Last but not least, one issue with GBS is the fact that it uses GNU Make and nothing other than GNU Make as a build tool. You will frequently have inconsistent builds, you will have a hard time debugging an inconsistent or a broken build, you will not be able to build a program called "clean" or "install", etc. All these issues with GNU Make undermine the important achievements of autoconf.
The need to configure the sources before compilation is quite old, and solutions have grown in almost all large C/C++ codebases, many being ad-hoc solutions. Some codebases have been migrated to GBS, some kept their own because of some isolated advantages or just lack of resources for migration. Many build tools evolved to provide some form of source configuration, either as an add-on or as a part of the same tool. For example, Boost.Jam has its "feature normalization", SCons has its Configure, etc. I would like to mention here two stand-alone solutions used with Make, iffe and metaconfig.
If you are familiar with the AT&T labs Open Source software, you've met iffe. This is the configuration tool of their source distribution packages. Like "configure" in GBS, iffe is a shell script and is distributed with the C sourcecode. Unlike "configure", iffe is not generated. Another important difference is that iffe doesn't focus on build descriptions. It generates only source files (to be precise, it generates header files that your C files are supposed to include). It does that by processing input files named "feature test files". The feature test files are written in a specific language that is interpreted by iffe to generate header files. Because, in one configuration process, several headers are generated according to your decisions to group probes in feature test files, iffe's authors claim that their system is more flexible than GBS, and they are probably right. In the end, the fundamental mechanism to adapt the sourcecode is the same: conditional compilation with the help of the C preprocessor.
If you are familiar with the Perl software source distribution, you've met metaconfig and the dist package. Metaconfig is a shell script compiler (older than autoconf), and the configuration tool it generates is called Configure. Unlike the one generated by autoconf, Configure is mainly an interactive tool. The probes used by metaconfig are called units. These units are shell code snippets. Metaconfig was probably the very first tool to do a decent job scanning the sourcecode base to automatically detect points of customization. By comparison, autoconf still has to rely on a helper tool, autoscan, to do this with more or less success. Automatic scanning of sourcecode is great, but it requires consistent compliance with coding guidelines decided by the configuration tool. That may not be the case for existing code.
There are several other alternatives to GBS available today, both commercial products and Open Source programs. I would like to mention two of them: CMake and Qmake. Do not to be misled by their names; they are not replacements for the Make tool, but are Makefile generators. Also, do not confuse the Qmake tool from TrollTech with the qmake tool from the Sun GridEngine, which is just a parallel GNU Make.
As with GBS, the main issue addressed by these tools is the portability of the build description. CMake and Qmake have a lot in common in their design and in their spirit. They are both Open Source and they are both implemented in C++ (Qmake had a predecessor, tmake, implemented in Perl). They both support more than one build tool. (Most notably, they support the classic Make as well as the Microsoft IDE. They provide dynamic linking library abstraction to cover both Unix .so files and Microsoft .DLL files.) They both introduce their own high-level format for the build description. They both take basically the same approach to hierarchical builds as the original Make. Not betraying their high-level nature, both tools also provide support for generating some sourcecode (wrappers or mock objects). My preference goes to the CMake tool. It supports a wider set of build tools, and it offers a choice between commandline and graphical user interfaces.
For both of these tools, you have to distribute the binary executable of the tool with the sourcecode package, and this raises a bootstrapping issue. Also, in my opinion, having the knowledge embedded in C++ classes as in CMake is a shortcoming. I certainly agree that C++ classes are much better than M4 macros, but C++ will limit the number of contributions from the community. People who can write good C++ and want to support new toolchains are required to contribute their changes in a consistent and systematic way. Otherwise, over time, we will get several incompatible versions floating around for the executable of the tool. Storing the knowledge about the toolchains in some kind of configuration files so that additions don't require C++ recompilation may outweight the disadvantage of a more complex distribution of the tool (a set of files to distribute instead of a monolithic executable). Finally, like GBS, these two build systems share the disadvantages of the build tools they use. As already mentioned, that is mainly poor checking that results in high chances of getting inconsistent builds during active development of the software.
My point in the previous section is that, despite what some people hoped, a good build system will not spare you the need for a good build tool. It is certainly helpful, but if the build tool is weak, the build system will have a hard time hiding it. Fortunately, the community of developers has been busy making not only smarter build systems, but also better build tools. In the following section, I describe another category of build tools. I call this category "script-based" build tools, but it is not easy to find a name that will adequately describe them in two words. It is important to understand the new paradigm shift introduced by this category of tools. The approach taken by their authors says, "Let's not invent a new syntax. We are not in the business of writing text parsers. Others already did that, and they did it better then we will do it in a fraction of our time." They also say, "Let's not write and maintain the portability layer for our build tool. Let's reuse some virtual machine already available." So these people focus on the build design and on build design-related issues. This seems to me a sound choice when one aims to provide a better build tool. I will mention only two tools in this category, Ant and SCons, but there are many others.
The Ant build tool is well on the way to become the next Make. It is an Open Source project from the Apache developer community. It is a Java program, and it uses XML for the build descriptions. It has spread quite quickly because it faced no competition (certainly not from the old Make). Too Java-centric at the beginning, it has grown today into a mature build tool that can put to shame many of its competitors. One sign that a piece of software is mature and fulfills a real need is the number of new projects that are taking that software as a base, and there are many software projects based on Ant, including:
It fact, no other build tool is gathering so many development efforts today. This gives an important head start to Ant in the race for the next Make.
Like the original Make, Ant was the primary source of inspiration for a set of new build tools like Nant and, more recently, MSBuild. They don't aim at file compatibility for build descriptions, but their spirit is the same: Describe the build as a set of tasks to carry out, each task coded as an XML element.
Of course, Ant is no silver bullet. Ant without extensions has a scalability problem. A lot of it comes from the choice of XML as a file format. XML is verbose. XML does not have a convenient way to include a file in another file (the XML Fragments standard is not really supported by XML parsers, XML entities includes work but have limitations, etc.). You need some kind of inclusion because you want to factor out common parts in several build descriptions. Of course, an application using XML files is free to add "include" semantics to one chosen XML element (and that is what Ant finally did with the "import" task in version 1.6).
Also, XML is not really a programming language. You may want to have the content of one XML element computed from sibling elements (computed strings occur often in build descriptions; see Make macros). Or you may want one element to be taken from the parent or the parent if not found at some place (which would correspond to variable scoping in programming languages). I say "you may want", but for projects of real-life size, you will need that. Otherwise, the build description grows very big and redundant. This is not a hard limitation; the Ant tool does give you the power to express whatever you want. It is just the fact that you have to go back and forth between a purely descriptive part of the build specification (build.xml) and a purely procedural part (the Ant tasks implementation) that I don't like very much. People have contributed Ant tasks like <if>, but this will not turn XML into a programming language overnight.
Another issue is that Ant has a noticeable overhead for startup and initial XML processing. This is an important psychological factor in the case of null builds (builds in which everything is up-to-date and nothing has to be done). The Make tool was fast in such builds. Make was fast for a bad reason, but the fact remains that people have a philosophical problem today with accepting that the build is slow when nothing is to be done. As with other build tools in this category, it would be nice to have the ability to "compile" the build description, and maybe the entire dependency tree, into a speed-efficient format.
SCons is a build tool that uses Python for the syntax of the build descriptions. The origin of SCons is Cons, a similar tool using Perl syntax for the build descriptions. That older tool is also still available. SCons is my preferred tool in this category and also my preferred build tool overall. It comes preconfigured with a fair set of rules ("builders" in SCons parlance). Even more compelling, it can be extended in an easy and natural way. Unlike Ant, the build description and the extensions use the same syntax (in this case, Python). SCons allows flexibility in the compromise of speed versus safety (by using either content signatures or time stamps). It detects changes on the commandline used to build something, and this makes it outstandingly reliable compared to its competitors. There are many other features I like in SCons: the transparent use of code repositories, the wink in from cached binaries, the ability to accurately document the build, the support for extraction of snapshots of the code and, finally, the promise to grow into a better build system (autoconf-style probes, the ability to generate projects files for IDEs, etc.).
As for its shortcomings, SCons is a young piece of software, at least when compared with some of the tools mentioned above. Another issue is that, despite not being recursive, SCons is slow. This seems to be a price to pay when focusing on high-level build design issues. A time will come when the authors of SCons will focus on speed optimizations. As with Perl and Java, Python allows moving the speed-critical parts to a native compiled code implementation (when the software is stable enough).
From another perspective, despite the fact that Python has a clear syntax, some people perceive SCons as being too low-level. It is perfectly possible for the release engineer to describe the build of a 50-developer project for several target platforms in only a few dozen lines of SCons code. The release engineer will find this a great time saver, but the average developer will find that code cryptic, like shell code in the early Unix days, or worse. The developers will soon ask for easier ways to add new components to the build.
An overview of build tools in one article has to be a limited one. We didn't even mention all the areas in which build tools needed and received improvements.
One such area is better integration with the SCM tools. This is important because it directly addresses a long-standing issue: the reliability of the build. There are many SCM tools providing improved build tools, both commercial (Omake in IBM Rational ClearCase) and Open Source software (Vesta). But comparing the build tools integrated with the SCM systems goes far out of the scope of this discussion.
Another area we didn't touch is automation and reporting of the build (CruiseControl and Dart as Open Source tools, VisualBuilder and FinalBuilder as MS Windows commercial software). Yet another area not discussed here is the acceleration of the build when parallel build machines are available (distcc and ccache as Open Source tools, IncrediBuild and ElectricCloud as commercial software).
Finally, there is one last area that we don't want to discuss here: the software distribution tools. Many tools today allow you to connect as a client to some software repository, download the software you want, automatically download other needed parts, and install everything on your system while locally building parts that need building. Those systems include a build tool or act as a build tool when needed. They have to act as a configuration tool also, and they have to do some form of dependency tracking, much like build tools do. The best known is probably Ports of the BSD OS, and its followers like Gentoo Portage. Take a look at the A-A-P tool Web site for a list of such systems.
The final conclusion is up to you. The considerations in this paper reflect only my experience. But I hope that you have now some useful background to choose the build tool and the build system that best fit your own requirements.