Articles / Stop the autoconf insanity!…

Stop the autoconf insanity! Why we need a new build system.

Any veteran GNU/Linux user has, at one point or another, run across a package which used the autoconf/automake toolset. There is a lot to be said in favor of this emerging standard. Running "./configure && make && make install" usually results in a working installation of whatever package you are attempting to compile. The autoconf tools are also portable to almost every *nix platform in existence, which generally makes it easier to release your program for a large variety of systems. However, despite these few pluses, the auto* tools are constantly a thorn in the side of users and developers alike.

Problems for Users

Let's take a typical autoconf package. I'll call it package-xyz. Joe GNU/Linux User has just downloaded and untared this package. Like most users, the first thing he does is change directory to the newly-unpacked source tree and run a quick "ls" to see what files are there. To his delight, he discovers a "configure" script, indicating that he probably doesn't have to do any editing of Makefiles or other such craziness. Little does he realize the troubles awaiting him.

He again does the typical thing and runs "./configure --prefix=/opt". The configure script runs for a while, then exits with an error which basically translates to "You have an autoconf version which is three weeks old; please upgrade", but this is displayed in the most cryptic manner possible. He won't realize this is indeed what the error message means until he runs a few quick Google searches. He really wants to install this program, so he doesn't give up quickly. A few minutes later, he's run apt-get upgrade (or run whatever auto update his distribution uses).

He decides that he would like to customize his package, so he runs "./configure --help". A list of options puke themselves all over his screen. Undeterred, he runs configure again, but this time pipes the output to his favorite pager. He's forced to parse this output because there is no real standard for the options to pass to configure. He wants to use GTK, but how does he do it? Is it "--with-gtk" or "--enable-gtk"? Perhaps this time it's "--enable-gnome", or maybe "--with-toolkit=gtk". In any case, he finally "gets" it and runs the configure script, customizing the package to his heart's delight.

Then he discovers he's missing lib-lzw-3.2.3.4. He takes the time to go through the same rigamarole with that package and, after a bit of tinkering, gets it to install. Then he discovers that configure still thinks that the library is not there. He does a bit of investigation and discovers that he needs to delete config.cache. That's if he's lucky and the "cached" option isn't in some other random directory or file (in which case he would remove the source tree and start again from scratch).

The configure script now runs properly and outputs a nice Makefile, so Joe runs "make". To his surprise, configure, for no apparent reason, decides to run again. That's ok; Joe sits patiently through the same tests as they run over and over (this time, hopefully, cached). While looking through the log messages, he happens to wonder if there is, perhaps, maybe, some way to not have to run the same 50 tests over and over and over. After a while, make runs, starts compiling the program, and errors out. Joe reruns make to see what the error is, redirecting stderr so he can scroll up and see the original error message of the 95 presented. To his amazement, while configure picked up the fact that he did indeed have lib-lzw-3.2.3.4 installed, it failed to realize that the header files were located in /usr/include/lzw, not in /usr/include. At this point, Joe has a real problem.

Our friend Joe is a seasoned *nix user, so his gut reaction is to start Emacs (one thing the FSF got right) and edit the makefile. He does this only to discover a 50,000-line-long monstrosity. He greps it, looking for the right variable to edit. Of course, there are 500 different $(INCLUDE) settings spattered throughout. Little does he realize that the variable he's looking for is three directories down and called $(YOUD_NEVER_GUESS_THIS_VARIABLE_NAME_HAHA). He finally finds it and edits it to the proper value. configure again decides to run for no apparent reason (even though "make" skipped it the last five times) and overwrites all this hard work.

Now, Joe is presented with an interesting problem. He realizes that he needs to edit something besides the Makefile. But what does he look at? configure.in? Makefile.am? Makefile.in? Makefile.yoyo? Makefile.banana?

By now, the average user has done one of the following:

  1. Given up and tried a different package.
  2. Shot himself in the head with a twelve gauge shotgun.

Joe is feeling masochistic, and continues trying. I'll save you the pain of talking about his later problems and the pain of hearing about Sally FreeBSD user, who wants to configure a package to run under GNU/Linux on her iPaq.

Problems for Developers

Your average developer has no clue what m4 is or how it works. Most Unix people (except the old diehards) have not even heard of m4. That didn't stop the autoconf guys from using it.[1]

Here's the issue: In order to be able to write your autoconf setup files properly (without a considerable amount of pain and suffering), you must first learn m4. Let's follow the path of a GNU/Linux developer, a creative lady named Jane.

Jane is a seasoned developer. She is accustomed to writing Makefiles, but decides she wants the extra portability that autoconf allows. She isn't familiar with m4, but thinks she can wade through building her files anyway.

First, she looks for a tutorial on how to use autoconf. Sadly, she will spend a lot of time looking. If she's lucky, she'll discover a five-year-old online copy of a book that might help her accomplish 5% of what she needs to do.

The GNU info system isn't much help, either. The documentation she needs is spread over 50 info pages, divided into three packages. These divisions don't help much, as configure.in sometimes makes Makefile.am do strange things. Eventually, she gives up writing the files and does what every other developer does with the auto* tools: copy someone else. There are perhaps 30 developers worldwide who understand the autoconf tool chain. (I doubt there is anyone who quite understands the braindead Makefile that is puked out.) This very problem is what caused many of Joe's issues. Well-written configure scripts are rare things of beauty. (They are mostly found in GNU projects, but I've seen a few others.)

The Root of the Problem

The autoconf/configure system is clever, but in the end, it is just a very creative hack to work around the deficiencies of Makefiles. Those who have used *nix for a long while have seen several such workarounds. There's the Sleepy-Cat libdb approach of having 50 Makefiles, one for each architecture. There's the "config.h" approach seen in several packages. There's "imake", used by X11 projects.[2]

It's easy to see why autoconf resorts to outputting 50,000-line-long Makefiles. Makefiles lack support for complex dependencies and complicated error checking. Many of the above tools use the same hacks, like using "touch" to create files used for dependencies, with complicated build rules for making those files. A paper on the various Makefile techniques could fill volumes.

Solutions

There are several unique and innovative projects being developed to help sort out the quagmire in build processes.

Two such solutions are "SCons" and "Cons". Both try to replace Make with something far more flexible. However, they both depend on tools which are a bit less standard than sh and m4: Perl and Python. Still, both languages are quickly becoming standard on the major Unixes. SCons is a bit less mature, but is preferred by the author of this paper. Using either allows a much clearer build process for complicated software. There is no requirement to generate a set of build instructions from a template.

Another interesting piece of software is A-A-P. It allows a user to create "recipes" for building software.

Any of these is cleaner and easier than autoconf. I like the familiarity that Make brings to the table, but it's time to face up to the fact that Makefiles introduce far more complexity than needs to be.

  1. To be fair, m4 is a nice tool if you know how to use it.
  2. imake is perhaps the only thing that makes autoconf look sane, but I digress.

Recent comments

22 Mar 2013 06:39 Avatar igor2code

I've had the same problems before I sat down and wrote scconfig (http://repo.hu/projects/scconfig) - no m4 or other script dependency, no 50 layers, doesn't try to automatically generate 50000 lines of Makefile. It detects, then lets you do whatever you want with the result, be it generating a Makefile.inc with some variables you will include from your plain Makefile, config.h or a monolith Makefile.

12 Mar 2013 20:10 Avatar ilya239

>He again does the typical thing and runs "./configure --prefix=/opt". The configure script runs for a >while, then exits with an error which basically translates to "You have an autoconf version which is >three weeks old; please upgrade"

By design, configure scripts generated by autoconf do not require autoconf when run.
To avoid the above behavior, add
AM_MAINTAINER_MODE([disable])
to configure.ac

20 Jul 2007 04:28 Avatar Vintermann

Re: Portability is always hard to achieve


> Developers should

> learn to use the autotools properly

> instead of agreeing with users who bash

> against them.

>

I think that if almost no one manage to make a "proper" autotools project, that says more about the project than the developers. If it was easy to learn to use the autotools properly, people would have done it, and every open source project of any size would pretty soon run into a guy who said "here, I'll fix your build scripts, they're too ugly right now". That just isn't happening...

23 Oct 2006 09:33 Avatar jcalcote

Missing features in scons
I've been working with autotools for about 4 years now - the first 3 years I didn't know what I was doing, and just tried to hack my way through when there was a problem. This last year, I decided to really understand it. It's been painful, but worth the effort. The really sad part is that it didn't have to be that painful. If there were a really good tutorial on not just the mechanics of autotools, but on the underlying motivation for it, and if all of this documentation were gathered into a single place, it would have been almost painless. Here's the basis for understanding autotools, step by step:

1. Learn what the intent behind autotools is

2. Learn how the toolchain works - what generates what?

3. Learn the underlying language (m4).

Now you have a half a chance at understanding what it does. Quit trying to think you can hack your way through autotools input files without understanding what they are for and how they work.

Before you decide to look at other tools, please try to compare apples to apples. Don't sit there and tell me that scons is a great replacement for autotools. For the things that scons does, it's a wonderful replacement. But if you need the additional functionality provided by autotools, then you just can't do it well in scons. What are these things? Mostly they have to do with package building, maintenance, and distribution.

I'm a packager for SuSE, as well as an open source software project administrator (for multiple projects). Scons is great if I want to build, but it does nothing to help me package and distribute my software. I've been on the mailing lists for scons for some time now, and I've commented on this missing feature set. One of the originators of scons (Steven Knight) has responded to my comments, and the crux of his responses are this: You're right, we need to add these features - why don't you start such a project and add them?

If you want to create a project for inhouse use, then by all means, use scons. If, however, you want to create a project to be packaged for distribution in a GNU/Linux distribution, you'd better use autotools, or be prepared to emulate all of the functionality that autotools gives you with a custom makefile. But don't even dream that a packager for a major distro is going to pick up your project and add it to a distro unless you've done just this. You must have support for all of the following targets in your custom makefile:

all dist distcheck install check installcheck

And all of these targets have to build cleanly on most *nix platforms in order to be considered as a candidate for packaging with a distro such as SuSE, RedHat, Ubuntu, or Debian.

Even now, there is a recent (may 2006) news posting on the scons web site - quote:

"Google's Summer of Code will fund a project proposed by Philipp Scholl to add support for packaging and release dependencies to SCons. Stefan Seefeld will be mentoring Philipp's proposal for the next several months. Congratulations to Philipp, and thanks to Stefan for mentoring."

Summary: Let's not be premature in trying to replace autotools with other build management tools - they're not ready yet - and don't forget, this article was written in 2003 - it's now 2006, and we STILL don't have a proper replacement tool chain for autotools. Perhaps soon...

30 Jul 2006 16:12 Avatar jasen_13

Re: hum


> It won't solve the problem, though,

> because then include files move,

> libraries move, etc. We would also need

> something like KDE's 'kde-config' script

> for every single package on the system.

something like pkg-config perhaps...

it works quite well here (linux), it's a bit of a pain

to get it working right for doing cross-compiles (MinGW)

Where working right requires it to completely ignore the native config data.

I haven't managed to get autoconf to work for cross compiling, scons can be convinced though.

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.