Articles / Build and Release Managemen…

Build and Release Management

So, you want to write software? Don't forget that you'll need to build or package it, test it, fix some stuff, test it again, and ultimately release it... somehow. The "somehow" is the art and science of Build and Release Management.

Let's get some terms straight.

Building software, as any 15-year-old geek knows, is the generic term applied to the compilation or aggregation of sources into a usable utility or application. Sources needn't be compiled (built) in the classic sense, which relies upon a compiler linking files together; modern systems may consist entirely of interpreted source files (such as those for Perl or PHP). Still, putting everything together in a usable format may be considered "building".

Releasing software refers to the process of providing some named (or otherwise uniquely identified) files to others for use. The others may be your department at work, your classmates, or The World.

Managing the release means you know:

  • What went into it,
  • Where it went,
  • Why it went there,
  • And how to deal with it when bugs are reported. (They will be.)

Why is any of this important? I mean, if it compiles, SHIP IT! Right?

Wrong.

The larger and more complex the software project becomes, the greater the need for an extremely well-managed Build and Release function.

What characterizes a well-managed Build and Release function?

Several features are present, some quite basic. You may even be doing some of these things on your project. If you're only doing one -- or none at all -- walk away from the keyboard now before you hurt yourself.

Overnight builds

This sounds so basic, but I've seen many really big shops that weren't doing this. Building the software overnight accomplishes two primary objectives:

  1. It establishes the code-wise integration of the project.
  2. It immediately alerts you to code-wise disintegration.

These may sound like the same thing, but they aren't. You expect to see code-wise integration as the project matures; the overnight build confirms you have it. You do NOT expect to see code-wise disintegration (that old "it built yesterday, but not today" problem), and thus want to know immediately when it is encountered. Engineers should be the only ones who see legitimate code-wise disintegration. They should catch it during their unit tests and fix it prior to committing to the project. By running overnight builds, you will never have more than 24 hours of source code changes to review and/or backout if the build breaks.

Overnight builds also give your QA department something to throw test cases at, and to target for specific bug regressions. If you don't establish overnight builds very early on in your project, your QA team will suffer schedule slippage late in the project. This is almost guaranteed.

Competent Build Numbering

If you're going to build the project every night (and more often, some days), you will need to establish a build numbering system before you do much of anything else, along with some rules for build publication. Build publication rules refer to what builds get published for QA consumption and where. This usually just means a special directory structure on a specific machine. You should determine this up front, start using it, and not change it. This way, throughout the entire project cycle, any interested party can quickly look at any build. Use a sequential numbering scheme that is flexible enough to provide specific, built-in meaning in the number itself. For example:

freepository20110

This build "number" allows for visible identification of its contents. While the exact numbering scheme I use for Freepository isn't important, the fact that I can look at this number and know the build contains:

  • Freepository
  • Version 2 source
  • Internal build # 110

is very useful. Ideally, come up with a numbering scheme that is easily incremented via automated build scripts. The last thing you want to do is remember what build number to use, and then screw things up by accidentally reusing a number or skipping a number. (But hey, I've never done that...)

Automate, automate, automate

Human error suchs. I mean sucjs. Oh shti. See? Human error is an acceptable risk on trivial, performed-once operations such as writing an email message. Human error is NOT an acceptable risk when building and releasing your software. If you can type the steps into the keyboard, you can automate the process. Trust me, it is worth the trouble.

I automate virtually everything. In fact, an extremely complex Perl script is writing this article for me. I'm really asleep...

Which brings me to Perl. I began using it in 1996 and am still learning some neat tricks. This is really the Magic Belt. Using Perl, I have automated tasks that at first seemed completely unapproachable. If you don't like Perl, try Python or even one of the older scripting languages, like sed or awk. Heck, some of my earliest automation (1985) was accomplished using DOS batch files. Just use something.

Auto-Build Numbers

Got a build number embedded in your source? This is pretty common. Open up virtually any application and click "About". You'll see something like "Version 12.3 Build 1234". Those strings came from the source code, and someone probably had to hand-edit them prior to that build, commit them, tag the source, and then re-edit the strings to free the build number. Yuch! Automate that whole nasty process. Write a little script that reads your master build number file, increments and caches it, then opens the source file with the build and version string in it, greps or searches for the string in the file, does a replace, closes the file, commits it, and tags it.

Scheduling

I have encountered numerous problems over the years with cranky AT and WinAT schedulers, so I gave up and wrote my own. I use Perl; use whatever you like, but remember that Perl is extremely portable. It's very likely that the problem you resolve in Perl today will be a problem you'll face later on another project. By doing all your work in Perl, it is likely that you'll be able to reuse your work.

Reuse

Modularize your automation so it can be used over and over and over. Perl modules are an obvious Perl-centric solution. Create and export all your globals in a few well-thought-out modules. I have one module (now about 4 years old) that does NOTHING but figure out what today is and what yesterday was. Sound worthless? Not if you have to automatically roll back a release that begins to exhibit problems once in production. The concept of a build system-wide "Yesterday" is very useful.

Notification

I have my build system email me when it starts and finishes. I log everything, and, at the end, cull out the important stuff and format it into a message. This gets sent to the project team, and the text has a high-level "Build OK" message. There's a link for the details, but keeping the eye-catching message brief and to the point is extremely useful.

Since I know how long the build takes (to within a couple of minutes), I can be reasonably sure something is wrong if I don't receive the "completion" email on time. This allows me to proactively investigate and deal with whatever may have happened. This reduces schedule impact and, ultimately, project costs.

HTML

Web servers are everywhere. Use one as your presentation manager for builds and releases. All of my logs have the capacity to be displayed in HTML. We're not talking about flaming logos here, but rather the newest lowest common denominator: the browser. There's nothing fancy in my build logs, just lots of data that I must present in the most meaningful, cost-effective fashion possible.

Database storage

Once you discover how much data a build generates, you'll begin to consider all the really cool things you can do with that data, or, more properly, metadata. Examples of build metadata that I capture and key on the build number and date are:

  • Build number
  • Build date
  • Build version
  • Branch-tag used
  • Overnight build (Y/N)
  • QA tested (Y/N)
  • QA test results (Pass/Fail)
  • Location of full logs

I use PostgreSQL and MySQL. Each is freely available and can be quickly set up on a Linux box, and you can import your existing logs, after you massage them a bit. There's a practical limit to the amount of metadata you store. For example, even though you could capture and store things like "Number of binary files included in the build", you have to ask yourself whether the data is real. Just because a piece of information is available, it doesn't automatically become data.

Implementing one or more of the features mentioned above could have an immediate positive impact on your project. Start with the biggest ones, like the overnight build, and move on from there. Getting the framework in place isn't easy, but it only has to be done once.

So get to it.

Recent comments

04 Oct 2004 20:27 Avatar yjshen

Another tool concentrate on build automation
Just try luntbuild, an open source project.

It concentrates on build automation and management. I am sure it will flashes your eyes.

09 Sep 2003 10:56 Avatar banesong

Perl and Release cycle
We are developing an web application in perl, and due to an expansion of clients, we are having to look at a more complex release cycle. What tools would you recommend to use to manage releases? We currently us CVS on a linux box for vc, but looking at ant and anthill and their ilk, they don't exactly seem to fit us as we do a flat (ie no compile) distribution.

Thanks.

06 Dec 2002 13:44 Avatar urbancode

Re: Ant + AntHill
Here is the Anthill link:
http://www.urbancode.com/projects/anthill/default.jsp


23 Feb 2002 04:30 Avatar JornH

Re: Ant + AntHill
Yesterday i just stumbled over AntHill at www.urbancode.com/projects/anthill/ - it is a Tomcat based web-thing that makes the releases built with Ant available on a web server.... and its OSS :)

16 Feb 2002 03:09 Avatar mystran

I like to buid and release software with...
...ABSOLUTE NO WARRANTY.

No need to make any new releases.. once it works for me, well.. nobody can expect any response to bug reports.

Well.. NOT!!

But this actually how software industry works.. and how support works is "yeah we know.. it will hopefully fixed someday.."

OpenSource basicly works just because there is no actual need for any RELEASE .. everybody can make patches and it will just work.. release should just be the source with number of patches preapplied and probably some binary builds.

When you work with patches rather than files, you quickly find that it's actually possible to debug ONE thing rather than the whole release at the same time. You don't need to use any alpha code.. just the latest release and the patch.

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.