Articles / The Universal Source Packag…

The Universal Source Package

Bud Bruegger writes: "This paper discusses the need for extending the philosophy of GNU autoconf into the world of package management. Such an extension could be seen as a 'universal source package' standard and tools. It was written with the hope of stimulating a discussion on the feasibility of such an approach."

GNU Autoconf Solves Only Part of the Problem

Most open source software runs on a wide variety of UNIX-like platforms. To make life much easier for software authors, and as a great incentive to write multi-platform software, the GNU Project created Autoconf, which makes it possible to easily compile and install software on a multitude of platforms -- or does it?

Well, it does a great job for compilation and also for installation if we install the software manually (eg., using make install). But considering that the majority of installations use comfortable package management tools, autoconf really covers only the first part of the needs. It does not address the problem of packaging the software for a wide variety of package management systems.

The number of package management systems is very large, and it is neither possible nor desirable to standardize on a single one. This means that a multi-platform program has to be packaged in many different package formats -- an enormous effort!

The similarity of compilation environments that autoconf relies on to address the multi-platform problem seems to be very similar to the similarity between various package formats: there are slightly different ways of achieving the same goal. (While it is true that different package management systems may differ in their functionality, the similarity is still there in all supported functional areas).

The Duplication of Effort for Packaging

Let's look at the social cost of this. Assuming that packaging for different systems is really similar, an enormous amount of effort goes into doing highly similar tasks. This is a serious concern, since users are constantly faced with problems such as software that does not exist pre-packaged, or packages being out-of-date or lacking important security fixes.

I believe that a lot of valuable resources are wasted by this tremendous duplication of effort. Applying the open source philosophy of reusing and sharing to packaging could free resources that could be used to:

  • Greatly increase the choice of prepackaged software.
  • Do a better job of keeping packages up-to-date.
  • Look into related problems such as the design of package tools that allow central administration of large clusters of machines (see, for example, http://www.beowulf.org/ or http://www.infrastructures.org/).

The Universal Source Package as Solution

The solution to the problem seems to be to extend the autoconf approach to package systems. This requires providing the necessary tools, standards, and guidelines to software and/or package authors to enable a single source package to produce a wide variety of binary packages for the various platforms and packaging systems. Even for someone who wants to compile all software by hand, the consistency of a standardized source package makes this job much easier, faster, and less error-prone.

While there are some efforts in this direction, I believe they are too limited:

  • Alien ( http://kitenet.net/programs/alien/) operates at a binary level and is thus inherently limited to a single platform.
  • EPM ( http://www.easysw.com/epm/) seems to be a great first step, but I believe that the solution of the problem requires a somewhat wider problem definition to maximize effect and that, in order to be successful, an organizational effort of standardization has to go along with the technical effort of development.

Components of the Universal Source Package Approach

What should a potential project on universal source packages produce? The following list of components is not meant to be comprehensive or even thought through in detail; it is rather meant to give first ideas of what may be needed to achieve the goal:

  • Specification of a universal source package format.
  • Tools to help build such packages (that can be ported to multiple platforms).
  • Tools to build binary packages for multiple architectures, base-OSes, and distributions from this single source. (EPM may do some of this already.)
  • An API that abstracts platform/distribution specifics that has to be used in pre- and post-install scripts (how to start and stop daemons, how to register a cron job, how to get logs automatically managed, etc.)
  • A universal API to be used by original authors that covers things such as logging, management of configuration data, etc. (Note that this opens the way to new system administration tools that are simply too complex with the current diversity of approaches).
  • Platform/distrib/language-dependent implementations of these APIs
  • Backends for the configuration management (XML files, central LDAP, DBMS, interactive Qs, etc.) based on these APIs (similar to Debian's debconf).
  • Guidelines for original authors.
  • Guidelines for universal package authors.

Conclusions

While competition in the field of package management tools is unavoidable and perhaps even desirable, effort duplication during packaging seems undesirable and causes an unnecessary limitation of the availability of packaged open source software today. Just as autoconf addresses the multi-platform issue in the domain of the software author, source-level packaging seems to be best dealt with there.

It is the author's belief that the time is ripe to start working on a universal source package system. While the effort required for such a project is considerable, the enormous benefits it promises may make it well worth while.


Bud P. Bruegger (bud@sistema.it) received a Ph.D. from the University of Maine in Geographic Information Systems in 1994. His company Sistema (www.sistema.it) develops information systems for customers such as FAO (www.fao.org). The idea of universal source packages grew out of a discussion about managing infrastructures (large clusters of computers of different UNIX flavors) on the mailing list of www.infrastructures.org.


T-Shirts and Fame!

We're eager to find people interested in writing editorials on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an editorial gets a freshmeat t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.

Recent comments

23 Apr 2004 17:43 Avatar nekrad

We need a good modeling of the package structure in source


I'm doing some research on a completely different approach:

Instead of writing rules for building software and then writing

rules about how these rules should be written by the machine

(well, that's in fact what autoconf does!), we need go in from

another side: tell the machine just the (abstract) structure of

our software.

Most packages can be modeled in a tree-structure of nodes like

libraries, modules, executables, sources, manfiles, etc.

We give our buildsystem just the information which makes our

software individual, thus listing the executables, sources,

libs, etc.

Here's a little example i've posted to xouvert-general today:

(this could be an XML representation of the "bitmap" applet)

<PACKAGE class="xwindow:applet" name="bitmap">

<EXECUTABLE name="bitmap">

<IMPORT type="library" name="Xaw"/>

<SOURCE name="BitEdit.c"/>

<SOURCE name="CutPaste.c"/>

<SOURCE name="Graphics.c"/>

<SOURCE name="ReqMach.c"/>

<SOURCE name="Bitmap.c"/>

<INCLUDE name="Dashes" class="xbm"/>

<INCLUDE name="Down" class="xbm"/>

<INCLUDE name="Excl" class="xbm"/>

<INCLUDE name="FlipHoriz" class="xbm"/>

<INCLUDE name="FlipVert" class="xbm"/>

<INCLUDE name="Fold" class="xbm"/>

<INCLUDE name="Left" class="xbm"/>

<INCLUDE name="Right" class="xbm"/>

<INCLUDE name="RotateLeft" class="xbm"/>

<INCLUDE name="RotateRight" class="xbm"/>

<INCLUDE name="Stipple" class="xbm"/>

<INCLUDE name="Term" class="xbm"/>

<INCLUDE name="Up" class="xbm"/>

</EXECUTABLE>

<EXECUTABLE name="bmtoa">

<SOURCE name="bmtoa.c"/>

</EXECUTABLE>

<EXECUTABLE name="atobm">

<SOURCE name="atobm.c"/>

</EXECUTABLE>

</PACKAGE>

I've set up a mailing list for my project, which I've called

"treebuild". You can subscribe via majordomo's mail interface at majordomo@metux.de

regards

--

E. Weigelt, CEO, metux IT service == http://www.metux.de/

--

Software-Patente töten Innovation und entrechten Autoren.

Stand up for your rights! == http://www.softwarepatente.com/

03 Feb 2003 21:06 Avatar spiderbox

Another project
How about this?:

uPM - The micro Package Manager

http://u-os.org/upm.html

16 Feb 2000 14:13 Avatar lithos

From another direction..
Spaceman Spiff you may be right, the only problem is I've hit resistance like that before. And although I really like to be the nice helpful guy, sometimes a nice sharp smack upside the head yeilds better results than being nice (frightingly so).

On the topic of bringing together all the unified, etc.. packing discussions the easiest way would be to start a mailing list. Depending on if you want a project from it or not you can go to www.onelist.com (just a mailinglist no project) or to www.sourceforge.net (mailing list, project, forum, etc..) Both are free and do a really good job.

Now to jump back from my pre-emtive flaming view with a unified package manager there need to be a few things laid out (all of them obvious, but I'll reiterate them anyway [go figure none of this stuff is really new])

Things needed from a Unified Package Manager (UPM) user perspective:
-Must be able to install a package with as few keystrokes/mouse clicks as possible.
-Installation should not require more than 12 steps to complete (and no sub-steps).
-You must be able to list what you currently have installed.
-You must be able to uninstall what you have installed.
-Configuration should be automatically run after the install.

Things that a UPM must do:
-Keep track of installed packages and their respective versions (I'd personally push for a standard versioning scheme across software eg: use the date)
-Needs to know if package xyz is a newer version of zyx and it's dependancies, or is it just a similar package.
-Have one format across systems.
-Have a set of API's such that the front end and middle ware can be different across platforms but the backend has strict guidelines.
-Have a set of tools so those who write software can easily package their software into the UPM format.
-Be able to install and uninstall packages across linux distributions.
-Should have installation rules (you must do things in this order *period*) such as check for diskspace before trying to install.
-Should have installation guidelines (suggested things but not manditory), things like a default installation path (that can be changed).
-Be able to resolve versioning/symbolic linking problems.. which libgl.so am I really using?

Things that need to be discussed/agree'd upon:
-Package format/organization
-API prototypes
-Rules and guidelines
-Etc..

One of the best ways to understand what is needed is to look at (gasp) windows, UPM must be better than the windows system (I'd really like when I uninstall things for the files and directories to really be removed).

Once things like this have been decided upon, then a spec can be made, then a working model.. Then whether or not the distribution maintainers want to use it, being nice and kind and asking those who write and distribute software to use it will make it a standard. But the foundation must be made first and shown to work. I know that with a good well rounded system that a few people from different software companies that I've conversed with would be very supportive of a UPM project and standard.

16 Feb 2000 11:42 Avatar pechter

Packaging is definitely a problem and a solution.
I'm a big fan of the FreeBSD ports and packages method as mentioned above.

While it's a pretty good answer to the problem of keeping thousands of packages up to date and handling the installation of commercial software for Linux on *BSD -- it's not perfect and there are people looking to improve it.

I'd also like to see a package format and tools compatible with the commercial Solaris packages so software from commercial vendors can run under Linux and *BSD as well (since binary compatibility is possible within the same architecture).

Imagine a modified BSD ports makefile system that would make packages that could be installed by a Solaris-compatible pkgadd utility. We could also load Solaris compatibility packages with it.

Bill

16 Feb 2000 09:40 Avatar swingerman

Additional RPM features (applies to other packagers, too)
All too often, RPM packages appear to rely on a particular package name as a dependency. This is fine if you are only using packages from one package vendor, but once you start to mix them, things can get ugly. The purpose of packages is to bundle up related components (docs, binaries, libs) into one file for easy distribution and installation. If you install a RedHat RPM of Mesa and you install a SuSE RPM of a program that needs Mesa to work, I have seen the combination of differing packagers fail.


Here is what I propose. Instead of each package listing other package names as dependencies, have an association between each package that is installed on a system and the capability or capabilities that it provides. This sort of association is potentially cross-distribution. We could set up some sort of central repository of what capabilities (from a joint-proposed name) exist on a system and what package provides it as a DBM-style database file. Each packager (deb, rpm, etc...) will be responsible for updating this database with its capabilities and reading it for capability conflicts.


This will work towards elimination of that all-annoying rpm --force -Uvh .i386.rpm by removing the dependence on a particular package naming scheme and replacing it with a joint-approved capability based setup. Tools could also exist to allow people to register the capability of software that they installed that didn't use a package managing system (.tar, for example). That way, if someone wanted to download and install Mesa by .tar and then install a package that required libGL.so, they could register that the Mesa.tar.gz provided the OpenGL execution capability, version 1.1. Then, when they went to install GLQuake.i386.rpm, the package manager would check it dependent capabilities and then happily install or alert the user if capabilities were missing. In either case, the user wouldn't have to rpm --force -Uvh each package that wanted the OpenGL execution capability, version 1.1.


Comments on this are solicited to jasonp019@yahoo.com (mailto:jasonp019@yahoo.com).


Thanks!

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.