Articles / Editorial: The LSB is dead …

Editorial: The LSB is dead - Long live the LSB

Jim Pick originally intended to delve into some technical details about the differences between Debian and Red Hat and what obstacles lay in store for the LSB team. Due to the obscure activity that took place on the LSB mailing list (including Bruce Perens leaving the project), he changed his editorial to reflect the pros/cons of diversity in the Linux distribution space and what benefits there would be to have a LSB/LCS-style project. The LSB is dead! Long live the LSB!

I signed up to write an editorial on the LSB. Unfortunately, I picked this week to do it. Oh joy. :-)

I was originally going to delve into some technical details about the differences between Debian and Red Hat (both of which I am somewhat familiar with), and what obstacles lay in store for the LSB team.

But the political shenanigans of this week have convinced me to change my editorial to reflect on the pros/cons of diversity in the Linux distribution space, and what benefits there would be to having a LSB/LCS-style project. I'll try to dish out some of the dirt on the current situation (as I understand it) as well. :-)

I apologize for the length of this essay. I'm not very adept at compressing my arguments.

First, I'll clear up some information about myself. I'm a 28 year old programmer / electrical engineer who has been using Linux since 1995. I first started with Slackware, and then moved to Debian when the 1.1 release came out in mid-1996. That same year, I became a Debian developer -- so I've had about 2 years exposure to the inside dealings involved in building a Linux distribution (Debian is 5 years old this weekend). I'm also the new webmaster for LinuxHQ, which I am (very) slowly rebuilding.

  • Benefits of Diversity

    Why are there so many different Linux distributions? Because, "that's the way Linus wants it to be".

    When he wrote the Linux kernel, his goal could have been to put together a complete operating system distribution - much like what many other free OS's (FreeBSD, NetBSD, OpenBSD) have chosen to do. But as his interests lay primarily in hacking the kernel, he wisely left the politics of distribution building to others.

    As a result, there are many Linux distributions floating around out there. Each is different, some dramatically so.

    This is good, because they all can compete against each other. Think Darwin. Each distribution is evolving. The mature distributions (Red Hat, Debian, SuSE, Slackware, Caldera) have evolved to fit certain ecological niches. New distributions (ie. Stampede, Eonova, Mandrake) are born almost monthly. There are already many distributions that failed the "survival of the fittest" contest and have become extinct or morphed into something else (ie. SLS, Bogus, Lasermoon, Craftworks, LST).

    Some distributions target the mainstream. The vast majority of Linux users run a mainstream distribution, of which Red Hat currently has the most market share.

    But that's only a part of the story.

    There are a large number of distributions which target specific architectures which some of the mainstream distributions do not yet support (ie. Linux-ARM, MkLinux, LinuxPPC, Linux/SGI, UltraPenguin, and Extreme Linux for Beowulf class systems). There are even a bunch of distributions designed to be small (for use as rescue disks, Linux trial/demo disks, routers, etc.)

    There are also a number of Linux distributions that are localized for a particular non-English culture. I've found specialized distributions that cater to the Brazilian, French, German, Japanese, Russian, Spanish and Turkish markets. I've probably missed many. These "local" distributions are very important to the future of Linux, because the vast majority of the world's population does not speak English.

    Interesting as well are the variants of Linux that are being implanted in non-traditional computing devices. By the year 2001, IDG predicts that these devices will outnumber traditional computers. You can see examples of variants of Linux in action in Corel's NetWinder, Cobalt's Qube, and DEC's Itsy.

    There is competition in other areas than just the distributions. The Gnome vs. KDE rivalry is a good example. There are dozens of window managers, mail servers, web servers, web browsers, etc, etc, etc. All of these are competing against each other and rapidly evolving.

    As I write this, I've made a list of 50 active Linux distributions which I have posted on LinuxHQ. I'm hoping that some people will write in and give me pointers to a few more.

    Linux has evolved, and it's strength is in it's diversity. That's a large part of the reason that Linux is now one of the most popular OS's in the world, and will inevitably topple the mighty Microsoft empire.

  • Costs of Diversity

    All this diversity has come at a price however. That price is a massive duplication of effort, incompatibility, and a fragmented community.

    Is duplication of effort a real problem? I don't believe so. If it was a large problem, we would see the symptoms manifest themselves in the slow pace of development. The development of Linux I've witnessed has been anything but slow. Just look at the level of activity on Freshmeat! In fact, I tend to believe that our tendency to do everything at least twice in parallel is good. It has led to a high level of friendly (sometimes unfriendly) competition. This has resulted in better ways of doing things, as bad ideas and buggy software quickly becomes obsolete. The productivity gains we have achieved from competition have far outweighed the costs of duplication of effort.

    Is incompatibility a big problem? For the most part, it has been minimal. Most of the distributions are built from the same genetic material. They all use the Linux kernel, and they all come with the same libc and standard utilities you want (from GNU, BSD, and elsewhere) in order to have a full Unix-style environment. Open Source (tm) software is extremely portable, and uses tools like ./configure scripts to enable the software to compile just about anywhere.

    If you have the source code to a piece of Open Source software, there is very little stopping you from running that particular piece of software on any Linux distribution (or any Unix-style environment for that matter). After 3 years using Linux, I am now proud to use a system that contains nothing but Open Source software.

    Binary compatibility is trickier. Over time, major shifts do occur in the underlying libraries and the kernel itself. About 3 years ago, everybody moved from the a.out binary format to ELF. As we speak, distributions are still making the switch from libc5 to GNU libc. Next, some distributions will move from the FSSTND file layout to the FHS. These transitions will continue to occur, and they are very difficult to manage. Distributions need to control their own source code for the base system - they cannot rely on binaries from elsewhere.

    If you have old dynamically-linked binaries, you need to keep the old libraries around. If you want to run a piece of new software supplied as a binary, you need to have the newest libraries (forcing you to upgrade). Software distributed in binary format is sensitive to the way the different distributions name their libraries and the filesystem layout.

    Software distributed as source has none of these problems (provided that you are allowed to modify it). That is why the best Linux distributions come with full source. Free software is best distributed as source. Most distributions also provide pre-compiled binary packages. This is mostly a convenience to the users more than anything else. What distinguishes distributions from each other is how they plug their source together. Sharing binary packages across distributions is of limited value, because it is so easy to compile packages from source.

    People who write proprietary software have no choice but to distribute binaries. If they want to be portable across a wide variety of Linux distributions, they can do so quite easily by installing everything under /opt, dynamically linking with only a few well known libraries (such as libc), and by staticly linking (or avoiding altogether) libraries that aren't available everywhere (ie. termcap, curses, Motif). Most proprietary software is already designed to be portable across a wide range of Unix platforms, so they already do this.

    IMHO, binary incompatibility is not a big problem across the Linux distributions right now, and it won't be in the future.

    This leads us to the third cost of diversity - a fragmented community.

    Some fragmentation is inevitable. There are smaller communities within the larger "Linux" or "Free Software" community. Anything as big and global as this community is going to be fragmented by lots of things: national boundaries, language, choice of distribution, choice of windowing environment, choice of programming language, or choice of editors. I could go on.

    Personally, I don't think the Linux community is fragmented at all. The KDE vs. Gnome thing is the deepest split I see, and the two sides still talk at each other. The advocacy gets nasty at times, but that's the nature of electronic communication - and everybody is constantly learning. Almost everybody reads Slashdot, Freshmeat, c.o.l.a., and the LDP. And we all want to beat Microsoft.

  • LSB Project History

    The LSB project was Bruce Perens' baby. I'll go into a bit of history about Bruce (hopefully not too inaccurate) to give some insight into what I think the LSB was initially about.

    I first installed Debian in 1996, just after version 1.1 was released (there never was a 1.0 version). Bruce had recently taken over the leadership position from Ian Murdock, who had started Debian three years earlier. Bruce was very active in the development of the base system.

    He had gotten involved in Debian quite a bit earlier when it was still a GNU project. He was looking for a base system upon which he could build a custom "Linux for HAMs" distribution for amateur radio operators. I am speculating that Bruce believed that Debian would become the defacto base Linux system upon which all the distributions would be built, uniting the community, because it had the backing of the GNU project (which it later lost, before he became leader).

    In the two years when he was leader of Debian, Red Hat overtook Slackware and became the Linux distribution with the greatest market share. Linus uses Red Hat. I believe Bruce still wanted Debian to become a "standard base system" for other Linux distributions. He actively advocated moving to RPM and COAS within the project, which the rest of the developers showed little interest in. In the end he quit in frustration.

    A few months after quitting, he reappeared, soliciting Debian developers to jump ship with him to start a new RPM-based distribution. Soon after, he scrapped that idea, and announced the "Linux Standard Base" project, together with a list of mostly RPM-derived distributions that supported the idea. Conspicuously absent were Debian and Red Hat, who were later included.

    He put together a closed group of people representing several distributions to do the "engineering". This was to be done using a closed mailing list (with a public archive at lists.openprojects.net). Bruce's game plan for the LSB was to build a "reference" base system against which the different distributions could compare themselves against to judge their conformance. Bruce then set about building the base system by himself - alienating most of the members of the group.

    In a veiled act of mutiny, Dale Scheetz (of Debian) and Erik Troan (of Red Hat) started the "Linux Compatibility Standards" project to try to find some common ground (ie. library names) that could be written up into a paper standard.

    At the same time, Bruce was having a dispute with "Software in the Public Interest" (the legal front-end on Debian which he founded, then resigned from). It all ended this week with Bruce blowing up and resigning from the LSB and renouncing all involvement in Free Software. (I hope he has a change of heart, and comes back.) See Slashdot or the Linux Weekly News for the gory details. Yow!

    So now, we are left with picking up the pieces of the LSB, and reshaping it to become something that the Linux community needs.

  • Benefits to having an LSB/LCS style project

    There are almost as many Linux distributions as there are egos out there. Many people felt the LSB would be a good step towards unifying the Linux distributions. These people advocated that everybody should use the same packaging manager, use the file system hierarchy, share binary packages, etc, etc, etc. In essence, everybody would switch to using one distribution (closely resembling the distribution of whomever is espousing this view).

    That will never happen. Multi-million dollar companies such as Red Hat, Caldera, and SuSE aren't going to entrust their core product engineering to a community effort. Debian probably won't switch to rpm. And, as I argued previously, Linus doesn't want this to happen - this would reduce the diversity, and potential for world-domination, of Linux.

    But there are some good reasons for the distributions to at least talk to each other. If they could agree on some small policy decisions, they could make it easier to write source code that would run across Linux distributions. They could increase binary compatibility for commercial applications by agreeing on a common filesystem hierarchy (ie. the FSSTND/FHS) and collaborating on naming libraries (ie. sonames).

    Determining policy such as this is, well, political. Oftentimes, there is no single technical solution. What emerges is a compromise based on negotiation. If all the distributions used identical policy for everything, there would be no differences between them, and no diversity.

    Some common policy would still be very nice to have. Anybody who is developing a distribution already has a set of policies in place. In the case of Debian, which is a bunch of volunteers distributed around the globe, this policy is written up in a formal policy document. Debian even has a tool called "lintian" that will analyze packages and point out hundreds of places where they violate policy. And it has a bug system so that policy violations can be tracked. The end result is a very consistent, high-quality distribution.

    Other distributions, such as Red Hat, Caldera or SuSE, have similar set of internal policies that have been informally developed, but aren't written up anywhere. This works for them, because the developers physically work together, and can talk shop over the water-cooler. One problem with this approach is that the "contrib" maintainers from outside of the company have no idea what the policies are, so they make mistakes. Red Hat is taking some steps to move to a Debian-style system for "contrib" developers with their Contrib|Net system (see developer.redhat.com).

In conclusion, I do believe there is some benefit to having some common policy: increased source and binary compatibility (although perfect binary compatibility isn't really necessary or needed), and the existence of some formal policy documents for "contrib" developers to use (leading to higher quality contrib packages). It would be a good community-building exercise as well, as long as it is handled with some political tact.

Jim Pick

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.