Update: Developers from Conectiva have written to share their thoughts on the subject. Readying a distribution for a new release of the kernel at MandrakeSoft is, like other open projects, a constant process of refinement, testing, and interaction with non-MandrakeSoft Open Source developers.
Mandrake's bleeding-edge development distribution, "cooker", is always available to outside testers on the Internet. When a new kernel appears on the horizon, the first step is to package the new development kernel in a special RPM called "hackkernel". (Mandrake uses the "hack" prefix to indicate a development/unstable version of a package.) Once hackkernel is packaged, we can begin testing the distribution with the newer kernel. Testing involves hardware testing at MandrakeSoft's labs, but, even more importantly, it involves the cooker users out on the Internet. They are our biggest resource; no amount of internal testing can replace beta testing on the Internet at large.
Once the feedback starts arriving from cooker users and internal Q/A, the process of refining the distribution for the new kernel begins. This process involves the evaluation of each and every package to determine how each can best take advantage of the new kernel while still retaining full compatibility with the older kernels. The package changes which make the distribution better suited to the new kernel can come from many sources: internal developers, Open Source developers out on the Internet, and other distributions. These changes are all integrated, in patch form, into each RPM package. These updates are posted on cooker, and the process of refinement begins again.
The final step in readying a distribution for a new kernel is to update the installer. Since each distribution has its own installer (for the most part), this is typically a MandrakeSoft-only affair. (However, all our updates, including installer updates, are GPLed and available freely to all.) Eventually, based on cooker user and internal Q/A feedback and schedules, the cooker distribution will be frozen into a stable form and given a release code name like "oxygen" or "odyssey." After this freeze occurs, no more new additions occur, including kernel changes. Any final incompatibilities between the new kernel and other packages are smoothed out at this point. Finally, the CDs are pressed, and ISOs are uploaded to the Internet for all to enjoy.
The dialogue between MandrakeSoft and Open Source developers and users is constant. New changes to the kernel that appear on the linux-kernel mailing list and elsewhere on various project Web sites must be evaluated and tested. Bugfixes and features from MandrakeSoft must be fed back to the maintainers of the Open Source projects on the Internet. In the case of the kernel itself, this usually involves sending patches to Linus and/or the linux-kernel mailing list.
Linux-Mandrake is not unlike many established Open Source projects. The process of development, integration of new kernels, and testing is a constant cycle of refinement.
I'd like to hear any of your thoughts about the relationship between the kernel and the distributions built around it. If you're a user, does the wait for a new version of your distribution with the new kernel and features you need frustrate you, or do you just compile a new kernel on your own and hope the distribution is ready to handle it? If you're a distribution maintainer, what work do you have to do to accommodate a new version of the kernel? Are the communication channels between you and the kernel developers good enough that you get all the information you need in a timely manner? Is there anything that could be done to make the job easier for you? Do you find yourself spending much time talking to third parties who haven't made the necessary changes to their software to work with the new kernel, and convincing them to do it? Is there anything else the community could do to shorten the time between a kernel release and its inclusion in the distributions? Do you have any thoughts on distributions that use patches that haven't made it into the official kernel yet (journaling filesystem foo or USB support bar)? What about proprietary binary-only modules (video card drivers, etc.)?
The perfect integration of a new kernel in the distribution is, of course, one of the major concerns of every Linux distributor, and it involves expertise in kernel hackery as well as user feedback to find any problems not detected in internal testing. Internal testing just ensures that minimal quality standards are met; only the feedback of a larger circle of users can provide the real measurement of the distribution's quality.
Conectiva's kernel hackery expertise is supplied by a number of kernel developers including MM guy Rik Van Riel, Marcelo Tosatti (who actually builds the kernel packages), Arnaldo Melo, Aristeu Rozanski, and others. To extend its testing circles beyond Latin America, Conectiva is currently working to mirror its daily updated snapshot distribution in as many places as possible (and get as much community feedback as possible) through the bug tracking system and mailing lists.
The kernel bundled in the distribution in heavily patched. How many patches it carries is a compromise between features and maintenance costs. Keeping our tree close to the official tree avoids effort replication within the community and also it makes maintenance easier. However, there are some features which are interesting for us but not acceptable in the stable kernel tree for a variety of reasons. If these features are interesting enough for us to spend time testing and maintaining, they will be adopted. Examples are the USB backport, raw I/O, bigmem, ReiserFS, and others. This is basically the way all commercial distribution vendors work.
The drawback is that once you provide USB backports, LVM, ReiserFS, or any other extra functionality, you can't remove them and say to the users that it's not supported anymore. If something happens to the main development team, it's up to the distributor to keep maintaining this functionality. As for the latest stable release, Conectiva included all these patches as well as many others such as lm_sensors, I2C, DRDB, iBCS, IPVS, VM patches, supermount, raw I/O, fair scheduling, and dxr2. Locally developed patches are always sent upstream to the official maintainers.
Some of these addons require a set of userspace tools to work properly, and these tools also need maintenance. LVM, for example, has different, incompatible I/O protocol versions in different kernel releases, and the userspace tools must allow the user to boot the system using kernels with different IOP versions. In the snapshot release, Conectiva supports IOP6 (used in later 2.2 and early 2.4.0-test kernels) as well as IOP10 (used in the current 2.4). The use of wrappers and lvm-iop packages has been adopted by Conectiva and TurboLinux.
Additionally, the system should still work with plain, non-patched kernels. An administrator must be able to compile her own stock kernel and still have the system working.
Binary modules are not included in the kernel package. To be able to support the kernel code which is in the official distribution, we cannot accept unfixable and unmaintainable code. There may be binary modules on the additional CDs that are shipped with the distribution for user convenience, but they are not supported and the documentation is clear about that.