Users of certain other operating systems upgrade the software on their machines every few years (95, 98, 2000...). When you deal with software that moves at Open Source speeds and have a powerful package manager at your disposal, you can get in the habit of updating your system every morning while you sip the first cup of coffee. It's certainly convenient to be able to say "Grab anything new and install it for me", but do you know what procedures are place to ensure that you get what you were expecting and not an unwelcome surprise?
Today, I offer the results of an email discussion about package management security issues that I had with Jason Gunthorpe of Debian and Jeff Johnson of Red Hat.
|Jeff Covey:||The popularity of apt and rpm has led to a large number of users relying on automatic upgrades through their package management systems. Old timers who insist on compiling everything from source can be understandably concerned about the process of downloading a binary and installing it with minimal admin intervention. The convenience is bought at the price of trust in the system. How would you answer the following questions? Do you agree or disagree that the concerns they express are valid? If they are valid and are not currently addressed, do you have any ideas about how the problems could be fixed?|
|Jason Gunthorpe (Debian):||It depends who is expressing them :> If the 'old timers' use source packages with signature files and check those signatures, then they are pretty safe. If they audit all the source code they hand compile, then they are pretty safe. But if they just download random .tars and compile them without even looking, they are no better off than we are, possibly worse off.|
|Jeff Covey:||What facilities does your package manager (or a third party add-on such as autorpm) provide for automatic upgrading of installed packages?|
|Jason Gunthorpe (Debian):||APT! :>|
|Jeff Johnson (Red Hat):||
rpm (and all package managers based on rpmlib) depend on ordering
based on epoch:version-release comparison in order to identify the "newer"
of two instances of a package with the same name.
[Jeff Covey: What I meant was: Does Red Hat provide the ability for a user to issue a command that says, "Go get any new versions of the software I have installed, and install them for me" as Debian does with APT, or can this only be done with third-party tools such as autorpm?
Jeff Johnson (Red Hat): RPM has some rudimentary support for FTP and HTTP transfers, but does not try to download anything other than what was requested. Neither does dpkg, which is a closer analogue to rpm than APT.
Closer to APT in functionality is the Red Hat up2date package, which does dependency resolution using HTTP POSTs and is able to augment an update request with other packages that will be needed to complete an upgrade.]
|Jeff Covey:||Who controls the package archives from which new packages are downloaded? If it's possible for third party archives to be used, does your package manager warn the user that packages are being downloaded from somewhere other than the official source?|
|Jason Gunthorpe (Debian):||The user has choice here. Our default setup uses our top level 'official' mirrors. If third party archives are used, they would have to be manually configured by the user. No special warnings are given for these sources.|
|Jeff Johnson (Red Hat):||
Um, not applicable, as Red Hat packages are often the "official source".
[Jeff Covey: Red Hat only provides a limited subset of the software available in the RPM format. On http://www.rpmfind.net/linux/RPM/ , users will find all of these archives in addition to Red Hat versions and updates:
and many of these have multiple subdivisions.
Jeff Johnson (Red Hat): Um, almost all, if not all, binary software distributed by Red Hat (I do not speak about Cygnus, yet) is in rpm package format with signatures. Perhaps by "Red Hat" you mean "Red Hat Distribution"? Or do you mean that ftp.redhat.com has not just Red Hat software?
Jeff Covey: It seems you're reading what you want to see instead of what I wrote. I said that Red Hat's RPMs make up only a subset of the software which is available in RPM packages, and that people will therefore be, of necessity, downloading RPMs from sources other than you, possibly overwriting your own packages, and I was asking whether issues of security related to this are taken into account in RPM's design.
Jeff Johnson (Red Hat): I was confused by your rpmfind output, as you choose to, for example, include Red Hat Power Tools as a separate entity even though those are packages produced by Red Hat. Furthermore, many (but not all) of the rpmfind categories (e.g. Mandrake, LinuxPPC come to mind) that you chose to distinguish are closely related to Red Hat packages -- in fact, often *are* Red Hat source packages except for signatures or lack thereof -- that have been rebuilt and/or redistributed for other architectures and other purposes. I can speak meaningfully only about packages produced by Red Hat, not derivative works.
Addressing issues of heterogeneous mixtures of "You pick 'em" package installation is a difficult problem that will require administrative superstructure and standards development in order to be solved meaningfully, and none of that process is complete (although it has been started).
Jeff Covey: The question I was asking was: Since you obviously can't verify the thousands of RPM files packaged by all of these distributions and individuals, does rpm provide a warning like "This package has not been prepared by Red Hat. While it's probably fine, we cannot confirm that it will work with your system. Continue installation? [Y/n]"?
Jeff Johnson (Red Hat): The goal of rpm (and tools that use rpmlib, including the Red Hat installer), by design, is to not prevent unattended installs and automatic updates by blocking on user interaction. Please note that in the example above there is little information ("Not packaged by Red Hat") that is helpful in or pertinent to answering the question "Continue installation? [Y/n]". Therefore, rpm (and the Red Hat installer) do not ask these questions during package install.
This doesn't mean that better policy (e.g. "Install only packages from Red Hat") tools aren't needed or useful, just that rpm (and the Red Hat installer) is not where the implementation should be.
Again, the Red Hat up2date agent currently implements certain install policies (but not the example above) like "Don't permit /bin/sh to be replaced" or "Don't upgrade the kernel package".
Jason Gunthorpe (Debian): One thing I hear often about .debs is that [since] we basically are the only provider (particularly of the base system), all .debs 'work' with your system.]
|Jeff Covey:||Does your package manager support digital signatures that can confirm that the package is from the packager it claims to be from and has not been tampered with?|
|Jason Gunthorpe (Debian):||No. This is a very tricky topic given Debian's distributed nature.|
|Jeff Johnson (Red Hat):||rpm supports header/payload signatures using md5 as well as all algorithms implemented in either pgp/pgp5/gpg (e.g. RSA, DSS, Diffie-Hellman, ...).|
|Jeff Covey:||Are there procedures in place to check for trojans/virii/etc. in the original source package?|
|Jason Gunthorpe (Debian):||Depending on which maintainer you talk to, yes or no. Some packages are inspected carefully, some are not.|
|Jeff Johnson (Red Hat):||Checking for trojans/virii in sources is outside rpm's abilities and is solely the responsibility of the packager.|
|Jeff Covey:||Are there procedures in place to check for trojans/virii/etc. in the package itself (for example, in the scripts used to install the package)?|
|Jason Gunthorpe (Debian):||I don't think we have an official program for this. People do look occasionally, I'm sure.|
|Jeff Johnson (Red Hat):||Signed rpm packages cannot be altered without being able to detect the alteration. The scripts are part of the header, which is signed, and so cannot be altered without being able to detect the change.|
I'm not asking about them being altered after the fact; I'm just
confirming that a procedure is in place to double-check the official
signed packages to confirm that, for example, a disgruntled employee
on his last day of work doesn't add "/bin/rm -rf /" to the preinstall
script of the binutils package and place it in the errata FTP space.
(Debian folks: This is even more of a question for you, since you're accepting packages from people from all over, who may only have their reputations, not their jobs and the threat of prosecution, hanging over them and keeping them in line.)
|Jeff Johnson (Red Hat):||
Part of Red Hat QA involves repeated installs of packages before and after
signing that would easily detect the example you have given. Red Hat also
does not release unsigned packages as errata, and there is sufficient
process in place that no single employee, disgruntled or otherwise, is able
to put an errata on the FTP site.
There are, of course, time bombs, viruses, and many other forms of damage more sophisticated than your example, and more generally, Red Hat, like many distributions, relies on the scrutiny of the community at large to detect and correct problems promptly. Enhancing the ability of the community to detect and correct problems before damage becomes widespread is the single best approach to insuring the safety (and quality) of packages that I can think of.
|Jason Gunthorpe (Debian):||
We have no official auditing of packages, but before we make a stable
release, the packages are put through a lot of testing and
investigation; it would be hard for a simple attack to get
through. Smart Devilish attacks I think could pass into stable
undetected if one of our maintainers decided to make one.
People do monitor the upload list to make sure that the 'right people' are uploading the 'right packages', which tends to defuse the worst things (like libc6 trojans, etc.).
Actually, we go through a fairly intensive ID process before we accept a package from anyone. If someone does decide to do something nasty, we will know exactly who it was, and depending on local laws, they may face prosecution. Look at http://www.debian.org/devel/join/nm-checklist; it has some information about this process.
|Jeff Covey:||If someone were to sneak a trojan into a package, it could spread to thousands of machines overnight as admins performed automated upgrades on their systems. If this were to happen, would it be possible for you to prepare a package that would fix the problem on the next dist-upgrade (not everyone reads security bulletins, so not everyone will be aware that she's been compromised)?|
|Jason Gunthorpe (Debian):||Yes, barring the point below.|
|Jeff Johnson (Red Hat):||Um, yes, as Red Hat releases security errata as quickly as possible, and these updates are copied to mirror sites and up2date servers as part of the process of releasing an errata.|
|Jeff Covey:||The answer to the previous question is naturally somewhat dependent on the nature of the trojan. As a worst case scenario: Is it possible for someone to insert a trojan into your upgrade stream which would disable your package upgrade system on the client side, making it impossible for you to distribute a fix through the normal method?|
|Jason Gunthorpe (Debian):||Yeap. The packages install with root privilege; a well-written trojan can do anything.|
|Jeff Johnson (Red Hat):||Signed rpm packages cannot be used as a vector for trojan horses (assuming the installer checks the signature).|
Let's say Joe Packager uploads a new package of sendmail to a contrib
directory. Jane User runs her automatic update script. It sees that
she has sendmail installed, spots Joe's package, and offers to install
it for her. Jane checks the signature. Yes, it's from Joe and has
not been tampered with, so she installs it.
A couple of days later, someone notices that sendmail has been altered in this package to silently send copies of all mail to Joe and all his friends. You put out warnings about it and distribute a package with a version number higher than Joe's, so those people (like Jane) who don't bother to read security lists will at least get the fix when they run their update scripts.
Unfortunately, Joe's package also did something else: It replaced /bin/rpm with a version that will not install any version of sendmail or RPM other than Joe's. It will pretend to install your replacement, but won't actually do it, so Jane will never know that her system's been compromised.
[Jason Gunthorpe (Debian): Unless you sandbox the install scripts, this is impossible to prevent :<]
You might say this really isn't your problem, because if people want to be safe, they shouldn't install any packages that don't come from you, but it isn't reasonable to expect that, since there's a lot of software people want that Red Hat doesn't package (otherwise, Mandrake, etc. wouldn't exist). People *are* going to be getting their RPMs from other places.
[Jeff Johnson (Red Hat): Um, I question whether the example above illustrates anything but
Would you consider either of these valid solutions to the problem?:
Do you have any other ideas about what could be done?
[Jeff Johnson (Red Hat): Yes, but judging from the types of examples and questions you are asking, I don't believe that this is the correct forum to present other ideas.]
|Jeff Covey:||If the answer to the previous question [whether a trojan could disable the update stream] is "yes", do you think it would be beneficial to establish a class of protected packages which can only be upgraded with packages that come signed by you?|
|Jason Gunthorpe (Debian):||This would not really help prevent the attack above; you can always use some trivial package to modify the files of an important one.|
|Jeff Johnson (Red Hat):||Implementing better install policies based on explicitly verifying signatures is in everyone's interest.|
|Jason Gunthorpe (Debian):||
I think given what Jeff [Johnson of Red Hat] said, I'd just like to
mention basically the key point in the list thread I mentioned. [See
Red Hat has a single (hopefully) well-secured signing key that can only be used for packages that they produce in house. This is feasible for them because their development is concentrated in one physical location, and they don't have the constantly-changing archive like we do
Debian has a similar key kept by the Security Team, but it is infeasible to sign every official package that is created with this key, particularly since there are hundreds of new packages produced every single day. (Though we have been talking about signing full releases with this key.)
So, in the Debian situation, the next logical option is to use the maintainer's personal key for signatures. This brings up the really interesting question of 'who should sign a package'. With some 500 signing keys, the security of the whole scheme is in question. It is entirely possible to trojan an important package like libc using the most weakly secured key out of the 500.
This same problem can be applied to upstream source packages, too. If someone downloads a package, he can check the signature, but he also has to *check the key*.
The main point here is that just slapping a signature on packages does not necessarily make them as safe as the cryptosystem being used, or any safer than not having a signature.
Jeff Covey received his degree in classical guitar performance but
spent so much time with his computer that he fell in with a bad crowd
and ended up working for Andover.net. He currently works on freshmeat
and runs a computer lab for the
kids in his neighborhood in his spare time.
We're eager to find people interested in writing editorials on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an editorial gets a freshmeat t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let firstname.lastname@example.org know what you'd like to write about.