Articles / Security Issues of Auto-upg…

Security Issues of Auto-upgrades

Package managers with download capabilities make it easy to download and install the latest software releases, bugfixes, and security patches. Could they also make it easy to download and install the latest exploits without your knowing about it? In today's editorial, I put that question to representatives of Red Hat and Debian, makers of the two most widely-used Linux package management systems.

apt-get install trojan

Users of certain other operating systems upgrade the software on their machines every few years (95, 98, 2000...). When you deal with software that moves at Open Source speeds and have a powerful package manager at your disposal, you can get in the habit of updating your system every morning while you sip the first cup of coffee. It's certainly convenient to be able to say "Grab anything new and install it for me", but do you know what procedures are place to ensure that you get what you were expecting and not an unwelcome surprise?

Today, I offer the results of an email discussion about package management security issues that I had with Jason Gunthorpe of Debian and Jeff Johnson of Red Hat.

Jeff Covey: The popularity of apt and rpm has led to a large number of users relying on automatic upgrades through their package management systems. Old timers who insist on compiling everything from source can be understandably concerned about the process of downloading a binary and installing it with minimal admin intervention. The convenience is bought at the price of trust in the system. How would you answer the following questions? Do you agree or disagree that the concerns they express are valid? If they are valid and are not currently addressed, do you have any ideas about how the problems could be fixed?
Jason Gunthorpe (Debian): It depends who is expressing them :> If the 'old timers' use source packages with signature files and check those signatures, then they are pretty safe. If they audit all the source code they hand compile, then they are pretty safe. But if they just download random .tars and compile them without even looking, they are no better off than we are, possibly worse off.
Jeff Covey: What facilities does your package manager (or a third party add-on such as autorpm) provide for automatic upgrading of installed packages?
Jason Gunthorpe (Debian): APT! :>
Jeff Johnson (Red Hat): rpm (and all package managers based on rpmlib) depend on ordering based on epoch:version-release comparison in order to identify the "newer" of two instances of a package with the same name.

[Jeff Covey: What I meant was: Does Red Hat provide the ability for a user to issue a command that says, "Go get any new versions of the software I have installed, and install them for me" as Debian does with APT, or can this only be done with third-party tools such as autorpm?

Jeff Johnson (Red Hat): RPM has some rudimentary support for FTP and HTTP transfers, but does not try to download anything other than what was requested. Neither does dpkg, which is a closer analogue to rpm than APT.

Closer to APT in functionality is the Red Hat up2date package, which does dependency resolution using HTTP POSTs and is able to augment an update request with other packages that will be needed to complete an upgrade.]

Jeff Covey: Who controls the package archives from which new packages are downloaded? If it's possible for third party archives to be used, does your package manager warn the user that packages are being downloaded from somewhere other than the official source?
Jason Gunthorpe (Debian): The user has choice here. Our default setup uses our top level 'official' mirrors. If third party archives are used, they would have to be manually configured by the user. No special warnings are given for these sources.
Jeff Johnson (Red Hat): Um, not applicable, as Red Hat packages are often the "official source".

[Jeff Covey: Red Hat only provides a limited subset of the software available in the RPM format. On http://www.rpmfind.net/linux/RPM/ , users will find all of these archives in addition to Red Hat versions and updates:

PowerTools
Perl CPAN
RedHat Contrib Net
Gnome Desktop Environment
Libc6 Contribs
Libc5 RedHat Contribs
Arch Independent RedHat Contribs
No Sources RedHat Contribs
RawHide
Conectiva Linux
HelixCode Gnome distribs
Mandrake
Mandrake Cooker
XBF X-Windows servers
SuSE
Linux/PPC
Yellow Dog PPC
OpenLinux
Caldera Contribs
TurboLinux
Archives for blind users
DLD
UltraPenguin
SGIlinux
LinuxM68k
Freshmeat
Coda
Beowulf
RPM.Org
Stuff on Rufus.W3.Org
Solaris packages on Real-Time.Com
RPMs for Cygwin32/Windows

and many of these have multiple subdivisions.

Jeff Johnson (Red Hat): Um, almost all, if not all, binary software distributed by Red Hat (I do not speak about Cygnus, yet) is in rpm package format with signatures. Perhaps by "Red Hat" you mean "Red Hat Distribution"? Or do you mean that ftp.redhat.com has not just Red Hat software?

Jeff Covey: It seems you're reading what you want to see instead of what I wrote. I said that Red Hat's RPMs make up only a subset of the software which is available in RPM packages, and that people will therefore be, of necessity, downloading RPMs from sources other than you, possibly overwriting your own packages, and I was asking whether issues of security related to this are taken into account in RPM's design.

Jeff Johnson (Red Hat): I was confused by your rpmfind output, as you choose to, for example, include Red Hat Power Tools as a separate entity even though those are packages produced by Red Hat. Furthermore, many (but not all) of the rpmfind categories (e.g. Mandrake, LinuxPPC come to mind) that you chose to distinguish are closely related to Red Hat packages -- in fact, often *are* Red Hat source packages except for signatures or lack thereof -- that have been rebuilt and/or redistributed for other architectures and other purposes. I can speak meaningfully only about packages produced by Red Hat, not derivative works.

Addressing issues of heterogeneous mixtures of "You pick 'em" package installation is a difficult problem that will require administrative superstructure and standards development in order to be solved meaningfully, and none of that process is complete (although it has been started).

Jeff Covey: The question I was asking was: Since you obviously can't verify the thousands of RPM files packaged by all of these distributions and individuals, does rpm provide a warning like "This package has not been prepared by Red Hat. While it's probably fine, we cannot confirm that it will work with your system. Continue installation? [Y/n]"?

Jeff Johnson (Red Hat): The goal of rpm (and tools that use rpmlib, including the Red Hat installer), by design, is to not prevent unattended installs and automatic updates by blocking on user interaction. Please note that in the example above there is little information ("Not packaged by Red Hat") that is helpful in or pertinent to answering the question "Continue installation? [Y/n]". Therefore, rpm (and the Red Hat installer) do not ask these questions during package install.

This doesn't mean that better policy (e.g. "Install only packages from Red Hat") tools aren't needed or useful, just that rpm (and the Red Hat installer) is not where the implementation should be.

Again, the Red Hat up2date agent currently implements certain install policies (but not the example above) like "Don't permit /bin/sh to be replaced" or "Don't upgrade the kernel package".

Jason Gunthorpe (Debian): One thing I hear often about .debs is that [since] we basically are the only provider (particularly of the base system), all .debs 'work' with your system.]

Jeff Covey: Does your package manager support digital signatures that can confirm that the package is from the packager it claims to be from and has not been tampered with?
Jason Gunthorpe (Debian): No. This is a very tricky topic given Debian's distributed nature.
Jeff Johnson (Red Hat): rpm supports header/payload signatures using md5 as well as all algorithms implemented in either pgp/pgp5/gpg (e.g. RSA, DSS, Diffie-Hellman, ...).
Jeff Covey: Are there procedures in place to check for trojans/virii/etc. in the original source package?
Jason Gunthorpe (Debian): Depending on which maintainer you talk to, yes or no. Some packages are inspected carefully, some are not.
Jeff Johnson (Red Hat): Checking for trojans/virii in sources is outside rpm's abilities and is solely the responsibility of the packager.
Jeff Covey: Are there procedures in place to check for trojans/virii/etc. in the package itself (for example, in the scripts used to install the package)?
Jason Gunthorpe (Debian): I don't think we have an official program for this. People do look occasionally, I'm sure.
Jeff Johnson (Red Hat): Signed rpm packages cannot be altered without being able to detect the alteration. The scripts are part of the header, which is signed, and so cannot be altered without being able to detect the change.
Jeff Covey: I'm not asking about them being altered after the fact; I'm just confirming that a procedure is in place to double-check the official signed packages to confirm that, for example, a disgruntled employee on his last day of work doesn't add "/bin/rm -rf /" to the preinstall script of the binutils package and place it in the errata FTP space.

(Debian folks: This is even more of a question for you, since you're accepting packages from people from all over, who may only have their reputations, not their jobs and the threat of prosecution, hanging over them and keeping them in line.)

Jeff Johnson (Red Hat): Part of Red Hat QA involves repeated installs of packages before and after signing that would easily detect the example you have given. Red Hat also does not release unsigned packages as errata, and there is sufficient process in place that no single employee, disgruntled or otherwise, is able to put an errata on the FTP site.

There are, of course, time bombs, viruses, and many other forms of damage more sophisticated than your example, and more generally, Red Hat, like many distributions, relies on the scrutiny of the community at large to detect and correct problems promptly. Enhancing the ability of the community to detect and correct problems before damage becomes widespread is the single best approach to insuring the safety (and quality) of packages that I can think of.

Jason Gunthorpe (Debian): We have no official auditing of packages, but before we make a stable release, the packages are put through a lot of testing and investigation; it would be hard for a simple attack to get through. Smart Devilish attacks I think could pass into stable undetected if one of our maintainers decided to make one.

People do monitor the upload list to make sure that the 'right people' are uploading the 'right packages', which tends to defuse the worst things (like libc6 trojans, etc.).

Actually, we go through a fairly intensive ID process before we accept a package from anyone. If someone does decide to do something nasty, we will know exactly who it was, and depending on local laws, they may face prosecution. Look at http://www.debian.org/devel/join/nm-checklist; it has some information about this process.

Jeff Covey: If someone were to sneak a trojan into a package, it could spread to thousands of machines overnight as admins performed automated upgrades on their systems. If this were to happen, would it be possible for you to prepare a package that would fix the problem on the next dist-upgrade (not everyone reads security bulletins, so not everyone will be aware that she's been compromised)?
Jason Gunthorpe (Debian): Yes, barring the point below.
Jeff Johnson (Red Hat): Um, yes, as Red Hat releases security errata as quickly as possible, and these updates are copied to mirror sites and up2date servers as part of the process of releasing an errata.
Jeff Covey: The answer to the previous question is naturally somewhat dependent on the nature of the trojan. As a worst case scenario: Is it possible for someone to insert a trojan into your upgrade stream which would disable your package upgrade system on the client side, making it impossible for you to distribute a fix through the normal method?
Jason Gunthorpe (Debian): Yeap. The packages install with root privilege; a well-written trojan can do anything.
Jeff Johnson (Red Hat): Signed rpm packages cannot be used as a vector for trojan horses (assuming the installer checks the signature).
Jeff Covey: Let's say Joe Packager uploads a new package of sendmail to a contrib directory. Jane User runs her automatic update script. It sees that she has sendmail installed, spots Joe's package, and offers to install it for her. Jane checks the signature. Yes, it's from Joe and has not been tampered with, so she installs it.

A couple of days later, someone notices that sendmail has been altered in this package to silently send copies of all mail to Joe and all his friends. You put out warnings about it and distribute a package with a version number higher than Joe's, so those people (like Jane) who don't bother to read security lists will at least get the fix when they run their update scripts.

Unfortunately, Joe's package also did something else: It replaced /bin/rpm with a version that will not install any version of sendmail or RPM other than Joe's. It will pretend to install your replacement, but won't actually do it, so Jane will never know that her system's been compromised.

[Jason Gunthorpe (Debian): Unless you sandbox the install scripts, this is impossible to prevent :<]

You might say this really isn't your problem, because if people want to be safe, they shouldn't install any packages that don't come from you, but it isn't reasonable to expect that, since there's a lot of software people want that Red Hat doesn't package (otherwise, Mandrake, etc. wouldn't exist). People *are* going to be getting their RPMs from other places.

[Jeff Johnson (Red Hat): Um, I question whether the example above illustrates anything but

  • "Joe is a dangerous moron"
  • "Jane is too trusting"
  • "Thank God someone noticed"
  • "People are going to do whatever they want"
so a specifically informed reply is difficult.]

Would you consider either of these valid solutions to the problem?:

  1. Informing users when they're about to install a package that didn't come from you.

    [Jeff Johnson (Red Hat): Presenting repeated yes/no questions to users usually leads to simple carriage return answers to accept the default. That isn't exactly "informing users" in anything other than a (possible) legal sense.

    Not valid.]

  2. Making certain core files untouchable by non-Red Hat packages, or at least providing much stronger warnings like "WARNING! This package, which is not an official Red Hat package, wishes to overwrite /bin/sh, which is a protected Red Hat file. This could have dangerous consequences. Proceed at your own risk, and run rpm again with --force if you really want to install this package."

    [Jeff Johnson (Red Hat): Attempting to make files unremovable makes the package manager (rather than the end user) apparently responsible for the end user's safety, and, like shouting "WOLF! ..." too often or mistakenly, lulls the end user into a false sense of security. Adding "Proceed at your own risk" would keep the lawyers happy I'm sure, but would do little to protect the end user. Instrumenting overrides like "--force" is necessary for many reasons, but the functionality is more often abused than used correctly.

    Not valid.]

Do you have any other ideas about what could be done?

[Jeff Johnson (Red Hat): Yes, but judging from the types of examples and questions you are asking, I don't believe that this is the correct forum to present other ideas.]

Jeff Covey: If the answer to the previous question [whether a trojan could disable the update stream] is "yes", do you think it would be beneficial to establish a class of protected packages which can only be upgraded with packages that come signed by you?
Jason Gunthorpe (Debian): This would not really help prevent the attack above; you can always use some trivial package to modify the files of an important one.
Jeff Johnson (Red Hat): Implementing better install policies based on explicitly verifying signatures is in everyone's interest.
Jason Gunthorpe (Debian): I think given what Jeff [Johnson of Red Hat] said, I'd just like to mention basically the key point in the list thread I mentioned. [See References.]

Red Hat has a single (hopefully) well-secured signing key that can only be used for packages that they produce in house. This is feasible for them because their development is concentrated in one physical location, and they don't have the constantly-changing archive like we do

Debian has a similar key kept by the Security Team, but it is infeasible to sign every official package that is created with this key, particularly since there are hundreds of new packages produced every single day. (Though we have been talking about signing full releases with this key.)

So, in the Debian situation, the next logical option is to use the maintainer's personal key for signatures. This brings up the really interesting question of 'who should sign a package'. With some 500 signing keys, the security of the whole scheme is in question. It is entirely possible to trojan an important package like libc using the most weakly secured key out of the 500.

This same problem can be applied to upstream source packages, too. If someone downloads a package, he can check the signature, but he also has to *check the key*.

The main point here is that just slapping a signature on packages does not necessarily make them as safe as the cryptosystem being used, or any safer than not having a signature.

References


Jeff Covey received his degree in classical guitar performance but spent so much time with his computer that he fell in with a bad crowd and ended up working for Andover.net. He currently works on freshmeat and runs a computer lab for the kids in his neighborhood in his spare time.
http://pobox.com/~jeff.covey
jeff.covey@freshmeat.net


T-Shirts and Fame!

We're eager to find people interested in writing editorials on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an editorial gets a freshmeat t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.

Recent comments

15 May 2000 19:53 Avatar jmitchel

Yet another automated distribution system
AT&amp;T/Bell Labs/Lucent has had an internal automatic distribution system for some time. A variation on the current version was detailed in a September 1998 Dr. Dobbs article titled &quot;NSBD and Automatic Software Distribution&quot;, or http://www.bell-labs.com/project/nsbd/nsbdpaper1.html.

The solution to the problem of signed packages in a distributed development environment is for the central administrators to sign the public keys of the developers, and distributing them to the users, along with a signed list of packages that the developer has the right to distribute. This does not mean the package may not contain malware, but it does certify that the person who built the package had a right to do so.

More significantly here is to not do anything as root. This may render it useless for updating core parts of the environment, but a bit of creativity could, I suspect, at least nearly surmount this. The only program which must run in the context of this user is the automatic update program, which fetches modified files.

No untrusted program is ever run here in a context (other than by a careless root user) where it can possibly modify any system-wider resources. Further all files can be audited against a signed database of checksums, which highlights any discrepancies. This does remove the posibility of the usual pre and post install scripts, but in most circumstances this can be surmounted.

NSBD dosen't solve the problem that RedHat and Debian are trying to solve, updating 100% of the system without user intervention. It does solve 90% though, updating all non-suid/non-sgid programs without intervention, and requiring user intervention for the others. By lowering it sights slightly, it is able to acomplish this without the breathtaking security implications that rpm and dpkg have.

Further information is available at http://www.bell-labs.com/project/nsbd

14 May 2000 22:33 Avatar waldoj

Yup
I just want to point out that there's an auto-updater that wasn't mentioned in the discussion: yup, put out by Yellow Dog Linux. While it doesn't help with any of the problems discussed here, I feel I should at least point out that it exists. :)

14 May 2000 21:43 Avatar mattdm

A couple of responses to other comments.
First, why upgrade so often? Simple: to keep up to date with security issues. Not that there's necessarily one of these every week, let alone every night, but when one does come out, it's nice to have the fix applied as soon as possible.

Second, how could /bin/rpm be replaced by a different package? Simple. It's not done via the normal install-a-file mechanism of package manager -- instead, it's copied into place by a script.

14 May 2000 01:58 Avatar jtkirk

Re: Inspecting .deb files with ar

The next time you're starved for amusement (or worried that there might be a Trojan in your .deb), read the man page for 'ar' and play around unpacking a .deb with it to inspect the contents...

The command is ar -x PACKAGENAME


You could also try dpkg -x PACKAGENAME OUTPUTDIR which is likely to work even when the deb format changes, as I assume at sometime in the future it will. Not using bzip2 for these packages is silly, it would cut down package size and bandwidth requirements at the cost of a few extra cycles on local machines. Anyone know what compression/archiver redhat uses? I seem to recall using cpio and gzip (?) maybe.

14 May 2000 01:27 Avatar bdale

Inspecting .deb files with ar
The .deb format had as an explicit design criteria that it be possible to unpack and inspect a .deb without needing to trust any binary tools not already on your system. Back then, very few systems ran either Debian or Redhat, so even the package mangement tool started out as something you might want to inspect before installation.


The next time you're starved for amusement (or worried that there might be a Trojan in your .deb), read the man page for 'ar' and play around unpacking a .deb with it to inspect the contents...

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.