The Metapackage Problem and apt-get autoremove

Seems that this is a common problem among people using APT. During an apt-get run, it might tell you that there are a number of programs that can be removed using apt-get autoremove. Users that are too trusting may take these recommendations at face value and thus wind up removing too much – like all of GNOME and X for instance.

First, a little description about metapackages and package dependencies. A metapackage is one that only exists to provide dependencies: a metapackage requires a set of other packages, usually to provide a collective set of packages (such as a GNOME Desktop, a KDE Desktop, or other things). When you install a package like gnome-desktop-environment, it installs everything you need to have a complete set (in this case, a complete GNOME Desktop). This is expected behaviour.

The other thing to understand is the concept of an automatically installed package versus a manually installed package. When you manually install a package – that is, installed it explicitly by name – the package is considered a package that you, the user, wanted on the system. An automatically installed package is only there because a manually installed package required it. If you wanted one of the automatically installed packages, you would have installed it explicitly – right?

The problem comes when these two processes collide: when you install a metapackage, you are really saying that you want all of the dependencies installed, even though all of the dependencies are marked as automatically installed. A metapackage is an easy way of saying that you want all of these packages (listed as dependencies) installed without explicitly saying so. However, the system does not know this.

This problem then manifests itself when you remove one of the dependencies.

Let’s continue with this example: having installed the gnome-desktop-environment package, let’s say you wanted to remove GNOME Evolution – the evolution package. APT will warn you that the gnome-desktop-environment package will also be removed. It is at this point that you should pause and seriously consider the ramifications of what is about to happen – but we’ll continue onwards.

Once you have removed Evolution – and the “GNOME Desktop Environment” metapackage – there are a lot of automatically installed packages that are not required by any packages on the system. What does this mean exactly? Normally, an automatically installed package is not one that you wanted to have installed but it was required by something you did want. However, in this case, these automatically installed packages (such as vino, evince, and totem for example) are in actuality software that you want.

If you try to remove packages through the use of the apt-get autoremove command, you will see a list of packages that are marked as automatically installed and that are not needed by any package currently loaded. In our example, this is a long list of packages that you actually want to keep!

If you have already removed these packages…

There are a few ways to fix this problem once it has occurred. One is to mark all packages in the system as manually installed:

aptitude keep-all

This marks everything in the system as a manually installed package. This defeats the “autoremove” process entirely and may cause your system to contain unnecessary packages over time.

Another option is to use tools to mark packages as manually installed. There are a variety of ways to do this. One way is to actually try to install the package (again): APT recognizes this and just flags the already installed package as manually installed. Another way is to use the apt-mark utility or to use the aptitude unmarkauto command to change the marking on the package (such that the system labels it as manually installed).

However, the best way is to reinstall the metapackage: this will reinstall all of the bits that are needed for operation. If necessary, using the --reinstall option:

apt-get install --reinstall gnome-desktop-environment

This command can be done from the command line and can be done without X or other graphical environment. It can also be done from the rescue shell – and possibly without networking if you have never run apt-get clean.

If you are determined enough (and knowledgeable enough) you can get around this problem by building your own metapackages with the software you desire. Building a metapackage is simple enough and is made trivial through the use of the equivs package. However, if you want a package like gnome-desktop-environment but don’t want specific packages, a better idea might be to get the source to the package and rebuild it with just the software desired listed as dependencies.

You could also “fake it” by using the equivs tools to generate a metapackage that fulfils the role of the package or packages you wish to remove. This is not recommended, however.

An excellent article about using equivs was written in the Ubuntu Forums way back in March of 2008 by “epimeteo” from Portugal.

Even with the capabilities of the equivs package and other metapackages, the best thing to do is to keep the normal metapackage: this allows you to keep the system updated with current packages, prevents future surprises, and saves a lot of work.

Putting Debian packages on hold

When administering a Debian (or Ubuntu) system, putting packages on hold can be very useful. For example, if a critical part of the system is used by developers, and is continually updated, the developers will want to be aware of updates and will want to check their code in the new environment. Programs like Tomcat, Cocoon, MySQL will be in this category.

Similarly, if a critical portion of the system is to be updated, you wouldn’t want it to be part of the automatic updates – though you really shouldn’t automatically update, since you don’t know what can break until you test it.

To hold a package or packages, you should use dpkg --set-selections. If you run the command dpkg --get-selections you can see what is set already:

# dpkg --get-selections | head
acct                                            install
adduser                                         install
apparmor                                        install
apparmor-utils                                  install
apt                                             install
apt-transport-https                             install
apt-utils                                       install
aptitude                                        install
at                                              install
auditd                                          install

As an example, let’s consider the package dnsutils. Let’s see what would happen before we do anything:

# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  bind9-host dnsutils libbind9-60 libdns64 libisc60 libisccc60 libisccfg60 liblwres60
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,257kB of archives.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? n

Now let’s change this. We’ll put the package dnsutils on hold using dpkg --set-selections:

# echo dnsutils hold | dpkg --set-selections

Let us check the results:

# dpkg --get-selections | grep dnsutils
dnsutils                                        hold

Now, when we try the system update again, things have changed:

# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
  bind9-host dnsutils libbind9-60 libdns64 libisc60 libisccfg60 liblwres60
The following packages will be upgraded:
1 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 29.9kB of archives.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? n

Now, dnsutils – and its related packages – are being held back, just as we wanted. The other packages are being held back because they are only required by dnsutils; without upgrading dnsutils, they won’t be upgraded either.

Ubuntu: dpkg fails with “failed in buffer_read(fd)”

Recently, I was trying to update (and upgrade) Ubuntu Lucid, and received this error while running apt-get:

dpkg: unrecoverable fatal error, aborting:
failed in buffer_read(fd): files list for package `apparmor': Input/output error
E: Sub-process /usr/bin/dpkg returned an error code (2)

The solution was summarized nicely by Vivek Kapoor; he attributes the solution to C.M. Connelly (from 5 May, 2003). One of the nice things about Connelly’s entry is that he shows you how he debugged the problem he had, and how he fixed it; go read the post.

The error message is coming from dpkg, and refers to the “files list for package `apparmor'“. The files list is in /var/lib/dpkg/info; in this case, /var/lib/dpkg/info/apparmor.list. The problem being referred to in the error message is that, for some reason, this file cannot be read.

This file can be recreated if you have the package on hand; if not, you can fetch it with apt-get install -d package (possibly with the --reinstall option if necessary). The package will be downloaded to /var/cache/apt/archives, and even if a reinstall is attempted, the reinstall will fail (through dpkg) even though the download through apt succeeds.

The info file contains lines like this (using the top five lines of apparmor as an example):

drwxr-xr-x root/root         0 2010-03-30 14:59 ./
drwxr-xr-x root/root         0 2010-03-30 14:59 ./sbin/
-rwxr-xr-x root/root    783108 2010-03-30 14:59 ./sbin/apparmor_parser
drwxr-xr-x root/root         0 2010-03-30 14:59 ./etc/
drwxr-xr-x root/root         0 2010-03-30 14:59 ./etc/apparmor/

To recreate the file, pipe the output from dpkg -c debfile – like this:

dgd@cor:/var/cache/apt/archives$ dpkg -c apparmor_2.5-0ubuntu3_i386.deb |sudo tee /var/lib/dpkg/info/apparmor.list >/dev/null

After that, you should be good to go. You might want to check the disk (using fdisk) and perhaps reinstall the package to make sure all files are okay.

I don’t understand how this dpkg problem can last for seven years now; dpkg should be able to cleanly handle the recreation of this file if necessary, and shouldn’t be reporting obscure messages about its internal workings. From a user perspective – and a system administrator perspective – dpkg should automatically recreate the list file if there are problems with it, or even recreate all control files used by the package.

One very interesting tip was hidden in Connelly’s blog post from 2003: you can use less on a Debian package and it will report useful information (here’s an example from first lines of apparmor):

 new debian package, version 2.0.
 size 350314 bytes: control archive= 3944 bytes.
    2338 bytes,    61 lines      conffiles            
     360 bytes,    18 lines   *  config               #!/bin/sh
     662 bytes,    15 lines      control              
     708 bytes,    10 lines      md5sums              
    3577 bytes,   119 lines   *  postinst             #!/bin/sh
    2402 bytes,    90 lines   *  postrm               #!/bin/sh
    1186 bytes,    52 lines   *  preinst              #!/bin/sh
     959 bytes,    32 lines   *  prerm                #!/bin/sh
     421 bytes,     9 lines      templates            
 Package: apparmor
 Version: 2.5-0ubuntu3
 Architecture: i386
 Maintainer: Ubuntu Core Developers 
 Installed-Size: 2248
 Depends: libc6 (>= 2.8), debconf (>= 0.5) | debconf-2.0, lsb-base, initramfs-tools, debconf
 Suggests: apparmor-profiles, apparmor-docs
 Conflicts: libapache2-mod-apparmor (<< 2.5-0ubuntu2)
 Replaces: apparmor-parser, libapache2-mod-apparmor (<< 2.5-0ubuntu2)
 Section: admin
 Priority: extra
 Description: User-space parser utility for AppArmor
  AppArmor Parser is a user level programs that is used to load in program
  profiles to the AppArmor Security kernel module.

*** Contents:
drwxr-xr-x root/root         0 2010-03-30 14:59 ./
drwxr-xr-x root/root         0 2010-03-30 14:59 ./sbin/