The Metapackage Problem and apt-get autoremove

Seems that this is a common problem among people using APT. During an apt-get run, it might tell you that there are a number of programs that can be removed using apt-get autoremove. Users that are too trusting may take these recommendations at face value and thus wind up removing too much – like all of GNOME and X for instance.

First, a little description about metapackages and package dependencies. A metapackage is one that only exists to provide dependencies: a metapackage requires a set of other packages, usually to provide a collective set of packages (such as a GNOME Desktop, a KDE Desktop, or other things). When you install a package like gnome-desktop-environment, it installs everything you need to have a complete set (in this case, a complete GNOME Desktop). This is expected behaviour.

The other thing to understand is the concept of an automatically installed package versus a manually installed package. When you manually install a package – that is, installed it explicitly by name – the package is considered a package that you, the user, wanted on the system. An automatically installed package is only there because a manually installed package required it. If you wanted one of the automatically installed packages, you would have installed it explicitly – right?

The problem comes when these two processes collide: when you install a metapackage, you are really saying that you want all of the dependencies installed, even though all of the dependencies are marked as automatically installed. A metapackage is an easy way of saying that you want all of these packages (listed as dependencies) installed without explicitly saying so. However, the system does not know this.

This problem then manifests itself when you remove one of the dependencies.

Let’s continue with this example: having installed the gnome-desktop-environment package, let’s say you wanted to remove GNOME Evolution – the evolution package. APT will warn you that the gnome-desktop-environment package will also be removed. It is at this point that you should pause and seriously consider the ramifications of what is about to happen – but we’ll continue onwards.

Once you have removed Evolution – and the “GNOME Desktop Environment” metapackage – there are a lot of automatically installed packages that are not required by any packages on the system. What does this mean exactly? Normally, an automatically installed package is not one that you wanted to have installed but it was required by something you did want. However, in this case, these automatically installed packages (such as vino, evince, and totem for example) are in actuality software that you want.

If you try to remove packages through the use of the apt-get autoremove command, you will see a list of packages that are marked as automatically installed and that are not needed by any package currently loaded. In our example, this is a long list of packages that you actually want to keep!

If you have already removed these packages…

There are a few ways to fix this problem once it has occurred. One is to mark all packages in the system as manually installed:

aptitude keep-all

This marks everything in the system as a manually installed package. This defeats the “autoremove” process entirely and may cause your system to contain unnecessary packages over time.

Another option is to use tools to mark packages as manually installed. There are a variety of ways to do this. One way is to actually try to install the package (again): APT recognizes this and just flags the already installed package as manually installed. Another way is to use the apt-mark utility or to use the aptitude unmarkauto command to change the marking on the package (such that the system labels it as manually installed).

However, the best way is to reinstall the metapackage: this will reinstall all of the bits that are needed for operation. If necessary, using the --reinstall option:

apt-get install --reinstall gnome-desktop-environment

This command can be done from the command line and can be done without X or other graphical environment. It can also be done from the rescue shell – and possibly without networking if you have never run apt-get clean.

If you are determined enough (and knowledgeable enough) you can get around this problem by building your own metapackages with the software you desire. Building a metapackage is simple enough and is made trivial through the use of the equivs package. However, if you want a package like gnome-desktop-environment but don’t want specific packages, a better idea might be to get the source to the package and rebuild it with just the software desired listed as dependencies.

You could also “fake it” by using the equivs tools to generate a metapackage that fulfils the role of the package or packages you wish to remove. This is not recommended, however.

An excellent article about using equivs was written in the Ubuntu Forums way back in March of 2008 by “epimeteo” from Portugal.

Even with the capabilities of the equivs package and other metapackages, the best thing to do is to keep the normal metapackage: this allows you to keep the system updated with current packages, prevents future surprises, and saves a lot of work.

Using the Message of the Day (MOTD) in Ubuntu Linux

The message of the day file (/etc/motd) used to be so simple… but with great power often comes increased complexity. The Ubuntu motd file is very powerful and can be manipulated easily once you know how.

The actual file is composed in /var/run/motd so that the /etc file system can still be considered read only and the file /etc/motd is a symbolic link to /var/run/motd. To make the motd file a static file as before, just change the link to another file such as /etc/motd.static.

In older versions of Ubuntu, the file /etc/motd.tail was introduced for this purpose; however, it has been deprecated for some time in favor of the newer /etc/update-motd.d framework. The /etc/motd.tail file can still be used and is adapted into the new framework.

The update-motd framework was introduced with the update-motd package in Ubuntu Intrepid; it now resides in the libpam-modules package as part of the pam_motd module.

In earlier versions of update-motd, the /etc/update-motd.d directory contained multiple directories that would be executed hourly, daily, weekly, or monthly. The scripts would be executed by a daemon update-motd that was run from cron.

In Ubuntu Lucid, this structure was phased out in favor of a straight sequential operation. The motd file is now generated by the pam_motd module upon login.

Each of the files in the /etc/update-motd.d is executed in numerical order; those in a Ubuntu Lucid Server install are:

/etc/update-motd.d# ls -l
total 28
-rwxr-xr-x 1 root root  57 2010-04-23 04:45 00-header
-rwxr-xr-x 1 root root 248 2010-04-23 04:45 10-help-text
-rwxr-xr-x 1 root root  65 2010-04-13 15:45 20-cpu-checker
lrwxrwxrwx 1 root root  46 2010-12-02 12:26 50-landscape-sysinfo -> /usr/share/landscape/landscape-sysinfo.wrapper
-rwxr-xr-x 1 root root  71 2010-04-13 15:45 90-updates-available
-rwxr-xr-x 1 root root  61 2010-06-30 09:01 91-release-upgrade
-rwxr-xr-x 1 root root  69 2010-04-13 15:45 98-reboot-required
-rwxr-xr-x 1 root root 261 2010-04-23 04:45 99-footer

These files generate a motd like this:

Linux myserver 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.2 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc

  System information as of Tue Nov 22 11:01:48 CST 2011

  System load:  1.84                Processes:           167
  Usage of /:   80.6% of 115.84GB   Users logged in:     0
  Memory usage: 50%                 IP address for lo:   127.0.0.1
  Swap usage:   10%                 IP address for eth0: 10.6.7.1

  Graph this data and manage this system at https://landscape.canonical.com/

21 packages can be updated.
0 updates are security updates.

*** System restart required ***
Last login: Fri Nov 18 09:46:33 2011 from 192.168.15.1

SSH introduces some slight complexity: SSH has its own handling of the motd file and can produce it upon demand. In a Ubuntu Lucid install, this capability is turned off because the PAM framework shows the motd through the pam_motd module on login.

Despite the documentation for motd.tail stating otherwise, there is no /etc/init.d/bootmisc.sh that runs and creates /var/run/motd; this is done upon login as previously mentioned.

If logging in with SSH, the SSH display of motd may be turned off either by adding a file ~/.hushlogin or by reconfiguring the server with a PrintMotd No option. Note that this is separate from the pam_motd process; thus if a Ubuntu system has PrintMotd Yes as well as pam_motd you will see double messages. Likewise, if a ~/.hushlogin file is present – or the option PrintMotd No is set – the pam_motd module will still run and a message of the day will still be seen.

In the PAM configuration of a standard Ubuntu Lucid install, both /etc/pam.d/login and /etc/pam.d/ssh contain the pammotd module. The /etc/motd file is selectable here; a different file entirely could be used by modifying the pammotd line appropriately:

session optional pam_motd.so motd=/etc/motd.other

To modify the motd within the update-motd framework, just create (or delete) scripts in the /etc/update-motd.d directory. Each script must be in the format NN-xxxxxxx where the first two numbers NN specify the order and the last xxxxxxx specifies a reasonable name for the script. The script should be an actual file and not a link, according to the documentation (although the 50-landscape-sysinfo script is a symbolic link!).

Output should be to stdout and start with a blank line and end with a newline. Remember that the script will execute on every login, so it should be quick.

Using README Files for System Documentation

I’ve become a huge fan of README text files scattered throughout the filesystem of any managed systems. When creating directories for various purposes, such as SFTP-only users, home directories for non-person users, data directories, etc – the README will serve well to document the files and the directory in general.

There are many places you can use a README file for better clarity. Consider some of these locations:

• NFS mount: where is the directory actually located?
• Mountpoint root: what directory is this supposed to be normally?
• Network shares: like NFS, describe where the directory is actually located – and why it is shared.

Lots of things can be described in a README file.

What are these files? This is the most obvious question to answer. Are they backup files? Are they one-time backup files? Are these files part of a project?

How long is the directory or files needed? Can the directory be eliminated at another time?

Where is the directory located? For network shares – and for disk mounts – labelling the directory in the README can help. This way, when the disk is mounted elsewhere (for recovery for instance) its actual mount point will be recorded. In the case of a network share, it helps to identify where the share is coming from without having to resort to other tools.

What should be done in the future? Putting a modest amount of future plans in the file helps to show future admins what was going on at the time the file was written.

When was the file written? The file itself should contain the date; otherwise, no one will know when the file was written. The filesystem metadata (file time) cannot be relied on as it can change for various reasons.

How does this configuration or set of files deviate from the norm? If you have a different set up, then a description of the configuration is in order. Otherwise, a future admin – or yourself – will not realize that things are different and may spend time figuring that out when they don’t have to.

Give an overview. Many directories will be comparable to large “containers” (like /data); describe what each of the directories are used for in a short paragraph or two.

When you go about creating a text file like this, remember several things:

• Use text-only with no formatting. Don’t make it a WordPad document, nor an OpenOffice document. The document should be readable by a minimal environment.
• Check clarity. It won’t do to go to this trouble and then find later that the writing is incomprehensible.
• Use proper spelling, grammar, and punctuation. Reading something that requires basic fixes in grammar and spelling can be grating and is not good for your professional image.
• Revisit the README file once in a while. Updating the file with new information will prevent mistakes and errors from creeping into the document, and will prevent future admins from relying on out-of-date information.
• Make more than one if necessary. This is almost never necessary – but when it is, don’t be afraid to do so. Name the file README.something to help future readers.
• Do your research. If you have questions about what a directory is for, what the files are for, or why things are the way they are, ask. Someone will have to do this one day anyway – so do it now and document it.

If you make a regular habit of sprinkling a little documentation like snow, you won’t regret it.

Restricting Users to SFTP Only and to Home Directories Using a Chroot

A chroot jail is a miniature environment which provides only just enough resources for one or another program to run. In this case, the sftp program is the one that is chroot’ed. The process can be made easier with tools like rssh or scponly - both of whcih are available in Red Hat Enterprise Linux 5.

However, the problem comes up when you want to use multiple users in this environment and you want to restrict each user so that they cannot see the rest of the environment, including other users. Neither rssh nor scponly will handle this sort of thing.

There is the program MySecureShell but it is not included with either RHEL or Ubuntu Server. One would have to compile it from source or use their packages for these environments.

However, as of OpenSSH 4.9 this problem was resolved: with the addition of the Match and ChrootDirectory options to OpenSSH 4.9 – as well as an internal SFTP server (in contrast to the external helper program otherwise used) – this allows the kind of restrictions wanted here.

Ubuntu Server Lucid Lynx (10.04) and Red Hat Enterprise Server 6 both come with OpenSSH 5.3p1 which includes both of these options. However, Red Hat Enterprise 5 only comes with OpenSSH 4.3p2. To see which version you have, use a command like this:

$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
$

Red Hat did backport the ChrootDirectory option into OpenSSH in RHEL 5 back in 2009, but not the Match option. Thus the ChrootDirectory option is good for everyone or no one. It is possible to use the option on the command line or in a specially crafted configuration file, but this requires a special server for SFTP instead of using the normal one – meaning two instances of SSH running if you want standard SSH logins.

It doesn’t seem to be possible to pass such options into rssh; rssh doesn’t allow you to pass options to ssh.

If you are using RHEL 5, it appears that the best option is to compile the most recent OpenSSH from source. To build the source, unpack the archive and copy the file contrib/redhat/openssh.spec from the unpacked openssh directory to the /usr/src/redhat/SPECS directory. Edit the file to your desire (such as disabling all of the X11 capabilities) and then perform this command:

rpmbuild -ba openssh.spec

In my case, the SPEC file contained a error in 5.9p1: there is a line that says:

%doc CREDITS ChangeLog INSTALL LICENCE OVERVIEW README* PROTOCOL* TODO WARNING*

Since there are no files named WARNING* this fails. Remove that entry from the end of the line:

%doc CREDITS ChangeLog INSTALL LICENCE OVERVIEW README* PROTOCOL* TODO

Then it compiles without problem.

If you want to use a recent version of OpenSSH on RHEL 5, you’ll have to make some configuration changes – and watch out for some changes. Changes include:

• ServerKeyBits increased from 768 to 1024 (knock back down with specific entry if this worries you)
• Protocol only supports version 2 by default
• ShowPatchLevel not supported by standard OpenSSH (requires patch)
• New ECSDA keys not supported fully or properly; `ssh-keygen -A` generates errors (can be ignored)
• MaxSessions added to configuration file
• AuthorizedKeysFile changes meaning slightly
• UseDNS is included in the configuration file (instead of being hidden – set it to “no” for speed)
• Banner is set to “none” by default

Change the Subsystem configuration entry for sftp to look like this:

Subsystem sftp internal-sftp

Add this to the end of the configuration:

Match group sftp-only
 ChrootDirectory %h
 AllowTCPForwarding no
 X11Forwarding no
 ForceCommand internal-sftp

Don’t forget to restart the ssh server:

service sshd restart

When you do this, make sure to have a root shell already open on the system so that you don’t get locked out. The user should exist in the password file like so:

test:x:521:521::/home/test:/bin/nologin

The home directory must be owned by user root and group root – including all directories in the path. The shell does not matter, as SSH will take over before the shell is activated; however, if there are other ways to log in with this user aside from SSH, then a proper shell like /bin/nologin or /bin/false is necessary.

For the specified configuration the user must be a member of the sftp-only group – either primary or secondary, it doesn’t matter. The Match option can also be used against a single user instead.

If you want the user – or anyone else – to be able to write files in this tree, create a subdirectory in the home directory with the appropriate permissions. The user – and others – will not be able to write to the home directory, but will be able to write to the subdirectory.

Don’t forget to test the login first – and don’t forget to have a root shell open before you start changing OpenSSH configuration or you may be unable to SSH into the system!

Configuring the Bash Shell (and Korn Shell) to Permanently Use vi-mode Editing

The GNU bash shell, as one might expect, uses Emacs key bindings to perform history and editing in the shell. Vi key bindings are available, and can be set with this standard command (just like in ksh):

set -o vi

My fingers have gotten this sequence of letters ingrained in them so I don’t even have to think about it – and I type it almost automatically after login.

However, there is a way to set up ksh to automatically set editing to vi mode. If the VISUAL variable is set to a value like vi, then the editing mode will be set to vi mode automatically. Likewise, if the VISUAL variable is set to emacs, then emacs mode is used for editing.

If the VISUAL variable is not set, then the EDITOR variable is used.

In later versions of the AT&T Korn shell (such as 20100202) the recognized values have been expanded: the pattern *[Vv][Ii]* is recognized as a request for vi mode; the pattern *gmacs* results in gmacs mode; and the pattern *macs* results in emacs mode. (The only difference between emacs and gmacs mode is the handling of the ^T character.)

This means that using vim or gvim will now trigger vi mode – indeed, using almost any vi clone editor such as nvi or elvis will work with this method. This also means that the full path can be used, although this may have been true previously.

In bash however, this use of EDITOR and VISUAL is not available. However, since bash uses GNU readline, we can set up readline to use vi mode by default – and thus also affect all programs that use GNU readline besides.

Edit (or create) a file in your home directory called .inputrc and add this line to it:

set editing-mode vi

After this, any time you log in – or use anything else that uses GNU readline (such as CLISP for example) – you’ll automatically have vi mode editing. I can finally rest my fingers…

Rescuing an Interrupted Ubuntu Upgrade

Upgrading Ubuntu from one version to the next has always been amazingly reliable; yet, sometimes, things happen. How do you recover when your upgrade is interrupted and you can’t boot the system in the usual way?

First step is to boot the machine to a shell that you can use to recover. This can be done in the standard system by using the Recovery Mode. Press the left shift key as the system boots up and choose a recovery mode kernel from the menu that appears – it will be labeled with “(recovery mode)” on the end. The recovery mode kernel will present you with the option to use a recovery shell.

You could use the rescue mode by booting with an Ubuntu install CDROM and entering “rescue” at the boot prompt, but then you must remember to mount all of your partitions and to perform a chroot to /target where the root partition is mounted.

After the system has booted to the menu, select the item that says “Drop to root shell prompt with networking”. This provides you with the ability to update the software packages from the Internet.

When the prompt appears, start by making sure that all currently installed programs are configured:

dpkg --configure -a

This may take a long time, depending on how many programs were installed before the system stopped. Then go ahead and update the current list of packages:

apt-get update

Following this, upgrade the software on the system:

apt-get upgrade

This last step, again, may take a long time to complete. You should pay attention to the list of packages held back (listed at the top of the output from this command); these packages will have to be requested specifically. Usually, packages are held back because they require new software or other drastic changes: the Linux kernel is always held back. Use a command like this one to install these packages:

apt-get install some-package some-other-package

You may have to repeat this more than once until all of the packages have been installed and none are held back.

Then, you should repeat the update and upgrade in order to completely verify that the system is as updated as it can be:

apt-get update
apt-get upgrade

These last commands should execute quickly, as everything is probably complete – however, it is not impossible that the upgrades would have affected something that requires another upgrade. Not doing this probably won’t matter, but why not do it anyway?

Once done, a reboot is required to make sure that the old software is no longer being used and all that is used is the new upgraded software. Don’t just continue the boot: reboot.

One more thing can be done to clean up afterwards – possibly after the reboot. At a command shell, enter this command to remove unneeded software:

apt-get autoremove

Some general tips on making distribution upgrades as easy as possible: first, upgrade the system to the latest set of patches and software available. If there are bugs in the ugprading software, you want to fix it first.

Once upgraded to the most recent patch set (or the most recent minor level), then upgrade to the next major level (and no more!). Upgrades are normally tested only to the next major release. Any other upgrade is certainly not tested and not guaranteed to work.

If you want to upgrade beyond just the next major release, repeat the above steps for each major release: first upgrade to the most recent software, then upgrade to the next major release.

Make sure that you have reliable power throughout: if a system loses power in the middle of an upgrade process (or otherwise stops cold) then you will have to perform a recovery and hope that nothing was lost.