Using the Message of the Day (MOTD) in Ubuntu Linux

The message of the day file (/etc/motd) used to be so simple… but with great power often comes increased complexity. The Ubuntu motd file is very powerful and can be manipulated easily once you know how.

The actual file is composed in /var/run/motd so that the /etc file system can still be considered read only and the file /etc/motd is a symbolic link to /var/run/motd. To make the motd file a static file as before, just change the link to another file such as /etc/motd.static.

In older versions of Ubuntu, the file /etc/motd.tail was introduced for this purpose; however, it has been deprecated for some time in favor of the newer /etc/update-motd.d framework. The /etc/motd.tail file can still be used and is adapted into the new framework.

The update-motd framework was introduced with the update-motd package in Ubuntu Intrepid; it now resides in the libpam-modules package as part of the pam_motd module.

In earlier versions of update-motd, the /etc/update-motd.d directory contained multiple directories that would be executed hourly, daily, weekly, or monthly. The scripts would be executed by a daemon update-motd that was run from cron.

In Ubuntu Lucid, this structure was phased out in favor of a straight sequential operation. The motd file is now generated by the pam_motd module upon login.

Each of the files in the /etc/update-motd.d is executed in numerical order; those in a Ubuntu Lucid Server install are:

/etc/update-motd.d# ls -l
total 28
-rwxr-xr-x 1 root root  57 2010-04-23 04:45 00-header
-rwxr-xr-x 1 root root 248 2010-04-23 04:45 10-help-text
-rwxr-xr-x 1 root root  65 2010-04-13 15:45 20-cpu-checker
lrwxrwxrwx 1 root root  46 2010-12-02 12:26 50-landscape-sysinfo -> /usr/share/landscape/landscape-sysinfo.wrapper
-rwxr-xr-x 1 root root  71 2010-04-13 15:45 90-updates-available
-rwxr-xr-x 1 root root  61 2010-06-30 09:01 91-release-upgrade
-rwxr-xr-x 1 root root  69 2010-04-13 15:45 98-reboot-required
-rwxr-xr-x 1 root root 261 2010-04-23 04:45 99-footer

These files generate a motd like this:

Linux myserver 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.2 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc

  System information as of Tue Nov 22 11:01:48 CST 2011

  System load:  1.84                Processes:           167
  Usage of /:   80.6% of 115.84GB   Users logged in:     0
  Memory usage: 50%                 IP address for lo:   127.0.0.1
  Swap usage:   10%                 IP address for eth0: 10.6.7.1

  Graph this data and manage this system at https://landscape.canonical.com/

21 packages can be updated.
0 updates are security updates.

*** System restart required ***
Last login: Fri Nov 18 09:46:33 2011 from 192.168.15.1

SSH introduces some slight complexity: SSH has its own handling of the motd file and can produce it upon demand. In a Ubuntu Lucid install, this capability is turned off because the PAM framework shows the motd through the pam_motd module on login.

Despite the documentation for motd.tail stating otherwise, there is no /etc/init.d/bootmisc.sh that runs and creates /var/run/motd; this is done upon login as previously mentioned.

If logging in with SSH, the SSH display of motd may be turned off either by adding a file ~/.hushlogin or by reconfiguring the server with a PrintMotd No option. Note that this is separate from the pam_motd process; thus if a Ubuntu system has PrintMotd Yes as well as pam_motd you will see double messages. Likewise, if a ~/.hushlogin file is present – or the option PrintMotd No is set – the pam_motd module will still run and a message of the day will still be seen.

In the PAM configuration of a standard Ubuntu Lucid install, both /etc/pam.d/login and /etc/pam.d/ssh contain the pammotd module. The /etc/motd file is selectable here; a different file entirely could be used by modifying the pammotd line appropriately:

session optional pam_motd.so motd=/etc/motd.other

To modify the motd within the update-motd framework, just create (or delete) scripts in the /etc/update-motd.d directory. Each script must be in the format NN-xxxxxxx where the first two numbers NN specify the order and the last xxxxxxx specifies a reasonable name for the script. The script should be an actual file and not a link, according to the documentation (although the 50-landscape-sysinfo script is a symbolic link!).

Output should be to stdout and start with a blank line and end with a newline. Remember that the script will execute on every login, so it should be quick.

Configuring the Bash Shell (and Korn Shell) to Permanently Use vi-mode Editing

The GNU bash shell, as one might expect, uses Emacs key bindings to perform history and editing in the shell. Vi key bindings are available, and can be set with this standard command (just like in ksh):

set -o vi

My fingers have gotten this sequence of letters ingrained in them so I don’t even have to think about it – and I type it almost automatically after login.

However, there is a way to set up ksh to automatically set editing to vi mode. If the VISUAL variable is set to a value like vi, then the editing mode will be set to vi mode automatically. Likewise, if the VISUAL variable is set to emacs, then emacs mode is used for editing.

If the VISUAL variable is not set, then the EDITOR variable is used.

In later versions of the AT&T Korn shell (such as 20100202) the recognized values have been expanded: the pattern *[Vv][Ii]* is recognized as a request for vi mode; the pattern *gmacs* results in gmacs mode; and the pattern *macs* results in emacs mode. (The only difference between emacs and gmacs mode is the handling of the ^T character.)

This means that using vim or gvim will now trigger vi mode – indeed, using almost any vi clone editor such as nvi or elvis will work with this method. This also means that the full path can be used, although this may have been true previously.

In bash however, this use of EDITOR and VISUAL is not available. However, since bash uses GNU readline, we can set up readline to use vi mode by default – and thus also affect all programs that use GNU readline besides.

Edit (or create) a file in your home directory called .inputrc and add this line to it:

set editing-mode vi

After this, any time you log in – or use anything else that uses GNU readline (such as CLISP for example) – you’ll automatically have vi mode editing. I can finally rest my fingers…

Turning off NET-SNMP overlogging

In the normal configuration – both on Red Hat and Ubuntu – you’ll find that SNMP is filling your logs with an endless amount of log entries, especially if you have monitoring tools that use SNMP every five minutes. They’ll generate messages like this:

Jan  8 13:45:02 example snmpd[2048]: Connection from UDP: [10.0.0.1]:51890
Jan  8 13:45:02 example snmpd[2048]: Received SNMP packet(s) from UDP: [10.0.0.1]:51890
Jan  8 13:45:02 example last message repeated 2 times

To get rid of these, change the priority levels that are logged by NET-SNMP. This can be done by changing the options sent to SNMP.

Look for a file /etc/default/snmpd or /etc/sysconfig/snmpd or similar. There should be a set of SNMP options – probably with an option like this one:

-Ls d

Change this option to be:

-LS5d

This will log everything at level NOTICE or higher (that is, severity level 5 down to severity 0). The severity levels used are those used by syslog; they are described in syslog(3).

This works because the messages being seen are logged at level INFO; by not logging items at that severity level the log entries no longer clutter the syslog files.

However, there is another set of messages that are common to NET-SNMP logs:

Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 9 request
Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 10 request
Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 11 request

This is the result of a bug (Red Hat Bugzilla #474093 – login required) which causes these “errors” when the SNMP diskIOTable is traversed. Red Hat fixed this bug back in September of 2009.

According to this message by Chris Rizzo, Red Hat stated:

The message that you see here is a result of querying for statistics
that are not available on the linux system. Requests 9, 10, and 11 are
defined as:

#define DISKIO_LA1 9
#define DISKIO_LA5 10
#define DISKIO_LA15 11

You can see where these statistics pop up by querying the SNMP diskIOTable using this command:

snmptable -v 2c -c public host diskIOTable

The output will look like this:

SNMP table: UCD-DISKIO-MIB::diskIOTable

 diskIOIndex diskIODevice diskIONRead diskIONWritten diskIOReads diskIOWrites diskIOLA1 diskIOLA5 diskIOLA15 diskIONReadX diskIONWrittenX
           1         ram0           0              0           0            0         ?         ?          ?            0               0
           2         ram1           0              0           0            0         ?         ?          ?            0               0
           3         ram2           0              0           0            0         ?         ?          ?            0               0
           4         ram3           0              0           0            0         ?         ?          ?            0               0
           5         ram4           0              0           0            0         ?         ?          ?            0               0
           6         ram5           0              0           0            0         ?         ?          ?            0               0
           7         ram6           0              0           0            0         ?         ?          ?            0               0
           8         ram7           0              0           0            0         ?         ?          ?            0               0
           9         ram8           0              0           0            0         ?         ?          ?            0               0
          10         ram9           0              0           0            0         ?         ?          ?            0               0
          11        ram10           0              0           0            0         ?         ?          ?            0               0
          12        ram11           0              0           0            0         ?         ?          ?            0               0
          13        ram12           0              0           0            0         ?         ?          ?            0               0
          14        ram13           0              0           0            0         ?         ?          ?            0               0
          15        ram14           0              0           0            0         ?         ?          ?            0               0
          16        ram15           0              0           0            0         ?         ?          ?            0               0
          17        loop0           0              0           0            0         ?         ?          ?            0               0
          18        loop1           0              0           0            0         ?         ?          ?            0               0
          19        loop2           0              0           0            0         ?         ?          ?            0               0
          20        loop3           0              0           0            0         ?         ?          ?            0               0
          21        loop4           0              0           0            0         ?         ?          ?            0               0
          22        loop5           0              0           0            0         ?         ?          ?            0               0
          23        loop6           0              0           0            0         ?         ?          ?            0               0
          24        loop7           0              0           0            0         ?         ?          ?            0               0
          25          sr0           0              0           0            0         ?         ?          ?            0               0
          26          sda  2840214016     1178369536     8946299      2080062         ?         ?          ? 990682692096     18358238720
          27         sda1      598016         208896          82            8         ?         ?          ?       598016          208896
          28         sda2        2048              0           2            0         ?         ?          ?         2048               0
          29         sda3     2286592        4649984         449          332         ?         ?          ?      2286592         4649984
          30         sda5  2836463104     1173473792     8945636      2079527         ?         ?          ? 990678941184     18353342976
          31          sdb  2960873984     1173473792     1940422      2079476         ?         ?          ? 990803352064     18353342976
          32         sdb1      638976              0          83            0         ?         ?          ?       638976               0
          33         sdb2      798720              0         153            0         ?         ?          ?       798720               0
          34         sdb3        6144              0           2            0         ?         ?          ?         6144               0
          35         sdb5  2958672384     1173473792     1940080      2079284         ?         ?          ? 990801150464     18353342976
          36          md0  3051716608     1727252992       93765      1641387         ?         ?          ?   3051716608     14612154880
          37          sdc  3067452416      425118208     1640751      3638614         ?         ?          ? 101851700224    438511782400
          38         sdc1  3067317248      425118208     1640721      3638613         ?         ?          ? 101851565056    438511782400
          39          sdd      307712              0          76            0         ?         ?          ?       307712               0
          40         sdd1      147968              0          37            0         ?         ?          ?       147968               0
          41         sdd2      131072              0          32            0         ?         ?          ?       131072               0

Towards the right side of center, you can see the metrics diskIOLA1, diskIOLA5, diskIOLA15; these are unsupported on Linux (as marked by the ? in each column). These are the 1 minute average disk load (as a percentage), the 5 minute average disk load, and the 15 minute average disk load respectively.

The three have SNMP OIDs of .1.3.6.1.4.1.2021.13.15.1.1.9 and .1.3.6.1.4.1.2021.13.15.1.1.10 and .1.3.6.1.4.1.2021.13.15.1.1.11 respectively – thus, the logged complaint of not knowing how to handle request 9 (or 10 or 11).

Without changing the code, there doesn’t seem to be any way to eradicate this message if you are querying the diskIOTable. Red Hat fixed the bug, perhaps others will? The bug remains on Ubuntu Lucid Lynx, unfortunately.

Instant 10-20% boost in disk performance: the “noatime” option

Many people already know about this option, but it is worth mentioning again. However, a description of what atime is is in order.

The atime is one of the three times associated with a UNIX file: the three are: the ctime (or change time – that is, when the inode was last changed); the mtime (or modified time – that is, when the file is changed); and lastly, the atime (or access time).

It is the continual changes to the atime that cause so much grief. Compared to the mtime and the ctime, the atime changes with alarming frequency. Every single time a file is accessed, the atime is updated to match the current time – whether the file is opened, read, written, or accessed in any manner whatsoever.

There was a Linux kernel mailing list discussion thread that gave rise to some interesting quotes on this topic.

The discussion became quite heated when Ingo Molnar suggested that atime should be the kernel default. He had this to say:

Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_.

and:

It’s also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: ‘For every file that is read from the disk, lets do a … write to the disk! And, for every file that is already cached and which we read from the cache … do a write to the disk!’

and later, this:

Measurements show that noatime helps 20-30% on regular desktop
workloads, easily 50% for kernel builds and much more than that (in
excess of 100%) for file-read-intense workloads.

and, this:

Give me a Linux desktop anywhere and i can
tell you whether it has atimes on or off, just by clicking around and
using apps (without looking at the mount options). That’s how i notice
it that i forgot to turn off atime on any newly installed system – the
system has weird desktop lags and unnecessary disk trashing.

Linus had this to say:

yeah, it’s really ugly. But otherwise i’ve got no real complaint about
ext3 – with the obligatory qualification that “noatime,nodiratime” in
/etc/fstab is a must. This speeds up things very visibly – especially
when lots of files are accessed. It’s kind of weird that every Linux
desktop and server is hurt by a noticeable IO performance slowdown due
to the constant atime updates, while there’s just two real users of it:
tmpwatch [which can be configured to use ctime so it’s not a big issue]
and some backup tools. (Ok, and mail-notify too i guess.) Out of tens of
thousands of applications. So for most file workloads we give Windows a
20%-30% performance edge, for almost nothing. (for RAM-starved kernel
builds the performance difference between atime and noatime+nodiratime
setups is more on the order of 40%)

Changing a file system to run without atime is simple; use this command on a mounted filesystem:

# mount -o remount,noatime /disk

Don’t forget to change the /etc/fstab to match this.

The Current State of Window Maker

I’ve always liked the Window Maker window manager. However, the current state of Window Maker is in some turmoil.

Development on the original Window Maker window manager has ceased, and new development has been taken up by wmaker-crm (a fork). Nightly builds of wmaker-crm had been available for Debian from a user-created repository, but no new builds have been put up since 29 April 2011 – and the repository information hasn’t been rebuilt since 26 May 2011. Building a Debian package seems to be problematical in any case. According to this mailing list thread, Andreas Metzler and Martin Dietze are responsible for changes therein, but the changelog hasn’t reflected any changes in wmaker-crm.

It was recommended to the Debian Window Maker package maintainer, John H. Robinson IV, to use the wmaker-crm sources for the package; he was receptive but nothing has happened since. As of 6 July 2011, he stepped down as maintainer of the package, leaving the Debian package orphaned. This event did not go unnoticed; a thread was taken up on the wmaker-crm development list.

The person behind wmaker-crm, Carlos R. Mafra, also created a git repository for numerous Window Maker dockapps that are no longer maintained.

The standard Window Maker display manager, the WINGs Display Manager, apparently is much less desired than Window Maker itself: according to the popularity contest statistics, at a peak during January 2011, 2000+ people had Window Maker installed while less than 200 currently have WDM installed.

Statistics about the installed base (and user base) of Window Maker and other packages can be seen over at popcon.debian.org; these statistics come from people who have installed the popularity-contest package.

Over at ArchLinux, their wiki has an excellent write-up on Window Maker (including basic technical details and information on wmaker-crm) which certainly makes one think that Window Maker is vibrant in that community. Both Window Maker and wmaker-crm are packaged (as windowmaker and windowmaker-crm-git respectively) and available to ArchLinux users from the Extra Repository as well.

If you want to see Window Maker in action (more or less – it does show a lot of Tux Commander too…) you can check out this video showing Window Maker on a Duron 850MHz system with 256Mb.

I may post a short video of my own Window Maker desktop; having forced myself to run with Window Maker as my default desktop has made me a complete convert – and helped me to force myself to research and resolve problems with making Window Maker a default desktop. Recently, I wrote about just what it took to make Window Maker fully capable and up-to-date.

If GNOME or KDE are finally just too much – and XFCE isn’t quite what you want – try Window Maker instead.

 

Why I use Window Maker

I keep coming back to Window Maker as a desktop environment. Why? Sometimes I ask myself the same question. However, there are a number of reasons.

Simplicity. There’s not a ton of things happening on the Window Maker screen; basically, there is the Dock and the Clip and minimized application windows. That’s it. This simplicity also translates into lower memory usage.

Light-weight. This is a biggie for me. First time I truly used Window Maker in any depth was on OpenBSD installed onto a Macintosh Quadra 800. It took a while to build, but it built and was comfortable – and in that environment, GNOME or KDE was out of the question.

Unique. This doesn’t sound like a positive aspect – but to me, I love learning a new environment. After a while, the other environments can blend together. The dock is also much better than any that have followed it; others like the MacOS X dock are too small and don’t work well in a user interface. Window Maker is very clean and simple and easy to use.

Since it is included as an available package on almost every Linux installation, what does it take to make a good Window Maker desktop? With Ubuntu, KDE can be installed by installing the kubuntu-desktop; XFCE can be installed by installing the xubuntu-desktop. There is no equivalent Window Maker desktop unfortunately.

For Ubuntu, you might want to install the following packages using Synaptic or apt-get:

The menu application makes sure that all of your applications that are installed with Ubuntu are also available in the Window Maker menus.

There are other packages you may like; search for packages that end in “.app” or begin with “wm” for starters.

For wireless management, you’ll want to stop the network-manager and then install wicd instead. This is because network-manager requires a KDE/GNOME style desktop. Do this with the following:

sudo service network-manager stop
sudo apt-get install wicd wicd-gtk
sudo service wicd start
wicd-gtk &

When you do this, you should be able to configure the wireless connections as you see necessary. Note that there is no graphic controls for VPNs at this time, but you can control them from the command line.

So what do I have in my dock? Here is a list:

  • firefox (application)
  • wmbattery
  • wicd (application)
  • wmdiskmon
  • wmclockmon
  • wmcpuload
  • wmwave
  • wmmaiload
  • wmcalclock
  • wmdrawer

All of these are part of the standard Ubuntu repositories; unfortunately, wmbatteries is not. However, you can get that dock applet (or “dockapp”) and more from dockapps.org.

There are a lot of resources for learning Window Maker, although some are dated; these are all good places to go:

Hopefully, this won’t be too many resources; most are quick overviews. Don’t be afraid to try out Window Maker today!

/etc/shadow format

Recently, I found myself trying to understand what was going on in the shadow file. For one thing, the “password” field in Ubuntu Lucid Lynx is much longer than it is in Red Hat Enterprise Linux 5. Then there were the exclamation points in front of some password entries.

Originally, there was no /etc/shadow file – the passwords and salts were contained with other user information in /etc/passwd. The problem was that /etc/passwd was readable by anyone, and that attacks became more common and more powerful. It was necessary to move the password into a separate file that would only be readable by root.

Here is an example of a shadow file line:

ddouthitt:$6$QR3drPrQ$JLolPKyiVuXvea1F2IpbPx9F9PEV0s/IGcNCpm6ZrBA6AFDvwHPQG7EsTQHuUqxfCvlsuKRb.O7w5RLPyj8nS/:15119:0:99999:7:::

Each field is separated by a colon (:). There are a number of fields:

  1. User name. This must match up with a user name in /etc/passwd; it is the key.
  2. Password. This field is a hash of the password (not the actual password itself) combined with an id (specifying which form of encryption was used) and with a salt.
  3. Last changed date. This is the day that the password was last set (in days since January 1, 1970).
  4. Minimum days. This is the minimum number of days that a password must be active before a password can be changed again. A user’s password cannot be changed by them until this number of days has elapsed.
  5. Maximum days. This is the maximum number of days for a password to be valid. Once the maximum number of days has elapsed, the user is forced to change their password.
  6. Warn days. Before a password expires, warn the user this many days ahead of time.
  7. Inactive days. Once a password has expired, it will be disabled after this many days.
  8. Expire date. On this date (in days since January 1, 1970), the account is disabled and the login can no longer be used.

Rather than trying to compute the date from the number of days since 1970, use the chage command:

$ chage -l ddouthitt
Last password change					: May 25, 2011
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

This also explains what each number means, and incorporates any defaults.

The password is more complicated. The actual password is not stored anywhere; the stored value in the /etc/shadow file is a hash generated from the password.

Originally, the password hash was just the hash itself; later, a two-character salt was added at the beginning.

The password hash today comes in three parts, separated by dollar signs ($):

  • Id. This identifies the encryption hash method used. A value of 1 denotes MD5; 2 or 2a is Blowfish; 3 is NT Hash; 5 is SHA-256; and 6 is SHA-512.
  • Salt. This is used by the encryption algorithms, and could be up to 16 characters.
  • Hash. The actual “password” (or hash) is last. MD5 uses 22 characters, SHA-256 uses 43, and SHA-512 uses 86.

Picking apart the example password entry above shows this:

  • Id = 6 (SHA-512)
  • Salt = QR3drPrQ
  • Hash = JLolPKyiVuXvea1F2IpbPx9F9PEV0s/IGcNCpm6ZrBA6AFDvwHPQG7EsTQHuUqxfCvlsuKRb.O7w5RLPyj8nS/

Lastly, the password entry could be missing entirely (which means no password) or it could have an invalid entry (locked account).

Traditionally, locking was done with the asterisk (*) by convention or by the word LOCKED. However, this has been extended – by convention – to a variety of new suggestions. System logins that never had a password use a double exclamation mark (!!) and accounts that have been locked have the valid password entry invalidated by a exclamation mark prefix. Using the prefix in this way allows the password to be reinstated as it was.

FreeBSD uses a slightly different setup, with a file called /etc/master.passwd, an extended /etc/passwd file with password aging included, and support for an extended DES format (which starts with an underline). Extended DES is more secure than the original DES, but is no longer used in favor of the new modular format described above. FreeBSD standardizes on MD5 – like Red Hat, and unlike Ubuntu.

By the way – that’s not my real password – so don’t look at it.

Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTCONTROL=
export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTIGNORE=
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTFILESIZE
typeset -r HISTIGNORE
typeset -r HISTSIZE
typeset -r HISTTIMEFORMAT

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

Linux Kernel sync() Bug

There is a bug in the Linux kernel (up to 2.6.37) that can result in a sync() command taking many minutes (instead of several seconds).

The exact nature of the bug is unclear, but it shows up in the use of dpkg – since dpkg now uses sync() instead of fsync(). The result is updating an Ubuntu or Debian system can hang when dpkg goes to synchronize the file system.

The symptom also shows up by using the sync command directly.

The dpkg command was updated in 1.16.0 to include a force-unsafe-io option which re-enables the previous behavior which bypasses this bug. This version is not yet available in Ubuntu Lucid Lynx (10.04) but should be showing up in 10.04 LTS sooner or later. The option can be added to /etc/dpkg/dpkg.cfg or to a file in /etc/dpkg/dpkg.cfg.d to make it a default setting.

There was also a suggestion by Theodore T’so that upgrading to dpkg 1.15.8.7 would be sufficient to work around this kernel bug even without the force-unsafe-io option.

It is possible to upgrade to dpkg 1.16.0 on Lucid, but it requires pulling in the Natty Narwhal repositories and setting apt tp prefer the Lucid repositories, then pulling in the Natty version of dpkg specifically.

Suggestions on how to fix the problem (aside from replacing the kernel) now include:

None of these answers are definitive and the search continues. Ubuntu bug #624877 is on this topic, as is Linux kernel bug #15426.

Correction: bug #15426 is a Linux kernel bug report, not a Debian bug report. We apologize for the error.

Current State of Berkeley Automounter (amd)

Well, this is really sad. I have always believed in using the Berkeley Automounter (amd) over the automounters otherwise available, given the power and portability of amd. However, in investigating automounters for Linux, it seems that amd has been stagnating for some time.

Apparently, this has happened before. The current maintainer, Erez Zadok, found that amd was stagnating and took over development many years ago. He wrote a book on Linux NFS and Automounter, which is still a fantastic book.

Now the git repository for amd shows that the last development on amd happened around November of 2010 – and no development in any other month of 2010. Version 6.2rc1 was tagged four months ago.

It would be a true shame if amd were to die from stagnation; it has strong capabilities that aren’t present elsewhere.