Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTCONTROL=
export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTIGNORE=
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTFILESIZE
typeset -r HISTIGNORE
typeset -r HISTSIZE
typeset -r HISTTIMEFORMAT

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)