Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTIGNORE
typeset -r HISTSIZE

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

12 thoughts on “Logging every shell command”

  1. “For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.”

    Bash 4 has a patch that enables bash to log all shell activity to syslog, and in 4.1 it is built in. I don’t see that as making logging of commands more difficult. And since 4.1 came out in 2010 and 4.0 and the logging patch in 2009, it should have been noted by the author.

    1. The problem is that neither Red Hat nor Ubuntu package the shells with their auditing capabilities; if you look at bash or at ksh, neither will support this.

      Also, if you read the entire article, you’ll see that I mention both bash and ksh auditing features.

      1. RHEL 5.4 introduced ksh with the SHOPT_AUDIT flag enabled. You have to create /etc/ksh_audit yourself though. I recommend pointing it to an off-host syslog system using a destination like:
        /dev/udp/loghost/514 0

        (log effective UID 0, send to system named loghost, 514/udp).

        We have been in discussion with Red Hat over bash auditing features, but require it to be similar in output and style to ksh-93. We did allow them to add a field to denote the real UID of the caller to save us the trouble of digging through wtmp and sudo logs to see who had that tty at that time.

  2. This may be a little late and a little off-topic but I am searching for a shell or shell extension, that rivals or at least imitates PowerBroker, on the open source side. The shell, history of which at the least, that I am trying to protect, is root’s. There have been incidences among the sysadmin team of 20+ for which nobody wants to take the responsibility (and for a good reason, such as job security)
    These people are logging in as root from a trusted server, without a password or anything else and get root privileges right at the point where they signed in. SInce the root password on the trusted server is known by everyone, it makes the forensic analysis impossible or damn near to impossible.
    What I want this shell to do is to send all keystrokes and everything that passes thru the terminal buffer to a remote log server, manager of which will be someone from outside the sysadmin team. I know it is a tall order and may not be easy to implement but I am okay, if every command issued from the any shell process running on the system to be logged to the remote log server, as typed with the typos in it and everything. This way, I can give sysadmins their personal logins, with the default shell being this auditable shell and they can do what the hell ever they want, knowing well that “i didn’t do nothing” will not be a defense if things go south on them.
    Is there such a product ? I found the traces of something called EAS (enterprise audit shell) in my google search but looks like it was on HP-UX and it is now abandoned.
    Any help is greatly appreciated.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: