Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTIGNORE
typeset -r HISTSIZE

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

sudo – bane or benefit?

The commonly requested tool sudo is often maligned by younger administrators and venerated by older administrators.

But how much security does sudo actually provide? How useful is it really?

Running sudo gives you some benefits – so one hears:

  • Using sudo records everything someone does as root.
  • Using sudo prevents a user from running continually as root, preventing errors.
  • Using sudo prevents a user from having to know the root password.
  • Using sudo prevents a user from executing anything they are not permitted to.

The problem is that sudo does none of these things very well.

Sudo does indeed record everything one does as root. How does this increase our security? How does this prevent a hacker from compromising a system?

The answer is it doesn’t; recording what someone is doing through this method provides no security at all. A hacker will not use sudo; a regular admin who is using sudo can just use sudo ksh to get a shell that will do what they want. In this way, sudo is merely security through obscurity: if you don’t know how to bypass it, you’ll be recorded. Sudo is like a security camera, watching what you are doing.

Certainly, if you are using sudo, you are not continually running as root – and this is indeed a benefit – and perhaps the primary one. However, there are many things that must be done that cannot be done with sudo. Changing into a restricted directory is just one of them.

It is also true that a user using sudo does not have to know the root password; this also can be a benefit. However, ssh should already be in use and can serve this purpose even better: using an individual’s key, that individual can be granted rights to the root account without knowing the root password – and revoked at will.

The last item is the most dangerous: that using sudo, users can be completely limited in what they can do. The best example of how this can be abused is the attempt of some to prevent the running of a shell by a sudo user. No matter what the sudo file says, one can always do this:

$ cd /bin
$ cp ksh ~/bugbear
$ chmod 700 ~/bugbear
$ sudo ~/bugbear

Instant root shell for anyone with sudo access, regardless of the restrictions in the sudo file. This can be done for any program, as long as the user can read the file it can be changed to a different name and run elsewhere – and if the file can’t be read it can’t be run.

So what are the answers? There are several that, when taken together, will do what sudo wants to do but better:

  • Use SSH. Using ssh one can limit a user to a specific command, and prevent the user from knowing the root password.
  • Use rksh. This may be too restrictive, but can permit users to execute only certain commands appointed by the administrator.
  • Use a chroot jail (or better yet, a BSD jail). Again, this may be too restrictive for most, but will permit a user to only do what is allowed – on a much more restrictive basis than rksh.
  • Use an auditing shell. Ksh93 provides this capability – though there may be a reason that this specific capability is not in the standard ksh supported on today’s system. Using an auditing shell, every command from every user, including system administrators, is logged and retained – possibly to another system on the network. Here is a good article on ksh93 auditing. I would posit, however, that if you can’t trust your system administrators then they shouldn’t be your system administrators: often auditing like this merely provides “a throat to choke.”

In particular, if you utilize the ssh public key encryption capabilities to their fullest with a logging ksh93 shell – and with a “captive” menu perhaps – you can provide the capabilities of sudo without the drawbacks. If you do use sudo, realize how serious its shortcomings are and be aware of them to increase your security elsewhere.