Configuring the Bash Shell (and Korn Shell) to Permanently Use vi-mode Editing

The GNU bash shell, as one might expect, uses Emacs key bindings to perform history and editing in the shell. Vi key bindings are available, and can be set with this standard command (just like in ksh):

set -o vi

My fingers have gotten this sequence of letters ingrained in them so I don’t even have to think about it – and I type it almost automatically after login.

However, there is a way to set up ksh to automatically set editing to vi mode. If the VISUAL variable is set to a value like vi, then the editing mode will be set to vi mode automatically. Likewise, if the VISUAL variable is set to emacs, then emacs mode is used for editing.

If the VISUAL variable is not set, then the EDITOR variable is used.

In later versions of the AT&T Korn shell (such as 20100202) the recognized values have been expanded: the pattern *[Vv][Ii]* is recognized as a request for vi mode; the pattern *gmacs* results in gmacs mode; and the pattern *macs* results in emacs mode. (The only difference between emacs and gmacs mode is the handling of the ^T character.)

This means that using vim or gvim will now trigger vi mode – indeed, using almost any vi clone editor such as nvi or elvis will work with this method. This also means that the full path can be used, although this may have been true previously.

In bash however, this use of EDITOR and VISUAL is not available. However, since bash uses GNU readline, we can set up readline to use vi mode by default – and thus also affect all programs that use GNU readline besides.

Edit (or create) a file in your home directory called .inputrc and add this line to it:

set editing-mode vi

After this, any time you log in – or use anything else that uses GNU readline (such as CLISP for example) – you’ll automatically have vi mode editing. I can finally rest my fingers…

Why doesn’t my /bin/sh script run under Ubuntu?

This is a very interesting question – and the resolution is simple. In Ubuntu 6.10 (known as Edgy Eft) the decision was made to replace the Bourne Again Shell (bash) with the Debian Almquist Shell (or dash) as /bin/sh in Ubuntu. There was considerable uproar in Ubuntu brainstorm (community ideas) and in Ubuntu bug reports, as using dash instead of the original bash caused numerous scripts to break.

In particular, the entire reasoning given for this change was efficiency: dash is more efficient (i.e., faster) than bash. According to the explanatory document created by the Ubuntu developer team, Debian has required scripts to work on POSIX-compliant shells for some time (even pre-dating the Ubuntu project). Thus, any scripts that broke were, in essence, not “following directions” and deserved what they got.

To undo this change by the Ubuntu team, one can do this:

sudo dpkg-reconfigure dash

When this command executes, specify that you do not want dash to act as /bin/sh. This will make every script that runs /bin/sh run bash as has traditionally been the case.

You can also make your scripts run /bin/bash instead of /bin/sh; this provides all of the bash capabilities without any concern as to whether /bin/sh will change again.

Making the boot process faster is a laudable goal, but like the removal of OSS from the kernel, it caused a lot of problems for users.

In both cases, it appears that the Ubuntu team is more focused on doing the technologically “right” thing rather than providing a stable and reliable platform. Unfortunately, this means that you cannot rely on Ubuntu to stay reliable – at least from one version to the next. The response of Ubuntu to such system failures has always been that they are doing the “right” thing and the problem must be fixed by someone else (i.e., it’s not Ubuntu’s problem).

Users – many of them system administrators – take the brunt of this: they don’t care whose fault it is, nor do they care whether the boot process is faster or whether the Linux sound environment is “cleaner”; they care about the stability of their systems. A system that boots faster doesn’t matter if it crashes during the boot process because of a broken script.

If the focus of Ubuntu were to provide a stable and unchanging environment, then their decisions would be different – and would result in an improved customer experience.

Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

• rc – a small shell used by Plan 9
• scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
*
$ echo '$AB'
$AB

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTCONTROL=
export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTIGNORE=
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTFILESIZE
typeset -r HISTIGNORE
typeset -r HISTSIZE
typeset -r HISTTIMEFORMAT

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

Why I Use Korn Shell Everywhere

The first thing I do when I log into a system, including Solaris, HP-UX, FreeBSD, and Linux is exec ksh. Whatever for?

Consider this fact: the root shell on FreeBSD defaults to C shell; HP-UX defaults to the POSIX shell (without history); Linux almost everywhere defaults to bash. All of these shells are different in various ways. It is possible you might log into three separate machines and get three separate shells with three different ways of handling things.

Using Korn Shell means that all of these systems will be standardized on one shell, and every system will act the same when you interact with it. There will be no surprises – and surprises at the root command line often translate into disastrous errors.

On HP-UX, using ksh has the additional benefit of enabling history for root – although the security risks of this make this a dangerous benefit: best to erase history after you log out and to make sure that history is independent for every root shell.

What makes this possible is that the Korn Shell is available virtually everywhere, including FreeBSD, Linux, Solaris, and HP-UX – whereas other shells are not (which includes C shell, Bourne shell, and bash).

Argument list too long?

Well, what now? We got the dreaded “argument list too long” error. What to do?

To explain the problem, let’s consider what the shell does (we won’t get into system calls, to make things simple). The shell (Bourne compatible shells, actually) will first scan the line typed in. One of the first things to do is to expand file globs. For example, the command:

ls a*

will be translated by the shell into:

ls andrew apple alex allen alfred almonzo august axel albert

(and so forth). If the expansion expands beyond the system limitations, then it cannot be processed and the error “argument list too long” results.

The limit could be the number of arguments or the number of characters; however, the fix is the same. The key is to get the list (usually files) out of the argument list and into something else – such as stdout. Once the overlong list is being sent via stdout, the command xargs can be used to place all of the items on the command line, observing all relevant limits as it goes (and as efficient as possible to boot).

There are a variety of quick answers which will all fail, because the argument list would remain too long:

ls a* >savefile.txt
for i in a* ; echo $i ; done
echo a*

However, all is not lost: there are a number of ways to get a long list of files into stdout; the most common is to use find:

find . -name "CACHE.DAT" | xargs ls -ld

This may not be possible, if the desired list of files doesn’t fit neatly into a find command.

Another possibility, related to the previous example, would be this:

ls -1 | sed '/^a/!d' | xargs ls -ld

Yet another possibility might be to use a language like Perl; since it does not scan and process the same way, it would work without limitations:

perl -e 'opendir(DIR, "."); @all = grep /^a/, readdir DIR; closedir DIR; print "@all\n";' | xargs ls -ld

I would only recommend using Perl or other such if you are quick and snappy with your knowledge of the language; otherwise, such a long line will have you looking things up repeatedly.

If the arguments are coming from a file, then things become even easier; instead of something like this:

ls -ld $(cat mylist)

You can simply use:

cat mylist | xargs ls -ld

Of course, any binary command can be used with xargs, not just ls.

Shell history

The Korn shell (as well as bash and the POSIX shell) has a history mechanism that can be very useful. This history also can be used with line-editing in order to edit the command before entry – and either vi editing or emacs editing can be used.

There are two environment variables that control the history:

• HISTFILE (stores commands; default $HOME/.sh_history)
• HISTSIZE (number of commands to keep)

Ksh-93 instroduces two new variables to go with these:

• HISTEDIT (replaces FCEDIT)
• HISTCMD (number representing current command)

Neither of those are of much if you use command-line editing.

The location of the history file (contained in HISTFILE) is of some importance. When ksh is used in an environment with NFS-mounted home directories, then the history file will be stored on the NFS volume. This has been known to cause problems in some environments (HP-UX, for one). In these cases, the HISTFILE can be changed to a local directory such as /tmp/.$$_hist_file.

This also brings up another thing of importance: this history file is read by all of the shell logins of that user. So if multiple people are logged into the same account (root for instance), then the same history file is used. This can be confusing, so it may be useful to change the HISTFILE setting for that session to avoid interference.

This history file is also preserved across logins – so some root sessions will disable the history mechanism entirely, preventing others from reading the history file. However, history is quite useful, so a compromise would be to limit the number of commands kept (by changing the HISTSIZE variable).

Line-editing is what makes command history so eminently useful. The mode used (vi or emacs) is chosen based on the setting of the EDITOR and VISUAL environment variables (if set to vi, emacs, or gmacs). Alternately, the option may be chosen directly by using the command:

set -o vi

or the command to choose emacs:

set -o emacs

Once this is done, then standard editing commands can be used. The current line is treated like a single-line window into the shell history file; so going up goes up a line (or command) in the history, and going down goes down a line (command) in the history. In emacs this translates to ^P (previous line) and ^N (next line); in vi it translates to the commands k (up) and j (down). The only special commands are file-completion commands. In both vi and emacs modes, they are similar.

• ESC-\ (filename completion – or as much as possible; emacs uses META-ESC and vi has an alternate ESC-ESC)
• ESC-= (outputs list of possible completions)
• ESC-* (completes file with all matches and enters editing mode)
• ESC-_ (enter nth word from end of last command – e.g., first from end, second from end, third from end – default is first from end)

In emacs mode, META translates into ESC normally.

Here is an example of a list of possible completions (from ESC-= entered at end of first line):

$ ls -l s
1) sec-2.4.1.tar.gz
2) sec-2.4.1/
$ ls -l s

The command is presented a second time for editing at the end of the list. If ESC-* is pressed the command line becomes:

# ls -l sec-2.4.1 sec-2.4.1.tar.gz

Alternately, if ESC-\ is pressed, the command becomes:

# ls -l sec-2.4.1

It was the lack of understanding of the shell history and command-line editing that held me back from adopting the Korn shell over the C shell for a number of years (I used csh interactively, but wrote scripts in ksh).  I made the switch and never went back.