Configuring the Bash Shell (and Korn Shell) to Permanently Use vi-mode Editing

The GNU bash shell, as one might expect, uses Emacs key bindings to perform history and editing in the shell. Vi key bindings are available, and can be set with this standard command (just like in ksh):

set -o vi

My fingers have gotten this sequence of letters ingrained in them so I don’t even have to think about it – and I type it almost automatically after login.

However, there is a way to set up ksh to automatically set editing to vi mode. If the VISUAL variable is set to a value like vi, then the editing mode will be set to vi mode automatically. Likewise, if the VISUAL variable is set to emacs, then emacs mode is used for editing.

If the VISUAL variable is not set, then the EDITOR variable is used.

In later versions of the AT&T Korn shell (such as 20100202) the recognized values have been expanded: the pattern *[Vv][Ii]* is recognized as a request for vi mode; the pattern *gmacs* results in gmacs mode; and the pattern *macs* results in emacs mode. (The only difference between emacs and gmacs mode is the handling of the ^T character.)

This means that using vim or gvim will now trigger vi mode – indeed, using almost any vi clone editor such as nvi or elvis will work with this method. This also means that the full path can be used, although this may have been true previously.

In bash however, this use of EDITOR and VISUAL is not available. However, since bash uses GNU readline, we can set up readline to use vi mode by default – and thus also affect all programs that use GNU readline besides.

Edit (or create) a file in your home directory called .inputrc and add this line to it:

set editing-mode vi

After this, any time you log in – or use anything else that uses GNU readline (such as CLISP for example) – you’ll automatically have vi mode editing. I can finally rest my fingers…

Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

• rc – a small shell used by Plan 9
• scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
*
$ echo '$AB'
$AB

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTCONTROL=
export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTIGNORE=
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTFILESIZE
typeset -r HISTIGNORE
typeset -r HISTSIZE
typeset -r HISTTIMEFORMAT

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

5 Programs That Should be in the Base Install

There are a number of programs that never seem to be installed with the base system, but should be. In this day and age of click-to-install, these programs will often require an additional install – I maintain that this should not be.

Most of these will be relevant to Linux, but the programs will often be missing on other commercial UNIXes also.

• Ruby. This is the first that comes to mind. I have been installing ruby onto systems since 1.46 – and ruby is still a fantastic scripting language, and one of the best implementations of object-orientated programming since Smalltalk.
• m4. I recently wrote about m4, and thought it was already installed on my Ubuntu Karmic system – not so. I used it to create a template for the APT sources.list file.
• ssh. This should be installed everywhere automatically, and not as an add-on. For many UNIX systems, ssh is an add-on product that must be selected or compiled from source.
• rsync. Rsync is a fabulous way to copy files across the network while minimizing traffic – even though it is not designed to be a fast way.
• ksh. This will surprise most commercial UNIX administrators. However, Linux does not come with ksh installed – and the emulation by GNU bash is weak. Now you can install either AT&T ksh-93 (the newest version!) or the old standby, pdksh (which is close to ksh-88).

OpenVMS is a different animal – and some of the things that should be installed by default would be perl, ruby, java, SMH, and ssh. I’m not sure if perl or ssh is installed by default, but they should be. OpenVMS should also support compliant NFS v3 and v4 support out of the box – without making it difficult to connect to other NFS servers.

What programs do you think should be in the base install?

Why I Use Korn Shell Everywhere

The first thing I do when I log into a system, including Solaris, HP-UX, FreeBSD, and Linux is exec ksh. Whatever for?

Consider this fact: the root shell on FreeBSD defaults to C shell; HP-UX defaults to the POSIX shell (without history); Linux almost everywhere defaults to bash. All of these shells are different in various ways. It is possible you might log into three separate machines and get three separate shells with three different ways of handling things.

Using Korn Shell means that all of these systems will be standardized on one shell, and every system will act the same when you interact with it. There will be no surprises – and surprises at the root command line often translate into disastrous errors.

On HP-UX, using ksh has the additional benefit of enabling history for root – although the security risks of this make this a dangerous benefit: best to erase history after you log out and to make sure that history is independent for every root shell.

What makes this possible is that the Korn Shell is available virtually everywhere, including FreeBSD, Linux, Solaris, and HP-UX – whereas other shells are not (which includes C shell, Bourne shell, and bash).

Avoiding catastrophe!

All UNIX administrators, if they’ve not been bit by a typo like this, have heard of it (or something like it) happening:

cd /
rm -rf foo *

Notice the extra space, and the current working directory. Instead of deleting directories beginning with the name “foo” it will delete everything!

How can this be avoided? There are number of practices that I have ingrained which have saved me from such disasters.

Before executing dangerous commands, check which files will be affected… For example, you may find a need for a command such as:

find . -size +10000 | xargs rm -f

However, before doing this, do a command like this one:

find . -size +10000 | xargs ls -ld

This will save you from potentially removing important files.

Instead of executing disastrous commands automatically, use a computed script. For example, the previously mentioned example:

find . -size +10000 | xargs rm -f

This example could be replaced by the following instead:

find . -size +10000 | xargs -i{} echo rm -f {} > scriptfile.sh

Then the resulting script can be scanned for files that one wants to keep, and the script edited to reflect the desired actions. Once the script is the way you want it, it can be run:

sh ./scriptfile.sh

This method is particularly good for times when you want to delete or modify hundreds of files, but want to prevent a few from being affected.

Notice where you are! Perhaps you think you are in a particular directory, but you aren’t: double-check. Take a moment and do a pwd command and check.

If it affects the entire system, make sure you are on the right system! You don’t want to shut down a system only to realize it was the wrong one! Do a uname -n just to check – every time. Don’t use the system prompt; if you rely on that you may realize too late that the system prompt is just a string (and it could lie to you!).

If you manage many diverse systems, double-check the syntax of the command first. If you do the command in the wrong way who knows what could happen? When I managed four or five types of UNIX, I did a man 1m shutdown before every shutdown. Most were similar – some were not (such as Unixware and Solaris for example).

Notice dangerous commands and stop and double-check them! When doing an rm -rf I always double-check the command before pressing enter – even just pausing to look – rather than just typing through. This goes just as much for the find command piped into a disastrous command (such as rm or mv).

Notice dangerous commands and pause before pressing enter. I always pause to check the command – even just pausing, knowing that this is the moment – the last chance to back out. With that knowledge in the back of your mind, it will cause you to double-check if you haven’t already.

Argument list too long?

Well, what now? We got the dreaded “argument list too long” error. What to do?

To explain the problem, let’s consider what the shell does (we won’t get into system calls, to make things simple). The shell (Bourne compatible shells, actually) will first scan the line typed in. One of the first things to do is to expand file globs. For example, the command:

ls a*

will be translated by the shell into:

ls andrew apple alex allen alfred almonzo august axel albert

(and so forth). If the expansion expands beyond the system limitations, then it cannot be processed and the error “argument list too long” results.

The limit could be the number of arguments or the number of characters; however, the fix is the same. The key is to get the list (usually files) out of the argument list and into something else – such as stdout. Once the overlong list is being sent via stdout, the command xargs can be used to place all of the items on the command line, observing all relevant limits as it goes (and as efficient as possible to boot).

There are a variety of quick answers which will all fail, because the argument list would remain too long:

ls a* >savefile.txt
for i in a* ; echo $i ; done
echo a*

However, all is not lost: there are a number of ways to get a long list of files into stdout; the most common is to use find:

find . -name "CACHE.DAT" | xargs ls -ld

This may not be possible, if the desired list of files doesn’t fit neatly into a find command.

Another possibility, related to the previous example, would be this:

ls -1 | sed '/^a/!d' | xargs ls -ld

Yet another possibility might be to use a language like Perl; since it does not scan and process the same way, it would work without limitations:

perl -e 'opendir(DIR, "."); @all = grep /^a/, readdir DIR; closedir DIR; print "@all\n";' | xargs ls -ld

I would only recommend using Perl or other such if you are quick and snappy with your knowledge of the language; otherwise, such a long line will have you looking things up repeatedly.

If the arguments are coming from a file, then things become even easier; instead of something like this:

ls -ld $(cat mylist)

You can simply use:

cat mylist | xargs ls -ld

Of course, any binary command can be used with xargs, not just ls.

Shell history

The Korn shell (as well as bash and the POSIX shell) has a history mechanism that can be very useful. This history also can be used with line-editing in order to edit the command before entry – and either vi editing or emacs editing can be used.

There are two environment variables that control the history:

• HISTFILE (stores commands; default $HOME/.sh_history)
• HISTSIZE (number of commands to keep)

Ksh-93 instroduces two new variables to go with these:

• HISTEDIT (replaces FCEDIT)
• HISTCMD (number representing current command)

Neither of those are of much if you use command-line editing.

The location of the history file (contained in HISTFILE) is of some importance. When ksh is used in an environment with NFS-mounted home directories, then the history file will be stored on the NFS volume. This has been known to cause problems in some environments (HP-UX, for one). In these cases, the HISTFILE can be changed to a local directory such as /tmp/.$$_hist_file.

This also brings up another thing of importance: this history file is read by all of the shell logins of that user. So if multiple people are logged into the same account (root for instance), then the same history file is used. This can be confusing, so it may be useful to change the HISTFILE setting for that session to avoid interference.

This history file is also preserved across logins – so some root sessions will disable the history mechanism entirely, preventing others from reading the history file. However, history is quite useful, so a compromise would be to limit the number of commands kept (by changing the HISTSIZE variable).

Line-editing is what makes command history so eminently useful. The mode used (vi or emacs) is chosen based on the setting of the EDITOR and VISUAL environment variables (if set to vi, emacs, or gmacs). Alternately, the option may be chosen directly by using the command:

set -o vi

or the command to choose emacs:

set -o emacs

Once this is done, then standard editing commands can be used. The current line is treated like a single-line window into the shell history file; so going up goes up a line (or command) in the history, and going down goes down a line (command) in the history. In emacs this translates to ^P (previous line) and ^N (next line); in vi it translates to the commands k (up) and j (down). The only special commands are file-completion commands. In both vi and emacs modes, they are similar.

• ESC-\ (filename completion – or as much as possible; emacs uses META-ESC and vi has an alternate ESC-ESC)
• ESC-= (outputs list of possible completions)
• ESC-* (completes file with all matches and enters editing mode)
• ESC-_ (enter nth word from end of last command – e.g., first from end, second from end, third from end – default is first from end)

In emacs mode, META translates into ESC normally.

Here is an example of a list of possible completions (from ESC-= entered at end of first line):

$ ls -l s
1) sec-2.4.1.tar.gz
2) sec-2.4.1/
$ ls -l s

The command is presented a second time for editing at the end of the list. If ESC-* is pressed the command line becomes:

# ls -l sec-2.4.1 sec-2.4.1.tar.gz

Alternately, if ESC-\ is pressed, the command becomes:

# ls -l sec-2.4.1

It was the lack of understanding of the shell history and command-line editing that held me back from adopting the Korn shell over the C shell for a number of years (I used csh interactively, but wrote scripts in ksh).  I made the switch and never went back.

Writing Portable Administration Scripts

Writing portable scripts for UNIX and Linux is fairly easy – Korn shell is everywhere, and ksh scripts work the same and have the same basic tools available (sed, awk, pipes, etc.).

What about writing portable scripts to work on UNIX, Linux, and OpenVMS? UNIX and Linux are similar enough that things will work across the different platforms – the same holds for the BSD platforms and for Windows with the Cygwin utilities.  But radically different platforms such as OpenVMS require a different approach.

The first thing I did in looking at OpenVMS was to search out the languages and utilities that were available.  HP offers a number of open source tools, and has Freeware CDROMs available as well. SAIC has a large OpenVMS archive, including the contents of the HP Freeware CDROMs.

Under OpenVMS, I found these languages available:

• Java (built-in)
• GNU awk
• Perl
• Perl/tk
• tcl/tk
• Python

I doubt that Java would be used for scripting purposes, but it is becoming ubiquitous and if it is well-known by the scripter, it is possible that it could be used.

However, the other (add-on) alternatives seem to be much more likely.  Perl, Python, GNU awk, and tcl have extensive capabilities, and with tk visual displays are possible.  My main choice would probably be Perl.

The next step is to make sure that any coding that is done is truly portable.  Perl, with its extensive documentation, includes documentation specifically for portability and for OpenVMS as well.

The Perl portability documentation goes into complete detail about the various points that may trip a programmer up; in short, several of the main points cover:

• Data representation (high-endian order? low-endian order? line terminator?)
• File path representation
• Character sets and encoding (including order)
• Time and date representation

The best thing to do is to following the guidelines in the Perl portability document (even if using other languages) and to then test the portable code on all systems affected. Only in extreme circumstances should code be written specifically to the target system and selected based on target OS type.  Better to make it portable at its core.

Why does my ksh login hang? (HP-UX)

Recently I had the problem of ksh logins under HP-UX hanging. The symptom was that the login process would appear to work, all of the system profile and the user’s profile would be executed, then the login would hang.

This happened in Korn shell (ksh), but not the default POSIX shell (sh).

Turns out this is a FAQ (!) even though it’s something I’ve not run into these many years. In the comp.sys.hp.hpux faq it is question and answer 8.7. The answer to both symptoms is the fact that the history file ($HOME/.sh_history) is on an NFS mount. In the book Optimizing NFS Performance: Tuning and Troubleshooting NFS on HP-UX Systems (by Dave Olker) he discusses (in section 6.4: Avoiding NFS File Lock Hangs in Your Environment) how NFS locks can cause hangs such as the Korn shell hanging. A small portion of section 6.4 can be read online.

This also was discussed in this message thread on a LDAP mailing list. Turns out that the Korn shell was hanging as it was trying to open the history file ($LOGNAME/.sh_history). To fix the problem, move the history file to /tmp or some other local directory. One suggestion was /tmp/.sh_history.$$; there is probably a better idea which is more secure.

One solution was to make the NFS-mounted file mode 777; but this is definitely insecure and not a good idea.