Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

  • rc – a small shell used by Plan 9
  • scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
$ echo '$AB'

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Avoiding catastrophe!

All UNIX administrators, if they’ve not been bit by a typo like this, have heard of it (or something like it) happening:

cd /
rm -rf foo *

Notice the extra space, and the current working directory. Instead of deleting directories beginning with the name “foo” it will delete everything!

How can this be avoided? There are number of practices that I have ingrained which have saved me from such disasters.

Before executing dangerous commands, check which files will be affected… For example, you may find a need for a command such as:

find . -size +10000 | xargs rm -f

However, before doing this, do a command like this one:

find . -size +10000 | xargs ls -ld

This will save you from potentially removing important files.

Instead of executing disastrous commands automatically, use a computed script. For example, the previously mentioned example:

find . -size +10000 | xargs rm -f

This example could be replaced by the following instead:

find . -size +10000 | xargs -i{} echo rm -f {} >

Then the resulting script can be scanned for files that one wants to keep, and the script edited to reflect the desired actions. Once the script is the way you want it, it can be run:

sh ./

This method is particularly good for times when you want to delete or modify hundreds of files, but want to prevent a few from being affected.

Notice where you are! Perhaps you think you are in a particular directory, but you aren’t: double-check. Take a moment and do a pwd command and check.

If it affects the entire system, make sure you are on the right system! You don’t want to shut down a system only to realize it was the wrong one! Do a uname -n just to check – every time. Don’t use the system prompt; if you rely on that you may realize too late that the system prompt is just a string (and it could lie to you!).

If you manage many diverse systems, double-check the syntax of the command first. If you do the command in the wrong way who knows what could happen? When I managed four or five types of UNIX, I did a man 1m shutdown before every shutdown. Most were similar – some were not (such as Unixware and Solaris for example).

Notice dangerous commands and stop and double-check them! When doing an rm -rf I always double-check the command before pressing enter – even just pausing to look – rather than just typing through. This goes just as much for the find command piped into a disastrous command (such as rm or mv).

Notice dangerous commands and pause before pressing enter. I always pause to check the command – even just pausing, knowing that this is the moment – the last chance to back out. With that knowledge in the back of your mind, it will cause you to double-check if you haven’t already.

Argument list too long?

Well, what now? We got the dreaded “argument list too long” error. What to do?

To explain the problem, let’s consider what the shell does (we won’t get into system calls, to make things simple). The shell (Bourne compatible shells, actually) will first scan the line typed in. One of the first things to do is to expand file globs. For example, the command:

ls a*

will be translated by the shell into:

ls andrew apple alex allen alfred almonzo august axel albert

(and so forth). If the expansion expands beyond the system limitations, then it cannot be processed and the error “argument list too long” results.

The limit could be the number of arguments or the number of characters; however, the fix is the same. The key is to get the list (usually files) out of the argument list and into something else – such as stdout. Once the overlong list is being sent via stdout, the command xargs can be used to place all of the items on the command line, observing all relevant limits as it goes (and as efficient as possible to boot).

There are a variety of quick answers which will all fail, because the argument list would remain too long:

ls a* >savefile.txt
for i in a* ; echo $i ; done
echo a*

However, all is not lost: there are a number of ways to get a long list of files into stdout; the most common is to use find:

find . -name "CACHE.DAT" | xargs ls -ld

This may not be possible, if the desired list of files doesn’t fit neatly into a find command.

Another possibility, related to the previous example, would be this:

ls -1 | sed '/^a/!d' | xargs ls -ld

Yet another possibility might be to use a language like Perl; since it does not scan and process the same way, it would work without limitations:

perl -e 'opendir(DIR, "."); @all = grep /^a/, readdir DIR; closedir DIR; print "@all\n";' | xargs ls -ld

I would only recommend using Perl or other such if you are quick and snappy with your knowledge of the language; otherwise, such a long line will have you looking things up repeatedly.

If the arguments are coming from a file, then things become even easier; instead of something like this:

ls -ld $(cat mylist)

You can simply use:

cat mylist | xargs ls -ld

Of course, any binary command can be used with xargs, not just ls.

Assorted Tips and Tricks

First, there is one that I learned recently myself. This trick is ingenious!

One of the most challenging things to explain is why this doesn’t work (when the outfile is write-restricted to root):

sudo command > outfile

This will fail because when the shell tries to open outfile, it is not running as root – and thus does not have access. Solving this problem is not simple because of the shell’s quoting mechanisms and when it opens (and doesn’t) the file in question.

However, there is a simple solution that I’d never considered before:

sudo command | sudo tee outfile

This takes care of all the problems involved – and if the command itself is not restricted to root, then the first sudo isn’t necessray either.

Another thing that can be seen often in shell scripts is something like the following:

cmd >> $logfile
cmd2 >> $logfile
cmd3 >> $logfile
print "New stuff...." >> $logfile

This entire section can be replaced like this:

( cmd
print "New stuff...." ) >> $logfile

In the first example, the $logfile is opened four times – and many situations would include many more than just that. The last only opens $logfile once.

Another tip – this time in the find command. A sequence like this:

find dir1 -mtime +1 -type f
find dir2 -mtime +1 -type f
find dir3 -mtime +1 -type f

…can be replaced by a much more succinct command, like so:

find dir1 dir2 dir3 -mtime +1 -type f

Thus, instead of three process invocations, there is just one.

One more tip: if you find yourself with a .tar.gz file (or whatever) and want to unpack it somewhere else, you don’t have to move the file at all. If you utilize this general sequence, the archive can be anywhere and the unpacked data can go anywhere. Assume that the working directory contains the archive, and the unpacking is to be done in another directory (such as /tmp):

gunzip -c myfile.tar.gz | ( cd /tmp ; tar xvf - )

Using the parenthesis allows you to change the working directory temporarily, and thus to utilize the tar command in a different directory. Inversely, if you were in this same situation but were located in the /tmp directory (and unpacking the archive located in your home directory) – you can do this:

( cd $HOME; gunzip -c myfile.tar.gz ) | tar xvf -

Why not: yet one more tip. Let’s say you want to go to this directory:


Rather than having to type that in (and try and get the spelling right!) use something like this to get there:

cd /etc/super*/expi*/atro*/

First, these will match the appropriate directories (assuming there is only one that matches all wildcards). However, with the final slash character in place, that means that only directories that begin with “atro” will be matched – files will not. Nifty, eh?

What’s more, once you’ve gotten to that directory with the nasty name – you can switch to another then back simply:

cd /etc/foo
cd -

That last command switches back to the previous directory – the very long-named directory mentioned before, all compressed down to a single character.