Instant 10-20% boost in disk performance: the “noatime” option

Many people already know about this option, but it is worth mentioning again. However, a description of what atime is is in order.

The atime is one of the three times associated with a UNIX file: the three are: the ctime (or change time – that is, when the inode was last changed); the mtime (or modified time – that is, when the file is changed); and lastly, the atime (or access time).

It is the continual changes to the atime that cause so much grief. Compared to the mtime and the ctime, the atime changes with alarming frequency. Every single time a file is accessed, the atime is updated to match the current time – whether the file is opened, read, written, or accessed in any manner whatsoever.

There was a Linux kernel mailing list discussion thread that gave rise to some interesting quotes on this topic.

The discussion became quite heated when Ingo Molnar suggested that atime should be the kernel default. He had this to say:

Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_.

and:

It’s also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: ‘For every file that is read from the disk, lets do a … write to the disk! And, for every file that is already cached and which we read from the cache … do a write to the disk!’

and later, this:

Measurements show that noatime helps 20-30% on regular desktop
workloads, easily 50% for kernel builds and much more than that (in
excess of 100%) for file-read-intense workloads.

and, this:

Give me a Linux desktop anywhere and i can
tell you whether it has atimes on or off, just by clicking around and
using apps (without looking at the mount options). That’s how i notice
it that i forgot to turn off atime on any newly installed system – the
system has weird desktop lags and unnecessary disk trashing.

Linus had this to say:

yeah, it’s really ugly. But otherwise i’ve got no real complaint about
ext3 – with the obligatory qualification that “noatime,nodiratime” in
/etc/fstab is a must. This speeds up things very visibly – especially
when lots of files are accessed. It’s kind of weird that every Linux
desktop and server is hurt by a noticeable IO performance slowdown due
to the constant atime updates, while there’s just two real users of it:
tmpwatch [which can be configured to use ctime so it’s not a big issue]
and some backup tools. (Ok, and mail-notify too i guess.) Out of tens of
thousands of applications. So for most file workloads we give Windows a
20%-30% performance edge, for almost nothing. (for RAM-starved kernel
builds the performance difference between atime and noatime+nodiratime
setups is more on the order of 40%)

Changing a file system to run without atime is simple; use this command on a mounted filesystem:

# mount -o remount,noatime /disk

Don’t forget to change the /etc/fstab to match this.

The Current State of Window Maker

I’ve always liked the Window Maker window manager. However, the current state of Window Maker is in some turmoil.

Development on the original Window Maker window manager has ceased, and new development has been taken up by wmaker-crm (a fork). Nightly builds of wmaker-crm had been available for Debian from a user-created repository, but no new builds have been put up since 29 April 2011 – and the repository information hasn’t been rebuilt since 26 May 2011. Building a Debian package seems to be problematical in any case. According to this mailing list thread, Andreas Metzler and Martin Dietze are responsible for changes therein, but the changelog hasn’t reflected any changes in wmaker-crm.

It was recommended to the Debian Window Maker package maintainer, John H. Robinson IV, to use the wmaker-crm sources for the package; he was receptive but nothing has happened since. As of 6 July 2011, he stepped down as maintainer of the package, leaving the Debian package orphaned. This event did not go unnoticed; a thread was taken up on the wmaker-crm development list.

The person behind wmaker-crm, Carlos R. Mafra, also created a git repository for numerous Window Maker dockapps that are no longer maintained.

The standard Window Maker display manager, the WINGs Display Manager, apparently is much less desired than Window Maker itself: according to the popularity contest statistics, at a peak during January 2011, 2000+ people had Window Maker installed while less than 200 currently have WDM installed.

Statistics about the installed base (and user base) of Window Maker and other packages can be seen over at popcon.debian.org; these statistics come from people who have installed the popularity-contest package.

Over at ArchLinux, their wiki has an excellent write-up on Window Maker (including basic technical details and information on wmaker-crm) which certainly makes one think that Window Maker is vibrant in that community. Both Window Maker and wmaker-crm are packaged (as windowmaker and windowmaker-crm-git respectively) and available to ArchLinux users from the Extra Repository as well.

If you want to see Window Maker in action (more or less – it does show a lot of Tux Commander too…) you can check out this video showing Window Maker on a Duron 850MHz system with 256Mb.

I may post a short video of my own Window Maker desktop; having forced myself to run with Window Maker as my default desktop has made me a complete convert – and helped me to force myself to research and resolve problems with making Window Maker a default desktop. Recently, I wrote about just what it took to make Window Maker fully capable and up-to-date.

If GNOME or KDE are finally just too much – and XFCE isn’t quite what you want – try Window Maker instead.

 

Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

• rc – a small shell used by Plan 9
• scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
*
$ echo '$AB'
$AB

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTCONTROL=
export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTIGNORE=
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTFILESIZE
typeset -r HISTIGNORE
typeset -r HISTSIZE
typeset -r HISTTIMEFORMAT

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

Are SSH Passwords Safer than Keys?

It almost goes without saying that servers should never run telnet or rlogin but rather should have SSH instead. Today’s UNIX and Linux server operating systems, whether commercial or open source, come with SSH (usually OpenSSH) installed.

The question is: is password authentication more secure than public key authentication? This question has been asked before; consider these two questions from serverfault.com, one from November 2010 and one from June 2010. The instinctive response from many will be “Of course not!” – but the question remains. What are the actual downfalls to password authentication – or are there any?

There are multiple problems that can exist with public key authentication that password adherents will point out.

Firstly, there is the possibility that a system can be compromised and the keys taken. This risk is not so high as it might sound; if you are running Linux or UNIX on the system and have it properly secured, the risk of system compromise is lower than you might think. It is also possible (and recommended) to encrypt the keys with a passphrase.

Secondly, those who recommend passwords will note that passwords exist only in your mind (as long as you don’t write it down). However, is an easily memorable password truly secure?

There are numerous advantages to using a key, including many not directly related to security: you can have different keys for different systems, and keys can be restricted in various ways by the server administrator. It is also possible to revoke access to a user without affecting any other user – no more having to tell everyone the new password after a password change.

Keys are also not susceptible to brute force attacks. A hacker can attempt to break into an SSH server by brute force by trying a variety of passwords and usernames; this is not possible with keys. This sort of attack can come from anywhere, which makes it easier for hackers to do and more likely to occur. If there are a lot of users using SSH, then the likelihood of someone using a weak password is much higher – making the risk of server compromise that much higher as well.

Passwords can also be stolen remotely. If a server is compromised, the next time you log in you may be connecting to a hacked SSH server which copies all of your passwords. If you used public key authentication, then your private key would never be seen (or compromised) by the remote server.

So how do you properly secure your private key? Here are a number of things you can do:

• Password protect your private key.
• Put the key on a removable device and remove after using.
• Put the key on an encrypted volume.
• Use ssh-agent to store the key instead of using it over and over.

If you have already created a key without a password, you can add a password by using ssh-keygen:

$ ssh-keygen -p -f id_mykey
Key has comment 'id_mykey'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$

There are a number of good sources of information on SSH – and OpenSSH in particular. Try some of these:

• Mick Bauer’s articles in Linux Journal, “The 101 Uses of OpenSSH”, Part 1 and Part 2
• The OpenSSH Wikibook
• Manual Pages from openssh.org
• Daniel Robbin’s articles on IBM developerWorks, “Common threads: OpenSSH key management”, Part 1, Part 2, and Part 3
• The book SSH: The Secure Shell and its associated web site
• An excellent article on using SSH keys from the Arch Linux wiki

A New Init for Fedora 14

Apparently, a new project (to replace init, inetd, and cron) named systemd is nearing release and will be used to replace upstart in Fedora 14 (to be released in November – with Alpha Release due today!).

There is a healthy crop of init replacements out there, and the field is still shaking out. Replacing init – or specifically, System V init and init scripts – seems to be one of those never-ending projects: everyone has an idea on how to do it, no one can agree on how.

Let’s recap the current crop (excluding BSD rc scripts and System V init):

• Upstart
• Apple’s launchd
• Solaris’s System Management Facility (SMF)
• runit
• Richard Gooch’s simpleinit (and introduced in a paper at the 2002 Ottawa Linux Symposium)
• initng
• daemontools
• NetBSD and rc.d
• Make

I am still waiting for the shakeout – it bugs me that there are dozens of different ways to start a system, and that none of them have taken over as the leader. For years, BSD rc scripts and System V init have been the standard – and both have stood the test of time.

My personal bias is towards SMF (OReilly had a nice article on it) and towards simpleinit – but neither has expanded like upstart has.

So where’s the replacement? Which is The One? It appears that no one is willing to work within a promising project, but rather starts over and creates yet another replacement for init, fragmenting the market further.

Lastly, if the current init scheme is so bad, why hasn’t anything taken over? Commercial UNIX environments continue to use the System V scheme, with the sole exception of Solaris which made the break to System Management Facility (or SMF). Why doesn’t HP-UX or AIX use SMF or Upstart if the current environment is horrible?

Sigh. It’s not that the current choices of replacement are bad – it’s just that there are so many – and more keep coming up every day. Perhaps we can learn something about the causes of this fragmentation from a quote from a paper written about the NetBSD rc.d startup scripts and their design:

The change [in init] has been one of the most contentious in the history of the [NetBSD] project.

Three Technologies We Wish Were in Linux (and More!)

Recently, an AIX administrator named Jon Buys talked about three tools he wishes that were available in Linux. Mainly, these technologies (not tools) are actually part of enterprise class UNIX environments in almost every case.

One was a tool to create a bootable system recovery disk. AIX calls the tool to do this makesysb; in my world – HP-UX – this is called make_tape_recovery. In HP-UX, this utility allows you to specify what part of the root volume (vg00) to save and other volumes. Booting the tape created from the make_tape_recovery utility will allow you to recreate the system – whether as part of a cloning process or part of a system recovery.

Another technology missing from Linux is the ability to rescan the system buses for new hardware. In Jon’s article, he describes the AIX utility cfgmgr. HP-UX utilizes the tool ioscan to scan for new I/O devices. Jon mentions LVM (which has its roots in HP-UX) but this does not preclude scanning for new devices (as any HP-UX administrator can attest).

Jon then discusses Spotlight (from MacOS X) and laments that it is missing from Linux. Linux has Beagle and Tracker, and all are quite annoying and provide nothing that locate does not – and on top of this, locate is present on AIX, HP-UX, Solaris, and others. I for one would like to completely disable and remove Spotlight from my MacOS X systems – Quicksilver and Launchbar are both better than Spotlight. In any case, all of these tools don’t really belong on an enterprise-class UNIX system anyway.

As for me, there are some more technologies that are still missing from Linux. One is LVM snapshots: while they exist in Linux, they are more cumbersome. In HP-UX (the model for Linux LVM) a snapshot is created from an empty logical volume at mount time, and the snapshot disappears during a dismount. In Linux, the snapshot created during logical volume create time (whatever for??) and then is destroyed by a logical volume delete. The snapshot operation should mirror that of HP-UX, which is much simpler.

Another thing missing from Linux which is present in every HP-UX (enterprise) system is a tool like GlancePlus: a monitoring tool with graphs and alarms (and the alarms include time-related alarms).

Consider an alarm to send an email when all disks in the system average over 75% busy for 5 minutes running. This can be done in HP-UX; not so in a standard Linux install. There are many others as well.

Personally, I think that Performance Co-Pilot could fill this need; however, I’m not aware of any enterprise class Linux that includes PCP as part of its standard supported installation. PCP has its roots in IRIX from SGI – enterprise UNIX – and puts GlancePlus to shame.

Perhaps one of the biggest things missing from Linux – though not specifically related to Linux – is enterprise-class hardware: the standard “PC” platform is not suitable for a corporate data center.

While the hardware will certainly work, it remains unsuitable for serious deployments. Enterprise servers – of all kinds – offer a variety of enhanced abilities that are not present in a PC system. Consider:

• Hot-swappable hard drives – i.e., hard drives that can be removed and replaced during system operation without affecting the system adversely.
• Hot-swappable I/O cards during system operation.
• Cell-based operations – or hardware-based partitioning.

For Linux deployment, the best idea may be to go with virtualized Linux servers on enterprise-class UNIX, or with Linux on Power from IBM – I don’t know of any other enterprise-class Linux platform (not on Itanium and not on Sparc) – and Linux on Power may not support much of the enterprise needs listed earlier either.

What are your thoughts?

Whither Sun Microsystems?

The recent fourth quarter reports from server manufacturers was dim, and Sun Microsystems was by far the worst (with a 35% loss compared to the same period last year). On top of this, Sun just announced in October (within their 8K filing for the SEC) intentions to lay off 3000 employees in the next 12 months. Infoworld also had a nice piece on this; according to Larry Ellison, CEO of Oracle, the company is losing $100 million each month the European Union regulators put off accepting the merger.

With the Oracle acquisition in progress, there are a lot of questions about the future viability of Sun Microsystems, and of some of its products.

I don’t think people realize just how important the Sun group of products are, and what an impact it would have if most – or even some – of the products were cancelled. Consider this list of Sun products:

• Java.
• Solaris – and OpenSolaris.
• SPARC – and OpenSPARC.
• MySQL. There was a recent report from the 451 Group that suggests that usage of MySQL may fall off based on the current uncertainty around Sun.
• StarOffice – and OpenOffice.
• VirtualBox (and associated virtual products). Virtualbox also has an open source version, and was just updated to version 3.1 with new features. InformationWeek and eWeek both had excellent articles on the new Virtualbox.
• Sun Modular Data Centers (originally with the codename Project Blackbox).
• The Lustre Filesystem.

Most of the most popular products were mentioned by Oracle in their Sun Acquisition FAQ (PDF), stating that they will increase money spent on each over what Sun spent. These products include: Java, Solaris, SPARC, StarOffice, NetBeans, virtualization products, Glassfish, and MySQL. Other products were not mentioned – such as Lustre, the Modular Data Center, and others.

The list above also does not list the technologies that were spearheaded by Sun – and some still are: ZFS, NFS, NIS (and NIS+), dtrace, containers, and smc.

It would be unfortunate – and materially significant – if Sun were to go under or if any of the majority of their products were to be cancelled. One can only hope this does not happen…

Why I Use Korn Shell Everywhere

The first thing I do when I log into a system, including Solaris, HP-UX, FreeBSD, and Linux is exec ksh. Whatever for?

Consider this fact: the root shell on FreeBSD defaults to C shell; HP-UX defaults to the POSIX shell (without history); Linux almost everywhere defaults to bash. All of these shells are different in various ways. It is possible you might log into three separate machines and get three separate shells with three different ways of handling things.

Using Korn Shell means that all of these systems will be standardized on one shell, and every system will act the same when you interact with it. There will be no surprises – and surprises at the root command line often translate into disastrous errors.

On HP-UX, using ksh has the additional benefit of enabling history for root – although the security risks of this make this a dangerous benefit: best to erase history after you log out and to make sure that history is independent for every root shell.

What makes this possible is that the Korn Shell is available virtually everywhere, including FreeBSD, Linux, Solaris, and HP-UX – whereas other shells are not (which includes C shell, Bourne shell, and bash).

The Dichotomy of a System Administration Career

When you choose to work in system administration, generally you have to focus on one operating system or another. The dichotomy comes in choosing a system to focus on for your career.

How do you go about choosing which system you want to administrate as a career? Do you go with a common system like Microsoft Windows or a relative rarity such as OpenVMS?

If you go with Microsoft Windows Server, for example, there will always be jobs available (relatively so, anyway). Every corporation seems to have at least one Microsoft Windows Server, and they all need to be taken care of by someone who knows what to do. However, there will be lots of other people that do the same thing. So even as there are jobs out there, there are lots of applicants and lots of competition. With this abundance of people who know how to administrate Windows servers (or think they do) comes a lower pay, as an employer can be selective in who they choose. This is the basic economic principle of supply and demand at work.

On the other side is administering UNIX servers – or even more so, OpenVMS servers. The number of people who can administrate these servers is less than those who work with Windows, which means their expertise is more expensive. For a variety of reasons, UNIX is present less in the average enterprise, and the number of UNIX servers is very likely dwarfed by the number of Windows servers. This is an advantage as the pay scale will be higher, but the disadvantage is that the jobs will be fewer.

When the market is tight, those with more specialized skills will find themselves having to move where the work is, and will have to search further afield for possible openings. It is a trade-off – and it’s your choice. Just be sure you have the facts first before you choose.