Puppet error: already in progress; skipping

Sometimes, you may try to run your puppet agent, and get an error like this:

# puppet agent --test
notice: Run of Puppet configuration client already in progress; skipping

If there is indeed another puppet agent running, it is simple enough to stop it and try again. However, what if this message appears, and there aren’t any other puppet instances running?

This happens because there is a flag stored in a file that didn’t get erased. Do this – but only if puppet is not in fact running:

# cd /var/lib/puppet/state
# rm -f puppetdlock

This will delete the lock, and puppet should start cleanly the next time. This tip works with puppet version 2.6.3.

Using Browsers with OpenLDAP cn=config Tree

The new OpenLDAP configuration is an LDAP tree like any other, and can be browsed as such. Using a graphical browser like Apache Directory Studio can simplify configuration (somewhat).

When setting up a new LDAP connection, specify the user as the admin user for the configuration tree – like so:

cn=admin,cn=config

Also specify the tree root as:

cn=config

With the appropriate settings, you should then get a list of the configuration tree, where you can view and edit entries. It would behoove you to be very careful about changing things in this tree: if you change the wrong thing, your LDAP server could stop functioning properly.

Tips and Tidbits About LDAP

Setting up and understanding LDAP is not easy. In my opinion, nothing is obfuscated more and unnecessarily so than LDAP. There are a number of tips that can help you to understand LDAP.

LDAP is not authentication. This was the number one problem I had when I started (a while back). The first time user might search for documents on setting up LDAP when in fact they are looking for documents on how to set up UNIX and Linux authentication using LDAP. An LDAP server at its most basic doesn’t understand UNIX uids, doesn’t understand GECOS fields, doesn’t understand UNIX gids, and doesn’t understand Linux shadow files. Support for all of this must be added.

Support for UNIX authentication must be added. You would think that the most common usage for LDAP would come bundled and ready to go with the server; however, often this is not the case. Even if it is the case, you may find that for certain capabilities you are expected to add new LDIF files to support the fields in LDAP.

LDAP is not just another database server. Virtually everything in LDAP has a different name; it is unlike anything you’ve done before. Take heart: X.500 (where LDAP comes from) was worse. You’ll have to slog through a pile of new terms, but after a while it will become easier to understand.

OpenLDAP is not synonymous with LDAP. There are other servers out there. OpenLDAP does come with virtually every Linux platform; there are however, many others – many of which may be easier to use. There is the 389 Directory Server from Red Hat, the ApacheDS (part of the Apache Directory Project) from Apache, and OpenDJ from ForgeRock. OpenDJ itself is a spin-off from OpenDS, originally from Sun.

OpenLDAP is known for making non-backwards-compatible changes. The most recent example is the complete replacement of the configuration system.

OpenLDAP no longer uses slapd.conf. This will cause you no end of problems: there are a lot of people trying to explain how to set up OpenLDAP, and with a single strike (as of version 2.3) OpenLDAP made all of that documentation obsolete and useless. This is incomprehensible, but it is a fact.

Using and administering LDAP requires command line expertise. This is basically true, but like many things, it is not the complete truth. There are many programs designed to make it easy to browse LDAP stores, along with editing capabilities. Some of the more interesting products include Jxplorer, Luma, and the Apache Directory Studio. Of these, the Apache Directory Studio is the most capable, robust, and actively developed – and by far the largest.

Some LDAP entries can be present more than once or have more than one value. If you are comparing LDAP to a database, then this will come as a surprise. One valuable example is UNIX groups: the original UNIX systems only had one group per user; later, secondary groups were added – thus presenting a single user with multiple groups. This is handled in LDAP in a variety of ways, but they all amount to having multiple entries with different values.

Limiting user logins by host is not available in LDAP. This capability is most likely to be done by using the client host. There are a number of ways to do it, but all require LDAP client configuration, and all are limited in their application. Without client configuration, all LDAP users will have authenticated access to the host.

Be prepared to do a lot of web searches for documentation and solutions. The best places to go for searches are: Google (of course) and Ubuntu Documentation.

There are also very good articles and documents on using LDAP for authentication. There is an article about OpenLDAP authentication on FreeBSD (FreeBSD articles tend to be very well-written). Similarly, Ubuntu documentation is well-written as well; each of the Ubuntu versions has a section in the documentation on using and configuring OpenLDAP for authentication. Ubuntu 11.04 documentation has a good article on OpenLDAP for example.

Ubuntu documentation also includes a lot of well-written (and current) articles. For example, there are articles on OpenLDAP Server (a general article), LDAP Client Authentication, Samba and LDAP (from the 10.04 Server documentation), and Securing OpenLDAP Conenctions. If you plan to use 389 Server instead, there are even a couple of articles on using it with Ubuntu: Fedora Directory Server (the original name of 389 Server) and Fedora Directory Server Client Howto.

A nice overview of LDAP comes from Brian Jones at O’Reilly: specifically, his 2006 articles on Demystifying LDAP and Demystifying LDAP Data. Linux Journal also has myriad different articles on LDAP (not to mention an OpenLDAP theme for the December 2002 issue). Linux Journal also has an article from 2007 on Fedora Directory Server (now 389 Server).

Lastly, an excellent resource is the “book” LDAP for Rocket Scientists from Zytrax.com. You simply must go and read portions of this book. One very apt quote from the introduction to the book which sums up the state of LDAP documentation generally:

The bad news is that [in our humble opinion] never has so much been written so incomprehensibly about a single topic with the possible exceptions of BIND and … and …

(It should be noted that the other book at Zytrax is about DNS. Is it any surprise?)

UPDATE:

Yet another trick to LDAP:

The cn= attribute is not solely a leaf attribute. This can be seen in OpenLDAP’s cn=config tree with OpenLDAP configuration. For example, a typical admin user can be designated like so:

cn=admin,dc=bigcorp,dc=com

However, when you use OpenLDAP’s configuration, the designation for the admin user is this:

cn=admin,cn=config

When you look into the configuration tree, there are more cn= entries – like this:

cn={4}misc,cn=schema,cn=config

Configuring the Bash Shell (and Korn Shell) to Permanently Use vi-mode Editing

The GNU bash shell, as one might expect, uses Emacs key bindings to perform history and editing in the shell. Vi key bindings are available, and can be set with this standard command (just like in ksh):

set -o vi

My fingers have gotten this sequence of letters ingrained in them so I don’t even have to think about it – and I type it almost automatically after login.

However, there is a way to set up ksh to automatically set editing to vi mode. If the VISUAL variable is set to a value like vi, then the editing mode will be set to vi mode automatically. Likewise, if the VISUAL variable is set to emacs, then emacs mode is used for editing.

If the VISUAL variable is not set, then the EDITOR variable is used.

In later versions of the AT&T Korn shell (such as 20100202) the recognized values have been expanded: the pattern *[Vv][Ii]* is recognized as a request for vi mode; the pattern *gmacs* results in gmacs mode; and the pattern *macs* results in emacs mode. (The only difference between emacs and gmacs mode is the handling of the ^T character.)

This means that using vim or gvim will now trigger vi mode – indeed, using almost any vi clone editor such as nvi or elvis will work with this method. This also means that the full path can be used, although this may have been true previously.

In bash however, this use of EDITOR and VISUAL is not available. However, since bash uses GNU readline, we can set up readline to use vi mode by default – and thus also affect all programs that use GNU readline besides.

Edit (or create) a file in your home directory called .inputrc and add this line to it:

set editing-mode vi

After this, any time you log in – or use anything else that uses GNU readline (such as CLISP for example) – you’ll automatically have vi mode editing. I can finally rest my fingers…

Instant 10-20% boost in disk performance: the “noatime” option

Many people already know about this option, but it is worth mentioning again. However, a description of what atime is is in order.

The atime is one of the three times associated with a UNIX file: the three are: the ctime (or change time – that is, when the inode was last changed); the mtime (or modified time – that is, when the file is changed); and lastly, the atime (or access time).

It is the continual changes to the atime that cause so much grief. Compared to the mtime and the ctime, the atime changes with alarming frequency. Every single time a file is accessed, the atime is updated to match the current time – whether the file is opened, read, written, or accessed in any manner whatsoever.

There was a Linux kernel mailing list discussion thread that gave rise to some interesting quotes on this topic.

The discussion became quite heated when Ingo Molnar suggested that atime should be the kernel default. He had this to say:

Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_.

and:

It’s also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: ‘For every file that is read from the disk, lets do a … write to the disk! And, for every file that is already cached and which we read from the cache … do a write to the disk!’

and later, this:

Measurements show that noatime helps 20-30% on regular desktop
workloads, easily 50% for kernel builds and much more than that (in
excess of 100%) for file-read-intense workloads.

and, this:

Give me a Linux desktop anywhere and i can
tell you whether it has atimes on or off, just by clicking around and
using apps (without looking at the mount options). That’s how i notice
it that i forgot to turn off atime on any newly installed system – the
system has weird desktop lags and unnecessary disk trashing.

Linus had this to say:

yeah, it’s really ugly. But otherwise i’ve got no real complaint about
ext3 – with the obligatory qualification that “noatime,nodiratime” in
/etc/fstab is a must. This speeds up things very visibly – especially
when lots of files are accessed. It’s kind of weird that every Linux
desktop and server is hurt by a noticeable IO performance slowdown due
to the constant atime updates, while there’s just two real users of it:
tmpwatch [which can be configured to use ctime so it’s not a big issue]
and some backup tools. (Ok, and mail-notify too i guess.) Out of tens of
thousands of applications. So for most file workloads we give Windows a
20%-30% performance edge, for almost nothing. (for RAM-starved kernel
builds the performance difference between atime and noatime+nodiratime
setups is more on the order of 40%)

Changing a file system to run without atime is simple; use this command on a mounted filesystem:

# mount -o remount,noatime /disk

Don’t forget to change the /etc/fstab to match this.

Logging transfers via SFTP

When you install OpenSSH, you also get SFTP for free: a replacement for FTP. (Note that SFTP is related to FTP by name only.)

SFTP is very useful, but as is normally configured, it does not provide any logging of user’s file transfers. For monitoring usage, this can be a problem.

The original setting in sshd_config might look like this:

Subsystem sftp /usr/libexec/openssh/sftp-server

This lack of logging is easily remedied; the details can be found in the man page for sftp-server:

Subsystem sftp /usr/libexec/openssh/sftp-server -l INFO

For best results, check your system’s man pages for the available options.

Why doesn’t my /bin/sh script run under Ubuntu?

This is a very interesting question – and the resolution is simple. In Ubuntu 6.10 (known as Edgy Eft) the decision was made to replace the Bourne Again Shell (bash) with the Debian Almquist Shell (or dash) as /bin/sh in Ubuntu. There was considerable uproar in Ubuntu brainstorm (community ideas) and in Ubuntu bug reports, as using dash instead of the original bash caused numerous scripts to break.

In particular, the entire reasoning given for this change was efficiency: dash is more efficient (i.e., faster) than bash. According to the explanatory document created by the Ubuntu developer team, Debian has required scripts to work on POSIX-compliant shells for some time (even pre-dating the Ubuntu project). Thus, any scripts that broke were, in essence, not “following directions” and deserved what they got.

To undo this change by the Ubuntu team, one can do this:

sudo dpkg-reconfigure dash

When this command executes, specify that you do not want dash to act as /bin/sh. This will make every script that runs /bin/sh run bash as has traditionally been the case.

You can also make your scripts run /bin/bash instead of /bin/sh; this provides all of the bash capabilities without any concern as to whether /bin/sh will change again.

Making the boot process faster is a laudable goal, but like the removal of OSS from the kernel, it caused a lot of problems for users.

In both cases, it appears that the Ubuntu team is more focused on doing the technologically “right” thing rather than providing a stable and reliable platform. Unfortunately, this means that you cannot rely on Ubuntu to stay reliable – at least from one version to the next. The response of Ubuntu to such system failures has always been that they are doing the “right” thing and the problem must be fixed by someone else (i.e., it’s not Ubuntu’s problem).

Users – many of them system administrators – take the brunt of this: they don’t care whose fault it is, nor do they care whether the boot process is faster or whether the Linux sound environment is “cleaner”; they care about the stability of their systems. A system that boots faster doesn’t matter if it crashes during the boot process because of a broken script.

If the focus of Ubuntu were to provide a stable and unchanging environment, then their decisions would be different – and would result in an improved customer experience.

Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

• rc – a small shell used by Plan 9
• scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
*
$ echo '$AB'
$AB

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Generating Passwords Using crypt(3)

When writing things like Red Hat Kickstart scripts, or using useradd, or in many other cases, a password is required in the format generated by crypt(3). However, how do we generate a password in this format? In fact, there are many, many ways – if you just know what they are.

One way is to use mkpasswd. This utility is made just for this very purpose; to generate a password with crypt(3) use the command:

mkpasswd

The program will ask for a password, and generates a crypt(3) output string.

If you have the Apache web server loaded, you can use htpasswd. Use this command

htpasswd -nd user

The name of the user doesn’t matter, as it is the password we want. The output will be in the format user:password; just copy the password and you’re set.

If you have OpenSSL available, you can use the openssl command:

openssl passwd -crypt myPassword

Replace myPassword with the password you want to encrypt.

The other methods all require putting the password into the command line in plain text. This can be a problem: the process list (seen using ps) will have the password in it as long as the program runs. The password will also go into the shell history.

One way around this – with programming languages – is to use a script or to use the language’s interpreter.

Using Perl:

perl -e "print crypt('password','sa');"

Perl requires a salt ('sa' in the example).

Ruby can generate a crypt-formatted password, but it requires a salt:

ruby -e 'print "password".crypt("JU"); print("\n");'

Using PHP, you can generate a password from the UNIX command line this way:

php -r "print(crypt('password','JU') . \"\n\");"

Note that if you do not provide a salt ('JU' in the example) then the string returned may not be in crypt format, but rather in MD5 format. Thus, while the salt is an optional parameter, it is necessary in this case.

Using Python requires importing the crypt library – and it requires a salt:

python -c 'import crypt; print crypt.crypt("password","Fx")'

Again, the salt in this case is the second parameter or "Fx".

Databases can generate crypt passwords too; using MySQL it can be done like this:

echo "select encrypt('password');" | mysql

For Tcl there are a number of options; if you are using Ubuntu, you could give the trf package a try. If using Lua, there is the lua-crypt extension.

Feel free to add other options below – with working command-line examples.

Using example domains

People have put example domains in all kinds of programs and servers, often using example.com, example.net, or example.org – along with such as test.com, anywhere.com, anywho.com, somewhere.com, anytownusa.com, and so on.

All of these are domains that resolve and have actual servers up and running on the Internet. Of those mentioned previously, only example.com, example.net and example.org are preserved for testing purposes by IANA in RFC 2606.

Better yet, when using example email addresses avoid any surprises by using one of these domains specified in RFC 2606:

• .example (for documentation)
• .test (for testing purposes)
• .localhost (for sending to the local host)
• .invalid (for creating guaranteed invalid domain names)

If you use these domains, you won’t have to worry about mail going out that wasn’t supposed to go out. I’ve seen this happen before – a configuration file sent out with an open source server sends mail to an example address – which address turns out to go to a valid domain on the Internet, where it is accepted by the mail host.

Don’t get caught by this mistake! Use the RFC 2606 domains wherever needed, and don’t make one up of your own.