A Rift Opens Between KDE and GNOME

This is unfortunate indeed. KDE developers are accusing GNOME developers of not conforming to standards and not collaborating, and Mark Shuttleworth, founder of Canonical, expressed agreement with this view.

The focus seems to be directly related to something called appindicators – and to a larger degree, over the Ubuntu Unity desktop.

The argument goes like this: Canonical and KDE have in the past both approached the GNOME project with ideas, and have been shot down for poor reasons; GNOME refuses to collaborate on projects; others are working together and GNOME refuses.

You can decide for yourself whether this is valid or not. Blog posts have been erupting everywhere on this topic: Dave Neary of GNOME: Has GNOME rejected Canonical help? and Lessons Learned – Aaron Siego of KDE: collaboration’s demise – Mark Shuttleworth of Canonical: Internal competition is healthy, but depends on strong and mature leadership.

Over at OSNews, Thom Holwerda has two very informative pieces on the conflict – one on 10 March and one on 14 March.

Where this conflict will hurt the user is when the user chooses an application: will it work with GNOME or KDE et al? It will also hurt application development as the applications will have to choose – and many will have to choose one technology or the other (not both). This means that applications may only work on one environment or the other – or will have reduced capabilities in one environment or the other. It’s really too bad that the developers can’t come together and work together instead of conflicts like this.

Ubuntu Kickstart Installs and Corrupt Packages File

When installing Ubuntu Lucid via Kickstart, you may run into the error:

Debootstrap warning http://server/ubuntu/lucid/dists/lucid/restricted/binary-amd64/Packages was corrupt

The problem is that the Ubunutu installer tried to decompress a file Packages.gz from that directory, and the result was a zero-length file. This result is being interpreted as a corrupt file.

To fix this, all that needs to be done is to unpack the Packages.gz file on the server end. With an unpacked Packages file, the Ubuntu installer retrieves the zero-length file from the server instead of getting it by unpacking Packages.gz – and it does not interpret the zero-length Packages file as corrupt.

If you are using a mounted Lucid Lynx CDROM, then you need to use a bind mount in order to supersede the mounted read-only directory with a fixed directory. Create a new directory and copy the contents of the appropriate directory into it – the directory containing Packages.gz.

Unpack Packages.gz, and mount the resulting “fixed” directory using bind:

UB=/var/www/ubuntu mount --bind $UB/lucid.bugfix $UB/lucid/dists/lucid/restricted/binary-amd64

This will take care of the problem. Hats off to Sascha Kauffman, from whose article I learned this solution.

Making the Case for Partitioning

What is it about partitioning? The old school rule was that there were separate partitions for /, /usr, /home, /var, and /tmp. In fact, default server installations (including Solaris) still use this partitioning set up.

Has partitioning outlived its usefulness?

This question has come up before. There are negative and positive aspects to partitioning, and the case for partitioning might not be as strong as it once was.

Partitioning means that you may have a situation with no space in one partition and lots in another. This, in fact, is the most common argument against partitioning. However, using LVM or ZFS where disks can be grown dynamically makes this a moot point. With technologies such as ZFS and LVM, you can expand a disk and filesystem any time you need to.

However, this still means that the system will require maintenance – but that is what administrators are for, right? If a filesystem fills – or is going to fill – it is up to a system administrator to find the disk space and allocate it to the disk.

Another argument against partitioning says “Disk is cheap.” Well, if this is true, then why do companies still balk at getting terabytes of disk into their SANs? The phrase “Disk is cheap” is trite but untested: in truth, buying 144Gb disks is not done in bulk. Companies still have to watch the budget, and getting more disk space is not necessarily going to be a high priority until disk runs out.

So, what are the benefits to partitioning disks? There are many.

Each partition can be treated seperately – so the /usr filesystem can be mounted read-only, the /home directory can be mounted with the noexec and nosuid options, which makes for a more secure and more robust system.

Also, if there are disk errors, then a single partition can be affected rather than the entire system. Thus, on a reboot, the system still comes up instead of being blocked because the root filesystem is trashed. In this same vein, if a filesystem requires a check, going through a 144Gb filesystem check could take a very long time – whereas, if the partition was 10Gb it would be not nearly as long and the system would come back up that much faster.

Backups – and restores – are another thing that is simplified by having multiple partitions. For example, when backing up an HP-UX system using make_tape_recovery, you specify which partitions to back up to tape. These partitions are then restored when the tape is booted. If you used a single partition for everything (data, home, etc.) then you would probably not be able to make this sort of backup at all.

One of the nicest reasons to partition is the ability to separate user data from system data. This allows the reinstallation of the system while keeping user data (and application data) untouched. This saves time and effort. I recently installed Ubuntu Server in place of Red Hat Enterprise Linux, and since the system was a single partition, there was no way to install Ubuntu Server without wiping out 200Gb of application data and restoring it – which took around 9 hours each way on a gigabit network link (if nothing else was sharing the network). Alternately, I converted my OpenSUSE laptop to using Xubuntu – and was able to keep all of my user settings because /home was on a separate partition. Keeping a system on a single partition cost the company somewhere in the order of a full day’s worth of time and effort – how much money would the company have saved by having a separate partition for /var/lib/mysql?

Performance is another reason for partitioning – but this is only relevant for separate disks. If you have multiple disks, you can segregate them into separate partitions, which means then that if a disk is heavily used for one purpose it can be dedicated to that purpose – you won’t have your database accesses slowing down because of system log writes, for example. However, this problem can be reduced or eliminated – or made moot – by moving data to a striped volume, and possibly with disk cache as well. Yet, as long as there are disk accesses, do you want them competing with your database?

Having said all of this, how does using a virtual machine change these rules? Do you need partitioning in a virtual machine?

A virtual machine makes all of the arguments relating to the physical disk moot – performance on a virtual disk doesn’t have a high correlation with true physical hardware unless the virtual machine host is set up with segregated physical disk. Even so, the disks may actually be separate LUNs created in a RAID.

However, the ability to secure a filesystem (such as /usr), save filesystem check time, prevent excessive /home usage, and other reasons suggest that the case for partitions is still valid.

Current State of Berkeley Automounter (amd)

Well, this is really sad. I have always believed in using the Berkeley Automounter (amd) over the automounters otherwise available, given the power and portability of amd. However, in investigating automounters for Linux, it seems that amd has been stagnating for some time.

Apparently, this has happened before. The current maintainer, Erez Zadok, found that amd was stagnating and took over development many years ago. He wrote a book on Linux NFS and Automounter, which is still a fantastic book.

Now the git repository for amd shows that the last development on amd happened around November of 2010 – and no development in any other month of 2010. Version 6.2rc1 was tagged four months ago.

It would be a true shame if amd were to die from stagnation; it has strong capabilities that aren’t present elsewhere.

Are SSH Passwords Safer than Keys?

It almost goes without saying that servers should never run telnet or rlogin but rather should have SSH instead. Today’s UNIX and Linux server operating systems, whether commercial or open source, come with SSH (usually OpenSSH) installed.

The question is: is password authentication more secure than public key authentication? This question has been asked before; consider these two questions from serverfault.com, one from November 2010 and one from June 2010. The instinctive response from many will be “Of course not!” – but the question remains. What are the actual downfalls to password authentication – or are there any?

There are multiple problems that can exist with public key authentication that password adherents will point out.

Firstly, there is the possibility that a system can be compromised and the keys taken. This risk is not so high as it might sound; if you are running Linux or UNIX on the system and have it properly secured, the risk of system compromise is lower than you might think. It is also possible (and recommended) to encrypt the keys with a passphrase.

Secondly, those who recommend passwords will note that passwords exist only in your mind (as long as you don’t write it down). However, is an easily memorable password truly secure?

There are numerous advantages to using a key, including many not directly related to security: you can have different keys for different systems, and keys can be restricted in various ways by the server administrator. It is also possible to revoke access to a user without affecting any other user – no more having to tell everyone the new password after a password change.

Keys are also not susceptible to brute force attacks. A hacker can attempt to break into an SSH server by brute force by trying a variety of passwords and usernames; this is not possible with keys. This sort of attack can come from anywhere, which makes it easier for hackers to do and more likely to occur. If there are a lot of users using SSH, then the likelihood of someone using a weak password is much higher – making the risk of server compromise that much higher as well.

Passwords can also be stolen remotely. If a server is compromised, the next time you log in you may be connecting to a hacked SSH server which copies all of your passwords. If you used public key authentication, then your private key would never be seen (or compromised) by the remote server.

So how do you properly secure your private key? Here are a number of things you can do:

• Password protect your private key.
• Put the key on a removable device and remove after using.
• Put the key on an encrypted volume.
• Use ssh-agent to store the key instead of using it over and over.

If you have already created a key without a password, you can add a password by using ssh-keygen:

$ ssh-keygen -p -f id_mykey
Key has comment 'id_mykey'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$

There are a number of good sources of information on SSH – and OpenSSH in particular. Try some of these:

• Mick Bauer’s articles in Linux Journal, “The 101 Uses of OpenSSH”, Part 1 and Part 2
• The OpenSSH Wikibook
• Manual Pages from openssh.org
• Daniel Robbin’s articles on IBM developerWorks, “Common threads: OpenSSH key management”, Part 1, Part 2, and Part 3
• The book SSH: The Secure Shell and its associated web site
• An excellent article on using SSH keys from the Arch Linux wiki

Nagios and the Ampersand Character in URLs

When configuring Nagios for notifications, you will have something like this:

# commands.cfg

# vim: se nowrap

#========================================
# NOTIFICATIONS
#========================================

# 'notify-host-by-email' command definition
define command{
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

# 'notify-service-by-email' command definition
define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
        }

Now, you might want to extend these by adding the host or service URL. With this addition, the receiver of the message can click on the URL for more data.

The URL for the notes and actions are available; they are in these macros

• $HOSTACTIONURL$
• $SERVICEACTIONURL$
• $HOSTNOTESURL$
• $SERVICENOTESURL$

It should be simple to add these to the command_line line in the Nagios configuration; unfortunately, it is not that simple. This is because ampersands are removed from the URL when it is inserted. Ampersands are used to separate arguments that are sent to web pages – and may be included in the URLs. PNP4Nagios does this, and sends arguments to its web pages separated by ampersands (as it should).

The Nagios documentation hints at this character cleansing process, but does not mention these macros as ones that are cleaned and stripped of shell characters.

The shell characters that are stripped can be set with the illegal_macro_output_chars setting in the main configuration file. According to the documentation, the affected macros are: $HOSTOUTPUT$, $HOSTPERFDATA$, $HOSTACKAUTHOR$, $HOSTACKCOMMENT$, $SERVICEOUTPUT$, $SERVICEPERFDATA$, $SERVICEACKAUTHOR$, and $SERVICEACKCOMMENT$.

There are ways to escape some characters in certain situations, but escaping the ampersand doesn’t work here – especially since the URLs are taken literally (and uninterpreted) in other situations.

The resolution to this problem is to bypass the macro altogether and use the environment variables instead. Replace the macro with its environment variable equivalent. For example, replace:

$HOSTACTIONURL$

in the command_line entry with this:

$${NAGIOS_HOSTACTIONURL}

With this, the string that is actually passed to the shell is:

${NAGIOS_HOSTACTIONURL}

which then is interpreted as a shell environment variable (which, in this case, is set by Nagios). The squiggly brackets are possibly irrelevant, but they don’t hurt.

Duplicate Passwords? Just Say No!

With the recent server compromise at Automattic (the fine people behind wordpress.com) and the compromise of the commenter accounts at Lifehacker not that long ago, I’ve been having to deal with changing passwords. Using Lastpass makes it much easier; I can just imagine how hard it would be without Lastpass.

The nicest thing about having Lastpass store accounts is that you don’t have to try and remember all the places you used a particular password. Using the Lastpass audit feature, it will scan your list of stored passwords and determine which web sites have duplicate passwords, and you can change them all – one by one, of course.

One other nice thing about Lastpass is their Android application – it provides all of the nice capabilities of their browser extensions but on the small Android platform and for all applications as well.

With Lastpass you can have a different random password for all of your web service accounts – and don’t even have to remember them. Just remember your Lastpass account password and you are good to go.

Without Lastpass… I dread the thought of having to remember all the web services, much less try to change all the passwords to go with them.

Side note to web service designers: Could you make it just a little easier to change passwords? Please? Several services had the change password selection hidden quite well – strangely, msn.com was one of these! Yahoo.com wasn’t any better – and Yahoo has the most exasperating password setup I’ve ever seen – requesting your password almost every two minutes. There simply must be a better way.

Making the Case for IPMI

IPMI is a system that allows you to maintain a system where it would not otherwise be able to. However, you have to convince others that it will be useful. As obvious as it is for a professional system administrator, there are others who will not see the usefulness of IPMI. This process – making the case for a technology – comes up more than most system administrators might realize.

There are numerous situations that require having a person to be actually present at a machine to do things – such as entering the setup, accessing the UNIX/Linux maintenance shell, changing boot devices, and so forth. So how do you prove how beneficial implementing full IPMI support is?

Provide a business case - and perhaps an informal user story - to show how IPMI can reduce the need for a person to actually be present.

To really make the case for IPMI, compute the actual costs in making a trip to the data center – the hourly cost of the administrator, the driving costs in gasoline (both to and from!), and the costs associated with handling the expense report. Report the other unquantifiable costs – the cost of an administrator unavailable for other tasks during the four hour round trip, including projects being delayed and problems not getting resolved. Combine this with a user story.

For example, create a user story around a possible kernel panic. A user story requires an actual user – an individual – whose story is followed. Here is our example continued as a user story:

Alma received an email that a system (db20) was unresponsive. Checking the system found that there was no response from the network at all, and checking the KVM showed that there was a panic on the display. No keystrokes were accepted by the system, and there was no way to power-cycle the system.

So Alma sends an email stating that she will be unavailable for the rest of the day, and calls her babysitter to take come and take care of her three children for the evening. Then she gets into her car and drives the two hours to the data center and parks in a lot ($10 for one hour). She power-cycles the machine by pressing its front panel power button, and checks the system response using her laptop. She finds that the server is responding and logs in.

Then Alma checks the server: logs show no problems restarting, the system has restarted cleanly, the subsystems are all running, and the monitoring system shows all systems are good.

Alma leaves the data center and drives the two hours back home.

If Alma is paid $60,000 yearly, the cost of her time spent on this event is US$144.23. If she drove 320 miles round trip at .76 cents a mile, she gets US$243.20 as an expense – in addition to the US$10 in parking fees. This makes a total direct cost of US$397.43.

If something like this happens six times a year, then the total yearly cost is US$2384.58 – and total downtime for the server is 24 hours for an uptime rate of 99.72%.

This account doesn’t include the indirect costs – such as projects being delayed because Alma was unable to work on them, nor does it include the personal costs involved such as babysitting and time away from family. It also doesn’t include the time that HR staff spent on yet another expense report. It also doesn’t include the costs associated with the server being unavailable for three hours.

On the other hand, Polly received word that another server in the data center was unresponsive, and also found that the kernel had panicked and there was no response from the console. She then used a command line tool to access the baseboard management controller (BMC) through IPMI. With an IPMI command, she rebooted the server, and watched the response on the KVM. Checking the system over, Polly found that the server had booted cleanly, subsystems were operational, and all looked good.

If Polly is paid the same amount as Alma, and her response took 15 minutes, we get a total cost of US$7.21.  Downtime was reduced by 92% (along with an associated reduction in costs tied to the server being down). If this happens to Polly six times a year, the total yearly cost is US$43.27 – and a downtime of 1.5 hours for an uptime rate of 99.98%.

Thus, IPMI and SoL would have saved Alma’s company US$2377.37 per year.

The strongest case can be made if a recent event could have been solved with the technology you are proposing. If you can point to a situation that could have been resolved through ten minutes instead of hours or days without – then the usefulness of the technology will be apparent.

With this user story and business case, the case for IPMI should be readily apparent to just about anybody. Similarly, the case can be made for other technologies in the same way.

Making Changes One Small Habit at a Time

I’ve been reading a book titled small change: Little Things Make a Big Difference and have enjoyed it tremendously. The book is written in a conversational style, with a couple as narrators.

The book focuses on the Japanese concept of kaizen, although they don’t really talk about kaizen very much. The idea is that we should make one small change each month, and do this on a regular basis. With a small change, it becomes a habit and can have a dramatic effect on the rest of our lives.

As an example, let’s say one changes from a soda a day to a glass of water every day. If you have a 12oz. soda (at US$1) each day – and switch to visiting the bubbler - then you will save US$365 in one year, and US$1825 in five years. What would you do with all that money?

Taking the same switch as an example, you would also save 140 calories each day – or 51,100 calories a year – or 255,500 calories in five years.

When you try to do too much, you can become overwhelmed and your attempts then suffer across the board. You find that you are failing in one area, and the negative reaction spills into all of the other habits you are trying to create – and none are successfully created.

Also, when you have several habits going at once, you may find that you are improving slightly – across the board – but not in any one area. None of the habits take, because you keep switching one habit for another and never completely creating a new habit.

When you succeed in one habit, the drive will propel you to succeed in another – and another.

Jason Thomas wrote an article in Lifehacker titled Practice Your Personal Kaizen which covers some of these areas. Leo Babauta in his book Less also covers the concept of changing just one habit each month. A related book (which I want to read) is One Small Step Can Change Your Life: the Kaizen Way by Robert Maurer.

What new habit are you going to create this month?

An Easier Way to Mount Disks in Linux (by UUID)

When mounting a disk, the traditional way has always been to use the name given to it by the operating system – such as hda1 (first partition on first hard drive) or sdc2 (second partition on third SCSI drive). However, with the possibility that disks may move from one location to another (such as from sdc to sda) what can be done to locate the appropriate disk to mount?

Most or all Linux systems now install with the /etc/fstab set up to use the Universally Unique Identifier (UUID). While the disk’s location could change, the UUID remains the same. How can we use the UUID?

First, you have to find out the UUID of the disk that you want to mount. There are a number of ways to do this; one traditional method has been to use the tool vol_id. This tool, however, was removed from udev back in May of 2009. There are other ways to find the UUID of a new disk.

One way is to use blkid. This command is part of util-linux-ng. The vol_id command was removed in favor of this one, and is the preferred tool to use instead of vol_id. Find the UUID of /dev/sda1 like this:

$ blkid /dev/sda1
/dev/sda1: UUID="e7b85511-58a1-45a0-9c72-72b554f01f9f" TYPE="ext3"

You could also use the tool udevadm (which takes the place of udevinfo) this way:

$ udevadm info -n /dev/sda1 -q property | grep UUID
ID_FS_UUID=e7b85511-58a1-45a0-9c72-72b554f01f9f
ID_FS_UUID_ENC=e7b85511-58a1-45a0-9c72-72b554f01f9f

Alternately, the tune2fs command (although specific to ext) can be used to get the UUID:

# tune2fs -l /dev/sda1 | grep UUID
Filesystem UUID:          e7b85511-58a1-45a0-9c72-72b554f01f9f

The tune2fs utility is also only available to the superuser, not to a normal user.

There are utilities similar to tune2fs for other filesystems – and most probably report the UUID. For instance, the XFS tool xfs_info reports the UUID (in the first line) like this:

$ xfs_info /dev/sda6
meta-data=/dev/disk/by-uuid/c68acf43-2c75-4a9d-a281-b70b5a0095e8 isize=256    agcount=4, agsize=15106816 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=60427264, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=29505, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

The easiest way is to use the blkid command. There is yet one more way to get the UUID – and it also allows you to write programs and scripts that utilize the UUID. The filesystem contains a list of disks by their UUID in /dev/disk/by-uuid:

$ ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 2011-04-01 13:08 6db9c678-7e17-4e9e-b10a-e75595c0cacb -> ../../sda5
lrwxrwxrwx 1 root root 10 2011-04-01 13:08 c68acf43-2c75-4a9d-a281-b70b5a0095e8 -> ../../sda6
lrwxrwxrwx 1 root root 10 2011-04-01 13:08 e7b85511-58a1-45a0-9c72-72b554f01f9f -> ../../sda1

Since a disk drive is just a file, these links can be used to “find” a disk device by UUID – and to identify the UUID as well. Just use /dev/disk/by-uuid/e7b85511-58a1-45a0-9c72-72b554f01f9f in your scripts instead of /dev/sda1 and you’ll be able to locate the disk no matter where it is (as long as /dev/disk/by-uuid exists and works).

For example, let’s say you want to do a disk image copy of what is now /dev/sda1 – but you want the script to be portable and to find the disk no matter where it winds up. Using dd and gzip, you can do something like this:

UUID=e7b85511-58a1-45a0-9c72-72b554f01f9f
dd if=/dev/disk/by-uuid/${UUID} of=- | gzip -c 9 > backup-$UUID.gz

You can script the retrieval of the UUID – perhaps from a user – this way:

eval `blkid -o udev ${DEV}`
echo "UUID of ${DEV} is ${ID_FS_UUID}"

In the /etc/fstab file, replace the mount device (such as /dev/sda1) with the phrase UUID=e7b85511-58a1-45a0-9c72-72b554f01f9f (or whatever the UUID actually is).

Be aware that the UUID is associated with the filesystem, not the device itself – so if you reformat the device with a new filesystem (of whatever type) the UUID will change.