Category: BSD

A New Init for Fedora 14

Apparently, a new project (to replace init, inetd, and cron) named systemd is nearing release and will be used to replace upstart in Fedora 14 (to be released in November – with Alpha Release due today!).

There is a healthy crop of init replacements out there, and the field is still shaking out. Replacing init – or specifically, System V init and init scripts – seems to be one of those never-ending projects: everyone has an idea on how to do it, no one can agree on how.

Let’s recap the current crop (excluding BSD rc scripts and System V init):

I am still waiting for the shakeout – it bugs me that there are dozens of different ways to start a system, and that none of them have taken over as the leader. For years, BSD rc scripts and System V init have been the standard – and both have stood the test of time.

My personal bias is towards SMF (OReilly had a nice article on it) and towards simpleinit – but neither has expanded like upstart has.

So where’s the replacement? Which is The One? It appears that no one is willing to work within a promising project, but rather starts over and creates yet another replacement for init, fragmenting the market further.

Lastly, if the current init scheme is so bad, why hasn’t anything taken over? Commercial UNIX environments continue to use the System V scheme, with the sole exception of Solaris which made the break to System Management Facility (or SMF). Why doesn’t HP-UX or AIX use SMF or Upstart if the current environment is horrible?

Sigh. It’s not that the current choices of replacement are bad – it’s just that there are so many – and more keep coming up every day. Perhaps we can learn something about the causes of this fragmentation from a quote from a paper written about the NetBSD rc.d startup scripts and their design:

The change [in init] has been one of the most contentious in the history of the [NetBSD] project.

Network Attached Storage (NAS)

Once you hear what a NAS appliance does, you might be tempted to think (as I did) what all the fuss might be about. But there are reasons for a NAS appliance, though a NAS isn’t for everybody.

Network Attached Storage is nothing more than a server with a pile of disks and a dozen different ways to access them. For most intents and purposes, the difference between a File Server of yesteryear and the Network Attached Storage of today is conceptually rather minimal.

NAS typically provides access to files via such methods as Windows shares, NFS, iSCSI, Appleshare and others.

So what does a NAS appliance provide that a NFS server does not? There are several benefits:

  • Special purpose. Since the system is solely for the purpose of serving up files for users, there is no need for any other facilities except those that deal with its specified purpose. Thus, a lot of potentially vulnerable or unreliable code can be removed, and the speed and reliability of the system can be increased. Some systems do not come with a general purpose operating system of any kind, but rather a specially designed operating system for serving files alone.

  • Extensive support. In many cases, since the system is specifically designed for serving up file storage, the innumerable variations of network storage protocols come supported out of the box.

  • Ease of use. With the system designed to serve one purpose – and to provide the customer with the best possible experience – the system is generally made much easier to configure and easier to use than having to configure the varying servers and protocols independently.

There are two different NAS products that are the heavy-weights in the free and open source arena: FreeNAS (freenas.org) and OpenFiler (openfiler.com).

The most obvious difference between these two is their base (and their associated licenses). The base for FreeNAS is FreeBSD, and like FreeBSD, is licensed using the BSD license. However, OpenFiler uses Linux as its base, and is likewise covered by the General Public License version 2.

This week, I’ll focus on FreeNAS with the assistance of a book entitled Learning FreeNAS by Gary Sims and published by Packt Publishing.

Powered by ScribeFire.

Using BSD jot

Most every Linux system comes with GNU seq (a utility to generate sequences of numbers or characters). However, a much older utility – and more flexible one – is the underutilized and unknown utility, jot. Every BSD system, including MacOS X, will come with jot (and not with seq).

Unfortunately, commercial UNIX tends to come with neither. So unless you are using an open source operating system, both of these utilities are unavailable. This is unfortunate.

To install jot under Debian Linux and variants (including Ubuntu), install the athena-jot package. FreeBSD, OpenBSD, and MacOS X should already have jot installed out of the box.

Using jot is easy:

  • Generate a count 1 to 6: jot 6
  • Generate a count 5 to 10: jot 6 5
  • Print “y” 100 times: jot -b y 100
  • Generate a count 1 to 6 separated by commas: jot -s "," 6
  • Generate abc1 to abc20: jot -w abc 20

This only begins to touch on the capabilities of jot. It can also generate random numbers and random character data.

In fact, to generate a large file (5Gb in this case), try this:

jot -r -c -s '' $(( 1024 * 1024 * 5 )) > file.5gb

If you do install the athena-jot package on Ubuntu or Debian, you may want to do this:


cd /usr/share/man/man1
sudo ln -s athena-jot.1.gz jot.1.gz

For some reason, the package – and the manpage – are called athena-jot but the utility is actually jot (not athena-jot). The FreeBSD man page for jot(1) (PDF)has more details.

Using OpenSSH public keys with OpenVMS (and SSH2)

Every Linux box comes with OpenSSH, as does every BSD (at least, I don’t know of any that don’t). UNIX almost invariably comes with OpenSSH as well (although Tru64 is a dying hold-out).

However, OpenVMS 8 comes with an implementation of SSH that appears to be very similar to the commercial SSH produced by F-Secure and distributed by Attachmate.

To use your OpenSSH keys with VMS, a conversion is required – and the configuration files are different as well.

The versions of SSH used are:

VMS $ ssh "-V"
$1$dga010:[sys1.syscommon.][sysexe]tcpip$ssh_ssh2.exe: SSH Secure Shell OpenVMS
(V5.5) 3.2.0 on HP rx2660 (1.40GHz/6.0MB) - VMS V8.3

UNIX $ ssh -V
OpenSSH_5.1p1, OpenSSL 0.9.8g 19 Oct 2007

First, the key must be converted. Using OpenSSH this is easy (assume that the OpenSSH key my_key.pub already exists):

$ ssh-keygen -e -f my_key.pub > my_vmskey.pub
$

Then copy this file to your VMS login directory, putting it into the directory [.ssh2] (relative to your VMS login directory). Let's assume that the UNIX host is named openssh:

VMS $ scp openssh:~/my_vmskey.pub .
VMS $

Then edit the file AUTHORIZATIONS to include this line:

key my_vmskey.pub

Then be sure to test it out from your OpenSSH host:

openssh $ slogin vms
The authenticity of host 'vms (10.6.4.4)' can't be established.
DSA key fingerprint is 46:43:2f:df:0a:4b:37:e2:4f:f9:63:05:21:84:c9:44.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vms' (DSA) to the list of known hosts.
Welcome to HP OpenVMS Industry Standard 64 Operating System, Version V8.3

$

Note that since you've not logged into the VMS machine previously, it must have its fingerprint added to the file known_hosts.

There is a good article about key conversions that seems to cover all the bases.

The Geek Stuff blog also has an excellent article on connecting to a SSH2 server using OpenSSH, as well as excellent articles on setting up SSH2 for logins and setting OpenSSH for logins.

Of course, the definitive book is SSH: The Definitive Guide by by Daniel Barrett, Richard Silverman, and Robert Byrnes. This book is also now online.

Why FreeBSD is (and isn’t) My Favorite Operating System

Over at Webmasters by Design there was a very interesting article by Scott Spear about Why FreeBSD is My Favorite *nix OS. Like him, I find FreeBSD to be wonderful and like it a lot. However, I find that I don’t want to use it for everything.

Why I Like BSD: Small Footprint

FreeBSD works in many, many, many more places than Linux – and even more places than Solaris. There just isn’t a lot of overloaded kernel involved. As kernels grow faster and faster, it is refreshing to be able to use something not so bloated.

Why I Like BSD: History

Unlike Linux, BSD goes back a very long ways (longer than Solaris even) and is UNIX. It is possible that with the exception of Unixware and NetBSD, no other UNIX system has as much of a history. Some of the original developers are still involved in FreeBSD (Marshall McKusick comes to mind).

Why I Like BSD: Cohesiveness

No matter how hard they try, a Linux distribution can’t match the overall cohesiveness of one of the BSD systems (such as FreeBSD). Some Linux distributions are very well done, but they still have “missing parts” – usually documentation.

Why I Like BSD: Documentation

There is nothing that matches FreeBSD documentation in the Linux world. Once, I ran a test: I wrote a script to test for the existance of man pages for every binary in the usual locations on the system (/bin /sbin /usr/bin /usr/sbin). Red Hat Linux come up with a number of programs that were undocumented; FreeBSD did not have a single missing man page.

All of the kernel tunables can also be found in man pages, and more.

The FreeBSD Handbook is phenomenal, and a valuable resource. Linux environments don’t have anything like it.

Why I Don’t Like BSD: Linux (In)Compatibility

Linux compatibility fails as often as it succeeds, and it is more of a simulated environment than it is just a compatibility layer. It doesn’t work, it’s bloated and it’s wrong to rely on it in any case.

Why I Don’t Like BSD: Flash et al

Getting to use Flash in FreeBSD is a nightmare. Even following the directions is no guarantee that it will work. Distributions such as OpenSUSE and Ubuntu come ready to plug in Flash support, and Adobe has specified that they will support Linux. That leaves out FreeBSD.

This may be better in PCBSD; I aim to try it soon.

Why I Don’t Like BSD: Installation

Sysinstall is not the easy install process that installation of distributions like Ubuntu and OpenSUSE is. There are way too many technical details to comprehend.

This probably has improved with FreeBSD 7; I’ve not yet tried FreeBSD 7.

Is FreeBSD a better choice for the desktop? (or dispelling myths)

It’s strange I should come across this article in one of my favorite blogs just after I switched from my FreeBSD desktop to Kubuntu. I’m also surprised at the lack of knowledge and the propagation of some long-standing myths about Linux and FreeBSD for that matter.

There are some ways that FreeBSD (or better put, BSD) is better than Linux – but the comparisons must be valid and appropriate without myths and falsehoods.

Perhaps the primary myth is that FreeBSD is a complete operating system and Linux is a boat-load of different distributions in all different flavors with different setups and so on. However, FreeBSD also has a large number of alternatives, including OpenBSD, NetBSD, PCBSD, DesktopBSD, PicoBSD, and Dragonfly BSD to name just a few.

Another comparison is that FreeBSD is put together by the FreeBSD Core team and that this is better than Linux (which has a “benevolent dictator” model). There’s no discussion of OpenBSD, for instance, which also follows this “benevolent dictator” model. There’s also no comparison to Red Hat Enterprise Linux, for example, which has a large number of people working towards putting together a complete distribution, not just the kernel.

The documentation is definitely an argument in favor of BSD – virtually everything that is in the system anywhere is documented in the online documentation, and the FreeBSD Handbook is without equal. It can be proven programmatically that there are commands in Red Hat (or other distributions) that are not documented. I daresay that the FreeBSD documentation beats other BSD variants as well.

Another benefit of FreeBSD specifically is the vast number of ports available. There are more ports for FreeBSD than any other system but Debian GNU/Linux. The sheer amount of packages available in both environments has made them appealing to me – and perhaps to others. Where else are you going to get Steel Bank Common Lisp for example? Both Debian and FreeBSD have it.

The article specifically asked about FreeBSD for the desktop: FreeBSD is definitely not ready for the desktop at all. When I installed it for my desktop (twice now), the basics are there certainly – but there were numerous problems that I had to overcome. Among them, I had to set up my own system bootsplash, and had to configure and set up my own login screen (kdm). USB devices plugged in weren’t properly recognized. Hibernation and sleep didn’t work. Flash doesn’t work. Unlike what has been said before, the drivers are much less available than they are for Linux: hardware manufacturers don’t see a need to support BSD, and many new UNIX users (and developers) don’t see a need to use anything but Linux. Wireless support is perhaps an exception, but that development is centered in OpenBSD, not FreeBSD.

There is also, in my mind, a benefit to BSD that goes often unmentioned: it has the smallest kernel of the open source UNIX and Linux kernels out there today. FreeBSD and OpenBSD will run in smaller environments that Linux won’t: on my 512M laptop, a Compaq Armada E500, Fedora 5 would crash during the install (not enough memory) – whereas the much more current FreeBSD 6.2 installed just fine.

Now, when I installed Kubuntu onto a Compaq nc4010 with 1G of memory, it went will – and it recognized everything – wireless, hibernate, bluetooth, USB devices, PCMCIA, video display, power capabilities, etc. – all without special configuration. (I might note that, here too, on this machine Fedora crashed – this time the Live USB Fedora 9 crashed during exit – sigh…) Preconfigured and tested support for Flash, Java, and MP3s was a click away.

When it comes to the desktop, FreeBSD has a long way to go (perhaps PCBSD is a lot better?). However, on the server end, I would propose that FreeBSD is a better way to go than Linux in many cases (except for OpenBSD might, in my opinion, be even better). It is unfortunate that none of the BSD variants are often considered for enterprise server use – especially considering FreeBSD is commonly found in NetCraft‘s list of top uptime.

Using sed: the next level

The utility sed is much more powerful than most people realize. It is underutilized by many.

Consider this: have you ever seen Tetris written in sed? Or Sokoban? Or Arkanoid?

While these games are not comparable to the originals or those that take full advantage of the computer, they prove one point: sed is very powerful.

Another example: the LEAF distributions which run on a floppy or CDROM all use sed to do a lot of the work in the distribution, emulating other much more powerful commands such as grep and who.

Here are some tips for using sed:

  • Don’t use grep, awk, and sed together. Either awk or sed can replace grep utterly; no need to spawn another process. Grep can be replaced with sed by using /pat/p inside of a sed command with the -n (don’t print by default) option.
  • Use the d command to sed to “delete” lines from the output. This will allow you to strip out lines that are not wanted.
  • Use multiple sed commands – by separating each with a semicolon (;) character. One sed process, multiple commands – don’t use multiple sed processes.
  • Want to do some substitutions and other things to only particular lines? Use a construct like this: /patt/{ ...do stuff.. }
  • Learn about the hold space and how to use it. This allows you to take matched patterns from one part of the file and put them elsewhere later.
  • Want to work on more than one line at once? Use the N command: this will append the next line into the current pattern space (with embedded newline). Thus, matching a line with EOF at the end and a blank line on the next would be like this: N; /EOF\n$/d; (this matches the two lines and deletes them…)

One more thing: get the sed and awk book: people are so used to wunderkind like Perl and Ruby that they forget what the originals are really capable of.

GNU find tips (again!)

I am reminded of one fact (by this post) about find that you may not be aware of (or have forgotten). The find command, by default, does not follow symbolic links: they are just another file for find to scan.

In order for a symbolic link to be recognized and used in its usual manner, the -follow option is needed.

This brings up another point about find that will clear up some confusion, even for experts. The find command mixes standard options (which change its overall behavior) with conditionals (picking and choosing files). Thus, some options can occur anywhere and change the operation of find as a whole; others are used in an implicit command line conditional. The man page calls these options “position-independent terms”.

So, even though a find option appears after the -a option (AND operator) it could actually be a true option to find, changing the command’s behavior (and thus not be relevant to the -a “option”).

The previously mentioned -follow option is a true option. However, options like -name, -a, -o, -size are all conditionals. If you keep this in mind, the occasional option in the middle of a conditional will not throw you.

The HP-UX man page for find lists the following position-independent terms:

  • -depth
  • -follow
  • -fsonly
  • -xdev

Let’s consider some more ideas on how we can use find to help us daily.

If this hasn’t been mentioned already, the -nouser and -nogroup options can be quite useful in a cron job for finding files that don’t have a proper user or group associated with them (these files are a security risk):

find / -nouser -o -nogroup -print

As we discussed earlier, using find in automated mode with an rm command is not a good idea – but at the command line, there is a replacement for -exec that will provide you with a safety net: -ok:

find . -name "f*" -ok rm -f {} \;

This command is functionally equivalent to:

find . -name "f*" | xargs rm -i

Both commands will prompt you as to whether you want to erase the file found or not.

The option -xdev is a favorite of mine: don’t follow any mount points. So if you want to search /tmp for various files but have things mounted in /tmp/mnt or other locations, this command will allow you to skip files under such mountpoints:

find /tmp -xdev -name "foo*"

This will find /tmp/foobar – but not /tmp/mnt/fooey.

An enhancement to the -exec command can improve performance – and reduce the need for xargs. Instead of terminating the command string with a semicolon (;), terminate it with a plus sign (+) right after the {} which signifies the location of the filenames in the command line:

find /tmp -name "f*" -exec ls -ld {}\+

The plus (+) is escaped, just like the semicolon would be, since the shell recognizes both characters as special characters; the backslash makes the shell treat them like any other character.

Using OPIE

Setting up OPIE (One-time Passwords In Everything) in OpenSUSE was easy: there is a opie RPM in the standard repository, and it installs cleanly and easily.  Then it is just a matter of initializing the database and modifying the PAM configuration to match.  Then each user is added to the database (/etc/opiekeys) one at a time.  I’ll describe the exact process on OpenSUSE at a later time.

Insufferingly, it appears that Fedora (and Red Hat) do not offer any form of one-time passwords anywhere – and certainly not OPIE.  RPMs for opie are exclusively for OpenSUSE and for the Polish PLD distribution (both of which seem to have everything).  How extremely frustrating!  This sounds like a good time to switch my home system from Fedora 5 to OpenSUSE 10.3.

OpenSUSE has supported LVM, XFS, KDE, and many other technologies when Red Hat staunchly refused to.  Even now, OpenSUSE support for all of these is much more integrated and time-tested than Red Hat’s.

Lest I sound like I hate Red Hat – I don’t – and that’s what makes it so frustrating.  Grrr….

The search for one-time passwords for HP-UX and for OpenVMS was even more fruitless.  HP-UX apparently has a third party skey package available; OpenVMS has nothing – though it could be added through programming the ACME interface (which provides similar capabilities to PAM – though perhaps not as flexible).

It looks like the BSDs aren’t a lot better: FreeBSD has OPIE built into the core (with a full section on OPIE in the FreeBSD Handbook on it); NetBSD and OpenBSD do not appear to have it (!).

Looks like my settling in to FreeBSD and OpenSUSE has paid off.  I don’t even need to suggest Debian – Debian has everything – and OPIE is no exception.  And of course, Ubuntu follows suit as well.

SystemTap (and DTrace)

SystemTap is one amazing piece of work – it is a programmer-friendy and admin-friendly interface to KProbes (which are included in the Linux 2.6 kernel).  When you compare its capabilities to what has gone before, it is truly amazing.  Here are some of the things you can do:

  • Quantify disk accesses per disk per process (or per user)
  • Quantify the number of context switches that are a result of time outs
  • List all accesses to a particular file and the process that accessed it

This is only the tip of the iceberg. There is a wiki with more details, including “war stories.”  There is a language reference there as well.

There was an excellent article in Red Hat Magazine, “Instrumenting the Linux Kernel with SystemTap” by William Cohen.

One controversy that came up was that the initial impetus for creating SystemTap was to implement something like Sun’s DTrace for Solaris but under the GNU Public License.  Solaris and DTrace are licensed with Sun’s Common Development and Distribution License (CDDL), which many feel makes DTrace incompatible with the GPL-licensed Linux kernel.

Apparently, the CDDL is also incompatible with the BSD-licensed FreeBSD, as FreeBSD 7.0 will not have DTrace either.  There appears to be some licensing issues.

According to the Wikipedia entry on the CDDL, it was designed to be both GPL-incompatible and BSD-incompatible.  With regard to the GPL, the entry suggests that Sun never clarified why; as to the BSD, Sun did not want Solaris to wind up in proprietary products – which the BSD license allows.

On a brighter note, Eugene Teo was able to get the SystemTap tool to work on the Nokia N800.  The article seems to be behind a wall at LiveJournal; the article is still in Google’s cache.  However, it does requires some amazing convolutions:

  • A kprobes-enabled kernel must be installed on the N800
  • The SystemTap programs (like stap) must be installed on the N800
  • Any traces must be cross-compiled on another host
  • The kernel module thus created must be moved to the N800
  • Once the kernel module is in place, then the trace can be done.

So every desired trace requires precross-compilation on a desktop (sigh)…  Oh, well.

There is even a GUI for SystemTap in the works.