UNIX Administratosphere

Things are working well with the combination of the new FreeBSD 6.3 and the Compaq Armada E500. The machine has a great feel to it, and despite the huge applications of today, 128M can still be used for a KDE environment.

There are a number of nice features, including a ton of connectors (10BaseT, WinModem, USB, serial, parallel, PS/2, PCMCIA). The machine just keeps going, though I have had a few (few!) lockups (normally with Amarok and something else running). With the appropriate tweaks, the KDE desktop can be as polished as any from Red Hat or Novell.

Some of the things I did:

• Replace the shutdown picture with something else; the picture of the dragon was too smarmy.
• Replace the background (of course!) - personalization to the max.
• Switch to the “Macintosh” version of menu layout; it’s the most user-friendly.
• Load KDM from ports, then activate and theme it.
• Load a splash screen for the boot loader
• Switch the boot loader to grub then use a splash screen to start it off.
• Configure the special buttons to work.
• Change the KDE menu for something easier to use (such as TastyMenu or KBFX).

When all of these are combined, the environment is very slick and professional. It still wants more than 128M though.

One pet peeve of mine I might mention with regards to menus (such as KMenu or KBFX): menus should respond instantly!! I absolutely despise hiccups and watch cursors because the menu is loading its stuff. It should just pop! into place, not thrash the hard disk. Maybe one day…

What was the best part of this? I learned a ton about themes, X keys, configuring KDM, configuring the boot loader, and using grub. And learning is the best part, right?

FreeBSD is a nice environment, and I tend to gravitate to it (though I love Linux and Solaris as well). It does tend to work better in smaller environments than either Linux or Solaris.

There was recently a discussion of FreeBSD on the EeePC; it appears that while some items do not work (to be expected) it runs nicely and works nicely (including wireless). There was recently posted a simple introductory article which also refers to a comprehensive article on FreeBSD on the EeePC.

There was also an article (with followup) about running FreeBSD on the fitPC; in contrast to the EeePC, this sounds like it is not as good. However, the fitPC has less memory and a slower processor; it is unclear as to whether the processor is “fast enough” (I still use Pentium IIIs for my use!) or if it really is slow. It is, however, very surprising that the default Ubuntu install would be a graphical installation that swaps badly and comes without SSH.

The fitPC forums have a nice Linux on fitPC section, which also includes the BSDs as well. The biggest problem with FreeBSD seems to be its lack of a USB CDROM driver in the base kernel; however, apparently OpenBSD loads fine. Since the system has only 256M of memory, it perhaps should not be swamped with heavy desktop applications.

Just after installing 6.3-RC2, I discovered that 6.3 was officially released!

This didn’t take much work to update to. The basic steps were:

freebsd-update upgrade
freebsd-update install

Then - after a reboot (for kernel updates, I presume) another:

freebsd-update install

Relatively painless, throughout.

Then, continuing my install over the weekend, there were a few niggling things to fix. First off, the buttons up on the top of the keyboard didn’t work (no surprise there). Using the xev utility helps to pin down the actual keycodes for these keys, then use the xmodmap tool to add the appropriate actions to the keys. In my case, xev reported that the keys left to right were:

• 163 (info key)
• 142 (home key)
• 154 (search key)
• 143 (mail key)

These can be configured using xmodmap and a .Xmodmap file configured this way:

keycode 163 = Help
keycode 142 = XF86HomePage
keycode 154 = XF86Search
keycode 143 = XF86Mail

The values on the right (Help, XF86HomePage, XF86Search, XF86Mail) show their XFree86 heritage, but apparently do not change for X.org. These are activated by using the command:

xmodmap .Xmodmap

A good place for this line would be in the .xinitrc. However, once this is set, it is still necessary to tie applications to the shortcuts listed. In KDE, this is done in the Keyboard Shortcuts section of the Regional and Accessibility pane in Settings. In this dialog, select the “Command Shortcuts” tab, then the application you desire to use. For example, “Find Files/Folders” could be attached to the shortcut XF86Search. Once the xmodmap has been modified using the command above, then click on the Custom radio button, and click the shortcut button. Press the actual shortcut button to define the shortcut for the application.

Do this for all four buttons, and all will be well.

Then there was the problem of the mouse not being operational when waking up from a suspend. Turns out that the moused( daemon is the culprit. Sending a HUP signal to the daemon fixes it, but having to do this all the time is not a desirable outcome. The utility acpiconf( describes how it uses /etc/rc.suspend and /etc/rc.resume before and after suspending the system. I placed the command:

pkill -HUP moused

into this script, but I don’t yet know if it is truly having the desired effect or if other things are causing failures.

Another thing: the CD player would not play CDs in Amarok. Apparently, this is due to HAL and DBUS not being available. HAL depends on DBUS, so both are necessary. The following packages were needed:

hal-0.5.8.20070909
dbus-1.0.2_2
dbus-glib-0.74
dbus-qt3-0.70_1

I don’t know that all the dbus packages are necessary, but I decided not to chance it. Of course, dbus is part of GNOME, but whatever. Once these packages are installed, add the following to the startup configuration in /etc/rc.conf:

hald_enable="YES"
dbus_enable=”YES”

Next time the system boots, these daemons will start.

Also, up until this point X.org had to be started using startx as a normal user. However, adding the login screen isn’t difficult. Edit the /etc/ttys file. In this file, there will be a line that specifies the command xdm.

Since kdm (the KDE display manager, the login screen) is actually a reworked xdm, switching one for the other is smooth and clean. Replace the xdm setting with /usr/local/bin/kdm (keeping the -nodaemon option) and set the tty to “on” (instead of “off”). Then the next time the system starts (or the ttys file is read) a login screen will activate on ttyv8 (if your file is like mine).

FreeBSD 6.2 has been on this machine for a while, but then I tried to upgrade all of the applications using the ports tree. This almost worked, except upgrading to Xorg turned out to be a massive headache and nothing worked.

It was then that FreeBSD 6.3 RC-2 was announced. I thought, why not? So off I went.

It installed well - if you don’t count my not providing enough room for /usr/local. With my “full-featured” (ha!) list of software, I wound up needing more than the original 2 Gb I originally alloted for /usr/local; with 4 Gb it worked. I also had to change the boot options, as it was still set to use 6.3-RC1 instead of 6.3-RC2. Changing the name in the options screen worked just fine.

Then after loading, I had to load the proper kernel - it couldn’t find the kernel. I selected /boot/GENERIC/kernel and all was well. At the boot loader prompt:

load /boot/GENERIC/kernel
boot

I had to configure Xorg. This was another headache. There was an excellent article from Julien Valroff about instaling Debian GNU/Linux on this machine. Despite the difference in operating systems, the fundamentals were similar. Another fantastic resource was this old page by Frank Steiner. Despite the age, the descriptions are relevant and useful (though, again, it is about Linux). There is a page on the Gentoo Wiki that describes the machine as well, though the other pages are more descriptive.

The screen display descriptions turned out to be the easiest; the problem was the mouse. Some descriptions suggest that the synaptics driver should work. However, this never did work for me. Using the standard PS/2 mouse driver and protocol worked just fine.

I also had to up the maximum files available, though for what reason I forget. Add this line to /etc/sysctl.conf to fix this problem:

kern.maxfiles=10000

Sound was another matter. It took a bit to figure out. First off, all the Linux directions suggested using lspci to see if it was there; this is Linux-specific. The FreeBSD counterpart is pciconf. Running pciconf -lv presents this:

pcm0@pci0:8:0:   class=0x040100 card=0xb1120e11 chip=0x1978125d rev=0x10 hdr=0x00
    vendor     = ‘ESS Technology’
    device     = ‘ES1978 Maestro-2E Audiodrive, ES1970 Canyon3D’
    class      = multimedia
    subclass   = audio

Thus, I knew that the sound was recognized. I just had to figure out how to get things to work with it. This means kernel support, da?

First attempts to load a driver turned up short; nothing is found in /boot/modules (!). The search path had to be changed to /boot/GENERIC:

kldconfig -i /boot/GENERIC

After this, load the snd_maestro driver:

kldload snd_maestro

After this, sound will work! Amarok is great…… and sound on this machine is excellent too!

Seeing as a I was trying to load KDE on here, the next step (once Xorg is working) is to add a startkde command to the .xinitrc file (in one’s home directory).

To make the system boot properly (and so you don’t have to load kernel modules manually all the time), the /boot/loader.conf file had to be created with this:

# Directory (in /boot) containing kernel and modules
kernel=”GENERIC”
 
# Load maestro driver
snd_maestro_load=”YES”

This then worked well.

I’m enjoying this machine again - though I am attempting to make it more of a usable desktop, which means more memory and all of the niggling setup work - like bootup splash screens, configuring kdm, and more - but hey, we’re system admins here, right?

I’ve been seriously considering the fitPC for my own purposes. Currently, the DHCP server (ISC) and the web proxy (Squid) in my home network are two separate servers running versions of Red Hat or CentOS. Both are slimline computers, but otherwise are standard in their configuration: standard hard drives, standard motherboards, etc. I think it might be safe to assume that this also means standard power consumption (200-250 watts, I’m guessing).

Along with the wattage is the noise as well. With the two servers and two desktops running (and a rackmountable switch), that is a lot of fans and a lot of noise (more than non-techies would realize!).  That’s four computers with 200-250 watts each (high estimate) - for 800-1000 watts of power consumption.

This doesn’t even take into account what it would be like with the Sun SparcStation 20 or the UltraEnterprise 1 started up - I shudder to think how much power they take up (and I know how noisy they are….).

A 5-watt computer with no fan (silence!) would be very nice. The things that are most appealing about the 5-watt fitPC are the silence, the minimal power drain, and the standard configuration. Many computers that fall into this category are in fact, embedded systems - small memory, small flash drives, and so on - often with specialized software (even if it is open source).

The fitPC is a standard system, so it will run anything that will run on an Intel 586-based system.

The company is in Israel, and until recently was the only source for their product. They appear to have been overwhelmed with orders sometime in November, but are going as fast as they can.  I’ve no idea if they are backed up now or not. They’ve also introduced resellers apparently: System Industrie Electronic AG in Austria, Anders Electronics in the United Kingdom, and CompuLab Embedded Systems in the United States.  None of the resellers has the fitPC listed as available, but presumably if you contact the right people from their list, everything will be just fine.

They have a set of forums that are excellent, and serve as an excellent resource for purchasers and owners of fitPC systems.

One of the questions that comes up on the forums is: Does the fitPC work well with X where X is:

• FreeBSD (only if USB installation is fixed)
• FreeDOS (unanswered so far)
• CentOS (one successful report)
• Ubuntu Server (still trying)
• PCLinuxOS (just fine!)
• Windows 2003 Server (not too badly)

It comes with either Windows XP or Ubuntu (an optimized Gentoo is also available). The listed cost is $285 US.

Setting up OPIE (One-time Passwords In Everything) in OpenSUSE was easy: there is a opie RPM in the standard repository, and it installs cleanly and easily.  Then it is just a matter of initializing the database and modifying the PAM configuration to match.  Then each user is added to the database (/etc/opiekeys) one at a time.  I’ll describe the exact process on OpenSUSE at a later time.

Insufferingly, it appears that Fedora (and Red Hat) do not offer any form of one-time passwords anywhere - and certainly not OPIE.  RPMs for opie are exclusively for OpenSUSE and for the Polish PLD distribution (both of which seem to have everything).  How extremely frustrating!  This sounds like a good time to switch my home system from Fedora 5 to OpenSUSE 10.3.

OpenSUSE has supported LVM, XFS, KDE, and many other technologies when Red Hat staunchly refused to.  Even now, OpenSUSE support for all of these is much more integrated and time-tested than Red Hat’s.

Lest I sound like I hate Red Hat - I don’t - and that’s what makes it so frustrating.  Grrr….

The search for one-time passwords for HP-UX and for OpenVMS was even more fruitless.  HP-UX apparently has a third party skey package available; OpenVMS has nothing - though it could be added through programming the ACME interface (which provides similar capabilities to PAM - though perhaps not as flexible).

It looks like the BSDs aren’t a lot better: FreeBSD has OPIE built into the core (with a full section on OPIE in the FreeBSD Handbook on it); NetBSD and OpenBSD do not appear to have it (!).

Looks like my settling in to FreeBSD and OpenSUSE has paid off.  I don’t even need to suggest Debian - Debian has everything - and OPIE is no exception.  And of course, Ubuntu follows suit as well.

SystemTap is one amazing piece of work - it is a programmer-friendy and admin-friendly interface to KProbes (which are included in the Linux 2.6 kernel).  When you compare its capabilities to what has gone before, it is truly amazing.  Here are some of the things you can do:

• Quantify disk accesses per disk per process (or per user)
• Quantify the number of context switches that are a result of time outs
• List all accesses to a particular file and the process that accessed it

This is only the tip of the iceberg. There is a wiki with more details, including “war stories.”  There is a language reference there as well.

There was an excellent article in Red Hat Magazine, “Instrumenting the Linux Kernel with SystemTap” by William Cohen.

One controversy that came up was that the initial impetus for creating SystemTap was to implement something like Sun’s DTrace for Solaris but under the GNU Public License.  Solaris and DTrace are licensed with Sun’s Common Development and Distribution License (CDDL), which many feel makes DTrace incompatible with the GPL-licensed Linux kernel.

Apparently, the CDDL is also incompatible with the BSD-licensed FreeBSD, as FreeBSD 7.0 will not have DTrace either.  There appears to be some licensing issues.

According to the Wikipedia entry on the CDDL, it was designed to be both GPL-incompatible and BSD-incompatible.  With regard to the GPL, the entry suggests that Sun never clarified why; as to the BSD, Sun did not want Solaris to wind up in proprietary products - which the BSD license allows.

On a brighter note, Eugene Teo was able to get the SystemTap tool to work on the Nokia N800.  The article seems to be behind a wall at LiveJournal; the article is still in Google’s cache.  However, it does requires some amazing convolutions:

• A kprobes-enabled kernel must be installed on the N800
• The SystemTap programs (like stap) must be installed on the N800
• Any traces must be cross-compiled on another host
• The kernel module thus created must be moved to the N800
• Once the kernel module is in place, then the trace can be done.

So every desired trace requires precross-compilation on a desktop (sigh)…  Oh, well.

There is even a GUI for SystemTap in the works.

There are several reasons to want to make the kernel dump core - the central one being there is some kernel or hardware based problem which continues to occur. What happens during a kernel panic (when properly configured) is that the kernel itself “dumps core” and the core can be used after reboot for analysis.

So here are some reasons:

• Intermittent kernel reboots
• Hard drive “lockups” (constant access, system frozen)
• Apparent hardware failures
• Speed problems in the kernel
• Kernel panic debugging

All except the last depend on a user (administrator) generated kernel panic with associated kernel dump. Of course, this is hard on filesystems, though Linux at least has the option of performing a “sync” from the same location as the user generated panic.

Most UNIX operating systems have the capability for the administrator to generate a kernel-based core dump. Linux users must have a kernel that supports the Magic SysReq key. Solaris on SPARC is set to go; Solaris on Intel processors requires booting the Solaris kernel with the kmdb kernel module loaded (through parameters and settings in the boot loader).

Applications will also generate core dumps, and a lot of the core dump analysis tools used for applications and the methods used can be useful in analyzing kernel dumps as well. BEA has an excellent (multi-platform) description of creating and analyzing core dumps - even though it is oriented towards their Tuxedo product, it seems still useful.

Sun has an excellent article, Core Dump Management on the Solaris OS, that covers both application core dumps and system kernel core dumps written by Adam Zhang at Sun.

For HP-UX, there isn’t as much on crash dump analysis, though the whitepaper Debugging Core Files using HP WDB (PDF) may be useful.

I don’t know AIX (nor z/OS) that well myself, but there are some free RedBooks that include core dump analysis as part of the book. There is z/OS Diagnostic Data Collection and Analysis for z/OS (if you just happen to have a mainframe in house) and Problem Solving and Troubleshooting in AIX 5L for AIX.

Likely I’ll be covering some of these tools in depth.  For most versions of UNIX and Linux, there are man pages for core(5).  Some systems offer the commands gcore and savecore as well.  As always, the FreeBSD man pages web page covers HP-UX (HP-UX 11.22), Solaris (Solaris 9), and Red Hat (Red Hat Linux 9) and others as well as FreeBSD. Unfortunately, it appears that other Linux and UNIX versions are not being updated (for whatever reason - space?).

This is just amazing: did everybody coordinate this? Within the last three weeks or so, we’ve seen these releases come out:

• Ubunutu 7.10 Gutsy Gibbon (and Kubuntu, Edubuntu, Xubuntu, et al)
• OpenBSD 4.2
• Fedora 8
• Red Hat Enterprise 5.1
• OpenVMS 8.3-1H1
• FreeBSD 7 Beta 2
• FreeBSD 6.3 Beta 1
• OpenSUSE 10.3
• Macintosh OS X 10.5
• NetBSD 4.0 RC3
• AIX 6.1

Several of these were released on the same day, November 1.

What next? Am I really supposed to choose just one? Sigh. And I just installed OpenBSD 4.1 and Fedora 7, too - not to mention installing FreeBSD 6.2 not too long ago.

From all the talk, I’ll have to try Kubuntu again. So many systems, so little time.

I have been using OpenSUSE 10.3 (with KDE). I just love it - and I love the new menu format, too.

Update: Sigh. I should have known. Microsoft Windows Vista celebrated its 1st Anniversary on Nov. 8.

Over at foogazi, Adam wrote about 5 reasons your parents should use Linux.  These are excellent reasons, and I can find nothing wrong with any of them.

However, when I tried to get my mother to use Linux, she eventually brought the Linux system back.

Here are some reasons that she and others won’t be using Linux any time soon:

1. Linux doesn’t work like the system they use at work (Microsoft Windows, of course).
2. Linux requires learning something new which they say they can’t do. Either they’ll tell you they don’t have time or they’ll tell you they just can’t learn new things.
3. The new system (or donated old system) already has Microsoft Windows on it.  It just works, and is something they already know.
4. Viruses?  Often, they may not use the system all that much, and may even leave it off most of the time.  With proper email and firewall etiquette, they won’t get any viruses.
5. It doesn’t run the software they want to use.  Virtually everything anybody wants to run will only run on Windows - nobody says “The system must be able to run Firefox” (or The Gimp - or Inkscape - or Thunderbird ….)  Linux won’t run Microsoft Outlook - or Microsoft Project - or Microsoft Excel - (notice a pattern?).

None of these are suitable reasons - but they are what users will tell you.  New adopters, people who like to learn something new, or are technically savvy will have no problems.  Most of these come down to one major complaint: They don’t want to learn something new.  Often times, “hard to use” translates into “doesn’t work like Windows.”

This is really unfortunate, but remains a problem that will not be solved by anything other than actually running Microsoft Windows.  Too bad, really.