• About

UNIX Administratosphere

~ UNIX and Linux System Administration

UNIX Administratosphere

Monthly Archives: May 2008

The EeePC and UNIX/Linux

30 Friday May 2008

Posted by ddouthitt in Linux, MacOS X, NetBSD, OpenSolaris, Solaris, Ubuntu, UNIX, Windows

≈ Leave a comment

Tags

asus, eeepc

Wandering about, it would appear that UNIX aficionados (geeks!) are putting everything imaginable onto the EeePC.

For example, there are people running MacOS X Tiger, MacOS X Leopard, OpenSolaris 2008.05, Ubuntu, and NetBSD on the EeePC. There is a video review of the EeePC which is very informative and well done (even if the video itself is slightly off-color). There is another video describing the EeePC and how to install Ubuntu Linux onto it. There’s even a video demonstration of MacOS X Tiger running on the EeePC. Doesn’t seem to be anything it can’t do.

There’s a NetBSD on EeePC page as well.

This all makes me want to go get one for myself; I’ve been holding off. One of these with a dual-bootable Windows/Linux or Windows/UNIX installation would be perfect – and as it boots off of flash cards as well, perhaps Windows 2000 on flash would be just the thing for me (every corporate doodad requires Windows – VPN, WPA, intranet – ack!).

These machines apparently continue to be quite popular, as this romp through UNIX on the EeePC shows.

Chicago Lisp Workshop (Introduction to Lisp!)

29 Thursday May 2008

Posted by ddouthitt in Personal Notes

≈ Leave a comment

Tags

chicago, lisp, workshop

This weekend there will be a workshop to introduce folks to Lisp; it will be held by the Chicago Lisp group. The workshop will be held Saturday 31 May 2008 at 3:00 pm at the Illinois Institute of Design (350 N. LaSalle) in central Chicago.

I plan to be there.

KDE4Daily

29 Thursday May 2008

Posted by ddouthitt in FreeBSD, Linux, UNIX

≈ Leave a comment

Tags

kde, kde4daily, qemu, Virtualization

If you are interested in the UNIX desktop and prefer KDE as I do, KDE4Daily may be just the thing for you. KDE4Daily is a QEMU virtual machine image built from the daily build of the most current development in KDE (currently at KDE 4.1).

This is what is often called “the bleeding edge” (an expansion of “the cutting edge”) – that is, the only guarantee is that there is no guarantee. The KDE build comes with a built-in bug reporting tool that activates when some part of KDE crashes.

There is an excellent (visual) review of KDE4Daily at /home/liquidat.

KDE4Daily is a neat way to get and use the latest KDE build in a relatively low hazard way. I plan to see how it runs for me – somewhere.

OpenSolaris 2008.05

28 Wednesday May 2008

Posted by ddouthitt in Nexenta, OpenSolaris

≈ Leave a comment

Tags

live cd, MacOS X, virtualbox, Virtualization

OpenSolaris 2008.05 (for x86) was released earlier this month, along with a new OpenSolaris web site (which to me bears more than a passing resemblence to the OpenSUSE web site). I don’t know the difference between that web site and OpenSolaris.org, but somebody must think its worth it.

I’ve already ordered my CD of OpenSolaris 2008.05; now to find something to run it on. Alright already, so that’s backwards – but I’m always trying the new UNIX versions. And why not? This is an excellent way to find out what is likely to be in Solaris 11.

The CD of OpenSolaris 2008.05 is also a live CD, which means you can try it out anywhere the operating system can run: just reboot with the CDROM in the drive.

Jason Perlow of ZDNet wrote a nice review article titled OpenSolaris: What Ubuntu wants to be when it grows up on May 5, 2008. It isn’t a step by step technical review, but it is good nonetheless. Another review was recently written by Ashwin Bhat K S about his experiences with OpenSolaris 2008.05. He used a laptop, so this review is doubly interesting. There is another review by Milind Arun Choudhary which talks about installing OpenSolaris (Nexenta, actually) into a VirtualBox virtual machine under MacOS X.

I can only hope that OpenSolaris driver support will improve; last time I installed it onto a current laptop it had no wifi connectivity and no network connectivity because of missing drivers. Is it any surprise then that Ashwin’s first recommendation is more network drivers and more wifi drivers?

Update: Another person has done up a nice article about installing OpenSolaris 2008.05 – their first and only post apparently!

Argument list too long?

24 Saturday May 2008

Posted by ddouthitt in Linux, Tips, UNIX

≈ 11 Comments

Tags

arguments, bash, ksh, sh, shell

Well, what now? We got the dreaded “argument list too long” error. What to do?

To explain the problem, let’s consider what the shell does (we won’t get into system calls, to make things simple). The shell (Bourne compatible shells, actually) will first scan the line typed in. One of the first things to do is to expand file globs. For example, the command:

ls a*

will be translated by the shell into:

ls andrew apple alex allen alfred almonzo august axel albert

(and so forth). If the expansion expands beyond the system limitations, then it cannot be processed and the error “argument list too long” results.

The limit could be the number of arguments or the number of characters; however, the fix is the same. The key is to get the list (usually files) out of the argument list and into something else – such as stdout. Once the overlong list is being sent via stdout, the command xargs can be used to place all of the items on the command line, observing all relevant limits as it goes (and as efficient as possible to boot).

There are a variety of quick answers which will all fail, because the argument list would remain too long:

ls a* >savefile.txt
for i in a* ; echo $i ; done
echo a*

However, all is not lost: there are a number of ways to get a long list of files into stdout; the most common is to use find:

find . -name "CACHE.DAT" | xargs ls -ld

This may not be possible, if the desired list of files doesn’t fit neatly into a find command.

Another possibility, related to the previous example, would be this:

ls -1 | sed '/^a/!d' | xargs ls -ld

Yet another possibility might be to use a language like Perl; since it does not scan and process the same way, it would work without limitations:

perl -e 'opendir(DIR, "."); @all = grep /^a/, readdir DIR; closedir DIR; print "@all\n";' | xargs ls -ld

I would only recommend using Perl or other such if you are quick and snappy with your knowledge of the language; otherwise, such a long line will have you looking things up repeatedly.

If the arguments are coming from a file, then things become even easier; instead of something like this:

ls -ld $(cat mylist)

You can simply use:

cat mylist | xargs ls -ld

Of course, any binary command can be used with xargs, not just ls.

Sparse files – what, why, and how

23 Friday May 2008

Posted by ddouthitt in HP-UX, Linux, OpenSolaris, Solaris, Tips, UNIX

≈ 16 Comments

Tags

cp, cpio, ls, rsync, sparse files, tar

Sparse files are basically just like any other file except that blocks that only contain zeros (i.e., nothing) are not actually stored on disk. This means you can have an apparently 16G file – with 16G of “data” – only taking up 1G of space on disk.

This can be particularly useful in situations where the full disk may never be completely used. One such situation would be virtual machines. If a virtual machine never fills the disk entirely, then a certain amount of the disk will never have anything but zeros in it – permitting the saving of disk space by using a sparse file.

The operating system (which supports sparse files) knows that the block “exists” but is null, so it provides the zero-filled block out of thin air. As soon as the block contains data, the data is written to disk in the appropriate way and the file on disk grows.

There are problems with using sparse files. The most egregious would be that any utility that does not recognize or utilize sparse files can replicate the entire file on disk, so that a 500M sparse file could suddenly balloon to 6G. Even utilities that can work with sparse files must be told to do so, so this trap is easy to fall into.

Another problem is that everything about disk management is based on how many sectors or blocks are used – and the disk size reported for a sparse file is the full size of the file (not the actual number of blocks on disk). This also means that any utilities that work solely with blocks on disk (e.g., du) will report different amounts than other utilities.

Yet another problem is whether backup programs recognize or preserve sparse files. A backup program that does not recognize (and store) sparse files may well be quite oversized. A restore of this flawed backup – or indeed, a restore that does not recognize properly backed up sparse files – will balloon in size as mentioned before. In a worst case, there would not be enough room on disk to restore the data, if the sparse file expands enough to fill the disk.

To get the ls command to report actual on-disk sizes (instead of just the file size typically reported) use these options:

# ls -ls
total 96
8 -rw-r--r-- 1 root root 2003 Aug 14 2006 anaconda-ks.cfg
68 -rw-r--r-- 1 root root 59663 Aug 14 2006 install.log
8 -rw-r--r-- 1 root root 3317 Aug 14 2006 install.log.syslog
12 -rw------- 1 root root 10164 Oct 25 2006 mbox

The blocks used are in the left side column; the bytes used are in their usual column just before the date. In this case, all of the files are using appropriate number of blocks – that is, none of these files are sparse.

Creating a sparse file basically amounts to a simple process: create a file, and seek to the desired end of the file. This can be done at the command line with a command like (to create a 1M sparse file):

dd if=/dev/zero of=sparse-file bs=1 count=1 seek=1024k

Here is an example, run on HP-UX:

# dd if=/dev/zero of=sparse bs=1 count=1 seek=1024k
1+0 records in
1+0 records out
# ls -ls sparse*
2 -rw-r--r-- 1 root sys 1048577 May 22 12:58 sparse
#

Looking at HP-UX (and perhaps other environments), there does not appear to be a wide amount of support for sparse files in the typical utilities (such as tar, cp, cpio, etc.), even as the operating system itself will dutifully create sparse files.

However, GNU cp supports the –sparse option. By default, GNU cp attempts to detect sparse files and recreates them as warranted. A file can be copied into a sparse format using –sparse=always, or into a nonsparse format using –sparse=never. The default, alluded to earlier, is –sparse=auto.

GNU tar uses the –sparse option (or the equivalent -S option) to make tar store sparse files appropriately.

GNU cpio supports the –sparse option, which operates similarly: any suitable length of zeros is recorded as a “empty” and the file is stored or created as a sparse file.

The command rsync, while not a GNU project, does support sparse files. The option –sparse (or -S) will attempt to handle sparse files efficiently, that is, not creating file blocks full of only zeros.

It appears that utilities pax, scp, sftp, and ftp do not in general support sparse files. Using such a utility, then, would make a sparse file balloon in size.

Given this utility support for sparse files, the best native environment is proably Linux or MacOS X, or other open platform. More specifically, any platform where the GNU utilities are available, along rsync.

Recognizing a sparse file can be done at the command line with the previously mentioned ls -ls command. To see a sparse file (right down to the block level), one way would be to write a C program (or other language) to start at the end of the file, and truncate the file one block shorter – if the file on disk becomes shorter, then that block was on disk; if not, then it was a sparse block. Of course, any block that contains data is on disk; it is only blocks with zeros with which there is any question as to whether it is on disk or not. Jonathan Corbet had an article in December of 2007 in the Linux Weekly News that described a method that the Solaris ZFS developers had proposed, and posed the question as to whether Linux should support similar system calls that will seek to the next hole or the next set of data in a file.

Using QEMU

19 Monday May 2008

Posted by ddouthitt in FreeBSD, Linux, Virtualization

≈ Leave a comment

Tags

Emulation, qemu, virtual machines

Using QEMU allows you to run a virtual machine on a UNIX system. Unlike other emulators, QEMU is a generic emulator – which means that it is portable, and that it emulates more than one system. There are versions for Windows, MacOS X, Linux, and OpenSolaris.

The QEMU emulator allows you to emulate an i386, an AMD-64, PowerPC, SPARC, and others. Not all are usable, but the i386, AMD-64, and PowerPC are usable.

Running QEMU on a Pentium III results in a slow emulated machine, but it is usable. Running Windows 2000 in 128M or 256M of memory is not too bad if you are patient. Running it on a modern machine is much more useful and much easier on the patience.

QEMU has some nifty features. The monitor, for instance, can be attached to a virtual console, to a pseudoterminal, or to a serial port. Likewise, the emulated serial port can be attached to a pseudoterminal, to a host serial port, or other locations – and more than one serial port is possible.

One very interesting feature is that the emulated machine can use a VNC session for its display, allowing you to access the host machine from anywhere on the network that you have a VNC viewer. This also suggests that you could have multiple emulated machines running, with all of them available over the network to VNC clients.

Unfortunately, there are no reliable control panels, and only two session managers for standard installations (or perhaps only one if you’ve not installed QT 4 yet). The only innate method of controlling QEMU emulators is through the text console. There is Qtemu (which, on my FreeBSD system, required all of QT4 to be installed), and there is qemu-launcher (which only requires the basics that are on any system). Unfortunately, QTEMU does not provide a way to open an existing virtual machine, nor does it offer a way to utilize machines and info created by qemu-launcher.

The only control panel I could find was the one used by qemu-launcher, qemuctl – which crashes.

Why you should know vi

15 Thursday May 2008

Posted by ddouthitt in Linux, UNIX

≈ 2 Comments

Tags

emacs, vi, vim

I won’t make any excuses; I love vi. Folks at The Register called vi Bill Joy’s greatest gift to mankind (who am I to argue?) However, being able to use vi well goes beyond whether you like it or not.

Why should you know vi (well)? For one thing, it is on every UNIX and Linux system. Emacs certainly isn’t, and neither are the other worthy competitors like pico, elm, nano, and others. None of these editors, good though they may be, are ubiquitous enough. In fact, every last one requires special installation on virtually every platform – that is, there are no default installations that include them.

Even vim is not ubiquitous – although it may be available for lots of environments, it is not necessarily installed by default. For a long time, I used vim in vi-compatible mode for just this reason. (My favorite vim-specific idea? :set nowrap).

Vi has an interesting history. Vi was written by Bill Joy, starting with the basics of the em editor from George Coulouris of Queen Mary College, University of London. Bill wrote vi to be able to work at 300 baud (as he says, that’s all he had and the fellows at MIT doing EMACS had the equivalent of fiber for that day and age). UNIX Review did an interview with Bill in August of 1984.

The original vi is now available under a BSD license. In contrast to EMACS and vim, both of which number in the multi-megabytes, vi was 160k on the i386.

So what editor does Bill use? ed.

UPDATE: Eric Raymond in his book The Art of UNIX Programming (also available online) has a very interesting analysis in Chapter 11 of the complexity and contrast (and psychology) between EMACS and vi.

Fedora 9 Announced

13 Tuesday May 2008

Posted by ddouthitt in Fedora, Linux, Red Hat

≈ Leave a comment

Tags

gnome, jigdo, kde, upstart, xfce

Yesterday Fedora 9 was announced. Using Fedora can give you a look at what may be in Red Hat Enterprise Linux down the road – and give you an exciting Linux distribution to boot.

There are a number of new exciting features to be found in Fedora 9. First, everything is updated to the latest versions, including GNOME 2.22, KDE 4.0.3, and Xfce 4.4.2.

Fedora 9 introduces the new filesystem ext4 as an option. While ext4 remains an experimental filesystem, it may be good to try it out. Like ext3, it remains compatible in both directions (an ext4 filesystem can be mounted as ext3, and vice versa).

Fedora 9 also replaces the System V initd process with an event-based replacement, upstart. Upstart was created and developed for Ubuntu Linux, and has spread to Fedora and Debian. Each process is started through a response to an event, and each process may generate another event.

Fedora 9 has several different spins or variations based on different sets of packages. For example, there could be a KDE spin, a GNOME spin, and a Xfce spin for example. The Fedora project has a page tracking spins for those who might be interested in custom spins.

This version of Fedora introduces support for Jigdo, which is a CD distribution mechanism that the Debian project has used for years. I’ve not used Jigdo, but the description given in the release notes suggests a large speedup if you have most of the data already.

It sounds like a very exciting distribution; I’ll be looking around my electronic wasteland to see where to install it.

Book Review: Disk and File Management Tasks on HP-UX

12 Monday May 2008

Posted by ddouthitt in HP-UX

≈ Leave a comment

Tags

lvm, onlinejfs, vxfs

Using LVM on HP-UX can be troublesome, and any deviation from the usual steps can cause problems. Specifically, if the data on disk and the data in the system vary, then there can be a variety of problems. The usual symptom is that the LVM environment tools won’t let you do something that seems obvious (like removing a volume group when the relevant disk volume is missing).

The book Disk and File Management Tasks on HP-UX (by Tom Madell) is basically a guide to all things LVM. Despite its age (1996), it is still relevant, and covers LVM, OnlineJFS, and VxFS. The book is also not standard publishing quality, but is a bound set of facsimile pages – this doesn’t detract from the contents however.

The book covers all aspects of disk and file management, including such advanced topics as creating your own bootable root volume, recovering from loss of /etc/lvmtab, and many other details. It covers commands such as vgcreate, vgexport, vgimport, vgreduce, vgremove (and their logical volume counterparts).

It also covers converting to VxFS (from older filesystems), mirroring, and creating striped volumes.

This is a worthy addition to any HP-UX administrator’s library.

← Older posts

Mei Douthitt

Mei is an experienced UNIX and Linux system administrator, a former Linux distribution maintainer, and author of two books ("Advanced Topics in System Administration" and "GNU Screen: A Comprehensive Manual").
  • Mei's Books
  • LEAF Project (home to the Oxygen distribution)
  • GNU Screen Home Page



View David Douthitt's profile on LinkedIn

Use OpenDNS

Bloggers' Rights at EFF

The Internet Traffic Report monitors the flow of data around the world. It then displays a value between zero and 100. Higher values indicate faster and more reliable connections.

Recent Posts

  • Running Icingaweb2 on Ubuntu 16.04.1 LTS
  • AppStream Error in Ubuntu 16.04 Xenial
  • Return to Window Maker (on Xubuntu 15.10)
  • Sharing Music from Xubuntu 15.10 using Tangerine
  • Mono on Xubuntu 15.04 and 15.10

Top Posts

  • The wheel Group
  • Generating Passwords Using crypt(3)
  • Resetting the MacOS X 10.4 (Tiger) Admin Password (without disk!)
  • Rescuing an Interrupted Ubuntu Upgrade
  • Logging every shell command
  • AppStream Error in Ubuntu 16.04 Xenial
  • DOS Partitions (fdisk) and the 2TB Limit
  • When root is locked out...
  • Apport uses 100% CPU in Xubuntu 15.04 Vivid Vervet
  • Generating a coredump (gcore)

Calendar

May 2008
M T W T F S S
« Apr   Jun »
 1234
567891011
12131415161718
19202122232425
262728293031  

Recent Comments

Sweekrith on AppStream Error in Ubuntu 16.0…
ddouthitt on Return to Window Maker (on Xub…
Missing internet and… on Rescuing an Interrupted Ubuntu…
cgnkev on Running Icingaweb2 on Ubuntu 1…
An Archy on Return to Window Maker (on Xub…

Category Cloud

BSD Career Data Centers Debian Debugging Disaster recovery Fedora FreeBSD Hardware HP-UX Industry Linux MacOS X Mobile Computing Monitoring Networking OpenSolaris Open Source OpenVMS Personal Notes Productivity Programming Red Hat Scripting Security Solaris Tips Ubuntu UNIX Virtualization

Archives

  • January 2017 (1)
  • December 2016 (1)
  • July 2016 (1)
  • January 2016 (4)
  • December 2015 (1)
  • August 2015 (1)
  • September 2014 (2)
  • August 2014 (1)
  • June 2012 (2)
  • May 2012 (2)
  • April 2012 (5)
  • March 2012 (3)
  • February 2012 (5)
  • January 2012 (9)
  • December 2011 (2)
  • November 2011 (6)
  • September 2011 (4)
  • August 2011 (2)
  • July 2011 (10)
  • June 2011 (3)
  • May 2011 (12)
  • April 2011 (10)
  • March 2011 (4)
  • February 2011 (3)
  • January 2011 (7)
  • December 2010 (1)
  • November 2010 (5)
  • September 2010 (11)
  • August 2010 (12)
  • July 2010 (8)
  • June 2010 (9)
  • May 2010 (13)
  • April 2010 (12)
  • March 2010 (23)
  • February 2010 (18)
  • January 2010 (14)
  • December 2009 (14)
  • November 2009 (11)
  • October 2009 (6)
  • September 2009 (12)
  • August 2009 (7)
  • July 2009 (8)
  • June 2009 (16)
  • May 2009 (13)
  • April 2009 (3)
  • March 2009 (13)
  • February 2009 (5)
  • January 2009 (16)
  • December 2008 (10)
  • November 2008 (9)
  • October 2008 (5)
  • September 2008 (10)
  • August 2008 (17)
  • July 2008 (6)
  • June 2008 (3)
  • May 2008 (14)
  • April 2008 (11)
  • March 2008 (10)
  • February 2008 (18)
  • January 2008 (17)
  • December 2007 (15)
  • November 2007 (30)
  • October 2007 (25)
  • September 2007 (11)
  • August 2007 (21)
  • July 2007 (9)
  • June 2007 (2)

Blogroll

  • …Details…
  • Aaron's OpenVMS Hobby Site
  • Brazen Careerist
  • Cuddletech
  • Debian Admin
  • Eight-Cubed
  • FreeBSD Diary
  • Hack a Day
  • Hoffman Labs
  • Librenix
  • Linux Kernel Newbies
  • Linux Kernel Weather Forecast
  • Living with IPv6
  • Mission Critical Computing
  • nixCraft
  • PaulDotCom (of Security Weekly Podcast)
  • root prompt
  • Standalone Sysadmin
  • The Book of Dead:[Systems]
  • Tim's Blog
  • Transparent Uptime
  • WordPress.com
  • WordPress.org

RSS Sharky’s Column

  • Funny ha-ha
  • Throwback Thursday: Can I see me now?
  • How much knowledge is a dangerous thing?
  • Time-Machine Tuesday: It’s all in how you ask
  • So much learning in one day
  • Flashback Friday: A little TOO efficient
  • Throwback Thursday: Pick a card, any card ...
  • And that, kids, is how you boot a system
  • Not-so-Big Gulp
  • Memory-Lane Monday: Do as I say, not as I ...

Pages

  • About

Meta

  • Register
  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.com

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy