Using LVM on HP-UX can be troublesome, and any deviation from the usual steps can cause problems. Specifically, if the data on disk and the data in the system vary, then there can be a variety of problems. The usual symptom is that the LVM environment tools won’t let you do something that seems obvious (like removing a volume group when the relevant disk volume is missing).

The book Disk and File Management Tasks on HP-UX (by Tom Madell) is basically a guide to all things LVM. Despite its age (1996), it is still relevant, and covers LVM, OnlineJFS, and VxFS. The book is also not standard publishing quality, but is a bound set of facsimile pages - this doesn’t detract from the contents however.

The book covers all aspects of disk and file management, including such advanced topics as creating your own bootable root volume, recovering from loss of /etc/lvmtab, and many other details. It covers commands such as vgcreate, vgexport, vgimport, vgreduce, vgremove (and their logical volume counterparts).

It also covers converting to VxFS (from older filesystems), mirroring, and creating striped volumes.

This is a worthy addition to any HP-UX administrator’s library.

My entry on using find turned out to be popular; I thought I’d throw out some tidbits on using find for various things. So sit back and enjoy the list!

find . -type f | xargs ls -ld

List all standard files in the current hierarchy starting at the working directory.

find . -type d | xargs ls -ld

Get a list of directories, showing the tree structure starting at the working directory.

find . -mtime +2

Show all files older than two days.

find . -size +10000

Show all large files (possibly for freeing up some space?)

find /*bin /usr/*bin /usr/local/*bin -name "somebin"

Find a executable file by the name given in any of the usual locations (including both bin and sbin).

find . -cpio dev

Find files (as specified) and write to a cpio archive specified in the cpio parameter. This is a newer option; check if your find has it first.

find / -nogroup -o -nouser

Find all files on the system that have no group or owner; files such as these are a security risk and should be associated with groups and owners in /etc/group and /etc/passwd. Remember to run a fidn command that starts at / at a time when users will not be inconvenienced by the massive search.

find / -perm -s

Find all files that are suid or guid - again, these may be a security risk. You should know which binaries are (and need to be) suid and guid.

On using the -exec parameter: if you use the “old school” form, with {} and \; (such as find . -exec rm -f {}\;) then the command will be executed once for each file found. If you use the “new school” form with {} followed by \+ (such as find . -exec rm -f {}\+) then the command will be executed for all files found in one go.

It is unclear whether it will limit the number of parameters (to keep the command line to within acceptable lengths). The xargs command is the usual way to do this; the biggest advantage of xargs over -exec is that no matter how many times the command is executed, it is never pulled again into memory: xargs restarts the command from the beginning without paging in any code. It also allows you to specify the line size limit as well, or a limit on the number of arguments for that matter.

Newer xargs appear to work with shell scripts as well; traditionally, xargs required binaries and shell scripts (or other scripts) would not work.

How much memory is in this machine?

It would seem that answering this question ought to be easy; it is - but every system has the answer in a different place. Most put an answer of some sort into kernel messages reported by dmesg (AIX apparently does not).

Most systems have a program for system inventory which reports a variety of things, including memory.

Rather than go into great detail about each one, we’ll just put these out there for all of you to reference. Each environment has multiple commands that give available memory; each command is listed below.

Without further ado, here are a few answers to this burning question:

Solaris

1. dmesg | grep mem
2. prtdiag | grep Memory
3. prtconf -v | grep Memory

AIX

1. bootinfo -r
2. lsattr -E1 sys0 -a realmem
3. getconf REAL_MEMORY

HPUX

1. dmesg | grep Physical
2. /opt/ignite/bin/print_manifest | grep Memory
3. machinfo | grep Memory

Linux

1. dmesg | grep Memory
2. grep -i memtotal /proc/meminfo
3. free

OpenVMS

1. show mem /page

Update:

FreeBSD

1. dmesg | grep memory
2. grep memory /var/run/dmesg.boot
3. sysctl -a | grep mem

The new version of Core Linux comes with packages and appears to be fully comprised of packages (like Red Hat Linux, and unlike FreeBSD which has a core application set). These packages are simple: they are just tar.bz2 files that contain files relevant to the application, and a set of files that go under /etc/coretools/pkg.

The directory /etc/coretools contains everything related to core packages; the pkg directory has the details on each package, and the directory exec.d has plugins for the program corepkg. Plugins are just scripts that are called by corepkg.

The program corepkg lists its help if called with no parameters. Some of the more common usages might be:

• corepkg --list (list current plugins)
• corepkg --exec=info --pkgname=pkg (package information by name: pkg)
• corepkg --exec=list (list all installed packages)

The plugins as installed are:

• contents - list files created by package named
• count - count packages matching specified options
• info - information on specified package
• install - install specified package
• list - list installed packages
• remove - removed specified package

The packaging system is simple and driven fully by shell scripts. It should be possible to ignore it without adverse effects. There don’t seem to be any packages beyond the basic system, but that may not be the case. Anyway, the goal of the original Core Linux - and its descendents - is to build your own system through compiling your own code.

To prepare for installing grub, it is necessary to chroot into the system mounted on /mnt. To do this, we need a working /proc and /dev under /mnt; this is accomplished like thus:

mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev

Then perform the chroot - making the new environment mounted under /mnt our new environment:

chroot /mnt /bin/bash

Note that this makes bash the current shell - not my general choice (I prefer ksh) but things might break if you use the wrong shell….

Now we can install and configure grub:

grub-mkdevicemap
grub-install –grub-setup=/boot/grub/grub.cfg /dev/sda
/sbin/grub-install: line 223: /boot/grub/grub.cfg: Permission denied
grub-setup –directory=/boot/grub –device-map=/boot/grub/device.map /dev/sda
cat /boot/2.6.21.1/grub >> /boot/grub/grub.cfg

The error comes from grub-install apparently trying to write to grub.cfg (which has permissions 644). The directions say this error can be ignored - so that’s what we’ll do.

The resulting grub.cfg looks like this:

# Set timeout
set timeout=30

# Set default entry
set default=0

# Grub configuration for linux-2.6.21.1
menuentry “Core 2.0 GNU/LInux (2.6.21.1)”{
set root=(hd0,1)
linux (hd0,1)/boot/2.6.21.1/bzImage root=/dev/hda1 vga=5
}

In this case, this is certainly wrong. The last stanza is preconfigured for hda and for a single volume root. Since I have two partitions, and am using /dev/sda, it should be:

# Grub configuration for linux-2.6.21.1
menuentry “Core 2.0 GNU/LInux (2.6.21.1)”{
set root=(hd0,1)
linux (hd0,1)/2.6.21.1/bzImage root=/dev/sda3 vga=5
}

Then edit /etc/fstab:

/dev/sda1 /boot ext3 defaults 0 0
/dev/sda3 / ext3 defaults 0 0
/dev/sda2 swap swap defaults 0 0

Then reboot:

exit
cd /
umount /mnt/dev
umount /mnt/proc
umount /mnt/boot
umount /mnt
shutdown -rn now

Next time…. the first boot.

Installing Core Linux is covered in this article (on the front page of coredistro.org as it happens!). The descriptions are clear; however, we will expand on the instructions listed there.

Let’s assume that you’ve already burned the ISO and booted with the CDROM. Once the system has booted, you’ll see this screen:

Log in as root (which has no password). First, figure out which disk to work with. The disk is most likely either /dev/hda (first IDE drive) or /dev/sda (first SCSI drive). You can see whether these disks exist by searching dmesg for sda and for hda.

Once the disk is determined, we must begin laying out the disk. In my configuration, I used fdisk to create three partitions:

• /boot partition of 500M
• Swap partition of 1G
• / partition of the rest (about 7G)

With this configuration, it is then necessary to (of course) create the filesystems:

mkfs -t ext2 -j /dev/sda1
mkfs -t ext2 -j /dev/sda3

The -j option creates what is commonly called an ext3 partition - which is in reality, and ext2 partition with a journal file attached. But that’s another post.

After the filesystems are created, then they must be mounted under /mnt as the complete environment would be layed out. For this example, that means:

mount /dev/sda3 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

After this is done, the actual process of installing Core Linux can begin. At the prompt, type:

install_core /mnt

Messages will appear like these:

Once this completes, a message comes back:

Install complete - the next step is to chroot into /mnt and install grub

Next installment… setting up grub.

If you want to create your own Linux distribution from scratch (and learn a lot during the process), you could do no better than to start with Linux From Scratch. However, many of us will not want to go to the extremes that LFS requires: there is an alternative - Core Linux.

Core Linux provides just enough of Linux to boot into a shell and compile. Thus it comes with bash and gcc and associated tools. If you want Apache - or PHP - or X11 - or KDE, you’ll have to compile itself - which is the point.

Core Linux comes in a small ISO, and provides a small shell script to install the basics of the system. The ISO is 150M and is available from Sourceforge.

When you burn the ISO, don’t forget to use the image capabilities of your CD burning application; don’t copy the file into a CDROM directory - but then you knew that already, didn’t you?

In upcoming parts of this series, I’ll show you how to install Core and how to get started with some applications. Of course, if you can’t wait…..

Downtime is inevitable: servers need to be updated, patches applied, hardware upgraded or fixed, and so on. But how do you choose when to take the server down (if you have a choice)?

First of all is to ask the affected users of the system. This can range from getting the supervisor or person in charge to give an OK to having meetings with all affected. In some fashion, the users who are affected need to know and need to have input into the best time to take the server down. When the server is down, they may not be able to do much of their work; minimizing impact on them is important.

To also minimize impact, the downtime can often be arranged during off-hours. In the U.S., this is typically outside of the normal workday of 8 a.m. to 5 p.m. (800-1700). However - if the server is in use in other time zones, this window gets bigger - and if the server is used world-wide, the window gets bigger still.

The other thing to think about is which day to take down the server. One can do it during the work week (Mon. through Fri.); however, all of those nights (except Friday) are constrained by the fact that the availability deadline would be something like the next day at 7:00 a.m. When downtime is scheduled for Friday night, there are two entire days more to get things right. If a downtime is necessary for a major upgrade or extensive changes, then scheduling for Friday night gives that much more time to get everything working before Monday morning.

Planning for a disaster is not necessarily as easy as it sounds. It helps if you have a rampant imagination. Throughout disaster planning, the dominant question is What if…? Following the planning, testing is required: the best plans are worthless if they don’t work in practice.

Consider an Internet server serving web pages. Let’s assume that downtime is not an option: this is a typical point to start at. The best thing to do is to start with the most specific to the system (the complete environment) and work out:

• What if… a disk goes bad?
• What if… the software stops?
• What if… memory runs out?
• What if… the power goes out?
• What if… the kernel panics?
• What if… the cluster failover fails?
• What if… the network switch fails?
• What if… the network firewall fails?
• What if… the internet link goes down?
• What if… the internet provider drops off the grid?

Each one of the questions must be answered and the results tested. To test for power outage, pull the power. For a failed network switch, pull the network cable - and so forth.

Most of the answers will include some form of redundancy - clusters, dual facilities (such as power and network and internet providers), and so on. However redundancy is only one solution; there is prevention and alerts as well.

Each risk must be weighed against the cost to mitigate that risk. However, assuming that the risk is minimal does not eliminate the risk; the biggest problem is not accounting for a risk that eventually happens. There is nothing like downtime of a critical server to get an unforeseen risk taken care of; better to handle the risk before it happens.

It also does not matter if the plans have not been tested. If tests are not done, then the actual event will be the first time things have been put to the test - and what if something was missed and the system goes down? During a test, preventive measures can be taken to make sure that things work as they should - during an unexpected event, it is not possible to back out or prepare; if things go down they go hard. Don’t let that happen to you!

And disaster planning is not limited to servers (or virtual servers) - what about the possibility of a server hosting multiple virtual servers going down? What if your server is hacked into? What about your monitoring system failing? What about getting paged? Have you planned contingencies for all of these events?

Plan, then test, test, test - and you will make it.

The utility lsof is a relatively new (well, compared to UNIX anyway) that has more options than even ls. These options provide for some extremely powerful capabilities, some of which we aim to illuminate here.

My favorite use for lsof is for networking: all sockets can be seen with the following options:

lsof -n -i

The -n option prevents lsof from being slowed down by a large number of DNS lookups, and the -i option returns all TCP/IP connections (with process numbers, user ids, file descriptor ids, and so on). To narrow it down, utilize options like the following - to list all SMTP connections, for example:

lsof -n -i :25

It is also possible to list only certain processes (such as process 25 and process 45):

lsof -n -p 25 -p 45

Alternately, the process can be selected by name:

lsof -n -c perl

However, suppose one wants to list all TCP/IP sockets held open by perl processes. The obvious choice does not work! This is because the options are combined together as an OR function; to combine them as an AND function (that is, all options must be satisfied) use the -a option - such as this:

lsof -a -n -c perl -i

This lists, as desired, all TCP/IP sockets held open by perl processes.

Another that might be useful in a security context is listing all files that are open but have no links to them: that is, they’ve been deleted, but one or more processes are keeping the file open, which means the file itself (and its blocks) are being preserved even though it appears to be deleted from the filesystem. To see these files, use this option:

lsof +L1

The utility lsof is indeed very useful, and reading the man page for lsof is recommended.