UNIX Administratosphere

How much memory is in this machine?

It would seem that answering this question ought to be easy; it is - but every system has the answer in a different place. Most put an answer of some sort into kernel messages reported by dmesg (AIX apparently does not).

Most systems have a program for system inventory which reports a variety of things, including memory.

Rather than go into great detail about each one, we’ll just put these out there for all of you to reference. Each environment has multiple commands that give available memory; each command is listed below.

Without further ado, here are a few answers to this burning question:

Solaris

1. dmesg | grep mem
2. prtdiag | grep Memory
3. prtconf -v | grep Memory

AIX

1. bootinfo -r
2. lsattr -E1 sys0 -a realmem
3. getconf REAL_MEMORY

HPUX

1. dmesg | grep Physical
2. /opt/ignite/bin/print_manifest | grep Memory
3. machinfo | grep Memory

Linux

1. dmesg | grep Memory
2. grep -i memtotal /proc/meminfo
3. free

OpenVMS

1. show mem /page

Update:

FreeBSD

1. dmesg | grep memory
2. grep memory /var/run/dmesg.boot
3. sysctl -a | grep mem

A disk snapshot is a snap in time, a picture of what a disk looked like “back then”. This can be very useful for maintenance.

For example, being able to freeze a Caché instance, take a disk snapshot, then thaw the Caché instance will permit you to take backups or copies of a Caché database with minimal downtime.

For HP-UX Online JFS and Veritas VxFS the commands are the same (since these are actually the same product - or close to it). To actually do a snapshot:

mount -F vxfs -o snapof=/var/cache/db /dev/snap01 /snap

The first file system presented in the command line is key: it is the source of the snapshot. Note that it can be either a device or a current mount point. The second (device) is a filesystem prepared to hold a snapshot, and the last is the usual mount point.

Once this is done, the normal filesystem can continue to be used while the snapshot retains the older data as it was taken. In the example above, /var/cache/db could be used normally while the snapshot resides on /snap. If there was a directory /var/cache/db/db01 then there would also be a /snap/db01 available as well.

One caveat is that as long as the snapshot is mounted and in use, the changes to the original filesystem are being saved - it is possible that the snapshot volume can run out of space. When this happens, you will receive what may appear to be mysterious disk full errors unless you realize what is happening. So don’t keep your snapshots around forever.

In Solaris 9, two new utilities were added: pkill and pgrep. These tools are perhaps old news to Solaris admins. However, these utilities were then quietly added to Linux in short order, and now show up in HP-UX 11i v3 and perhaps others. What do these tools do?

First of all, pkill is just a wrapper for pgrep. Well, what does pgrep do? It searches the current processes for a match based on arguments that you give to it.

The most common use would be to search for a command. The pgrep utility will then return each process ID, one per line. The pkill utility will send the default kill signal to each pid. The pgrep utility can search based on a large array of factors, including userid, groupid, virtual size, effective user ids, command name, full command string, and more.

Here’s an example:

# pgrep cache
13
12
14
15
33
49
22006
21950
21973
21976
#

When combined with scripting, these commands can be quite useful. Consider this ksh snippet:

for i in $(pgrep cache) ; do
  // do some commands here
done

HP-UX has some nice features in their initialization scripts, but you have to be aware of them in order to take advantage of them.

One good starting point is the rc(1m) man page. The rc program is the actual program that does all of what we can initialization: that is, all of the startup processes, the rc.d directories, the symbolic links, the run levels - rc does all this. It is rc that init(1) calls to make it happen.

Also, the init scripts are not where you would first look - or second. The scripts live in /sbin/init.d. This directory contains all of the scripts (and no links). Then look at one of the scripts in order to determine what can be done with your own scripts - and to see if there are any new features.

There are several features of the HP-UX init script mechanisms that you can take advantage of:

• The /etc/rc.config.d directory - to set configurable parameters for each individual subsystem (usually as represented by a script)
• The startup and shutdown messages: these are formatted nicely and make for quick viewing of the startup or shutdown process
• Results: not just SUCCEED or FAIL but also N/A (Not Applicable) and others.

To use the /etc/rc.config.d directory, just source the appropriate configuration “script” into your environment. Then the appropriate variables will be set.

The startup and shutdown messages are exquisitely simple: instead of start and stop, the routine is called with start_msg and stop_msg. Respond to these subcommands by printing the appropriate text for a descriptive message. With these set, your init script will display its appropriate message when it is started or stopped during startup or shutdown.

Then the results of the script - which show up in the HP-UX startup or shutdown in the far right hand side of the line. The possible results are:

• (0) Subsystem was brought up (or down) successfully.
• (1) Subsystem encountered errors.
• (2) Subsystem was skipped, such as from configuration in the /etc/rc.config.d file or other reasons, and did nothing.
• (3) A reboot of the system is requested; rc(1m) will perform the actual reboot. /etc/rc.bootmsg will be displayed on the console prior to reboot, then deleted.
• (4) Subsystem was successful and started a process in the background.

Any return value larger than 4 is the same as 1: subsystem encountered errors.

Note, too - there is nothing that mandates that these must be shell scripts (except perhaps for rc.config.d file syntax). If you must, it could be a perl script - although a shell script is best. Just make sure to have the proper header so that the system doesn’t try to interpret your perl script as a shell script.

And a quickie tip for you! If you find yourself using a shell and are expecting to edit the history using vi - only to find that there is no history - you can find that your display no longer responds (!). In reality, the system is responding just fine, except you can’t see it (small things, right?). This is because the editing sequence for vi is ^[k (ESCape-K). This mucks up the display as it is not being handled by the shell but rather by the terminal display.

How to get out? I use the following command sequence: first, do a set command - then another. How does this work? Well, I can’t explain everything, but the first output from set will contain the ^[k sequence in it (look in the variable $_). My theory is that that is not the only place to find it. It also does not seem to matter what the second command is, just one with output (and not just a carriage return).

Anyone who feels the need for a challenge can research this - I just know it works, and can be typed very fast.

There is recurring error that keeps putting a message into the syslog.log file every time a cron job runs under HP-UX 11i v3. This message can safely be ignored, but if you run a lot of cron jobs it can keep filling your syslog with useless messages. The message like thus:

Feb 28 16:26:00 mysys syslog: Cron daemon - could not change the compartment for the job /usr/bin/foo

These messages will continue to appear for every cron job. It refers to a product which in this case is not installed on the machine (that is the “compartments” of which it speaks).

There is a patch available from HP to fix this, but even after a year this patch remains off of the recommended list. The patch is PHCO_36539 and it will take care of the problem, returning your cron messages to a sane level.

A few versions back, HP started including Veritas VxVM (Veritas Volume Manager) with HP-UX 11i. There was some consternation that VxVM was a third-party non-HP product, and that it would mean the demise of LVM.

However, now in HP-UX 11i v3 HP has turned the tables on Veritas (now Symantec) so to speak. In essence, this is another replay of the operating system vendor shooting down the third-party vendor.

In the past, HP-UX did not support load balancing on HP-UX with the native LVM disks; only VxVM supported multipathing with fail-over and load balancing. Now, in HP-UX 11i v3, the native drivers support full load balancing and failover with multipathing, and VxVM has been eviscerated: under 11i v3, VxVM’s multipathing DMP is put into straight-through mode (that is, no failover, no load balancing) and the new 11i native multipathing takes over.

When the disks are presented to HP-UX 11i v3 properly, the following will be output for the ioscan -m dsf command:

# ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
...
/dev/rdisk/disk16        /dev/rdsk/c7t0d0
                         /dev/rdsk/c6t0d0
/dev/rdisk/disk17        /dev/rdsk/c7t0d1
                         /dev/rdsk/c6t0d1
/dev/rdisk/disk18        /dev/rdsk/c7t0d2
                         /dev/rdsk/c6t0d2
/dev/rdisk/disk19        /dev/rdsk/c7t0d3
                         /dev/rdsk/c6t0d3
/dev/rdisk/disk20        /dev/rdsk/c7t0d4
                         /dev/rdsk/c6t0d4

With the proper multipathing in place, the “persistent DSFs” (DSFs are Device Special Files) are matched to multiple paths (which are reflected in the legacy DSFs). If you see this output from ioscan -m dsf, then multipathing is enabled for the disks as shown.

If ioscan -m dsf reports a syntax error, then you are not running on HP-UX 11i v3 - 11i v2 and below did not support the -m option.

It would appear that HP was serious about supporting LVM on HP-UX after all.

HP-UX tried using a different approach to loading packages and binaries to run. The traditional method uses /usr/local - the BSD variants use this directory to its fullest extent. The original idea behind /usr/local was that the system file system could be upgraded - or even replaced - and it would not affect added system programs. This extends to /home as well. If /home and /usr/local were separate file systems, then the operating system could be upgraded with impunity.

HP-UX adopted a newer version of this, the /opt directory (for optional). The way HP-UX adopted it (which seems to be not all that unusual) every added program stuffs all of its information under its own directory. For example, /opt/ldapux contains the etc files, lib files, man files, sbin files and so forth for HP-UX LDAP.

In addition to the “new” HP-UX standard (introduced about HP-UX 10) the /usr/local directories remain available and very occasionaly used - but most of what was not to go into /opt but into /usr/local actually wound up going into /usr/contrib/bin. This means that there were generally three locations for all HP-UX binaries:

• /usr/local
• /opt
• /usr/contrib

What’s more, because of the layout of the /opt directories (with a great many /opt/*/bin and /opt/*/sbin directories - as well as a great many /opt/*/lib directories as well) the system PATH variable could be very long.

The actual system PATH is stored in /etc/PATH; each added program option was to update /etc/PATH (and other files in /etc such as SHLIBPATH) to match their directories. The easiest way to reset the path in HP-UX is to use this command:

export PATH=$(cat /etc/PATH)

Then add in any extra locations you want (separated by colons).

If you wish to examine a runaway program outside of its element, you may choose to use the utility gcore. This utility is found in Solaris, Linux, and HP-UX, and perhaps others. The program syntax is:

gcore [ -o corename ] pid

The pid is the process id of the process to dump core, and the corename is the base of the filename to use for the core dump - the full name is the base name plus period (”.”) and the process id number. The default is to use “core“.

HP-UX systems will accept multiple process ids instead of just one. Solaris has several additional flags (as well as multiple pids). The additional Solaris flags won’t be covered here.

Once core has been dumped, the program continues operation; it does not stop. Thus, gcore is especially useful for taking a snapshot of a running process.

For example, consider a program with the process id 6674:

gcore 6674

This command generates a core file in the current directory with the name “core.6674“. This file then can be read by the GNU debugger gdb. Solaris also provides the dbx(1), mdb(1), and pstack(1) utilities. HP-UX provides gdb as well as the HP adb(1) utility. Both Solaris and HP-UX provide a core management utility coreadm(1m) - which is a topic for another day.

This article has an excellent description of working with core files in Solaris.

Each UNIX-based operating system seems to take a turn at creating a unique and different startup process, and HP-UX is no different than the others.

The /etc/inittab is a typical inittab file, but as installed doesn’t contain any comments to explain its entries. There is, however, a decent description in the man page inittab(4).

To configure the startup, all the parameters can be found in files in the /etc/rc.config.d directory; this directory is read by /etc/rc.config. Some startup programs read their configuration files directly, and some use /etc/rc.config to read all files.

The startup initialization files are found in /sbin/init.d (which is perhaps unlike most). /sbin/rc is run to begin the startup scripts. As mentioned, these files are in /sbin/init.d, but they are run from /sbin/rc[0-9].d directories. These directories will contain links to files in /sbin/init.d. Across all of these directories, there only needs to be two files: one to start the daemon in the right runlevel, and one in the next lower runlevel to stop it.

For example, consider the cron daemon:

# ls -1Fd /sbin/rc*.d/*cron
/sbin/rc1.d/K270cron@
/sbin/rc2.d/S730cron@

(First a note about the options to ls: -1 means list only filenames, -F adds the filetype designator at the end of the filename, and -d lists directories and not their contents.)

When HP-UX starts up (or changes to a higher runlevel) it runs start scripts (the S### scripts) as it goes from one level to the next until it reaches the desired level. When going to a lower runlevel, the system runs the kill scripts (the K### scripts) in order to stop processes.

The two listed files above (K270cron and S730cron) follow the [SK][0-9][0-9][0-9]script convention, and the file these link to is accordingly /sbin/init.d/cron. The numbers help to define the order that the scripts are run in; scripts are run in alphabetic order as defined by the (system) shell.

The startup scripts themselves are somewhat typical: a first parameter of start will start the service; a first parameter of stop will stop the service. Two other expected parameters are start_msg and stop_msg; these are used during the startup to provide a decently readable account of what is going on.

The scripts should return one of the following values as its exit value:

• 0. Success!
• 1. Failure.
• 2. Noop - translates to N/A (not applicable) during startup; script did nothing.
• 3. Script will cause a reboot (and will display /etc/rc.bootmsg if it contains anything).
• 4 and up. Treated same as 1.

All of the output from the startup and shutdown scripts is stored in /etc/rc.log (and the previous rc.log is moved to rc.log.old). Only the last two startup logs are kept. Here is a snippet of an rc.log:

Old /etc/rc.log moved to /etc/rc.log.old
 
**************************************************
HP-UX Start-up in progress
Thu Dec 13 13:56:28 CST 2007
**************************************************
 
Configure system crash dumps
Output from “/sbin/rc1.d/S080crashconf start”:
—————————-
crashconf: concurrent mode not supported on this platform
EXIT CODE: 0
 
Removing old vxvm files
Output from “/sbin/rc1.d/S090sw_clean_vxvm start”:
—————————-
 
VxVM INFO V-5-2-3360 VxVM device node check
Output from “/sbin/rc1.d/S091vxvm-nodes-check start”:
—————————-
 
VxVM INFO V-5-2-3362 VxVM general startup
Output from “/sbin/rc1.d/S092vxvm-startup start”:
—————————-

More information can be had on the rc(1m) and rc.config(4) man pages.

When root is locked out of a system, it can be a real problem. It’s even worse when you are root, yes?

There are many legitimate reasons this can happen:

• You purchased a used system with operating system installed, and the root login is unknown.
• The root password expired. (This has happened!)
• You have taken on a new system as a system administrator, and no one knows what the root password is (or what the box is, or what it does…)
• You have taken on a job as a new system administrator, and the previous administrator didn’t leave all of the passwords.
• You just plain forgot the system password (uhoh…).

We are going to assume that single user mode is locked out or not available. We are also going to skip talking about taking the system down (which is going to be necessary in any case).  We are also going to assume that you have legitimate physical access to the system.

In most situations, the system admin’s adage holds true: if you have physical access to the box, then there’s no stopping you from breaking in.  That’s why many operating systems make it fairly easy to access the system with physical access - and this is the reason you must physically secure your servers.  Putting them behind locked doors is the easiest and best way to go.  With physical access, it generally becomes easy to get into the system.

If you can access single user mode (such as booting HP-UX with “hpux -is” or booting Linux with “linux single“) then you can do all of the password recovery steps quite easily.

No matter what the operating system, if you have physical access to the system itself, then it is generally possible to crack it. There is a method that will work with any system:

1. Convert the disk into a “secondary” data disk in some fashion.
2. Boot into another operating system that can read and write the now secondary data disk.

There are several ways to do this. One way is to use a bootable floppy, CDROM, or DVDROM to enter into a maintenance mode, then access the original boot disk in the system. Sun Solaris has this capability using the install disk, as does Red Hat Enterprise and OpenSUSE. There are also bootable floppies and CDROMs that run on Linux all over everywhere.

Another way is to physically extract the hard disk from the system, and to use it as a data disk on another (identical) system. Macintosh laptops have this capability “builtin”: it’s called Target Mode, and turns the laptop into a large disk on a Firewire connection to a desktop or other system.

Remember, however, that the Macintosh uses NetInfo to manage its data; you’ll probably have to manipulate the NetInfo database on the other system in some fashion, or at least, turn it off.  That’s currently beyond our knowledge (but who knows…).

Under HP-UX, the boot disk is probably a single volume group, vg00: this volume group would have to be vgexported and vgimported in some fashion. The LVM structures would have to be accounted for.

Once the once bootable system is correctly mounted (read and write) as a data disk on another system, the easiest thing to do (with UNIX and Linux) is to change root’s password to null, by making the password entry empty in the /etc/password file. With NT, you may be able to extract the SAM file and use a cracker like L0pht to reveal the password.

With OpenVMS, the steps are more complicated, but the concept is the same: use a different security environment (an alternate UAF) and change the password in the original environment (the original UAF). First, alter the booting process to make OpenVMS boot into a sysboot> prompt. At this prompt, enter the command to use an alternate UAF:

sysboot> set uafalternate 1
sysboot> continue

The username and password to login will be null, so login by pressing enter until the prompt appears. Then we need to go about changing the password entry for SYSTEM:

$ define/system/executive sysuaf sys$system:sysuaf.dat
$ set def sys$system
$ run authorize
Authorize> modify system/password=newpass/nopwdexp
Authorize> exit

Then restore the alternate UAF setting back to zero so that the system UAF will be used on reboot:

$ run sys$system:sysman
sysman> param use current
sysman> param set uafalternate 0
sysman> param write current
sysman> exit

Then reboot the system and the SYSTEM password should be newpass.

This sequence hints at one reason a root password might “go bad” - if your root password expires, then you are undone. These steps still work, but instead of changing the password, one will have to reset the expired account so you can login again.