How much can you find out about a HP-UX process?

The answer to this question can be important many times. Let’s take some examples of what can be done to find out all we can about a particular process.

There are, of course, simple things that can be done. Let’s take midaemon as an example. From the command line, we can find out where it is, what it is, and some description of it:

# type midaemon
midaemon is /opt/perf/bin/midaemon
# what `which midaemon`
/opt/perf/bin/midaemon:
        midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
# file `which midaemon`
/opt/perf/bin/midaemon: ELF-32 executable object file - IA64
# ldd `which midaemon`
        libpthread.so.1 =>      /usr/lib/hpux32/libpthread.so.1
        libIO.so =>     /opt/perf/lib/hpux32/libIO.so
        libc.so.1 =>    /usr/lib/hpux32/libc.so.1
        libdl.so.1 =>   /usr/lib/hpux32/libdl.so.1
# man midaemon
# cd /sbin/init.d
# grep midaemon
# cd /etc/rc.config.d
# grep -i midaemon *
# swlist -l file | grep midaemon
  MeasurementInt.MI: /opt/perf/bin/midaemon
  MeasurementInt.MI: /opt/perf/man/man1/midaemon.1
  MeasurementInt.MI-JPN: /opt/perf/man/ja_JP.SJIS/man1/midaemon.1
#

This tells us a lot already: it’s part of the performance system (/opt/perf) and is 32-bit and is part of the MeasurementInt package (and has a Japanese man page!). The man page explains the program in detail.

But there’s more. Let’s suppose that lsof is on hand (as it should be!); then we can do this:

# lsof -c midaemon
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
midaemon 2198 root  cwd    DIR 64,0x3     8192     2 /
midaemon 2198 root  txt    REG 64,0x5   828932 13799 /opt/perf/bin/midaemon
midaemon 2198 root  mem    REG 64,0x8    19799   956 /usr/lib/tztab
midaemon 2198 root  mem    REG 64,0x8    87900    78 /usr/lib/hpux32/libnss_dns.so.1
midaemon 2198 root  mem    REG 64,0x8   169104   722 /usr/lib/hpux32/libnss_files.so.1
midaemon 2198 root  mem    REG 64,0x8    76236 19454 /usr/lib/hpux32/libdl.so.1
midaemon 2198 root  mem    REG 64,0x8  4929272   695 /usr/lib/hpux32/libc.so.1
midaemon 2198 root  mem    REG 64,0x5   115124 13809 /opt/perf/lib/hpux32/libIO.so
midaemon 2198 root  mem    REG 64,0x8  1505144   734 /usr/lib/hpux32/libpthread.so.1
midaemon 2198 root  mem    REG 64,0x8  1065976 19453 /usr/lib/hpux32/dld.so
midaemon 2198 root  mem    REG 64,0x8   176988 19535 /usr/lib/hpux32/uld.so
midaemon 2198 root    2u   REG 64,0x9     1174 17923 /var (/dev/vg00/lvol9)
midaemon 2198 root    3u   REG 64,0x9     1174 17923 /var (/dev/vg00/lvol9)
midaemon 2198 root    4u   REG 64,0x9    11303 17949 /var (/dev/vg00/lvol9)
midaemon 2198 root    5u   REG 64,0x9    11303 17949 /var (/dev/vg00/lvol9)
midaemon 2198 root    7r   REG 64,0x9    13689  1620 /var/opt/perf/parm

This shows that the working directory is / (root); stdin and stdout are closed (0u and 1u in the FD column); stderr is still open and tied to /var; and there are four other file descriptors open: three on /var and one is the /var/opt/perf/parm file (configuration). We can also deduce that there was another file descriptor opened which is now closed (and would have been 6u).

There is also no network connections open, or pipes, or other things.

The ps output provides more details:

# ps -elf | sed -n '1p; /midaem[.]*on/p;'
  F S      UID   PID  PPID  C PRI NI             ADDR   SZ            WCHAN    STIME TTY       TIME COMD
541 R     root  2198     1  0 -16 20 e00000060de31b80  524                -  Jan 15  ?        28:55 /opt/perf/bin/midaemon

From this we can see it is relatively small (SZ = 524). This example also shows a couple of tricks: using sed this way keeps the header intact (1p) and also matches midaemon without matching the search string.

Using glance, we can find out even more. Using the text mode command glance, first select the process (using the command key s and entering the pid – 2198). Then a view of the current activity by the process is given. In this case, we can see the total size is 51.6Mb (VSS) and in memory size is 44.8Mb (RSS). We can also see that the process appears to be switching voluntarily almost all of the time – that is, it never utilizes its full time slice when scheduled.

From that process summary display, enter the command key M. This provides a detailed memory display of the process – very useful. The various types of memory used by the process are broken down at the bottom in summary: text refers to the program code; data is program data; stack is a working area as well as where function calls are stored; shmem refers to shared memory (memory shared between processes); and other, which is everything else. All these areas are shown explicitly above in the main display.

Using the command key F, we can see again what lsof showed us. With an inode number, we can search for the file explicitly. Using lsof:

# lsof  | sed -n '1p;  / 17949 /p'
COMMAND     PID     USER   FD   TYPE             DEVICE    SIZE/OFF    NODE NAME
scopeux    2150     root    0u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    1u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    2u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    4u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    5u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
midaemon   2198     root    4u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
midaemon   2198     root    5u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
# lsof  | sed -n '1p;  / 17923 /p'
COMMAND     PID     USER   FD   TYPE             DEVICE    SIZE/OFF    NODE NAME
midaemon   2198     root    2u   REG             64,0x9        1174   17923 /var (/dev/vg00/lvol9)
midaemon   2198     root    3u   REG             64,0x9        1174   17923 /var (/dev/vg00/lvol9)
#

It would appear that scopeux (another command) is sharing a file with midaemon (inode 17949) on /var, and that inode 17923 is not shared. Since there is no file listed, it is likely that these files were created, then deleted after opening. (The inode remains, but the file is not listed in the directory).

Another useful tool is tusc:

sybil # tusc 2198
( Attached to process 2198 ("/opt/perf/bin/midaemon") [32-bit] )
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. [sleeping]
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080b50, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080bf0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080c90, 0x80000, 0x7ffff860) ............................................................... = 8
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
( Detaching from process 2198 ("/opt/perf/bin/midaemon") )

The tusc command will show you what the process is doing, and what system calls it is making. If the process can be started from scratch (by restarting the program binary) then a lot of information can be gathered using tusc.

A summary view of this same data can be gotten from glance, using the L command key to show the system calls made and the time spent in each one. Just ask tusc related, in this case ki_call(), ksleep(), and kwakeup() are the three system calls be done.

Again using glance, if you want to see the wait states for the process (reasons the process gives up the CPU to other processes) use the W key command. For midaemon, it shows sleep as the reason for 85% of wait states in this process.

We can look through the binary for even more detail:

# strings `which midaemon` | head -n 7
/var/opt/perf/status.mi
/var/opt/perf/status.mi
/dev/ptym/
@$Header: miflock.c,v 1.2 95/09/27 08:43:20 thierry Exp $
@(#)midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
-pstat_freq
        4p
# tail -n 30 /var/opt/perf/status.mi
midaemon: Tue Oct 28 23:53:34 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Wed Oct 29 03:31:41 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Wed Oct 29 03:39:56 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Tue Nov 11 19:10:11 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Tue Nov 11 19:21:32 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Fri Nov 21 21:30:21 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Fri Nov 21 21:38:29 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Fri Nov 28 10:15:28 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Wed Dec 10 11:41:26 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Thu Jan 15 21:31:06 2009
Stop midaemon - Commanded MI termination
midaemon: Thu Jan 15 21:42:42 2009
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Thu Jan 15 21:55:53 2009
Stop midaemon - Commanded MI termination
midaemon: Thu Jan 15 22:03:59 2009
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=