How much can you find out about a HP-UX process?

The answer to this question can be important many times. Let’s take some examples of what can be done to find out all we can about a particular process.

There are, of course, simple things that can be done. Let’s take midaemon as an example. From the command line, we can find out where it is, what it is, and some description of it:

# type midaemon
midaemon is /opt/perf/bin/midaemon
# what `which midaemon`
/opt/perf/bin/midaemon:
        midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
# file `which midaemon`
/opt/perf/bin/midaemon: ELF-32 executable object file - IA64
# ldd `which midaemon`
        libpthread.so.1 =>      /usr/lib/hpux32/libpthread.so.1
        libIO.so =>     /opt/perf/lib/hpux32/libIO.so
        libc.so.1 =>    /usr/lib/hpux32/libc.so.1
        libdl.so.1 =>   /usr/lib/hpux32/libdl.so.1
# man midaemon
# cd /sbin/init.d
# grep midaemon
# cd /etc/rc.config.d
# grep -i midaemon *
# swlist -l file | grep midaemon
  MeasurementInt.MI: /opt/perf/bin/midaemon
  MeasurementInt.MI: /opt/perf/man/man1/midaemon.1
  MeasurementInt.MI-JPN: /opt/perf/man/ja_JP.SJIS/man1/midaemon.1
#

This tells us a lot already: it’s part of the performance system (/opt/perf) and is 32-bit and is part of the MeasurementInt package (and has a Japanese man page!). The man page explains the program in detail.

But there’s more. Let’s suppose that lsof is on hand (as it should be!); then we can do this:

# lsof -c midaemon
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
midaemon 2198 root  cwd    DIR 64,0x3     8192     2 /
midaemon 2198 root  txt    REG 64,0x5   828932 13799 /opt/perf/bin/midaemon
midaemon 2198 root  mem    REG 64,0x8    19799   956 /usr/lib/tztab
midaemon 2198 root  mem    REG 64,0x8    87900    78 /usr/lib/hpux32/libnss_dns.so.1
midaemon 2198 root  mem    REG 64,0x8   169104   722 /usr/lib/hpux32/libnss_files.so.1
midaemon 2198 root  mem    REG 64,0x8    76236 19454 /usr/lib/hpux32/libdl.so.1
midaemon 2198 root  mem    REG 64,0x8  4929272   695 /usr/lib/hpux32/libc.so.1
midaemon 2198 root  mem    REG 64,0x5   115124 13809 /opt/perf/lib/hpux32/libIO.so
midaemon 2198 root  mem    REG 64,0x8  1505144   734 /usr/lib/hpux32/libpthread.so.1
midaemon 2198 root  mem    REG 64,0x8  1065976 19453 /usr/lib/hpux32/dld.so
midaemon 2198 root  mem    REG 64,0x8   176988 19535 /usr/lib/hpux32/uld.so
midaemon 2198 root    2u   REG 64,0x9     1174 17923 /var (/dev/vg00/lvol9)
midaemon 2198 root    3u   REG 64,0x9     1174 17923 /var (/dev/vg00/lvol9)
midaemon 2198 root    4u   REG 64,0x9    11303 17949 /var (/dev/vg00/lvol9)
midaemon 2198 root    5u   REG 64,0x9    11303 17949 /var (/dev/vg00/lvol9)
midaemon 2198 root    7r   REG 64,0x9    13689  1620 /var/opt/perf/parm

This shows that the working directory is / (root); stdin and stdout are closed (0u and 1u in the FD column); stderr is still open and tied to /var; and there are four other file descriptors open: three on /var and one is the /var/opt/perf/parm file (configuration). We can also deduce that there was another file descriptor opened which is now closed (and would have been 6u).

There is also no network connections open, or pipes, or other things.

The ps output provides more details:

# ps -elf | sed -n '1p; /midaem[.]*on/p;'
  F S      UID   PID  PPID  C PRI NI             ADDR   SZ            WCHAN    STIME TTY       TIME COMD
541 R     root  2198     1  0 -16 20 e00000060de31b80  524                -  Jan 15  ?        28:55 /opt/perf/bin/midaemon

From this we can see it is relatively small (SZ = 524). This example also shows a couple of tricks: using sed this way keeps the header intact (1p) and also matches midaemon without matching the search string.

Using glance, we can find out even more. Using the text mode command glance, first select the process (using the command key s and entering the pid – 2198). Then a view of the current activity by the process is given. In this case, we can see the total size is 51.6Mb (VSS) and in memory size is 44.8Mb (RSS). We can also see that the process appears to be switching voluntarily almost all of the time – that is, it never utilizes its full time slice when scheduled.

From that process summary display, enter the command key M. This provides a detailed memory display of the process – very useful. The various types of memory used by the process are broken down at the bottom in summary: text refers to the program code; data is program data; stack is a working area as well as where function calls are stored; shmem refers to shared memory (memory shared between processes); and other, which is everything else. All these areas are shown explicitly above in the main display.

Using the command key F, we can see again what lsof showed us. With an inode number, we can search for the file explicitly. Using lsof:

# lsof  | sed -n '1p;  / 17949 /p'
COMMAND     PID     USER   FD   TYPE             DEVICE    SIZE/OFF    NODE NAME
scopeux    2150     root    0u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    1u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    2u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    4u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
scopeux    2150     root    5u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
midaemon   2198     root    4u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
midaemon   2198     root    5u   REG             64,0x9       11303   17949 /var (/dev/vg00/lvol9)
# lsof  | sed -n '1p;  / 17923 /p'
COMMAND     PID     USER   FD   TYPE             DEVICE    SIZE/OFF    NODE NAME
midaemon   2198     root    2u   REG             64,0x9        1174   17923 /var (/dev/vg00/lvol9)
midaemon   2198     root    3u   REG             64,0x9        1174   17923 /var (/dev/vg00/lvol9)
#

It would appear that scopeux (another command) is sharing a file with midaemon (inode 17949) on /var, and that inode 17923 is not shared. Since there is no file listed, it is likely that these files were created, then deleted after opening. (The inode remains, but the file is not listed in the directory).

Another useful tool is tusc:

sybil # tusc 2198
( Attached to process 2198 ("/opt/perf/bin/midaemon") [32-bit] )
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
In user-mode ......................................................................................................... [sleeping]
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. [sleeping]
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080b50, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080bf0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
ki_call(KI_TRACE_GET, 0x40080c90, 0x80000, 0x7ffff860) ............................................................... = 8
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ki_call(KI_TRACE_GET, 0x40080ab0, 0x80000, 0x7ffff860) ............................................................... = 8
kwakeup(PTH_CONDVAR_OBJECT, 0x400108b0, WAKEUP_ONE, 0x7ffff7c0) ...................................................... = 0
ksleep(PTH_CONDVAR_OBJECT, 0x400108b0, 0x400108b8, NULL) ............................................................. = 0
( Detaching from process 2198 ("/opt/perf/bin/midaemon") )

The tusc command will show you what the process is doing, and what system calls it is making. If the process can be started from scratch (by restarting the program binary) then a lot of information can be gathered using tusc.

A summary view of this same data can be gotten from glance, using the L command key to show the system calls made and the time spent in each one. Just ask tusc related, in this case ki_call(), ksleep(), and kwakeup() are the three system calls be done.

Again using glance, if you want to see the wait states for the process (reasons the process gives up the CPU to other processes) use the W key command. For midaemon, it shows sleep as the reason for 85% of wait states in this process.

We can look through the binary for even more detail:

# strings `which midaemon` | head -n 7
/var/opt/perf/status.mi
/var/opt/perf/status.mi
/dev/ptym/
@$Header: miflock.c,v 1.2 95/09/27 08:43:20 thierry Exp $
@(#)midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
-pstat_freq
        4p
# tail -n 30 /var/opt/perf/status.mi
midaemon: Tue Oct 28 23:53:34 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Wed Oct 29 03:31:41 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Wed Oct 29 03:39:56 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Tue Nov 11 19:10:11 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Tue Nov 11 19:21:32 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Fri Nov 21 21:30:21 2008
Stop midaemon - non-permanent/no-client, normal MI termination
midaemon: Fri Nov 21 21:38:29 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Fri Nov 28 10:15:28 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Wed Dec 10 11:41:26 2008
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Thu Jan 15 21:31:06 2009
Stop midaemon - Commanded MI termination
midaemon: Thu Jan 15 21:42:42 2009
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=
midaemon: Thu Jan 15 21:55:53 2009
Stop midaemon - Commanded MI termination
midaemon: Thu Jan 15 22:03:59 2009
Start midaemon       C.04.70.000  10/03/07 HP-UX 11 =*=

Tips on using lsof

The utility lsof is a relatively new (well, compared to UNIX anyway) that has more options than even ls. These options provide for some extremely powerful capabilities, some of which we aim to illuminate here.

My favorite use for lsof is for networking: all sockets can be seen with the following options:

lsof -n -i

The -n option prevents lsof from being slowed down by a large number of DNS lookups, and the -i option returns all TCP/IP connections (with process numbers, user ids, file descriptor ids, and so on). To narrow it down, utilize options like the following – to list all SMTP connections, for example:

lsof -n -i :25

It is also possible to list only certain processes (such as process 25 and process 45):

lsof -n -p 25 -p 45

Alternately, the process can be selected by name:

lsof -n -c perl

However, suppose one wants to list all TCP/IP sockets held open by perl processes. The obvious choice does not work! This is because the options are combined together as an OR function; to combine them as an AND function (that is, all options must be satisfied) use the -a option – such as this:

lsof -a -n -c perl -i

This lists, as desired, all TCP/IP sockets held open by perl processes.

Another that might be useful in a security context is listing all files that are open but have no links to them: that is, they’ve been deleted, but one or more processes are keeping the file open, which means the file itself (and its blocks) are being preserved even though it appears to be deleted from the filesystem. To see these files, use this option:

lsof +L1

The utility lsof is indeed very useful, and reading the man page for lsof is recommended.

Dealing with “Hidden” Files

Files can be “hidden” on a filesystem in several ways; some are just attempts to “hide in plain sight.” This is not hiding data in files (such as steganography) but rather hiding the mere existence of files themselves.

There are several ways to do this. The basic ideas are:

  • Make reporting tools lie about the existence of files. This falls into the realm of “rootkits.” Utilities such as ls (and even the kernel itself) may modified in order to not report the existance a particular file. A way to avoid this is to use programs (during examination) that are statically linked and precompiled and saved to read-only media: such programs are immune to future modifications, except for kernel resources (if the kernel is taken over, then all other resources are suspect).
  • Make files “disappear” in file listings. Files of this sort include “.. ” (two dots followed by a space) and other non-standard characters. These files look normal but are not. One way to see files like these is to use the ls command and its -b and -q options. The -b option prints the octal equivalent of non-printing characters; the -q option uses the “?” character instead of non-printing characters. The -F option may also be of use: a file with a terminating space will show up when the -F places its “type” character at the end of the file name.
  • Open the file, then delete it. As long as the file is in use, its disk blocks remain in use, and the file remains available. No other processes can access it – or even see it – but the program (or programs) that opened it still has access to it. Of course, this only works for running programs, but files disappear this way almost entirely. To see the files, use the utility lsof with the +L1 option. The +L option lists files with less than that many links; thus, +L1 lists files with less than one link (or zero links).

As mentioned previously, a static build of all relevant utilities to CDROM can be a very useful tool during investigation of a possible system break-in. All general reporting tools should be included: ls, ps, top, etc. Also included should be any desired programming interpreters: ksh, perl, ruby, tclsh, lua (whatever one finds useful). Also included should be any more intensive tools as well: tcpdump, nmap, etc.

Listing shared libraries in running processes

The utility lsof is a very useful utility, and can be used to list the shared libraries being used by a running process. It can be important to know if a running process is using a particular library, perhaps for forensics reasons or for library upgrades.

To list all the libraries in a particular process, try this command:

lsof -a -c name +D /usr/lib

This will list all files used by name in /usr/lib. To list all files used by name, just use:

lsof -c name

Alternately, to find all processes using a file (library) in /usr/lib, use this command:

lsof /usr/lib/libname

The -c option specifies the beginning of a name of a process to list. The -a option is used to create a boolean AND set; otherwise, lsof assumes a boolean OR set of options. With the +D option (which scans for files recursively down the directory tree), the first example looks for the process name that also has open files from the /usr/lib directory tree.

Another good use of lsof has to do with finding files that are open but deleted. Such a situation could potentially happen with a shared library if the library was deleted while a file was using it. This could perhaps happen during a library upgrade. Use this command to do this:

lsof +L1

The +L option specifies files with a specific number of links; here, any file with less than one link (that is, zero links) will be listed. Files with zero links are not listed in the filesystem but are open and in use by a file. The blocks from such files remain marked as in use by the filesystem, but the file cannot be found by name anywhere and has no inode.

There is a nice concise article by Joe Barr at Linux.com about what you can do with lsof. Lsof is available for download.