OpenSolaris on a MacBook

OpenSolaris is very interesting, and since the introduction of dtrace and ZFS has enthralled many. I tried to install it onto my HP Compaq E300 laptop (which it was unsuitable for), and tried to install it onto an HP Compaq 6910p laptop. In this case, the networking was unsupported: both the ethernet and the wireless drivers were not included with OpenSolaris Express (Developer Edition).

In any case, I expect I might just be shopping for a laptop in the next year – and it’s nice to see that OpenSolaris does run on the Apple MacBook.  This article goes into detail about how the writer got it to work, and each of the steps that were taken to make it happen.  Paul Mitchell from Sun discusses dual-partitioning a MacBook in this context as well.  Alan Perry (also from Sun) had done the same thing with a Mac Mini, and Paul extended it to the MacBook.  Both entries are detailed and have to do with MacOS X and Solaris dual-booting.

An a different note, check out the graph of library calls from dtrace in this article.  From what I’ve heard of dtrace, it’s the ultimate when it comes to debugging…

Researching the Dynamic Loader on Any System

In finding information about the dynamic loader and shared libraries in general, there are several places to look. The obvious Internet location is Google; however, the system itself has a lot of information about shared libraries if one only knows where to look.

The first place is the information on the dynamic loader’s man page. Unfortunately, most loaders have names that are unique among their UNIX peers (Linux is almost universal). You can start by looking at the /lib directory for a program containing the string “ld” (or perhaps, “dl”). On Linux, this produces:

# ls -d *ld*
ld-2.6.so ld-linux.so.2

The proper Linux loader is ld-linux.so.

On HP-UX, this produces:

# ls -d *ld*
dld.sl* libdld.2* libldap_send.1*
libdld.0@ libdld.sl@ libldap_send.sl@
libdld.1* libldap.sl@ libnss_ldap.1*

Here, the proper loader is dld.sl.

Looking at the man pages for ld-linux.so or dld.sl or whatever was found gives a vast amount of information directly related to the dynamic loader and how it loads shared libraries, as well as debugging tools to report on how the libraries are found and loaded.

This man page will also mention utilities that will help you manipulate shared libraries. For example, the Linux man page for ld-linux.so mentions ldconfig(8); the HP-UX man page for dld.sl mentions the utilities fastbind(8) and chatr(8).

There are other utilities that remain fairly generic and which can help, though these tend to be specific to machines that are configured for development. If the development tools are not loaded, these tools may be missing. These utilities may include:

  • ldd – list libraries used by a binary
  • nm – list symbols from program binaries and/or libraries
  • objdump – display information from binaries and/or libraries
  • readelf – display information from ELF-formatted binaries

Always look at the SEE ALSO section in order find more information.

Shared Libraries (Linux)

The loading of shared libraries in Linux is somewhat complicated (though not any more than other UNIX systems). Firstly, Linux has gone through tremendous growth; as a result, the method and format of loadable libraries has changed several times, necessitating a change in the library loader:

  • ld.so – for a.out formatted binaries
  • ld-linux.so.1 – for ELF binaries that used libc5
  • ld-linux.so.2 - for ELF binaries that use glibc

Nearly everything today will use ld-linux.so.2. All three use the same support files, configuration files, and so forth. The loaders ld-linux.so.1 and ld.so will not be discussed further.

The search for shared libraries goes in this order:

  • Use the binary’s internally stored RPATH (deprecated).
  • Use the environment variable LD_LIBRARY_PATH (except for suid and sgid binaries).
  • Use the binary’s internally stored RUNPATH.
  • Use the cache file /etc/ld.so.cache.
  • Search /lib then /usr/lib.

If the binary being loaded was linked with the -z nodeflib (that is, no default libraries) option then the last two steps are ignored.

Conveniently, the loader is an actual program which can be executed and with a variety of options. There are only two that will be of assistance when debugging the loading of system libraries. To use a specified path instead of LD_LIBRARY_PATH, use the loader this way:

/lib/ld-linux.so.2 --library-path path binary

…where path is the library path, and binary is the program binary to be loaded and executed. To keep the program binary’s preconfigured RPATH and RUNPATH from being used, use this invocation:

/lib/ld-linux.so.2 --inhibit-rpath binary

The loader also recognizes a lot of different environment variables; only a few are relevant for most purposes:

  • LD_LIBRARY_PATH: a path of directories to search for ELF libraries. Formatted like PATH.
  • LD_PRELOAD: a list of libraries (separated by whitespace) to be loaded before all others. For suid and sgid binaries, only libraries in standard search directories that are also suid will be loaded.
  • LD_BIND_NOW: bind (and load) all necessary libraries at startup instead of on demand. This is useful when using a debugger.
  • LD_TRACE_LOADED_OBJECTS: produces output like ldd(8).
  • LD_DEBUG: output debugging information related to the linker. If set to help will provide help for using this variable. If set to all, will provide full and detailed debugging of all portions of the dynamic loading process and of the dynamic loader.

The variable LD_DEBUG can be quite useful when debugging a missing library. The help for LD_DEBUG looks like this:

# LD_DEBUG=help nm --help
Valid options for the LD_DEBUG environment variable are:
.
libs display library search paths
reloc display relocation processing
files display progress for input file
symbols display symbol table processing
bindings display information about symbol binding
versions display version dependencies
all all previous options combined
statistics display relocation statistics
unused determined unused DSOs
help display this help message and exit
.
To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment variable.

To debug a program’s shared libraries, try invoking the program with this environment variable set. As the program executes, each library search and library invocation will be reported. For example:

# LD_DEBUG=libs,versions ls -m
28927: find library=librt.so.1 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/librt.so.1
28927:
28927: find library=libselinux.so.1 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libselinux.so.1
28927:
28927: find library=libacl.so.1 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libacl.so.1
28927:
28927: find library=libc.so.6 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libc.so.6
28927:
28927: find library=libpthread.so.0 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libpthread.so.0
28927:
28927: find library=libdl.so.2 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libdl.so.2
28927:
28927: find library=libsepol.so.1 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libsepol.so.1
28927:
28927: find library=libattr.so.1 [0]; searching
28927: search cache=/etc/ld.so.cache
28927: trying file=/lib/libattr.so.1
28927:
28927: checking for version `GLIBC_2.2' in file /lib/librt.so.1 [0] required by file ls [0]
28927: checking for version `ACL_1.0' in file /lib/libacl.so.1 [0] required by file ls [0]
28927: checking for version `GLIBC_2.2.3' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.4' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.3' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.3.4' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.1.3' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0] required by file ls [0]
28927: checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0] required by file ls [0]

[...Snip...]

The output can be quite verbose, but you can get the idea. This output can be most useful when a library is not being loaded by the loader (when you think it should be).

Warning: be aware, too, that the C compiler (gcc) as of version 4.0 requires a shared library for every program by default. If you are trying to create a static program, it will not truly be static until the gcc library is actually built into the binary. This requires a special gcc option for linking in addition to the standard -static option: -static-libgcc. The -static option alone is not enough. This library can be seen by using the -print-libgcc-file-name option:

# gcc -print-libgcc-file-name
/usr/lib/gcc/i386-redhat-linux/4.1.2/libgcc.a

Note, too, that any library required by a program has its own requirements: thus, if the library requires another library, and cannot find a static build version, then that library will be used as a shared library.

The program ldd can be used (as always) to find the libraries that a program requires:

# ldd $(which ls)
linux-gate.so.1 => (0x00ef8000)
librt.so.1 => /lib/librt.so.1 (0x0585b000)
libselinux.so.1 => /lib/libselinux.so.1 (0x004b2000)
libacl.so.1 => /lib/libacl.so.1 (0x05c0c000)
libc.so.6 => /lib/libc.so.6 (0x00101000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00282000)
/lib/ld-linux.so.2 (0x00dc2000)
libdl.so.2 => /lib/libdl.so.2 (0x00de1000)
libsepol.so.1 => /lib/libsepol.so.1 (0x0046a000)
libattr.so.1 => /lib/libattr.so.1 (0x05c05000)

The ldd program will also report if a library is missing.

Update: I forgot to mention these resources for further information about using (and creating) shared libraries in Linux:

In particular, I found Russ Allbery’s description most illuminating.

Shared Libraries (HP-UX)

On occasion, it may happen that trying to run a binary may provide an error message like the following:

# screen -DR
/usr/lib/hpux32/dld.so: Unable to find library 'libtermcap.so'.
Killed

When a binary is loaded in UNIX (and Linux), shared libraries may need to be loaded (like the above example, libtermcap.so). Each version of UNIX has its own specific method of searching for libraries; specifically, this process is done by a program called ld.so or (on HP-UX) dld.so or similar. The Linux dynamic loader is part of the GNU libc libraries; the loader thus generally works the same across Linux distributions that use GNU libc.

Each program has an embedded library search path, in addition to the system search process. In HP-UX, there are several places that may influence the search for dynamic libraries:

  • LD_LIBRARY_PATH (environment variable – standard PATH format)
  • SHLIB_PATH (environment variable – standard PATH format – from /etc/SHLIB_PATH)
  • LD_PRELOAD (environment variable – specifies library names)
  • rpath (the executable’s embedded path, embedded at link time by ld)
  • /etc/dld.sl.conf (one library directory per line)

The search can also be influenced by a number of factors:

  • How the executable was linked and with which options
  • Whether the executable is setguid or setuid
  • Whether dld.sl.conf has the right permissions or not (must be writable only by root)
  • Whether the system is PA-RISC or Itanium
  • Whether the system is 32-bit or 64-bit
  • What the settings of _HP_DLDOPTS are

On HP-UX for PA-RISC, the loader was dld.sl(5) and all the libraries had .sl extensions. On Itanium, the loader is dld.so(5). Both loaders are loaded and run by a “starter” program called crt0.o (although there are different versions for each architecture).

To get a view of the search process, use the ldd(1) command. Here is an example of successful ldd output:

# ldd `which screen`
libtermcap.so => /opt/termcap-1.3.1/lib/hpux32/libtermcap.so
libelf.so.1 => /usr/lib/hpux32/libelf.so.1
libc.so.1 => /usr/lib/hpux32/libc.so.1
libdl.so.1 => /usr/lib/hpux32/libdl.so.1

Note that if a library cannot be found by the HP-UX loader, an error will be returned about the missing library and nothing else will be reported. In HP-UX, the -s option to ldd provides a detailed list of what directories are being searched.

When you add libraries to the system, make sure to check the following:

  • Does the library directory need to be in /etc/dld.sl.conf? The dld.sl.conf file is used as a very basic search path; it is used when security or other factors mandate that virtually no shared libraries be used. Note that both Itanium and PA-RISC systems use this file (that is, both dld.so and dld.sl check /etc/dld.sl.conf). If you put a directory into dld.sl.conf, it must also be in either SHLIB_PATH (through the use of /etc/SHLIB_PATH) or in LD_LIBRARY_PATH.
  • Check file and path permissions. Libraries must not be world-writable, and the directories in the path must also not be world-writable – and almost certainly, none of these should be group-writable. Make sure that directories in the path are owned by root.
  • Check the executable for setuid and setgid flags. If testing a shared library, check if the executable is setgid or setuid. If these are set, then the loader will strip down the shared library search to almost nil. Check if the library search directory needs to be put in /etc/dld.sl.conf.
  • Check the permissions of /etc/dld.sl.conf. If this file is not writable only by root, then it will be ignored.
  • Check SHLIB_PATH for directories contained in /etc/dld.sl.conf. If a search directory contained in /etc/dld.sl.conf is not located in either SHLIB_PATH (from /etc/SHLIB_PATH) or in LD_LIBRARY_PATH, then it is ignored.
  • Is the directory fully specified from root? If a directory is not fully specified starting with ‘/’ then it wil be ignored.

When debugging the library search path, there are several utilities that can help. The aforementioned ldd -s command is one. Another is the _HP_DLDOPTS environment variable:

  • _HP_DLDOPTS=”-symtab_stat” – get statistical information
  • _HP_DLDOPTS=”-no_setuidpath” – disable all dynamic path lookup for setuid programs
  • _HP_DLDOPTS=”-warnings” – display additional dynamic loader warning messages
  • _HP_DLDOPTS=”-fbverbose” – related to fastbind(1)
  • _HP_DLDOPTS=”-nofastbind” – related to fastbind(1)

In addition, Itanium systems and PA-RISC 64-bit systems will recognize the variable DLD_VERBOSE_ERR: when set to true, all error messages will be displayed.

When programming, it is also possible to manipulate the loader, including loading and unloading of shared libraries. The functions are:

  • dlclose(3C)
  • dlerror(3C)
  • dlget(3C)
  • dlgetname(3C)
  • dlmodinfo(3C)
  • dlopen(3C)
  • dlsym(3C)
  • shl_load(3X)

Next week I’ll talk about shared libraries on Linux.

Follow

Get every new post delivered to your Inbox.

Join 43 other followers