SystemTap (and DTrace)

SystemTap is one amazing piece of work – it is a programmer-friendy and admin-friendly interface to KProbes (which are included in the Linux 2.6 kernel).  When you compare its capabilities to what has gone before, it is truly amazing.  Here are some of the things you can do:

• Quantify disk accesses per disk per process (or per user)
• Quantify the number of context switches that are a result of time outs
• List all accesses to a particular file and the process that accessed it

This is only the tip of the iceberg. There is a wiki with more details, including “war stories.”  There is a language reference there as well.

There was an excellent article in Red Hat Magazine, “Instrumenting the Linux Kernel with SystemTap” by William Cohen.

One controversy that came up was that the initial impetus for creating SystemTap was to implement something like Sun’s DTrace for Solaris but under the GNU Public License.  Solaris and DTrace are licensed with Sun’s Common Development and Distribution License (CDDL), which many feel makes DTrace incompatible with the GPL-licensed Linux kernel.

Apparently, the CDDL is also incompatible with the BSD-licensed FreeBSD, as FreeBSD 7.0 will not have DTrace either.  There appears to be some licensing issues.

According to the Wikipedia entry on the CDDL, it was designed to be both GPL-incompatible and BSD-incompatible.  With regard to the GPL, the entry suggests that Sun never clarified why; as to the BSD, Sun did not want Solaris to wind up in proprietary products – which the BSD license allows.

On a brighter note, Eugene Teo was able to get the SystemTap tool to work on the Nokia N800.  The article seems to be behind a wall at LiveJournal; the article is still in Google’s cache.  However, it does requires some amazing convolutions:

• A kprobes-enabled kernel must be installed on the N800
• The SystemTap programs (like stap) must be installed on the N800
• Any traces must be cross-compiled on another host
• The kernel module thus created must be moved to the N800
• Once the kernel module is in place, then the trace can be done.

So every desired trace requires precross-compilation on a desktop (sigh)…  Oh, well.

There is even a GUI for SystemTap in the works.