Turning off NET-SNMP overlogging

In the normal configuration – both on Red Hat and Ubuntu – you’ll find that SNMP is filling your logs with an endless amount of log entries, especially if you have monitoring tools that use SNMP every five minutes. They’ll generate messages like this:

Jan  8 13:45:02 example snmpd[2048]: Connection from UDP: [10.0.0.1]:51890
Jan  8 13:45:02 example snmpd[2048]: Received SNMP packet(s) from UDP: [10.0.0.1]:51890
Jan  8 13:45:02 example last message repeated 2 times

To get rid of these, change the priority levels that are logged by NET-SNMP. This can be done by changing the options sent to SNMP.

Look for a file /etc/default/snmpd or /etc/sysconfig/snmpd or similar. There should be a set of SNMP options – probably with an option like this one:

-Ls d

Change this option to be:

-LS5d

This will log everything at level NOTICE or higher (that is, severity level 5 down to severity 0). The severity levels used are those used by syslog; they are described in syslog(3).

This works because the messages being seen are logged at level INFO; by not logging items at that severity level the log entries no longer clutter the syslog files.

However, there is another set of messages that are common to NET-SNMP logs:

Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 9 request
Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 10 request
Sep  7 09:47:29 burp snmpd[19242]: diskio.c: don't know how to handle 11 request

This is the result of a bug (Red Hat Bugzilla #474093 – login required) which causes these “errors” when the SNMP diskIOTable is traversed. Red Hat fixed this bug back in September of 2009.

According to this message by Chris Rizzo, Red Hat stated:

The message that you see here is a result of querying for statistics
that are not available on the linux system. Requests 9, 10, and 11 are
defined as:

#define DISKIO_LA1 9
#define DISKIO_LA5 10
#define DISKIO_LA15 11

You can see where these statistics pop up by querying the SNMP diskIOTable using this command:

snmptable -v 2c -c public host diskIOTable

The output will look like this:

SNMP table: UCD-DISKIO-MIB::diskIOTable

 diskIOIndex diskIODevice diskIONRead diskIONWritten diskIOReads diskIOWrites diskIOLA1 diskIOLA5 diskIOLA15 diskIONReadX diskIONWrittenX
           1         ram0           0              0           0            0         ?         ?          ?            0               0
           2         ram1           0              0           0            0         ?         ?          ?            0               0
           3         ram2           0              0           0            0         ?         ?          ?            0               0
           4         ram3           0              0           0            0         ?         ?          ?            0               0
           5         ram4           0              0           0            0         ?         ?          ?            0               0
           6         ram5           0              0           0            0         ?         ?          ?            0               0
           7         ram6           0              0           0            0         ?         ?          ?            0               0
           8         ram7           0              0           0            0         ?         ?          ?            0               0
           9         ram8           0              0           0            0         ?         ?          ?            0               0
          10         ram9           0              0           0            0         ?         ?          ?            0               0
          11        ram10           0              0           0            0         ?         ?          ?            0               0
          12        ram11           0              0           0            0         ?         ?          ?            0               0
          13        ram12           0              0           0            0         ?         ?          ?            0               0
          14        ram13           0              0           0            0         ?         ?          ?            0               0
          15        ram14           0              0           0            0         ?         ?          ?            0               0
          16        ram15           0              0           0            0         ?         ?          ?            0               0
          17        loop0           0              0           0            0         ?         ?          ?            0               0
          18        loop1           0              0           0            0         ?         ?          ?            0               0
          19        loop2           0              0           0            0         ?         ?          ?            0               0
          20        loop3           0              0           0            0         ?         ?          ?            0               0
          21        loop4           0              0           0            0         ?         ?          ?            0               0
          22        loop5           0              0           0            0         ?         ?          ?            0               0
          23        loop6           0              0           0            0         ?         ?          ?            0               0
          24        loop7           0              0           0            0         ?         ?          ?            0               0
          25          sr0           0              0           0            0         ?         ?          ?            0               0
          26          sda  2840214016     1178369536     8946299      2080062         ?         ?          ? 990682692096     18358238720
          27         sda1      598016         208896          82            8         ?         ?          ?       598016          208896
          28         sda2        2048              0           2            0         ?         ?          ?         2048               0
          29         sda3     2286592        4649984         449          332         ?         ?          ?      2286592         4649984
          30         sda5  2836463104     1173473792     8945636      2079527         ?         ?          ? 990678941184     18353342976
          31          sdb  2960873984     1173473792     1940422      2079476         ?         ?          ? 990803352064     18353342976
          32         sdb1      638976              0          83            0         ?         ?          ?       638976               0
          33         sdb2      798720              0         153            0         ?         ?          ?       798720               0
          34         sdb3        6144              0           2            0         ?         ?          ?         6144               0
          35         sdb5  2958672384     1173473792     1940080      2079284         ?         ?          ? 990801150464     18353342976
          36          md0  3051716608     1727252992       93765      1641387         ?         ?          ?   3051716608     14612154880
          37          sdc  3067452416      425118208     1640751      3638614         ?         ?          ? 101851700224    438511782400
          38         sdc1  3067317248      425118208     1640721      3638613         ?         ?          ? 101851565056    438511782400
          39          sdd      307712              0          76            0         ?         ?          ?       307712               0
          40         sdd1      147968              0          37            0         ?         ?          ?       147968               0
          41         sdd2      131072              0          32            0         ?         ?          ?       131072               0

Towards the right side of center, you can see the metrics diskIOLA1, diskIOLA5, diskIOLA15; these are unsupported on Linux (as marked by the ? in each column). These are the 1 minute average disk load (as a percentage), the 5 minute average disk load, and the 15 minute average disk load respectively.

The three have SNMP OIDs of .1.3.6.1.4.1.2021.13.15.1.1.9 and .1.3.6.1.4.1.2021.13.15.1.1.10 and .1.3.6.1.4.1.2021.13.15.1.1.11 respectively – thus, the logged complaint of not knowing how to handle request 9 (or 10 or 11).

Without changing the code, there doesn’t seem to be any way to eradicate this message if you are querying the diskIOTable. Red Hat fixed the bug, perhaps others will? The bug remains on Ubuntu Lucid Lynx, unfortunately.

Securing your network traffic

If you want to start some exciting discussion in a security forum, just say you use telnet: you’ll find that every admin knows that telnet is insecure, that one should use OpenSSH or similar to encrypt the traffic, and that telnet should be banned from the server environment entirely.

However, telnet is not the only server that transmits its passwords in the clear. There are a lot of others. Here’s a list I came up with:

  • FTP
  • HTTP
  • IMAP
  • IPP
  • LDAP
  • LPD
  • NFS
  • POP3
  • rsync
  • SMTP
  • SNMP
  • syslog
  • VNC
  • X11
  • XDMCP

I won’t cover all of these here (more about these items can be found in my book) but I do want to cover just a few.

Consider, for example, the mail protocols: SMTP, POP3, and IMAP. SSL encryption is available with all three – but do you use it? And what about your logins to your mailbox at your ISP? Every time you login, your password to your mailbox goes across the wire in the clear.

What about NFS – particularly NFS home directories? If you have unencrypted secrets in your home directory, then these items will be transmitted across the network in the clear as well. What about private SSH keys? Unfortunately, there is no way to encrypt NFS traffic.

VNC is another one to watch for: if you type passwords for your root logins over VNC – even if you are using SSH in your VNC session – the passwords are in the clear. The only way to secure VNC entirely is to use an SSH tunnel to encrypt it.

X11 is insecure in the same way, but presents special problems. However, OpenSSH handles X transparently through the use of special tunnels just for X.

syslog is another unencrypted service; do you have passwords put into the system logs? What about secret doings of your servers? How much information leakage can you handle? Unfortunately, syslog is another service that cannot be secured unless you use something such as syslog-ng which permits you to use TCP (and thus, an OpenSSH tunnel).

Syslog: “could not change the compartment” (HP-UX)

There is recurring error that keeps putting a message into the syslog.log file every time a cron job runs under HP-UX 11i v3. This message can safely be ignored, but if you run a lot of cron jobs it can keep filling your syslog with useless messages. The message like thus:

Feb 28 16:26:00 mysys syslog: Cron daemon - could not change the compartment for the job /usr/bin/foo

These messages will continue to appear for every cron job. It refers to a product which in this case is not installed on the machine (that is the “compartments” of which it speaks).

There is a patch available from HP to fix this, but even after a year this patch remains off of the recommended list. The patch is PHCO_36539 and it will take care of the problem, returning your cron messages to a sane level.