Getting and Verifying SSH Fingerprints

Sometimes you may log into a machine, and find that the SSH key has changed. This can happen in a man-in-the-middle attack, but more likely the host has been rebuilt and new keys generated. (This is a good reason to save and restore the SSH keys during a rebuild.)

So how do you check the fingerprints?

You can do it over the network by pulling down the public keys from the remote server:

ssh-keyscan -p 22 -t rsa,dsa remote_host > /tmp/

(Replace remote_host with the appropriate name or IP address.) Then you generate fingerprints from this file:

ssh-keygen -l -f /tmp/

However, if you are investigating why your keys changed mysteriously, you don’t want to do this over the network. Log into the actual server through other means (such as a management console or a KVM console) and generate fingerprints directly:

ssh-keygen -lf /etc/ssh/ssh_host_dsa_key
ssh-keygen -lf /etc/ssh/ssh_host_rsa_key

Then compare this with what you got over the network. If they match, you should be alright; if not, there are some serious problems.

If the fingerprints don’t match, you should do a network scan for ARP requests and see who is answering the ARP requests for the relevant IP address. Scan for ARP requests as you do a ping. If there are two hosts, then they will “fight” each other for an ARP entry, and you should see both responding.

Once you know the Ethernet address of the mysterious host, then chase it down by tracking where the ARP traffic is going by looking at router (or switch) interfaces.

/etc/shadow format

Recently, I found myself trying to understand what was going on in the shadow file. For one thing, the “password” field in Ubuntu Lucid Lynx is much longer than it is in Red Hat Enterprise Linux 5. Then there were the exclamation points in front of some password entries.

Originally, there was no /etc/shadow file – the passwords and salts were contained with other user information in /etc/passwd. The problem was that /etc/passwd was readable by anyone, and that attacks became more common and more powerful. It was necessary to move the password into a separate file that would only be readable by root.

Here is an example of a shadow file line:


Each field is separated by a colon (:). There are a number of fields:

  1. User name. This must match up with a user name in /etc/passwd; it is the key.
  2. Password. This field is a hash of the password (not the actual password itself) combined with an id (specifying which form of encryption was used) and with a salt.
  3. Last changed date. This is the day that the password was last set (in days since January 1, 1970).
  4. Minimum days. This is the minimum number of days that a password must be active before a password can be changed again. A user’s password cannot be changed by them until this number of days has elapsed.
  5. Maximum days. This is the maximum number of days for a password to be valid. Once the maximum number of days has elapsed, the user is forced to change their password.
  6. Warn days. Before a password expires, warn the user this many days ahead of time.
  7. Inactive days. Once a password has expired, it will be disabled after this many days.
  8. Expire date. On this date (in days since January 1, 1970), the account is disabled and the login can no longer be used.

Rather than trying to compute the date from the number of days since 1970, use the chage command:

$ chage -l ddouthitt
Last password change					: May 25, 2011
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

This also explains what each number means, and incorporates any defaults.

The password is more complicated. The actual password is not stored anywhere; the stored value in the /etc/shadow file is a hash generated from the password.

Originally, the password hash was just the hash itself; later, a two-character salt was added at the beginning.

The password hash today comes in three parts, separated by dollar signs ($):

  • Id. This identifies the encryption hash method used. A value of 1 denotes MD5; 2 or 2a is Blowfish; 3 is NT Hash; 5 is SHA-256; and 6 is SHA-512.
  • Salt. This is used by the encryption algorithms, and could be up to 16 characters.
  • Hash. The actual “password” (or hash) is last. MD5 uses 22 characters, SHA-256 uses 43, and SHA-512 uses 86.

Picking apart the example password entry above shows this:

  • Id = 6 (SHA-512)
  • Salt = QR3drPrQ
  • Hash = JLolPKyiVuXvea1F2IpbPx9F9PEV0s/IGcNCpm6ZrBA6AFDvwHPQG7EsTQHuUqxfCvlsuKRb.O7w5RLPyj8nS/

Lastly, the password entry could be missing entirely (which means no password) or it could have an invalid entry (locked account).

Traditionally, locking was done with the asterisk (*) by convention or by the word LOCKED. However, this has been extended – by convention – to a variety of new suggestions. System logins that never had a password use a double exclamation mark (!!) and accounts that have been locked have the valid password entry invalidated by a exclamation mark prefix. Using the prefix in this way allows the password to be reinstated as it was.

FreeBSD uses a slightly different setup, with a file called /etc/master.passwd, an extended /etc/passwd file with password aging included, and support for an extended DES format (which starts with an underline). Extended DES is more secure than the original DES, but is no longer used in favor of the new modular format described above. FreeBSD standardizes on MD5 – like Red Hat, and unlike Ubuntu.

By the way – that’s not my real password – so don’t look at it.

Taking a screenshot in Linux

Taking a screenshot in Linux should be simple – and it is.  Pressing the PrintScreen key is all it takes – but what if….?

There is always more than one way to do things, and taking screen shots is no exception. In fact, it may be rather surprising how many ways it can be done. It can be done from the shell, from a keypress, and from various applications.

Over at Webmaster Tips, there is an excellent article that details most of them – and a followup article that describes a few more. The articles are worth reading – even if they focus on Ubuntu. The tips transfer to other Linux variants, though.

Running a finger daemon?

The finger daemon has been maligned for many years, and not without cause. The original daemons had bugs that caused them to dump core and became widely known as one of the ways the original Internet worm (the Morris Worm) spread across the Internet.

The only problem the finger daemon has today is that it provides user data to anyone on the network. This “information leak” will never be resolved as it is why finger exists in the first place.

However, if you put aside the fact that showing user information may be bad, finger can also be quite useful. It is also possible to run it without any network at all, using local data only. If there is no network access to finger (that is, no finger daemon is running), then the risk from finger is minimal.

There remains the ability to run a finger daemon on a network – cfingerd was written in order to provide secure finger services, including minimizing information leaks and giving people the ability to turn off finger access to particular users, or to create fake users. Those who wish to run a finger daemon would do well to choose cfingerd; it is enormously configurable.

Some systems include finger by default and some do not. Certainly, none come with a finger daemon active or present.

I will probably try cfingerd just to see how it works – but I wouldn’t recommend it on most production systems.

What is your take? I’d be interested in hearing from people who are actually running finger daemons and finding out why (and how).

Logging every shell command

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

export HISTFILE=$HOME/.bash_history
export HISTFILESIZE=2000
export HISTSIZE=1000
export HISTTIMEFORMAT="%a %b %Y %T %z "

typeset -r HISTCONTROL
typeset -r HISTFILE
typeset -r HISTIGNORE
typeset -r HISTSIZE

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 - all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories.  rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on

Mysterious IPs on the Network?

If you see mysterious IPs on the network – and they don’t seem to be actually be doing anything like SMTP or DNS or SSH, there’s an angle you may not have considered.

Do a Google search for the address and the word “default” and see if anything is using the IP address for a default setting. Many routers or other devices come with a default setting and you may have one or more of these on your network which has not yet been configured.

In my case, the culprit was the IP – which turned out to be a Dell iDRAC which was not yet configured.

What’s Wrong with Nagios? (and Monitoring)

Nagios (or its new off-spring, Icinga) is the king of open source monitoring, and there are others like it. So what’s wrong with monitoring? Why does it bug me so?

Nagios is not the complete monitoring solution that many think it is because it can only mark the passing of a threshold: there is basically only two states: good and not good (ignoring “warning” and “unknown” for now).

What monitoring needs is two things: a) time, and b) flexibility.

Time is the ability to look at the change in a process or value over time. Disk I/O might or might not be high – but has it been high for the last twenty minutes or is it just on a peak? Has disk usage been slowly increasing or did it skyrocket in the last minute? This capability can be provided by tools like the System Event Correlator (SEC). The biggest problem with SEC is that it runs by scanning logs; if something isn’t logged SEC won’t see it.

The second thing is what drove me to write: there is no flexibility in these good/not-good checks that Nagios and its ilk provide. There is also not enough flexibility in SEC and others like it.

What is needed is a pattern recognition system – one that says, this load is not like the others that the system has experienced at this time in the past. If you look at a chart of system load on an average server (with users or developers on it) you’ll see that the load rises in the morning and decreases at closing time. When using Nagios, the load is either good or bad – with a single value. Yet a moderately heavy load could be a danger sign at 3 a.m. but not at 11 a.m. Likewise, having 30 users logged in is not a problem at 3 p.m. on a Tuesday – but could be a big problem at 3 p.m. on a Sunday.

What we really need is a learning system that can match the current information from the system with recorded information from the past – matched by time.

It’s always been said that open source is driven by someone “scratching an itch.” This sounds like mine…

Linux Kernel sync() Bug

There is a bug in the Linux kernel (up to 2.6.37) that can result in a sync() command taking many minutes (instead of several seconds).

The exact nature of the bug is unclear, but it shows up in the use of dpkg – since dpkg now uses sync() instead of fsync(). The result is updating an Ubuntu or Debian system can hang when dpkg goes to synchronize the file system.

The symptom also shows up by using the sync command directly.

The dpkg command was updated in 1.16.0 to include a force-unsafe-io option which re-enables the previous behavior which bypasses this bug. This version is not yet available in Ubuntu Lucid Lynx (10.04) but should be showing up in 10.04 LTS sooner or later. The option can be added to /etc/dpkg/dpkg.cfg or to a file in /etc/dpkg/dpkg.cfg.d to make it a default setting.

There was also a suggestion by Theodore T’so that upgrading to dpkg would be sufficient to work around this kernel bug even without the force-unsafe-io option.

It is possible to upgrade to dpkg 1.16.0 on Lucid, but it requires pulling in the Natty Narwhal repositories and setting apt tp prefer the Lucid repositories, then pulling in the Natty version of dpkg specifically.

Suggestions on how to fix the problem (aside from replacing the kernel) now include:

None of these answers are definitive and the search continues. Ubuntu bug #624877 is on this topic, as is Linux kernel bug #15426.

Correction: bug #15426 is a Linux kernel bug report, not a Debian bug report. We apologize for the error.

Net-SNMP SMUX Fails on Ubuntu: A Fix

When trying to set up Dell OpenManage on some servers, I found that SMUX was not working. SMUX is a protocol that allows agents to connect to a SNMP daemon and provide answers to SNMP queries in a portion of the tree.

When working, SNMP should generate this log message in /var/log/daemon.log:

May 11 17:14:18 serverx snmpd[29678]: accepted smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager

In my case, I saw this instead:

May 11 17:06:59 serverx snmpd[29471]: /etc/snmp/snmpd.conf: line 370: Warning: Unknown token: smuxpeer.

After a long time fussing with SNMP and Dell OpenManage, it turned out that the problem was that the SMUX subsystem was being disabled at daemon startup by an option set in /etc/default/snmpd. Using the -I option will turn on (or off) a particular module used by snmpd. In this case, the line looked like this:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/'

With this configuration, the SMUX module is disabled. For snmpd to support SMUX, the line should look like this instead (removing the -I option and its argument):

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -p /var/run/'

After making the change, restart the daemon:

service snmpd restart

This should then fix problems in using Dell OpenManage (or other SMUX agents). You don’t have to restart Dell OpenManage to make this work, but it should have SNMP enabled (which provides the smuxpeer line in snmpd.conf).

To enable SNMP for Dell OpenManage, use this:

service dataeng enablesnmp

To restart the Dell OpenManage services, don’t use the usual services; use this command instead (which takes care of all Dell OpenManage services): restart

Why SMUX should be disabled I couldn’t say. The system is running Ubuntu 10.04.2 LTS with snmpd

Naming Those Servers

When it comes time to name a server, there are a few places to go for ideas – and a few things to remember.

One of my favorites is to use Google Sets. If you give Google several entries, Google will come back with more that match your list. This can be handy when you are trying to come up with a new name in a list.

An unknown champion in this area would be the site Once you’ve found this site, you’ll want to bookmark it – there’s nothing like it out there. If some scheme is missing – you can add it, since it is an open wiki.

Lastly, there is a question over at ServerFault which has some fabulous naming schemes that people have used (my all time favorite has to be the one that used names from the RFB list!). Another fantastic list from that question involves something like the players name’s from the famous Abbot and Costello skit Who’s on First.

There is also an IEEE RFC on Naming Your Computer (RFC-1178) which has good tips.

In my background, I’ve used (or seen used – or heard of) schemes based on the Horses of the Apocolypse (“war”, “famine”, etc.), colors, headache relief (“aspirin”, “acetominophen”, etc.), “test” and synonyms (“test”, “quiz”, “exam” – for testing servers!), cities, and fruits. The cities environment was fun – the hardware had two partitions, so naming the partitions was done with cities that were near to each other. The headache relief was fun until the pharmaceutical company found some of the names were of competing drugs…

Think about your names – and have some fun, too!


Get every new post delivered to your Inbox.

Join 43 other followers