Using SSH Agents with GNU Screen (and Byobu)

SSH is an encryption tool that allows you to connect to machines using an authenticated and encrypted connection. With an SSH agent, your authentication (and keys) can be “carried” from one system to the next. You load all of your keys on your local system into the agent, then connect to a remote system with the agent. Even though none of your keys are present on the remote system, they all exist and can be used to authenticate to another system.

This capability that the SSH agent gives you is very useful: you can keep all of your keys on a laptop or other personal system and only keep public keys on remote systems.

Running the agent is as simple as:

eval ssh-agent

This will load all keys that the agent can find (keys in your .ssh directory). You can add specific keys with:

ssh-add mykey

Replace mykey with your specific key name. If there is a password on the key, you only have to enter it once – at the very beginning.

Once the agent is configured, you can connect to a remote system with:

ssh -A host

The -A option tells SSH to use “Agent Forwarding” which is what allows us to take our keys “with” us from one host to the next.

Here is the really nice part: once you’ve connected to the place where your GNU screen sessions are located, copy the value of the SSH_AUTH_SOCK variable:

# set | grep SSH
SSH_AUTH_SOCK=/tmp/ssh-ttQal19039/agent.19039
SSH_CLIENT='192.168.6.181 42243 22'
SSH_CONNECTION='192.168.6.181 42243 192.168.6.161 22'
SSH_TTY=/dev/pts/1

Take the value of SSH_AUTH_SOCK and input it into GNU screen:

:setenv SSH_AUTH_SOCK /tmp/ssh-ttQal19039/agent.19039

After this command is executed, start new sessions to your remote hosts. For the local host, it may be easiest just to restart the session – but you could also just set the variable SSH_AUTH_SOCK in your shell – such as this command for the Korn shell:

export SSH_AUTH_SOCK=/tmp/ssh-ttQal19039/agent.19039

To verify that the agent now works, use the command

ssh-add -l

. You should see all of your keys; if instead you see

Could not open a connection to your authentication agent.

then you should check the setting of SSH_AUTH_SOCK.

With SSH agents, agent forwarding, and GNU screen, you will find your authentication difficulties eased considerably.

UPDATE: Added information about not always having to restart screen sessions.

Restricting Users to SFTP Only and to Home Directories Using a Chroot

A chroot jail is a miniature environment which provides only just enough resources for one or another program to run. In this case, the sftp program is the one that is chroot’ed. The process can be made easier with tools like rssh or scponly – both of whcih are available in Red Hat Enterprise Linux 5.

However, the problem comes up when you want to use multiple users in this environment and you want to restrict each user so that they cannot see the rest of the environment, including other users. Neither rssh nor scponly will handle this sort of thing.

There is the program MySecureShell but it is not included with either RHEL or Ubuntu Server. One would have to compile it from source or use their packages for these environments.

However, as of OpenSSH 4.9 this problem was resolved: with the addition of the Match and ChrootDirectory options to OpenSSH 4.9 – as well as an internal SFTP server (in contrast to the external helper program otherwise used) – this allows the kind of restrictions wanted here.

Ubuntu Server Lucid Lynx (10.04) and Red Hat Enterprise Server 6 both come with OpenSSH 5.3p1 which includes both of these options. However, Red Hat Enterprise 5 only comes with OpenSSH 4.3p2. To see which version you have, use a command like this:

$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
$

Red Hat did backport the ChrootDirectory option into OpenSSH in RHEL 5 back in 2009, but not the Match option. Thus the ChrootDirectory option is good for everyone or no one. It is possible to use the option on the command line or in a specially crafted configuration file, but this requires a special server for SFTP instead of using the normal one – meaning two instances of SSH running if you want standard SSH logins.

It doesn’t seem to be possible to pass such options into rssh; rssh doesn’t allow you to pass options to ssh.

If you are using RHEL 5, it appears that the best option is to compile the most recent OpenSSH from source. To build the source, unpack the archive and copy the file contrib/redhat/openssh.spec from the unpacked openssh directory to the /usr/src/redhat/SPECS directory. Edit the file to your desire (such as disabling all of the X11 capabilities) and then perform this command:

rpmbuild -ba openssh.spec

In my case, the SPEC file contained a error in 5.9p1: there is a line that says:

%doc CREDITS ChangeLog INSTALL LICENCE OVERVIEW README* PROTOCOL* TODO WARNING*

Since there are no files named WARNING* this fails. Remove that entry from the end of the line:

%doc CREDITS ChangeLog INSTALL LICENCE OVERVIEW README* PROTOCOL* TODO

Then it compiles without problem.

If you want to use a recent version of OpenSSH on RHEL 5, you’ll have to make some configuration changes – and watch out for some changes. Changes include:

  • ServerKeyBits increased from 768 to 1024 (knock back down with specific entry if this worries you)
  • Protocol only supports version 2 by default
  • ShowPatchLevel not supported by standard OpenSSH (requires patch)
  • New ECSDA keys not supported fully or properly; `ssh-keygen -A` generates errors (can be ignored)
  • MaxSessions added to configuration file
  • AuthorizedKeysFile changes meaning slightly
  • UseDNS is included in the configuration file (instead of being hidden – set it to “no” for speed)
  • Banner is set to “none” by default

Change the Subsystem configuration entry for sftp to look like this:

Subsystem sftp internal-sftp

Add this to the end of the configuration:

Match group sftp-only
 ChrootDirectory %h
 AllowTCPForwarding no
 X11Forwarding no
 ForceCommand internal-sftp

Don’t forget to restart the ssh server:

service sshd restart

When you do this, make sure to have a root shell already open on the system so that you don’t get locked out. The user should exist in the password file like so:

test:x:521:521::/home/test:/bin/nologin

The home directory must be owned by user root and group root – including all directories in the path. The shell does not matter, as SSH will take over before the shell is activated; however, if there are other ways to log in with this user aside from SSH, then a proper shell like /bin/nologin or /bin/false is necessary.

For the specified configuration the user must be a member of the sftp-only group – either primary or secondary, it doesn’t matter. The Match option can also be used against a single user instead.

If you want the user – or anyone else – to be able to write files in this tree, create a subdirectory in the home directory with the appropriate permissions. The user – and others – will not be able to write to the home directory, but will be able to write to the subdirectory.

Don’t forget to test the login first – and don’t forget to have a root shell open before you start changing OpenSSH configuration or you may be unable to SSH into the system!

Logging transfers via SFTP

When you install OpenSSH, you also get SFTP for free: a replacement for FTP. (Note that SFTP is related to FTP by name only.)

SFTP is very useful, but as is normally configured, it does not provide any logging of user’s file transfers. For monitoring usage, this can be a problem.

The original setting in sshd_config might look like this:

Subsystem sftp /usr/libexec/openssh/sftp-server

This lack of logging is easily remedied; the details can be found in the man page for sftp-server:

Subsystem sftp /usr/libexec/openssh/sftp-server -l INFO

For best results, check your system’s man pages for the available options.

Getting and Verifying SSH Fingerprints

Sometimes you may log into a machine, and find that the SSH key has changed. This can happen in a man-in-the-middle attack, but more likely the host has been rebuilt and new keys generated. (This is a good reason to save and restore the SSH keys during a rebuild.)

So how do you check the fingerprints?

You can do it over the network by pulling down the public keys from the remote server:

ssh-keyscan -p 22 -t rsa,dsa remote_host > /tmp/ssh_host_rsa_dsa_key.pub

(Replace remote_host with the appropriate name or IP address.) Then you generate fingerprints from this file:

ssh-keygen -l -f /tmp/ssh_host_rsa_dsa_key.pub

However, if you are investigating why your keys changed mysteriously, you don’t want to do this over the network. Log into the actual server through other means (such as a management console or a KVM console) and generate fingerprints directly:

ssh-keygen -lf /etc/ssh/ssh_host_dsa_key
ssh-keygen -lf /etc/ssh/ssh_host_rsa_key

Then compare this with what you got over the network. If they match, you should be alright; if not, there are some serious problems.

If the fingerprints don’t match, you should do a network scan for ARP requests and see who is answering the ARP requests for the relevant IP address. Scan for ARP requests as you do a ping. If there are two hosts, then they will “fight” each other for an ARP entry, and you should see both responding.

Once you know the Ethernet address of the mysterious host, then chase it down by tracking where the ARP traffic is going by looking at router (or switch) interfaces.

Speeding Up SSH Access

There have been numerous problems that have slowed down SSH connections over the years. All are resolvable today – and could be fixed in modern operating systems, but typically aren’t. Most or all of these resolutions are related to reverse IP lookups: that is, given an IP address find out what DNS name is associated with it. These fixes turn off reverse IP lookups for the SSH client and SSH server.

The most often suggested fix is to add the following to your server configuration (sshd_config):

UseDNS no

This prevents the SSH server from making a reverse DNS lookup – that is, looking up the IP address and finding what DNS name is associated with that IP. Since the reverse lookup often does not respond, then there is a time-out involved and this is what generates the normal delay seen in SSH connections.

However, there are other things as well. In some distributions (such as Debian and Ubuntu) the system is set up to use Avahi. When SSH attempts to connect to a host, it uses Avahi as part of the authentication process. When the system uses mDNS (part of Avahi) to resolve an IP address, this resolves in a large delay until the system comes back.

Fixing this is simple as well: add this line to the SSH client:

GSSAPIAuthentication no

This authentication method triggers Avahi; by disabling GSSAPIAuthentication this skips Avahi altogether. It is also possible to rewrite the /etc/nsswitch.conf file to do the same thing. Changing the nsswitch.conf file affects the entire system, so be sure this is really what you want.

On my Ubuntu 10.04 system, the original line in nsswitch.conf looks like this:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

To stop using mDNS entirely, change the /etc/nsswitch.conf file hosts line to this:

hosts: files dns

Reverse IP lookups are a common source of delays for all manner of services (including PPP, Sendmail, and many others!).

Are SSH Passwords Safer than Keys?

It almost goes without saying that servers should never run telnet or rlogin but rather should have SSH instead. Today’s UNIX and Linux server operating systems, whether commercial or open source, come with SSH (usually OpenSSH) installed.

The question is: is password authentication more secure than public key authentication? This question has been asked before; consider these two questions from serverfault.com, one from November 2010 and one from June 2010. The instinctive response from many will be “Of course not!” – but the question remains. What are the actual downfalls to password authentication – or are there any?

There are multiple problems that can exist with public key authentication that password adherents will point out.

Firstly, there is the possibility that a system can be compromised and the keys taken. This risk is not so high as it might sound; if you are running Linux or UNIX on the system and have it properly secured, the risk of system compromise is lower than you might think. It is also possible (and recommended) to encrypt the keys with a passphrase.

Secondly, those who recommend passwords will note that passwords exist only in your mind (as long as you don’t write it down). However, is an easily memorable password truly secure?

There are numerous advantages to using a key, including many not directly related to security: you can have different keys for different systems, and keys can be restricted in various ways by the server administrator. It is also possible to revoke access to a user without affecting any other user – no more having to tell everyone the new password after a password change.

Keys are also not susceptible to brute force attacks. A hacker can attempt to break into an SSH server by brute force by trying a variety of passwords and usernames; this is not possible with keys. This sort of attack can come from anywhere, which makes it easier for hackers to do and more likely to occur. If there are a lot of users using SSH, then the likelihood of someone using a weak password is much higher – making the risk of server compromise that much higher as well.

Passwords can also be stolen remotely. If a server is compromised, the next time you log in you may be connecting to a hacked SSH server which copies all of your passwords. If you used public key authentication, then your private key would never be seen (or compromised) by the remote server.

So how do you properly secure your private key? Here are a number of things you can do:

  • Password protect your private key.
  • Put the key on a removable device and remove after using.
  • Put the key on an encrypted volume.
  • Use ssh-agent to store the key instead of using it over and over.

If you have already created a key without a password, you can add a password by using ssh-keygen:

$ ssh-keygen -p -f id_mykey
Key has comment 'id_mykey'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$

There are a number of good sources of information on SSH – and OpenSSH in particular. Try some of these:

Installing an ISO for Use in VMware ESXi 4

Having a CDROM available for use in a VMware ESXi server can be very useful. However, what if there is no way to get the CDROM into the host? Better yet, what if you just want to avoid the hassle of having to go back and forth inserting and pulling CDROMs?

There is a way to get the ISO onto the VMware ESXi server and make it available to a virtual machine. First, you have to put the ISO on the server itself. Secondly, you have to make the ISO available to the host. That’s it.

First step: copy the ISO to the ESXi server. You could do this with scp (if you’ve activated ssh) or by creating the ISO on the fly from the original disk. You could also use scp on the ESXi server to copy an ISO into the server without activating ssh.

The best destination would probably be a directory in the datastore you set up during installation of the ESXi server. If the datastore is datastore1, then the location for ISOs could be /vmfs/volumes/datastore1/ISO Images/. You’ll have to create the directory ISO Images yourself. Note that “datastore1” (or whatever your datastore name is) is not used by VMware ESXi, but translates to a hex string: so don’t be alarmed when you see it instead of the datastore name you expect.

Lastly, make the ISO available to the virtual machine. Using vSphere Client, you can get to the appropriate place (“Edit Settings”) in a number of ways. Right click on the virtual machine in the list of machines (either on the tree to the left, or on the list in the Virtual Machines tab) and select “Edit Settings”. Alternatively, click on the virtual machine on the left, then click on the Summary tab – and on that tab, click on “Edit Settings”.

In Edit Settings, click on the CD/DVD Drive 1 entry and then click the button for Datastore ISO File (on the right). If you click on browse – then on the drop-down menu at the top, look for “Datastores” (at the bottom). Your datastores will be shown when you click on Datastores, then the directory you created will be in the datastore you used earlier. Select the ISO you want and click Open – then OK.

Once the CDROM is chosen, you can then install from it, or use it in any way you like from the virtual machine. Don’t forget: this is just like putting a new CDROM into the machine – so whatever your OS needs to have happen, you have to do it after “putting the CDROM” into the drive.

5 Programs That Should be in the Base Install

There are a number of programs that never seem to be installed with the base system, but should be. In this day and age of click-to-install, these programs will often require an additional install – I maintain that this should not be.

Most of these will be relevant to Linux, but the programs will often be missing on other commercial UNIXes also.

  • Ruby. This is the first that comes to mind. I have been installing ruby onto systems since 1.46 – and ruby is still a fantastic scripting language, and one of the best implementations of object-orientated programming since Smalltalk.
  • m4. I recently wrote about m4, and thought it was already installed on my Ubuntu Karmic system – not so. I used it to create a template for the APT sources.list file.
  • ssh. This should be installed everywhere automatically, and not as an add-on. For many UNIX systems, ssh is an add-on product that must be selected or compiled from source.
  • rsync. Rsync is a fabulous way to copy files across the network while minimizing traffic – even though it is not designed to be a fast way.
  • ksh. This will surprise most commercial UNIX administrators. However, Linux does not come with ksh installed – and the emulation by GNU bash is weak. Now you can install either AT&T ksh-93 (the newest version!) or the old standby, pdksh (which is close to ksh-88).

OpenVMS is a different animal – and some of the things that should be installed by default would be perl, ruby, java, SMH, and ssh. I’m not sure if perl or ssh is installed by default, but they should be. OpenVMS should also support compliant NFS v3 and v4 support out of the box – without making it difficult to connect to other NFS servers.

What programs do you think should be in the base install?

Using make and rsync for Data Replication

When maintaining a cluster environment (such as HP Serviceguard) there are often directories and configurations which need to be maintained on two different local disks (on different machines). Using make and rsync (with ssh) are excellent for this.

The rsync command allows you to replicate the local data onto the remote side copying only that which is necessary. This is not necessarily the fastest, but it is the most efficient: rsync was designed for efficiency over slow links, not speed over high speed links. Configure rsync to use ssh encryption automatically in the Makefile, then use rsync as the way to copy the files over:

RSYNC_RSH=/usr/bin/ssh -i /path/to/mykey

rsync -av $(LOCAL_FILES) remoteserver:$(PWD)

To automate this properly, an ssh key will have to be created using keygen and transfered to the other host. The private key (/path/to/mykey in this example) is used by ssh in the background during rsync processing; with the key in place, no interactive login is necessary.

For best purposes, create an “all” tag (at the top of the file) that explains the usable tags, and create a “copy” tag that does the relevant rsync.

I recommend copying only relevant files, not the entire directory: this way, some files can be retained only on one node – this is good for log files and for temporary files.

For example:

LOCAL_FILES=*.pkg *.ctl *.m4
RSYNC_RSH=/usr/bin/ssh -i /path/to/mykey

all:
    echo "To copy files, use the copy tag..."

copy:
    rsync -av $(LOCAL_FILES) remserver:$(PWD)

Make sure to verify the code before you use it in normal operation. Use the rsync option -n to perform a dry run which affects nothing. Also make sure that you don’t update files on different hosts; things might get interesting (and unfortunate…)

After performing the update, the Makefile can trigger a reconfiguration or a reload of the daemons to put the configuration in place.

SSH Key Conversions: ssh-keygen (OpenSSH)

I’ve discussed this before, but this time I’m focusing on another angle. When interacting with commercial SSH implementations (such as exists on OpenVMS and Tru64 implementations) it becomes useful to know how to convert your OpenSSH public keys to SSH2 format and vice versa.

These examples will assume that you are using OpenSSH and are on a UNIX system. Note that these are public keys, not private keys. All key types (DSA, RSA) should convert fine, but DSA is the stronger cryptographic algorithm.

One you have a public key in the appropriate format, you can add it to the authorized keys file (whatever that may be called). This is normally found in ~/.ssh or ~/.ssh2 depending on the SSH version.

The OpenSSH utility ssh-keygen is what makes this happen. This utility can do a lot more than just generate keys. It can be used to change passphrases of encrypted keys; convert keys; generate public keys from private OpenSSH keys; and read and write keys to smartcards.

Converting an SSH2 key to OpenSSH

To use an SSH2 public key in OpenSSH, it needs to be converted. Use the ssh-keygen utility in this manner:

ssh-keygen -e -f ~/.ssh/id_dsa_ssh2.pub > ~.ssh/id_dsa.pub

Converting an OpenSSH key to SSH2

Using an OpenSSH public key in SSH2 requires a conversion; ssh-keygen can do this:

ssh-keygen -i -f ~/.ssh/id_dsa.pub > ~/.ssh/id_dsa_ssh2.pub