UNIX Administratosphere

Planning for a disaster is not necessarily as easy as it sounds. It helps if you have a rampant imagination. Throughout disaster planning, the dominant question is What if…? Following the planning, testing is required: the best plans are worthless if they don’t work in practice.

Consider an Internet server serving web pages. Let’s assume that downtime is not an option: this is a typical point to start at. The best thing to do is to start with the most specific to the system (the complete environment) and work out:

• What if… a disk goes bad?
• What if… the software stops?
• What if… memory runs out?
• What if… the power goes out?
• What if… the kernel panics?
• What if… the cluster failover fails?
• What if… the network switch fails?
• What if… the network firewall fails?
• What if… the internet link goes down?
• What if… the internet provider drops off the grid?

Each one of the questions must be answered and the results tested. To test for power outage, pull the power. For a failed network switch, pull the network cable - and so forth.

Most of the answers will include some form of redundancy - clusters, dual facilities (such as power and network and internet providers), and so on. However redundancy is only one solution; there is prevention and alerts as well.

Each risk must be weighed against the cost to mitigate that risk. However, assuming that the risk is minimal does not eliminate the risk; the biggest problem is not accounting for a risk that eventually happens. There is nothing like downtime of a critical server to get an unforeseen risk taken care of; better to handle the risk before it happens.

It also does not matter if the plans have not been tested. If tests are not done, then the actual event will be the first time things have been put to the test - and what if something was missed and the system goes down? During a test, preventive measures can be taken to make sure that things work as they should - during an unexpected event, it is not possible to back out or prepare; if things go down they go hard. Don’t let that happen to you!

And disaster planning is not limited to servers (or virtual servers) - what about the possibility of a server hosting multiple virtual servers going down? What if your server is hacked into? What about your monitoring system failing? What about getting paged? Have you planned contingencies for all of these events?

Plan, then test, test, test - and you will make it.

The utility lsof is a relatively new (well, compared to UNIX anyway) that has more options than even ls. These options provide for some extremely powerful capabilities, some of which we aim to illuminate here.

My favorite use for lsof is for networking: all sockets can be seen with the following options:

lsof -n -i

The -n option prevents lsof from being slowed down by a large number of DNS lookups, and the -i option returns all TCP/IP connections (with process numbers, user ids, file descriptor ids, and so on). To narrow it down, utilize options like the following - to list all SMTP connections, for example:

lsof -n -i :25

It is also possible to list only certain processes (such as process 25 and process 45):

lsof -n -p 25 -p 45

Alternately, the process can be selected by name:

lsof -n -c perl

However, suppose one wants to list all TCP/IP sockets held open by perl processes. The obvious choice does not work! This is because the options are combined together as an OR function; to combine them as an AND function (that is, all options must be satisfied) use the -a option - such as this:

lsof -a -n -c perl -i

This lists, as desired, all TCP/IP sockets held open by perl processes.

Another that might be useful in a security context is listing all files that are open but have no links to them: that is, they’ve been deleted, but one or more processes are keeping the file open, which means the file itself (and its blocks) are being preserved even though it appears to be deleted from the filesystem. To see these files, use this option:

lsof +L1

The utility lsof is indeed very useful, and reading the man page for lsof is recommended.

The PWN to OWN Contest is a hacking contest at the CanSecWest security conference that pits a standard install of Linux, Windows, and MacOS X against all comers. Each laptop has a default installation on it, and has not been hardened at all. The successful hacker will not only win a cash prize, but the system in question as well.

The MacBook Pro was the first to fall, and the laptop running Microsoft Vista Ultimate second. However, there will be those that misinterpret the results by not realizing how the contest was conducted.

Each contestant gets 30 minutes to attempt to crack the machine, and can choose which machine to attack. The attacks are limited by the rules, and each day that went by the rules allowed a wider range of attack vectors. It was a third party application (Adobe Flash) that permitted the compromise of the Microsoft Vista machine.

No part of the contest can be considered a scientific study into which system is more secure than the other: contestants attacked a single machine of choice, and contestants were allowed their attempts one at a time - and the operating system was not hardened.

This is entirely different than, for example, the Capture the Flag contest at DEFCON. That contest consists of setting up a server and trying to capture the other teams “flag” through compromising the server in some way. In that contest, any and all comers are permitted to enter and to attack at will during the contest with whatever vulnerabilities and methods they have available.

Speaking of DEFCON, DEFCON 9 saw the entrance of an Alpha-based VMS machine - installed with the standard setup - which remained unscathed throughout the contest, though try they did. The VMS Team (the Green Team) had a writeup and also wrote a white paper afterwards.

If you are interested in DEFCON, DEFCON 16 will be August 8-10 in Las Vegas, Nevada.

Also, speaking of DEFCON - let’s not forget the similarly named but totally unrelated InterSystems DEVCON2008, which is just wrapping up. DEVCON, among other things, covers Caché development and related. It is interesting to note that InterSystems DEVCON began 15 years ago, whereas DEFCON began 16 years ago. I wonder how much Caché security is covered at DEVCON2008.

If you want to start some exciting discussion in a security forum, just say you use telnet: you’ll find that every admin knows that telnet is insecure, that one should use OpenSSH or similar to encrypt the traffic, and that telnet should be banned from the server environment entirely.

However, telnet is not the only server that transmits its passwords in the clear. There are a lot of others. Here’s a list I came up with:

• FTP
• HTTP
• IMAP
• IPP
• LDAP
• LPD
• NFS
• POP3
• rsync
• SMTP
• SNMP
• syslog
• VNC
• X11
• XDMCP

I won’t cover all of these here (more about these items can be found in my book) but I do want to cover just a few.

Consider, for example, the mail protocols: SMTP, POP3, and IMAP. SSL encryption is available with all three - but do you use it? And what about your logins to your mailbox at your ISP? Every time you login, your password to your mailbox goes across the wire in the clear.

What about NFS - particularly NFS home directories? If you have unencrypted secrets in your home directory, then these items will be transmitted across the network in the clear as well. What about private SSH keys? Unfortunately, there is no way to encrypt NFS traffic.

VNC is another one to watch for: if you type passwords for your root logins over VNC - even if you are using SSH in your VNC session - the passwords are in the clear. The only way to secure VNC entirely is to use an SSH tunnel to encrypt it.

X11 is insecure in the same way, but presents special problems. However, OpenSSH handles X transparently through the use of special tunnels just for X.

syslog is another unencrypted service; do you have passwords put into the system logs? What about secret doings of your servers? How much information leakage can you handle? Unfortunately, syslog is another service that cannot be secured unless you use something such as syslog-ng which permits you to use TCP (and thus, an OpenSSH tunnel).

It’s easy to be casual about confidentiality and to miss the finer aspects of what a system administrator needs to be quiet about in daily dealings with users and customers. There are obvious confidentiality agreements and the usual corporate trade secrets and new products - but there are other things that we as admins must be wary of.

The obvious corporate secrets include things such as trade secrets, secret formulas, research, new product development, and more. Perhaps almost as obvious are customer data and patient information (in healthcare related industries).

However, what about the server break-in? Certainly anyone affected will have to be told - but how much? And what can be told around the water cooler?

A server compromise is a perfect example of an area where we as admins should know a lot more than we tell. Telling all of the details could alert the cracker to what is known and not known, and could potentially compromise any future legal action against the cracker. Talking could also affect any public relations that the company may wish to do in the wake of a serious event like this.

Other items include user privacy. Certainly users expect a certain measure of privacy - perhaps too much given the realities of administration today. However, what does an email administrator do when they find out that someone is cheating on their spouse? Answer: nothing. What do you say to others? Nothing. A lot of things fall into this category.

However, what if you as an admin find something that demands action - perhaps legal action? Pornography on a system, for instance - or hate mail - or spam being sent? Best thing is to send it up the chain of command in the most secure way: face to face - on a walk in the clear air if necessary. Document (on paper) as much as possible, and pass that up the chain of command as well - perhaps keeping a second copy just in case. Sign and date both. And after all this, what do you tell your coworkers? Nothing. Absolutely nothing. Also, never ever identify the person until all action is done - and perhaps not even then.

What about the new fancy security feature that got installed? It doesn’t matter if it is digital, physical, or otherwise: say as little as possible. When working at a bank, one learns this first-hand and quickly; data center security and server security are the same way. Of course, among your cohorts in the trenches, security is a shared topic - but it is not for public (or staff) consumption.

It is better to err on the side of caution and silence than to say too much - and in this digital age, any thing in email or on disk is already too much and can be recovered. To maintain the strictest confidentiality, don’t use digital means to talk about confidential matters.

Initially, the virus writers among the programmers and hackers were hobbyists - or those engaged in research (though perhaps misguided or misapplied). Sometimes - or perhaps all the time - viruses would escape from their hosts and get sent into the wild.

One of the oldest had to do with the original game of Adventure: so many people wanted to play it, that it was decided in the local environment to automatically replicate Adventure on the user’s local machine before running. It was so wildly successful that it was everywhere - and then the counter-virus was written that would delete a copy of Adventure if the user no longer wanted it. (I’ll be durned if I can actually back up that story…. my memory doesn’t go back that far…)

Now it has been reported by The Register (and noted by darknet.org.uk and TrendLabs Malware Blog) that the virus writing club 29A is disbanding. Most virus writing groups of the past have been the equivalent of spray painters painting a building - or those that try to see how many places they can go in a building (building hackers?). Money was not the objective - prestige, honor, and popularity were all part of it.

Now with the demise of 29A, and the newly reported fact that adware has surpassed viruses as the largest current threat, it is becoming clear that the typical virus writer is changing - becoming more interested in profit and extortion.

Is this a fact worth bemoaning? Before, virus writers just wanted to wipe out a system - or propogate the virus as widely as possible. Now writers want to put the system into a botnet or to extort money from the owner. Which is better?

The hacker ethic states that you do no damage to a system. The earliest virus writers did their best to follow this - but virus writers haven’t followed that rule for many years.

It doesn’t make one sleep any easier at night knowing viruses are now the domain of the extortionist and not the spraypainter….

All I know is, I’ll spend whatever time and effort I can to keep them out. A production system cannot go down due to a virus, no matter if it is malignant or not.

The best documentation out there on OPIE focuses on using OPIE with FreeBSD. FreeBSD comes with OPIE as part of the core, and it works beautifully.

The FreeBSD Handbook contains a section on One-time Passwords (14.5) which discusses using OPIE, and how to use utilities associated with it.

The esteemed Dru Lavigne wrote excellent articles as well (way back in February of 2003!) about making OTP work in FreeBSD. The first was One-Time Passwords, and followed by PAM (which used OPIE access as an example).

There is no installation required, as all the appropriate bits already exist in the base install. To get started, follow this example with your own password:

$ opiepasswd -c
Only use this method from the console: NEVER from remote. If you are using
telnet, xterm, or a dial-in, type ^C now or exit with no password.
Then run opiepasswd without the -c parameter.
Using MD5 to compute responses.
Enter new secret pass phrase:
Again new secret pass phrase:
 
ID dgd OTP key is 499 xf4915
OTTO BRIM CAT PEN FAD PREEN
$

Now everything is set. However, the first password requested will be number 498: so let’s generate the key so we can record it:

$ opiekey 499 xf4915
Using the MD5 algorithm to compute response.
Reminder: Don’t use opiekey from telnet or dial-in sessions.
Enter secret pass phrase:
498: SLIP MAX ROT BLIP PIP FLIT
$

Note that this second step does not have to be done on the same machine. You do need to have your count (499 in the example) and the seed (xf4915 in the example). Combined with your secret password, you can generate your next password or passwords using any valid OTP password calculator.

Dru’s article on PAM discusses the conversion from Linux PAM to OpenPAM, and other things relevant to the introduction of FreeBSD 5.0. However, now with FreeBSD 6.3 out and FreeBSD 7 coming, most of what she discussed then is now well-tested and preconfigured. OPIE is built into PAM as she suggests (within the file /etc/pam.d/system file).

Now if you want to be forced to log in with OTP, touch the file .opiealways (note the leading dot) file in your home directory and it will be so.

There was also an interview done in June of 2007 in BSDTalk #117 (audio) which discussed OTP in depth. One interesting thing to note: FreeBSD uses OPIE, whereas NetBSD and OpenBSD both use S/Key for their implementations.

When a server is hacked, what do you do? That is the question. There are several questions:

• Do you want to prosecute the offender?
• Do you want to analyze the attack?
• Do you want to preserve the evidence?

For the moment, we’ll assume that you want to preserve the evidence in order to prosecute. Once this decision is made, the system administrator’s role starts to enter the legal realm. In order to preserve the evidence, a chain of custody must be maintained and preserved.

At its simplest, the chain of custody is a legal phrase that describes the provable knowledge of every one who handled the evidence and a provability that this evidence is the same as the one that was present at the crime scene. The question that the chain of custody attempts to answer is: Is this evidence (hard drive) the same one (and unaltered) that was at the crime scene (in the server)?

Before preserving the evidence, make a copy of the hard drive in order to have a copy for forensic analysis. This process should be noted on the evidence’s chain of custody, more than likely.

To make the chain of custody (and preserve the evidence), there are a number of things that can be done:

• Once the hard drive is no longer needed, wrap it and seal it (possibly along with an initial document stating its origins) - then sign across the seal itself (to prove that it hasn’t been opened since).
• Put the drive into a safe with a very limited access.
• Each time the drive is touched (i.e., moved, relocated, or reassigned) fill out an entry on a form with date, action, by who, and why the action was taken.

All through the process, there should always be a witness to what occurs. For example, when the system administrator removes a hard drive from a server, there should be additional witnesses present. Likewise, if anyone moves the hard drive - or gives it to someone, then witnesses should be present also. In all these cases, note the witnesses and other information in the chain of custody log (always with date).

Another excellent tip is to photograph the removal of the hard drive - specifically, photograph the location of the machine, what was done to the machine, and where was the hard drive physically located. The person removing the hard drive may also want to sign the hard drive itself, in order to prove that this was the actual hard drive removed from the system.

The wrap should be some sort of wrap that will show evidence of tampering - shrinkwrap may fit the bill.

The chain of custody is a legal phrase with a legal definition. However, I am not a lawyer (sheesh, do I really have to say that?). So for best advice, seek that of a lawyer (or forensic specialist).

In a big enough company, certainly you should consult with your in-house corporate attorney before taking any unnecessary action.

Here are some further resources:

• How to Keep a Digital Chain of Custody from CSO Online
• Proof of a Chain of Custody in a Case
• Forensic Science: the Basics by Jay A. Siegel (chapter 2)
• Computer Forensics: Computer Crime Scene Investigation by John R. Vacca (pp. 247 and following)
• Laws of Evidence by Thomas Buckles (pp. 81 and following)

Some of these resources are directed towards the legal profession instead of the computer professions, but if you have the willingness to learn more, they can only help.

Files can be “hidden” on a filesystem in several ways; some are just attempts to “hide in plain sight.” This is not hiding data in files (such as steganography) but rather hiding the mere existence of files themselves.

There are several ways to do this. The basic ideas are:

• Make reporting tools lie about the existence of files. This falls into the realm of “rootkits.” Utilities such as ls (and even the kernel itself) may modified in order to not report the existance a particular file. A way to avoid this is to use programs (during examination) that are statically linked and precompiled and saved to read-only media: such programs are immune to future modifications, except for kernel resources (if the kernel is taken over, then all other resources are suspect).
• Make files “disappear” in file listings. Files of this sort include “.. ” (two dots followed by a space) and other non-standard characters. These files look normal but are not. One way to see files like these is to use the ls command and its -b and -q options. The -b option prints the octal equivalent of non-printing characters; the -q option uses the “?” character instead of non-printing characters. The -F option may also be of use: a file with a terminating space will show up when the -F places its “type” character at the end of the file name.
• Open the file, then delete it. As long as the file is in use, its disk blocks remain in use, and the file remains available. No other processes can access it - or even see it - but the program (or programs) that opened it still has access to it. Of course, this only works for running programs, but files disappear this way almost entirely. To see the files, use the utility lsof with the +L1 option. The +L option lists files with less than that many links; thus, +L1 lists files with less than one link (or zero links).

As mentioned previously, a static build of all relevant utilities to CDROM can be a very useful tool during investigation of a possible system break-in. All general reporting tools should be included: ls, ps, top, etc. Also included should be any desired programming interpreters: ksh, perl, ruby, tclsh, lua (whatever one finds useful). Also included should be any more intensive tools as well: tcpdump, nmap, etc.

Passwords are the first line of defense into almost every computer and every server. Some do use One-time Passwords, Keycards, or biometric sensors - but they are the unusual ones.

Let’s go back to basics, and figure what our requirements are. If we wax a bit, and dream up the ultimate set of requirements for strong passwords, it might look something like this:

The passwords must be easy to remember. If you can’t remember it, you may be caught short and not be able to log in. If you have to write it down, someone could see it or snatch the sticky note. Also, if a user has to write their password down, they are likely to write it down where all the world can see it.

The password must have a random character distribution. This is different than being random you might note. A fully random password means that a password of all ‘A’s is equally possible to any other; what we really want is a random distribution of letters within the password (I hope I said that right!). In that case, all ‘A’s would be impossible as the distribution is all on the letter A. The ideal would be random letter distribution, but an English (or native language) letter distribution would probably be suitable.

Writing down the password must not give it away. This is one I’ve not seen anywhere, but it would be nice.

A new password must be easy to generate. Ever have to come up with a suitable password? Of course… but how long does it take to come up with one that is both secure and easy to remember?

So how do we come up with passwords and adhere to our given requirements?

My thoughts are: use a phrase you can remember (easy to remember) and then write it down in a series of boxes - that is, in a 6×6 grid for example.  With such a grid, you could then write in a sentence (left to right), then read the letters in a different order (perhaps top to bottom, over one, bottom to top, over one, top to bottom …).  In essence, you are memorizing a simple and easy password and a pattern - then typing in the resulting text.  Left over characters could be Xs or Zs or numbers or special characters.

I can’t speak to the strength of this method, but I suspect it is pretty strong.  You could even “clip” the password at a certain number of characters while using a large phrase - which would make it stronger perhaps, since the entire phrase would not be used.

If this interested you, you may also be interested in Diceware: a method of using dice to select words (for a passphrase) in a wordlist.  There are some interesting ideas on how to make the passphrase stronger, and so on.  However, my idea requires no additional materials at all and results in a seemingly random set of characters for a password.