Installing Core Linux

Installing Core Linux is covered in this article (on the front page of coredistro.org as it happens!). The descriptions are clear; however, we will expand on the instructions listed there.

Let’s assume that you’ve already burned the ISO and booted with the CDROM. Once the system has booted, you’ll see this screen:

Log in as root (which has no password). First, figure out which disk to work with. The disk is most likely either /dev/hda (first IDE drive) or /dev/sda (first SCSI drive). You can see whether these disks exist by searching dmesg for sda and for hda.

Once the disk is determined, we must begin laying out the disk. In my configuration, I used fdisk to create three partitions:

• /boot partition of 500M
• Swap partition of 1G
• / partition of the rest (about 7G)

With this configuration, it is then necessary to (of course) create the filesystems:

mkfs -t ext2 -j /dev/sda1
mkfs -t ext2 -j /dev/sda3

The -j option creates what is commonly called an ext3 partition – which is in reality, and ext2 partition with a journal file attached. But that’s another post.

After the filesystems are created, then they must be mounted under /mnt as the complete environment would be layed out. For this example, that means:

mount /dev/sda3 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

After this is done, the actual process of installing Core Linux can begin. At the prompt, type:

install_core /mnt

Messages will appear like these:

Once this completes, a message comes back:

Install complete - the next step is to chroot into /mnt and install grub

Next installment… setting up grub.

Core Linux

If you want to create your own Linux distribution from scratch (and learn a lot during the process), you could do no better than to start with Linux From Scratch. However, many of us will not want to go to the extremes that LFS requires: there is an alternative – Core Linux.

Core Linux provides just enough of Linux to boot into a shell and compile. Thus it comes with bash and gcc and associated tools. If you want Apache – or PHP – or X11 – or KDE, you’ll have to compile itself – which is the point.

Core Linux comes in a small ISO, and provides a small shell script to install the basics of the system. The ISO is 150M and is available from Sourceforge.

When you burn the ISO, don’t forget to use the image capabilities of your CD burning application; don’t copy the file into a CDROM directory – but then you knew that already, didn’t you?

In upcoming parts of this series, I’ll show you how to install Core and how to get started with some applications. Of course, if you can’t wait…..

Arranging for downtime

Downtime is inevitable: servers need to be updated, patches applied, hardware upgraded or fixed, and so on. But how do you choose when to take the server down (if you have a choice)?

First of all is to ask the affected users of the system. This can range from getting the supervisor or person in charge to give an OK to having meetings with all affected. In some fashion, the users who are affected need to know and need to have input into the best time to take the server down. When the server is down, they may not be able to do much of their work; minimizing impact on them is important.

To also minimize impact, the downtime can often be arranged during off-hours. In the U.S., this is typically outside of the normal workday of 8 a.m. to 5 p.m. (800-1700). However – if the server is in use in other time zones, this window gets bigger – and if the server is used world-wide, the window gets bigger still.

The other thing to think about is which day to take down the server. One can do it during the work week (Mon. through Fri.); however, all of those nights (except Friday) are constrained by the fact that the availability deadline would be something like the next day at 7:00 a.m. When downtime is scheduled for Friday night, there are two entire days more to get things right. If a downtime is necessary for a major upgrade or extensive changes, then scheduling for Friday night gives that much more time to get everything working before Monday morning.

Disaster recovery planning

Planning for a disaster is not necessarily as easy as it sounds. It helps if you have a rampant imagination. Throughout disaster planning, the dominant question is What if…? Following the planning, testing is required: the best plans are worthless if they don’t work in practice.

Consider an Internet server serving web pages. Let’s assume that downtime is not an option: this is a typical point to start at. The best thing to do is to start with the most specific to the system (the complete environment) and work out:

• What if… a disk goes bad?
• What if… the software stops?
• What if… memory runs out?
• What if… the power goes out?
• What if… the kernel panics?
• What if… the cluster failover fails?
• What if… the network switch fails?
• What if… the network firewall fails?
• What if… the internet link goes down?
• What if… the internet provider drops off the grid?

Each one of the questions must be answered and the results tested. To test for power outage, pull the power. For a failed network switch, pull the network cable – and so forth.

Most of the answers will include some form of redundancy – clusters, dual facilities (such as power and network and internet providers), and so on. However redundancy is only one solution; there is prevention and alerts as well.

Each risk must be weighed against the cost to mitigate that risk. However, assuming that the risk is minimal does not eliminate the risk; the biggest problem is not accounting for a risk that eventually happens. There is nothing like downtime of a critical server to get an unforeseen risk taken care of; better to handle the risk before it happens.

It also does not matter if the plans have not been tested. If tests are not done, then the actual event will be the first time things have been put to the test – and what if something was missed and the system goes down? During a test, preventive measures can be taken to make sure that things work as they should – during an unexpected event, it is not possible to back out or prepare; if things go down they go hard. Don’t let that happen to you!

And disaster planning is not limited to servers (or virtual servers) – what about the possibility of a server hosting multiple virtual servers going down? What if your server is hacked into? What about your monitoring system failing? What about getting paged? Have you planned contingencies for all of these events?

Plan, then test, test, test – and you will make it.

Tips on using lsof

The utility lsof is a relatively new (well, compared to UNIX anyway) that has more options than even ls. These options provide for some extremely powerful capabilities, some of which we aim to illuminate here.

My favorite use for lsof is for networking: all sockets can be seen with the following options:

lsof -n -i

The -n option prevents lsof from being slowed down by a large number of DNS lookups, and the -i option returns all TCP/IP connections (with process numbers, user ids, file descriptor ids, and so on). To narrow it down, utilize options like the following – to list all SMTP connections, for example:

lsof -n -i :25

It is also possible to list only certain processes (such as process 25 and process 45):

lsof -n -p 25 -p 45

Alternately, the process can be selected by name:

lsof -n -c perl

However, suppose one wants to list all TCP/IP sockets held open by perl processes. The obvious choice does not work! This is because the options are combined together as an OR function; to combine them as an AND function (that is, all options must be satisfied) use the -a option – such as this:

lsof -a -n -c perl -i

This lists, as desired, all TCP/IP sockets held open by perl processes.

Another that might be useful in a security context is listing all files that are open but have no links to them: that is, they’ve been deleted, but one or more processes are keeping the file open, which means the file itself (and its blocks) are being preserved even though it appears to be deleted from the filesystem. To see these files, use this option:

lsof +L1

The utility lsof is indeed very useful, and reading the man page for lsof is recommended.

Write it down!

It’s one thing when your missus (system administrator geeks are almost always guys – almost) wants to know what you do all day – what do you say when your boss wants to know what you did all day? You best have a suitable answer!

This is yet another benefit to the GTD maxim: Write it down! But it’s not just from GTD. Researchers will say If it isn’t written down, it didn’t happen. Lawyers say A verbal contract isn’t worth the paper it’s written on. Details matter.

When you write something down, in the words of David Allen, you get it out of your head. You don’t have to try and remember everything your boss asked you to do: you’ve got it written down.

To follow through with the GTD method, you should write down the Next Action: what is the very next thing you have to do? Set up server smith isn’t good enough. How about: Configure Kickstart server for new server smith. Much better!

UNIX commands and little known options

There are a number of options to standard UNIX commands that are not necessarily well-known or widely used.  As your expertise grows, these lesser known options can speed up your processing considerably, making you more productive.

For example, there is the ls command.  Of course, ls may have been the first command to list as a bug: “Has too many options.”  However, many of these will help you if you remember them.

To list files one per line, use the -1 (one) option.  This can be useful to override default options or to force a plain and generic single column output.

To list all arguments as if they were files, use the -d option.  This doesn’t sound like much at first, but can be very useful.  The best example would be a listing with a wildcard in it: the shell expands the wildcard, then any directory in that list will have the files in it listed in the output.  For the cleanest output, use the wildcard as desired and add the -d option: this will prevent ls from listing the directory.

If you want to see if two different files are the same file – i.e., hard-linked – use the -i option to list inodes.  The inode number for a hardlinked file will match for all of its links.

For looking for manual pages, use the -k option to find all man pages with that specific topic in the header.  The topic is specified as an argument.  For example, see what man -k wireless gets you.

Gzip is another one with some useful options.  The most useful gzip option that is perhaps not widely used is the -c option: this will generate the output onto the stdout stream – which permits it to be piped into another program, such as tar.  Since gzip and gunzip are the same program (with different settings), the -c option can be used for gunzip as much as for gzip.

The make program also has some handy options.  One particularly useful option is -C: when specified with a directory, -C tells make to change to that directory before running.  This means you can run make from anywhere and tell it to act on a makefile somewhere else.  For example, how about this for NIS map management?

make -C /var/yp

It doesn’t matter where you run that from, it will work.

The sudo program has some very useful options.  When you run sudo, you trigger a five-minute countdown (or whatever time is set in the configuration).  When this time runs out, you are asked for your password again when you run sudo.

However – the -v option allows you to restart that clock from the beginning, entering your password if necessary.  It also does not take any commands to run (as that would be superfluous).

The -k option to sudo can be just about as useful: the -k option stops the countdown and resets any clock back to zero.  The next time you use sudo, it will ask for password.

Blogged with the Flock Browser

Tags: ls, man, make, gunzip, gzip, tar, sudo

VxFS (or HP Online JFS) Snapshots

A disk snapshot is a snap in time, a picture of what a disk looked like “back then”. This can be very useful for maintenance.

For example, being able to freeze a Caché instance, take a disk snapshot, then thaw the Caché instance will permit you to take backups or copies of a Caché database with minimal downtime.

For HP-UX Online JFS and Veritas VxFS the commands are the same (since these are actually the same product – or close to it). To actually do a snapshot:

mount -F vxfs -o snapof=/var/cache/db /dev/snap01 /snap

The first file system presented in the command line is key: it is the source of the snapshot. Note that it can be either a device or a current mount point. The second (device) is a filesystem prepared to hold a snapshot, and the last is the usual mount point.

Once this is done, the normal filesystem can continue to be used while the snapshot retains the older data as it was taken. In the example above, /var/cache/db could be used normally while the snapshot resides on /snap. If there was a directory /var/cache/db/db01 then there would also be a /snap/db01 available as well.

One caveat is that as long as the snapshot is mounted and in use, the changes to the original filesystem are being saved – it is possible that the snapshot volume can run out of space. When this happens, you will receive what may appear to be mysterious disk full errors unless you realize what is happening. So don’t keep your snapshots around forever.

Learning resources

A lot of the stuff an admin deals with day to day isn’t based on what we know, but on what we can find out. We spend our days researching problems and looking for solutions. What are the best resources?

Each operating system vendor has their own knowledge base or forums. For example, HP has the HP ITRC (IT Resource Center) which provides a one stop source for forums, documentation, technical notes, patches, and more. Other vendors have similar portals for technical support.

Another lesser-known resource is Usenet. The people in the Usenet newsgroups include some very knowledgeable people, and often can help resolve problems faster than you can alone.

There are a number of books that one can turn to as well; however, books on certification may often be overlooked. These books cover more obscure areas of an operating system and its maintenance, and can point you in the right direction when nothing else will. Many times, these tomes are also written to be on-desk references as well, and as such include the author’s experiences and knowledge beyond what is needed for certification.

PWN to OWN Contest at CanSecWest 2008

The PWN to OWN Contest is a hacking contest at the CanSecWest security conference that pits a standard install of Linux, Windows, and MacOS X against all comers. Each laptop has a default installation on it, and has not been hardened at all. The successful hacker will not only win a cash prize, but the system in question as well.

The MacBook Pro was the first to fall, and the laptop running Microsoft Vista Ultimate second. However, there will be those that misinterpret the results by not realizing how the contest was conducted.

Each contestant gets 30 minutes to attempt to crack the machine, and can choose which machine to attack. The attacks are limited by the rules, and each day that went by the rules allowed a wider range of attack vectors. It was a third party application (Adobe Flash) that permitted the compromise of the Microsoft Vista machine.

No part of the contest can be considered a scientific study into which system is more secure than the other: contestants attacked a single machine of choice, and contestants were allowed their attempts one at a time – and the operating system was not hardened.

This is entirely different than, for example, the Capture the Flag contest at DEFCON. That contest consists of setting up a server and trying to capture the other teams “flag” through compromising the server in some way. In that contest, any and all comers are permitted to enter and to attack at will during the contest with whatever vulnerabilities and methods they have available.

Speaking of DEFCON, DEFCON 9 saw the entrance of an Alpha-based VMS machine – installed with the standard setup – which remained unscathed throughout the contest, though try they did. The VMS Team (the Green Team) had a writeup and also wrote a white paper afterwards.

If you are interested in DEFCON, DEFCON 16 will be August 8-10 in Las Vegas, Nevada.

Also, speaking of DEFCON – let’s not forget the similarly named but totally unrelated InterSystems DEVCON2008, which is just wrapping up. DEVCON, among other things, covers Caché development and related. It is interesting to note that InterSystems DEVCON began 15 years ago, whereas DEFCON began 16 years ago. I wonder how much Caché security is covered at DEVCON2008.