Virginia Experiences a Severe IT Outage

The Virginia Information Technologies Agency (VITA) has been dealing with an outage that resulted during a scan by storage technicians for failed components. The outage “only” affects 228 of 3,600 servers – which affects 24 different Virginia departments, including the Virginia Dept. of Motor Vehicles (DMV), the Governor’s Office, the Dept. of Taxation, the Dept. of Alcoholic Beverage Control, and even VITA itself.

No word on how many Virginians are affected, but certainly it will be in the thousands.

According to the Associated Press, the crash happened when a memory card failed and the fall-back hardware failed to operate successfully. From the sound of it, an entire storage array was affected by this – how else to account for the 228 servers affected?

This suggests that the storage array is a single point of failure, and that the memory card was not tested nor its fall-back. There should be some way of testing the hardware, or to have a clustered storage backup.

One of the biggest problems in many IT environments – including states – is budget: having full redundancy for all subsystems is expensive. States are not known for budgets that fill all departmental needs (despite large budgets, departments scrounge most of the time…). Many other data center owners consistently have tight budgets: libraries, non-profits, trade associations, etc.

The usual response to tight budgets is to consider the likelihood of failure in a particular component, and to avoid redundancy in such components. A better way to look at it would be to compare not the likelihood of failure, but the cost of failure. The failure in Virginia’s storage was supposed to be extremely unlikely, but has had a tremendous cost both to Virginia and to Northrup Grumman.

Virginia’s IT was outsourced in part to Northrup Grumman, and was supposed to be a model of privatizing state IT in the nation. However, Virginia’s experience shows how privatizing can fail, and how outsourcing companies do not necessarily provide the service that is desired.

The Washington Post reported on this failure, as well as others in the past. There have been other failures, such as when the network failed and there was no backup in place. Both Virginia and Northrup Grumman should have noticed this and rectified it before it was necessary.

The Richmond Times Dispatch has an article on this outage, and will be updating over the weekend.

Direct NFS in Solaris with Oracle 11g: Benchmarks

Over at Glenn Fawcett’s Oracle Blog, there is a write-up about the speed of Oracle’s Direct NFS (now a few years old) as compared to the traditional NFS client. Glenn wrote about how to set this up initially, then followed up with a report on how to monitor the environment, as well as the results of testing the environment.

Glenn worked with Kevin Closson, who is one of the minds behind Oracle’s DirectNFS. Kevin wrote about the colloboration and about some of the misunderstandings surrounding dNFS with Solaris and Sun storage.

Oracle has a nice whitepaper on this topic, going into detail as well.

There is also an old posting by the Oracle Storage Guy describing DirectNFS in detail, particularly in regards to using dNFS with EMC storage.

Adding a New Disk to HP-UX with Veritas Volume Manager

Adding a new disk to a HP-UX system that is using Veritas Volume Manager is not difficult. It does require some extra steps, but if you follow along there are no problems.

Note well that this discussion does not have anything to do with LVM whatsoever, and the use of the Veritas File System (VxFS) is a minor tangent – other filesytems (like HFS) can be used – but if you’re not using OnlineJFS (which is nearly all of VxFS and its capabilities), the question would be why?

First of all, the usual steps for adding a disk to HP-UX are necessary:

1. Run ioscan to scan for the new disk and to allocate resources in the kernel for it. My actual favorite invocation for this command is ioscan -fnC disk
2. Next, run the insf command to create device files. I prefer to use ioscan -eC disk in this instance.

Now we add the disk to the Veritas Volume Manager. Run these commands:

1. vxdctl enable
2. vxdisk scandisks

At this point, there is now an additional disk available for use by vxvm but which is not a part of any disk groups yet.

To add the disk to a disk group, use vxdiskadm and option 1. You can use "list" here to get the name of the unallocated disk. Enter the name of the disk and answer the prompts; in most cases you will want the default.

Now to extend a file system in the volume group to use the new disk, you have to figure the amount of space available to use. You can use the command vxdg -g diskgroup free to find the free space in the disk group "diskgroup" - but this does not tell you the actual amount of space you can extend by; that is, it does not include striping and other restrictions - it only includes a flat number of unallocated blocks.

To get the amount of disk space you can grow a volume by, use the command vxassist -g diskgroup maxgrow volume. This will give you the actual amounts (if it is possible to grow). Replace the diskgroup and volume with the actual names of your disk group and volume respectively.

To actually grow the filesystem, you can do the following:

vxresize -b -F vxfs -t mytag group 999999

Fill in with the appropriate information instead of mytag (a tag to identify background vxvm operations), instead of group (disk group), and instead of 999999 (use the max amount of growth - or other appropriate number - identified by the maxgrow operation).

VxVM is aware of VxFS, so resizing the volume will also resize the filesystem as well; it uses fsadm in the case of OnlineJFS.

Book review: Learning FreeNAS

I’ve been looking at the book Learning FreeNAS by Gary Sims, and trying out FreeNAS in the process. FreeNAS is now at 0.69.1, and is very stable and robust. FreeNAS is based on FreeBSD and thus is rock solid.

Writing a book about FreeNAS (or any Network Attached Storage system) is difficult for several reasons. The most obvious one is that entire books (big books!) have been written about each storage technology: Windows file-sharing (SMB/CIFS), NFS, iSCSI, FTP, backups, and more.

It is difficult to write a good book about NAS as it is not possible to cover all areas in depth – and alternately, it is not good to reduce the book to “click this button; click that button; next enter this data and click that button…” A NAS can make setting up and using a complicated server quite easy – and finding the right balance between describing all of how Samba works and just specifying which buttons to push can be a hard choice to make.

Learning FreeNAS tends slightly towards the simple end: if you discover any serious problems that require command-line knowledge, the book doesn’t really cover more than it must. In my case, I found that installing FreeNAS resulted in the lack of a default route. I had to add the default network route by hand, though the book never discusses this. This is not necessarily a deficiency, but one to be aware of.

One thing that I always look for in books is an in-depth index. These are simple to find: how many pages does the index contain? How many entries does each letter contain? How many entries can be found under U or X? This book contains 6 pages of index, compared to a similarly sized book that has 17 pages – and a smaller font size. As a reference work then, it will be harder to find items that are of interest.

Overall, this is a good book, worth getting. It could have been more in-depth, but as it stands it is still good. There is no comparable book for the only serious competitor in the open source NAS arena, OpenFiler (which is based on Linux).

The book is available from Packt Publishing in print or in a downloadable PDF.

Veritas Volume Manager (VxVM) and HP-UX

HP-UX comes with VxFS (the Veritas File System) and the Logical Volume Manager (LVM). Only the on-disk filesystem layout comes from Veritas; HP’s volume management is all their own. There’s nothing wrong with HP’s LVM – I tend to prefer it, but then that’s what I know.

Veritas (now Symantec) offers another, competing product called Veritas Volume Manger (refered to as VxVM). The tools are different, the layout is different, and the capabilities are different. Knowing LVM won’t help you much, though the most basic concepts are the same: collect a series of disks together and then parcel them out as a single large group, with user-defined subdivisions.

Veritas Volume Manager is now a part of the Veritas Storage Foundation.

An nice set of links to documents can be found at Aziz’s Blog. In particular, the Veritas Volume Manager Administrator’s Guide has been indispensable. Just about everything you can imagine you might need to do is located here.

Cuddletech offers the VxVM Quickstart with some older, but worthwhile documents that describe VxVM and its concepts. Likewise, Unixway offers a wide variety of documents on VxVM over several versions, as well as tutorials and more.

The AdminsChoice also has a good set of tutorials; there is Veritas Volume Manager part 1 and part 2 (focusing on vxassist).

There is a mailing list, but in recent months the activity has been rather sparse.

If you want to take your knowledge all the way, you can become Symantec Certified for the Veritas Storage Foundation (which mainly includes VxFS and VxVM).

Veritas VxVM works very well on HP-UX, and it is possible to create a root disk that utilizes VxVM and VxFS. When using VxVM, LVM is not used (unless a particular disk uses LVM instead of VxVM). The commands are the same across different platforms, and the on-disk layout is the same – so it should be possible to take a set of disks from a Solaris system and put them onto an HP-UX system and still read the data (but watch out for differing byte ordering!).

In the future I hope to discuss more on VxVM; we’ll see.

Powered by ScribeFire.

SheevaPlug: a Tiny Computer for $99

This computer introduced by Marvell is very tiny, and very interesting.  Despite the fact that Marvell’s wireless chipset has been closed to open source developers, it appears that the Sheeva Plug computer is being released as an open product: running Linux on an ARM processor, it is now available for $99 as a pre-release developer’s edition. There is already a place for developers to congregate and for documentation and so forth.

LinuxDevices had a delightful article on the technical aspects of the SheevaPlug, and it is very enlightening.

What would I use such a computer for?  I would quite possibly make it into a NAS solution with OpenFiler or FreeNAS; make it serve IP addresses via DHCP; make it into a web cache like squid; or make it serve music with subsonic.

This is one beautiful box.  One drawback I see is that with the way it is configured, there is no way to get it off the wall and out of the way.  Too many boxes plug right into the wall, which means there is no place for another box to plug in.

Another deficiency, which is silently ignored in a lot of applications shown: there is only one network connection. For the system to be a router of any type, it needs to have multiple network connections. If a SheevaPlug is to be a wireless router – or a cellular router – or other similar configurations, it needs to have more than one network connection. With the USB connection available, this is possible – but only if the USB isn’t taken with something else.

One nuisance to note, like others of its ilk: it requires added peripherals, so the “tiny” box could expand to include an external hard drive, and external USB hub with its own AC plug, a bluetooth USB plug, a USB cellular modem, a USB network port, and two network cables. This is the curse of tiny electronics today: one day, all of these extras will be included in a box the same size, and the cabling will be history.

One disadvantage that no one seems to have mentioned yet: the box is not grounded.  That’s right: only two prongs – no grounding plug.  This is totally baffling to me: no ground?

Still, these are really minor disadvantages: I want one – or even two!

It would be interesting to consider the use of these in the enterprise (although they are specifically designed for the home). The biggest places I could see these used in the enterprise would be for testing purposes, and for disaster recovery. If you had one of these ready as a DHCP server and DNS server, one as a NIS server – perhaps a medium-sized enterprise could run off of these until the real servers are built and ready to go.

They could also be used to support people in the field: preconfigured, ready to run: demonstration systems, VPN end points, presentation systems, security test launching points… What else can you think of?

Powered by ScribeFire.

Mirroring HP-UX Root Disks (LVM)

Mirroring the root disk in HP-UX is not necessarily as hard as it sounds. There are excellent directions out there. Firstly, go get the PDF from HP titled “When Good Disks Go Bad” (make sure you have the most recent!). This is frequently recommended and you should have it if you don’t already.

Appendix D: Procedures contains detailed instructions on both mirroring PARISC boot disks and Itanium disks. Here is a broad overview to get you prepped:

• 1. Prepare the disk for LVM. This means using the insf command to add the disk files as appropriate, and using pvcreate to lay down LVM structures.
• 2. Add the new volume to the root volume group. Use vgextend to add the new disk to the volume group.
• 3. Make the disk bootable. This is a hairy part. You have to copy the proper data from the original disk, and so forth. Follow the directions closely.
• 4. Mirror the logical volumes on the first disk to the new one. This has to be done in a specific order or it will fail. It’s not hard, just time consuming.
• 5. Update system data with new boot information. This involves configuring the machine to boot with the new disk.

Setting up Itanium boot mirrors requires a few additional steps: the disk must be partitioned into three parts: 1) EFI boot partition; 2) HP-UX root disk data; and 3) HP Service Partition. The second partition is equivalent to the entire disk in a PARISC system. The first partition (EFI boot) is like a mini-disk that contains all of the data needed to boot HP-UX.

The last partition – the HP service partition – is not configured unless you copy it from the original source. If you do not do this, you do not actually have a service partition – which could be a nasty surprise when you need it!

Network Attached Storage (NAS)

Once you hear what a NAS appliance does, you might be tempted to think (as I did) what all the fuss might be about. But there are reasons for a NAS appliance, though a NAS isn’t for everybody.

Network Attached Storage is nothing more than a server with a pile of disks and a dozen different ways to access them. For most intents and purposes, the difference between a File Server of yesteryear and the Network Attached Storage of today is conceptually rather minimal.

NAS typically provides access to files via such methods as Windows shares, NFS, iSCSI, Appleshare and others.

So what does a NAS appliance provide that a NFS server does not? There are several benefits:

• Special purpose. Since the system is solely for the purpose of serving up files for users, there is no need for any other facilities except those that deal with its specified purpose. Thus, a lot of potentially vulnerable or unreliable code can be removed, and the speed and reliability of the system can be increased. Some systems do not come with a general purpose operating system of any kind, but rather a specially designed operating system for serving files alone.
• Extensive support. In many cases, since the system is specifically designed for serving up file storage, the innumerable variations of network storage protocols come supported out of the box.
• Ease of use. With the system designed to serve one purpose – and to provide the customer with the best possible experience – the system is generally made much easier to configure and easier to use than having to configure the varying servers and protocols independently.

There are two different NAS products that are the heavy-weights in the free and open source arena: FreeNAS (freenas.org) and OpenFiler (openfiler.com).

The most obvious difference between these two is their base (and their associated licenses). The base for FreeNAS is FreeBSD, and like FreeBSD, is licensed using the BSD license. However, OpenFiler uses Linux as its base, and is likewise covered by the General Public License version 2.

This week, I’ll focus on FreeNAS with the assistance of a book entitled Learning FreeNAS by Gary Sims and published by Packt Publishing.

Powered by ScribeFire.

Expanding Integrity Virtual Machine Disk Volumes (HP-UX)

When an Integrity Virtual Machine (or IVM) is set up, the disks that the IVM will have internally can be backed with any number of things: a DVD device, an ISO file, a regular file, a physical disk, or a logical volume. Using a logical volume can be the most flexible option.

However, when expanding a disk volume for the Integrity Virtual Machine, the obvious solution is the wrong one. The logical volume on the VM host can be expanded – this, however, will all be in vain: there is no way to adjust the size of the disk inside the VM. Once pvcreate is done and the device is present, the volume cannot be expanded.

So even though the logical volume backing the VM disk has been expanded, there is no way to make the VM utilize the “new” space (which it can’t see).

It may be possible to do a pvremove on the disk, then remove the disk from the VM (using hpvmmodify) and add it back in as a new disk. It might also be wise to zero out the disk (being careful!) so no LVM structures remain in the VM disk. In this case, that would mean using a command like this:

dd if=/dev/zero of=/dev/vgvdisk/lvol1 bs=1024 count=50

This will lay down zeros onto the disk. Be very careful about which disk you are doing this to! Once this command is done, there will be no usable data on that disk – so if you choose the wrong one you could mess your system up completely.

The recommended way of increasing storage in an IVM is to create a new disk for the virtual machine, then add it to the VM (using hpvmmodify with a -a option). Once this new disk is presented to the guest HP-UX environment, add the new disk to the old volume group using vgextend, extend the logical volume with lvextend and extend the filesystem with fsadm.

No down time and no problems!

Powered by ScribeFire.

HP-UX Boot Disks on Integrity systems (in contrast to PARISC systems)

On PARISC-based HP-9000 systems, configuration of system boot disks was simple: the entire disk was used, split apart using logical volumes with LVM. Thus, an HP-9000 system (PARISC) will have a “standard” full disk for the boot disk – such as /dev/disk/disk56 (using the new disk labeling).

However, when using Integrity systems, space must be made at the beginning for EFI and at the end for an HP System Partition – which shows up in HP-UX as a disk with three partitions.

An Integrity system will have several more disks associated with the boot disk (using disk32 as the example):

• /dev/disk/disk32 – this is the full disk. The disk, however, is split into three parts as described below.
• /dev/disk/disk32_p1 – this is the EFI partition. When the system boots, it is this partition which loads the EFI data and runs the EFI shell.
• /dev/disk/disk32_p2 – this is where the HP-UX operating system data is stored. The logical volumes associated with HP-UX will be created here, and /dev/disk/disk32_p2 will be in volume group vg00.
• /dev/disk/disk32_p3 – this partition is an HP system partition of some sort. It is automatically created during installation.

Thus, if you are on an Integrity system and are attempting to follow some older directions, remember to use the appropriate disk label.

There are tools that are designed for Integrity systems with EFI that will help maintain or document these partitions. First is idisk:

# idisk -p /dev/rdisk/disk32
idisk version: 1.44

EFI Primary Header:
        Signature                 = EFI PART
        Revision                  = 0x10000
        HeaderSize                = 0x5c
        HeaderCRC32               = 0x30a62aae
        MyLbaLo                   = 0x1
        MyLbaHi                   = 0x0
        AlternateLbaLo            = 0x88bb991
        AlternateLbaHi            = 0x0
        FirstUsableLbaLo          = 0x40
        FirstUsableLbaHi          = 0x0
        LastUsableLbaLo           = 0x88bb93f
        LastUsableLbaHi           = 0x0
        Disk GUID                 = 43b615f6-a561-11dd-8000-d6217b60e588
        PartitionEntryLbaLo       = 0x2
        PartitionEntryLbaHi       = 0x0
        NumberOfPartitionEntries  = 0xc
        SizeOfPartitionEntry      = 0x80
        PartitionEntryArrayCRC32  = 0x97c6286c

  Primary Partition Table (in 512 byte blocks):
    Partition 1 (EFI):
        Partition Type GUID       = c12a7328-f81f-11d2-ba4b-00a0c93ec93b
        Unique Partition GUID     = 43b61920-a561-11dd-8000-d6217b60e588
        Starting Lba Lo            = 0x40
        Starting Lba Hi            = 0x0
        Ending Lba Lo              = 0xf9fff
        Ending Lba Hi              = 0x0
    Partition 2 (HP-UX):
        Partition Type GUID       = 75894c1e-3aeb-11d3-b7c1-7b03a0000000
        Unique Partition GUID     = 43b6195c-a561-11dd-8000-d6217b60e588
        Starting Lba Lo            = 0xfa000
        Starting Lba Hi            = 0x0
        Ending Lba Lo              = 0x87f37ff
        Ending Lba Hi              = 0x0
    Partition 3 (HPSP):
        Partition Type GUID       = e2a1e728-32e3-11d6-a682-7b03a0000000
        Unique Partition GUID     = 43b61970-a561-11dd-8000-d6217b60e588
        Starting Lba Lo            = 0x87f3800
        Starting Lba Hi            = 0x0
        Ending Lba Lo              = 0x88bb7ff
        Ending Lba Hi              = 0x0

Be careful in using idisk, as you can completely destroy your data easily with idisk, and even render your machine unbootable.

Then there are a number of utilities to work with the EFI partition; these are:

• efi_fsinit – initialize EFI partition;
• efi_cp – copy EFI files to and fro;
• efi_mkdir – make a directory on a EFI partition;
• efi_ls – list files on a EFI partition;
• efi_rm – remove files on an EFI partition; and
• efi_rmdir – remove a directory from an EFI partition.

These commands are further documented in efi(4) and in their respective man pages.