Mainframe Linux: Pros and Cons

Why would one want to move Linux to the mainframe (such as IBM’s z10)? There are many reasons – and many reasons not to. Computerworld Australia had a good article describing (in part) some of the reasons the insurance company Allianz did just that. IBM has been pushing Linux on the z series for some time, and Red Hat and SUSE offer Linux variants for that purpose.

One common reason to move to a mainframe is that Linux servers have proliferated in the data center, taking up valuable space and becoming quite numerous. When all you need for a server is the hardware and a low-cost or no-cost Linux, then servers start popping up all over the place.

A single mainframe such as the z10 can handle thousands of servers (a test done in 2000 put 41,400 Linux servers on one IBM mainframe). The replaced servers can then be eliminated from the data center, freeing up valuable space and reducing the workload of current system administrators.

A common instance is where the company already has a mainframe in-house, running COBOL applications. Thus, the purchase cost of a mainframe (in the millions of dollars) has already been absorbed. Such a scenario also makes the case for a new mainframe much more appealing, as it puts the enhanced power to work immediately.

Replacing thousands of Intel-based Linux servers with a single mainframe will reduce cooling costs, power costs, physical space requirements, and hardware costs.

So why would anyone not want to use a mainframe?

If there is not already a mainframe in the data center, purchasing a mainframe just for the purpose of consolidation can be too much – mainframes typically cost in the millions of dollars, and require specially trained staff to maintain. Adding a mainframe to the data center would also require training current staff or adding new staff. A new mainframe also requires a new support contract. All of this adds up to not just millions of dollars of additional cost up front, but additional costs every year.

Another consideration is the number of Linux servers in the data center that would be moved. If there are dozens – or a hundred or two – it may not be entirely cost-effective to focus a lot of energy on moving these servers to the mainframe.

A supercomputer such as HP’s Superdome (with its attendant iCap and Integrity Virtual Machine capabilities) would probably be a better choice to consolidate dozens of Linux servers. The costs are lower, and the power requirements are lower – and you can purchase as much or as little as you need and grow with iCap. Most companies also already have UNIX staff on hand, and adapting to HP-UX is not generally a problem if needed.

Another benefit is that a server such as the Superdome offers virtualization of not just Linux systems, but Microsoft Windows and HP-UX as well – and soon, OpenVMS as well.

Using a large Intel-based server can virtualize a large number of servers with software from companies like VMWare and Sun.

These options won’t necessarily allow you to virtualize thousands of servers – but then, do you need to?

HP Instant Capacity (iCap)

One of the things that may affect any clusters you have – or other systems – is that management does not want to spend enough to handle any possible load.  With a cluster, this means that you may not be able to handle a fail-over because there is not enough spare processing power to handle the extra load when it happens.

HP’s Instant Capacity (“capacity on demand”) is an answer to this dilemma.  The base idea is that you have extra hardware already in the data center that is not available for use until it is necessary.  The switch that will enable this expanded capacity can be automatic or manual; when some portion of the extra capacity is enabled, you pay for it and it can be used from then on.

Yet, Instant Capacity (iCAP) is more flexible than this.  The capacity may be enabled only temporarily instead of permanently – this is known as TiCAP (temporary iCAP).  Thus, you can save even more by buying extra hardware but enabling only a small portion of it.  During the recent HP Tech Days that I attended in San Jose, California, a situation was described where an HP Superdome could be purchased with a large amount of the hardware already in place – but only a small amount of the hardware enabled.  When the extra power is needed, for example, a cell in the Superdome could be enabled until such time as the power is no longer necessary.

There is also Global Instant Capacity (GiCAP) which even allows the movement of power from one system to another.  For example, if a CPU on one system is underutilized and another system needs the resource more – then the CPU resource can be “logically” moved from one system to the other through GiCAP.  Alternately, if one system dies and another system needs its power, the dead system’s resources can be used by the active system by moving them through GiCAP.

iCAP and TiCAP are available for HP-UX (on PARISC and Itanium) and for OpenVMS (only on Itanium). GiCAP is only available for HP-UX. 

I find iCAP and TiCAP to be very interesting.  From a cost perspective, you pay only a minimal amount to keep the resource; when it is enabled, you then pay for it for the duration – or buy the hardware outright for permanent use as needed.

Powered by ScribeFire.

HP Tech Day: HP Superdome

I was recently invited to take part in an HP Tech Day in San Jose, California, celebrating the 10th anniversary of the HP Superdome (the most advanced server in the Integrity line).  This was an event designed to introduce several of us in the blog world to the HP Superdome.  The attendees included David Adams from OSNews, Ben Rockwood from Cuddletech, Andy McCaskey from Simple Daily Report (SDR) News, and Saurabh Dubey of ActiveWin.  This was a quite eclectic and broad mix of perspectives: David Adams covers operating systems; Ben Rockwood covers Sun goings on (as he mentions in his article, he wore a Sun shirt: bold as brass!); Saurabh Dubey covers Microsoft goings on; and I, as loyal readers may know, cover system administration (with a focus on HP-UX, OpenVMS, and Linux – all of which will run on Superdome). Andy McCaskey over at SDR News also had a nice writeup on his experiences.

It is possible I was the most familiar with the architecture and with the capabilities, though I’ve not seen or worked with a Superdome in the past: the capabilities of the Superdome are largely based on the fact that it is cell-based.  The rp7420 cluster which I have maintained over the last several years uses the same technology, though cells from the rp7420 are incompatible with the Superdome.  The software is the same: prstatus, etc.  The System Management Homepage (SMH) was also shown, although it was almost shown as a benefit of the Superdome (it’s actually in every HP-UX since 11i v2, and is an option for OpenVMS 8.x).

There was a lot of talk about “scaling up” (that is, use a larger, more powerful system) instead of “scaling out” (using a massive cluster of many machines).  The Superdome is a perfect example of “scaling up” and is possibly one of the best examples.  I was impressed by what I saw as the capabilities of the Superdome.  There was a lot of comparison with the IBM zSeries, which is the epitome of the current crop of mainframes.  The presenters made a very strong case for using Superdome over zSeries.

They did seem to focus on running Linux in an LPAR, however; this creates a limit of 60 Linux installations.  Using z/VM as a hypervisor, one can run many more Linux systems.  I have heard of a test run in Europe (years ago) where a zSeries was loaded with one Linux installation after another – when the testers reached into the tens of thousands (30,000?) the network failed or was overloaded; the zSeries system was still going strong.  Trouble is, I’m not able to back this up with a source at the moment: I’m sure it was available as part of a print (Linux) journal – it may have been called “Project Charlie.”  Can anyone help?

The usability features of the Superdome were in prime display: for example, the power supplies were designed so that they could not be inserted upside-down.  Another example: the cells for the Superdome are in two parts: the “system” (including CPU, memory, and chip glue) and the power supply.  This makes it much easier to remove in the typical datacenter row and makes each part lighter, making it easier for users. There are innumerable items like this that the designers took into account during the design phase.  The engineering on these systems are amazing; usability has been thought of from the start.  In my opinion, both HP and Compaq have been this way for  a long time.

Speaking of the tour, this system that they showed us was a prototype of the original HP Superdome that shipped for the first time in 2000.  This system was still going and was using modern hardware: these systems are not designed for a 3-4 year lifecycle, but a much longer, extended lifecycle.

There were a lot of features of the system that I’ll cover in the next few days; it was enjoyable and educational.  I very much appreciate the work that went into it and hope to see more.

By the way, if you read Ben Rockwood’s article at Cuddletech, look at the first photograph: your author is center left, with the sweater.

Update: thanks to Michael Burschik for the updated information on Test Plan Charlie, which saw 41,000 Linux machines running on the IBM zSeries back in 2000. I’ll have a report on it soon. Strangely enough, I still haven’t found the article I was thinking of – but of course, a project like that isn’t reported in just one periodical…

Powered by ScribeFire.

User Interface Design: the Command Line

The command line is not immune from user interface design. Especially with the concept of language, one has to choose carefully the options and names and orders of the things in order to make things work just the way the user expects them to.

If the program is too different, people will be tripping over it all the time. The UNIX tar command comes to mind as one that failed here: options (or “actions”) specifically did not start with a dash. Likewise, UNIX find also failed: if you didn’t include the parameter -print at the end, you saw no output: your find command found nothing! (In reality, it just didn’t report it.) Both of these errors have been rectified in the last several decades: UNIX find has an implied -print, and tar often will make the dash optional – which makes it work both the way it always did and the way it should have.

As an example of what seems to be a colossal user interface failure – including poor writing – consider these articles from Scott Remnant which are absolutely a gem (albeit from way back in February 2009). He wrote an article titled Git Sucks – which was then followed by a second and then a third – followed by yet another titled Revision Control Systems Suck.

What Scott is railing about is how hard these systems are to learn (he targets not just git, but also GNU Arch and Bazaar). From his standpoint, he finds these systems to be complicated and hard to understand.

He also points out (rightly) that the most common actions should be the simplest, and finds that with git these common actions are rarely ever simple. He specifically mentions reviewing the changes that someone else has made compared to his own – and says that there’s not a revision control system that makes it easy.

An example of how user interface design can be incorporated into things like the command line and even programming is this quote from an interview with Yukihiro Matsumoto, the developer of the programming language Ruby about his guiding principle in developing Ruby:

[It's] called the “principle of least surprise.” I believe people want to express themselves when they program. They don’t want to fight with the language. Programming languages must feel natural to programmers.

and later in the same interview:

In addition, Ruby is designed to be human-oriented. It reduces the burden of programming. It tries to push jobs back to machines. You can accomplish more tasks with less work, in smaller yet readable code.

Another example: I was just rereading my copy of The Humane Interface written by Jef Raskin. In it, he had a section titled Noun-Verb versus Verb-Noun Constructions (section 3-3, p. 59). This mirrors a problem I have experienced with some command line software in the past: the command wants an action as the first argument, and the object of the action second. I despised it enough that it was the genesis of my writing a wrapper for the command that reversed the order: object first, action second. Imagine my surprise to find my troubles validated right there in Raskin’s book.

There are many examples of command line programs doing wrong things, and of programs doing right things. One of the right things comes from HP-UX and its software management tools such as swinstall: if the program can use an X display for a graphical display, it will: but if not, it goes to a text display instead.

There are many such examples, of programs just doing what you need and leaving you to think about other things. I wonder what would happen if a company like Apple decided to tackle the command line – although, in a way, they did already. In MacOS X, consider the open command for instance… absolutely brilliant, which is in contrast to the open command sometimes found in other UNIXes (never standard).

One very important point to remember: “It’s only hard until you learn it” is not a valid excuse. The learning curve for a program should not be any steeper than it has to be.

Renaming a host (UNIX, OpenVMS)

Renaming a host is not, in general, a pleasant experience. The general requirement is that you must find everywhere that your hostname is specified and change it.

In OpenVMS, this can be an extensive process, and even require relicensing if you have licenses that depend on your hostname. It can also require rebooting of an entire VMS cluster if you miss changing a parameter. The full details are in the VMS FAQ such as this one (dated from 2001) from faqs.org or this one from HP (undated). Hoffman Labs has a copy from September 2006; there is information on changing a node name in section 5.7.

Not that in changing the OpenVMS hostname in a cluster, you must change the SCSNODE parameter (which changes the cluster node’s nodename). If you change the SCSNODE parameter, you must change the SCSSYSTEMID as well or the entire cluster will refuse to function until it is reconfigured. The cluster tracks the pairing between these two parameters, and if the pair changes, then the cluster stops working normally.

For UNIX in general, one way to do it is to go to the /etc directory as root and run a search:

$ su -
Password:
# cd /etc
# find . -type f -print | xargs grep -i myhost

After running this, change all of the instances of myhost that is found.

This is the way to change hostnames in Solaris, including Solaris 9 and Solaris 10. Debian and derivatives (including Ubuntu and Linux Mint) and HP-UX make it simpler.

In Debian, there is a file called /etc/hostname. This will contain the current setting of the hostname. Change this to your desired new hostname, then run the shell script /etc/init.d/hostname.sh.

In HP-UX, change to root and run the program set_parms with the hostname option:

# set_parms hostname

For all of these possibilities, the best thing to do is to reboot afterwards: this will test the new setup as well as change any in-memory hostname settings.

Changing a hostname is a drastic measure, and will include much in the way of system modification and updates. Changing the actual hostname is very likely only the beginning; there may be clients that are set up to contact the host, and any services that the server provided (e.g., NTP server, FTP server, web server, NIS server, etc.) will require reconfiguration on the clients to use the new hostname.

In summary, the very best thing to do is to get the name right in the first place.

Powered by ScribeFire.

UNIX and OpenVMS Online Resources

It is possible to get free online access to UNIX or to OpenVMS; these can be useful in building up your experience on a platform when starting from scratch – or when a review is required.

One of the oldest public access systems in the country is the Super Dimension Fortress (or SDF as it is usually called). SDF offers free accounts, but does ask for US$1 to gain standard access. This isn’t because access is expensive, but because too many people have used the facilities for nefarious purposes (the process suggests that the new user is not a person who will strike and leave).

SDF runs NetBSD on DEC Alphas; this was driven mainly by security and stability. Previously, Stephen Jones, the proprietor, ran SDF using Linux on Intel for several years (which he describes as “the dark years”). BSDTalk had an interview with him back in 2006.

You could also try PolarHome – this shell provider provides access to hosts running Linux (Red Hat, Debian, SUSE, Ubuntu, or Mandriva), OpenVMS (Alpha or VAX), OpenBSD, FreeBSD, NetBSD, HPUX, IRIX, QNX, Solaris, Ultrix, AIX, Tru64, and OpenStep. Unfortunately it requires payment for shell accounts – again because of abuse. The payment is 10 units of your local currency or US$2, whichever is more – and this is per host as well. No other site provides this diverse of a selection.

For truly free UNIX shell accounts, one can try Grex, which is a more professionally-run system (Polarhome and SDF are sole proprietorships). Grex offers totally free shell accounts, but also has memberships (for people to help support the site). It is possible that Grex has the most users as well. Like the others, paid membership does have its privileges – but unlike the others, membership is mainly to provide support for Grex, rather as a security feature.

For OpenVMS, there is a very unique online shell provider: Deathrow Cluster. This is a cluster of three machines running OpenVMS 7.3 – one VAX, one Alpha, and one emulated VAX (SIMH) on a dual Xeon machine. This last is a perfect example of what can be done with an emulator, especially with SIMH which can emulate all manner of old Digital and IBM hardware. However, SIMH does not emulate the Digital Alpha, unfortunately. Like Grex, Deathrow provides completely free shell accounts; like SDF and Polarhome, it is (or appears to be) mainly one person’s purpose to keep it running with a lot of volunteer help.

Any of these will be good sources to keep your shell skills sharp – and in some cases, programming as well. They’re also good people to support; why not offer them some donations if you can?

Adding a New Disk to HP-UX with Veritas Volume Manager

Adding a new disk to a HP-UX system that is using Veritas Volume Manager is not difficult. It does require some extra steps, but if you follow along there are no problems.

Note well that this discussion does not have anything to do with LVM whatsoever, and the use of the Veritas File System (VxFS) is a minor tangent – other filesytems (like HFS) can be used – but if you’re not using OnlineJFS (which is nearly all of VxFS and its capabilities), the question would be why?

First of all, the usual steps for adding a disk to HP-UX are necessary:

1. Run ioscan to scan for the new disk and to allocate resources in the kernel for it. My actual favorite invocation for this command is ioscan -fnC disk
2. Next, run the insf command to create device files. I prefer to use ioscan -eC disk in this instance.

Now we add the disk to the Veritas Volume Manager. Run these commands:

1. vxdctl enable
2. vxdisk scandisks

At this point, there is now an additional disk available for use by vxvm but which is not a part of any disk groups yet.

To add the disk to a disk group, use vxdiskadm and option 1. You can use "list" here to get the name of the unallocated disk. Enter the name of the disk and answer the prompts; in most cases you will want the default.

Now to extend a file system in the volume group to use the new disk, you have to figure the amount of space available to use. You can use the command vxdg -g diskgroup free to find the free space in the disk group "diskgroup" - but this does not tell you the actual amount of space you can extend by; that is, it does not include striping and other restrictions - it only includes a flat number of unallocated blocks.

To get the amount of disk space you can grow a volume by, use the command vxassist -g diskgroup maxgrow volume. This will give you the actual amounts (if it is possible to grow). Replace the diskgroup and volume with the actual names of your disk group and volume respectively.

To actually grow the filesystem, you can do the following:

vxresize -b -F vxfs -t mytag group 999999

Fill in with the appropriate information instead of mytag (a tag to identify background vxvm operations), instead of group (disk group), and instead of 999999 (use the max amount of growth - or other appropriate number - identified by the maxgrow operation).

VxVM is aware of VxFS, so resizing the volume will also resize the filesystem as well; it uses fsadm in the case of OnlineJFS.

Using NET-SNMP and HP-UX SNMP together

Why would one want to do such a thing? A major reason is that the HP-UX SNMP daemon only supports the EMANATE protocol for subagents; this means that subagents that support the AgentX protocol (which NET-SNMP – provided as part of HP’s Internet Express – supports) are not supported and cannot be accessed via HP’s SNMP daemon.

However, the HP-UX specific information is only available via the HP-UX native SNMP daemon. What is the answer?

Change one or the other to run on a non-native port, that’s the answer. With the two daemons listening on different ports – in essence, acting like to discrete damons – the capabilities of both can be exploited. Since the native HP-UX snmp daemon does not provide the capability of specifying the port, the net-snmp daemon can be moved – and it is relatively trivial to do so as well.

There is probably already a line that says:

agentaddress 161

Change this line to a new port – I used 166:

agentaddress 166

Restart the daemon. Once the NET-SNMP daemon has been moved, enable HP’s SNMP daemon (if you’ve not already done so) and start it up again:

cd /sbin/init.d
SnmpMaster start

This should enable your two SNMP daemons on different ports. Now you can access whichever one holds the data you want. For example, using the command snmpwalk, getting Caché data can be as simple as:

snmpwalk -m ALL -v 2c -c public my:166 .1.3.6.1.4.1.16563

Whereas getting HP-specific data can be retrieved this way:

snmpwalk -m ALL -v 2c -c public my .1.3.6.1.4.1.11

Note the contrast between the two commands: one accesses the host my with the standard port (my); one uses the host my with the port 166 (my:166).

As a side note, note that Caché provides AgentX subagents, and note, too, that OpenVMS supports SNMP and AgentX as of v8.x. Thus, there’s no fighting with the SNMP daemon on OpenVMS.

Getting Passwords from Random Data (portably!)

Over at Mark Kolich’s blog, he wrote several months ago about using a source of randomness (/dev/urandom) to generate passwords. The idea is simple enough: take the random data, strip out only the printable characters, and then print the desired length of characters for a password.

Shortly thereafter, he described how to use a simple shell script to generate many passwords – such as for setting up many different accounts.

Working with HP-UX and OpenVMS as I do, I immediately thought: how could I do this in Perl, making the idea portable and making a program that will work on both UNIX and OpenVMS? It was easy – and easy to make it flexible as well. Here is the program that I came up with:

#!/usr/bin/perl

# code released by David Douthitt into the public domain

use Getopt::Long;

Getopt::Long::Configure('bundling');
GetOptions( 'l=i' => \$opt_l,
            'p=s' => \$opt_p,
            'm=i' => \$opt_m );

$pat{"ext"} = "[[:alnum:][:punct:]]";
$pat{"alnum"} = "[[:alnum:]]";
$pat{"alpha"} = "[[:alpha:]]";
$pat{"simple"} = "[a-km-z2-9]";
$pat{"normal"} = "[a-km-z2-9A-HJ-NPR-Z]";

if (defined($opt_p)) {
   if (defined($pat{$opt_p})) {
      $pat = $pat{$opt_p};
   } else {
      print "undefined pattern!\n";
      exit(1);
   }
} else {
   $pat = $pat{"normal"};
}

$max = (defined($opt_m) ? $opt_m : 1000);
$len = (defined($opt_l) ? $opt_l : 6);

$x = $len;

for $i (0..$max) {
   $c = chr(int(rand(255)));
   if ($c =~ /$pat/o) {
      $s .= $c;
      if (--$x == 0) {
         print "$s\n";
         $x = $len;
         $s = "";
      }
   }
}

Note that since OpenVMS does not use the “#!” notation, that this line will be ignored as a comment and the program needs to be invoked via direct invocation of perl itself.

As an aside, Mark says how he prefers random passwords. Me, I prefer “pronouncable” passwords – still random, but using phoenemes which makes the generation process just that more complicated – and complicates internationalization. Apple’s MacOS X comes with a password generator that can generate random and pronouncable passwords.

However, with the proper password storage system a fully randomized password is good – or is it? A completely random password of eight characters could be zzzzzzzz as much as anything else. Perhaps a password with a random distribution of characters (rather than a random selection of characters) would be better. I’m not aware of any password generators that guarantee a random distribution instead of a random collection.

Powered by ScribeFire.

Veritas Volume Manager (VxVM) and HP-UX

HP-UX comes with VxFS (the Veritas File System) and the Logical Volume Manager (LVM). Only the on-disk filesystem layout comes from Veritas; HP’s volume management is all their own. There’s nothing wrong with HP’s LVM – I tend to prefer it, but then that’s what I know.

Veritas (now Symantec) offers another, competing product called Veritas Volume Manger (refered to as VxVM). The tools are different, the layout is different, and the capabilities are different. Knowing LVM won’t help you much, though the most basic concepts are the same: collect a series of disks together and then parcel them out as a single large group, with user-defined subdivisions.

Veritas Volume Manager is now a part of the Veritas Storage Foundation.

An nice set of links to documents can be found at Aziz’s Blog. In particular, the Veritas Volume Manager Administrator’s Guide has been indispensable. Just about everything you can imagine you might need to do is located here.

Cuddletech offers the VxVM Quickstart with some older, but worthwhile documents that describe VxVM and its concepts. Likewise, Unixway offers a wide variety of documents on VxVM over several versions, as well as tutorials and more.

The AdminsChoice also has a good set of tutorials; there is Veritas Volume Manager part 1 and part 2 (focusing on vxassist).

There is a mailing list, but in recent months the activity has been rather sparse.

If you want to take your knowledge all the way, you can become Symantec Certified for the Veritas Storage Foundation (which mainly includes VxFS and VxVM).

Veritas VxVM works very well on HP-UX, and it is possible to create a root disk that utilizes VxVM and VxFS. When using VxVM, LVM is not used (unless a particular disk uses LVM instead of VxVM). The commands are the same across different platforms, and the on-disk layout is the same – so it should be possible to take a set of disks from a Solaris system and put them onto an HP-UX system and still read the data (but watch out for differing byte ordering!).

In the future I hope to discuss more on VxVM; we’ll see.

Powered by ScribeFire.