Are You Ready for the Onslaught? (or Scaling Your Environment)

Is your environment ready for the onslaught that you may or may not be aware is coming your way?

One commonly known example of this is what is called “the Slashdot effect.” This is what happens when the popular site Slashdot (or others like it) links to a small site. The combined effect of thousands of people attempting to view the site all at once can bring it to its knees – or fill up the traffic quota in a hurry.

Other situations may be the introduction of a popular product (the introductions of the Iphone and of Halo 3 come to mind), or a popular conference (such as EAA‘s Airventure, which had some overloading problems).

Examine what happens each time a request is made. Does it result in multiple database queries? Then if there are x requests, and each results in y queries, there will be x*y database queries. This shows that as requests go up, database queries go up dramatically.

Or let’s say each request results in a login which may be held for 5 minutes. If you get x requests per second, then in 5 minutes you’ll have 300x connections if none drop. Do you have buffers and resources for this?

Check your kernel tunables, and run real world tests to see. Examine every aspect of the system in order to see what resources it will take. Check TCP buffers for networking connections, number of TTYs allowed, and anything else that you can think of. Go end to end, from client to server to back-end software and back.

Some of the choices in alleviating pressure would be using caching proxies, clusters, rewriting software, changing buffers, and others.

James Hamilton already has collected a wide number of articles about how the big guys have handled such scaling problems already (although focused on database response), including names such as Flickr, Twitter, Amazon, Technorati, Second Life, and others. Do go check his article out!

AirVenture 2008 (Oshkosh, Wisconsin)

For part of last week, I was at AirVenture 2008 – the largest gathering of flyers and aviation buffs in the world. For this week, little Wittman Field in Oshkosh, Wisconsin becomes the busiest airport in the world.

Past conventions have featured the Harrier jumpjet, the Osprey, the Warthog, the Wright Flyer, many airshow teams, the Concorde, the Raptor, and many, many other aircraft. Every aviation related company makes an appearance, including such diverse companies as Piper, Cessna, the Air National Guard, the Federal Aviation Administration, the U.S. Border Patrol, Flying Magazine, the Aircraft Owner and Pilots Association, and much, much, much more.

Aviation luminaries also make an appearance, including those such as Burt Rutan, Patty Wagstaff, Buzz Aldrin, and many more.

One interesting thing for a systems administrator to see is how companies utilize computing, as well as how they handle such a large number of users.

At Airventure, the first thing I noticed was that wifi was unusable at the camping site nearby – because the captive portal was overwhelmed by the number of people using it (and not logging out). Part of a system administrator’s job is to predict utilization and to be prepared for it – and to be able to “scale” or handle a large influx of users without hiccups. How would you respond? How would you have prevented it?

The second thing of interest was in aircraft avionics – that is, the pilots array of dials and guages. The talk (old to aviators) is of the “glass panel”, or an LCD display which replaces multiple dials and guages. When I asked someone who was with Avionics Magazine about the use of open source and of operating systems such as Linux and/or Windows Embedded, it turns out that none of these products can be used in avionics as they are not certified for avionics. Interesting indeed.

Refresh yourself!

System administration can be a very stressful position. When systems go down, and CEOs and other executives start leaning over your shoulder, or a system fails and you suddenly (and temporarily) become one of the most sought-after people in the company – the stress is immense and undeniable.

Even if you release your stress by doing what I do – administering your systems instead of someone else’s – it is even better to give up the computer, give up the Internet, and find something radically different to do – something which requires a different set of skills and thought processes.

In my case, when I do this, one thing I might do is to pick up a camera. Currently I am using a Pentax PZ10 (film) but I hope to augment it with a Pentax digital SLR sometime in the future. (Yes, I know Canon is great – and so is Nikon – but they won’t fit my Pentax lenses…)

Another thing I do is attend my favorite conferences. The biggest conference I like to attend is happening this week (aviators already know which one!) – Airventure. To us in the neighborhood of this world-wide gathering of aviators, we just call it “Oshkosh” (Airventure is located in Oshkosh, Wisconsin) or “EAA” (the Experimental Aircraft Association, or EAA, are the sponsors and conference operators).

Events and hobbies like these force you to use other skills, and other thought patterns – relieving stress and perhaps even lengthening your life. Stress is a killer, and learning new things is thought to lead to a longer life as well.