Rescuing an Interrupted Ubuntu Upgrade

Upgrading Ubuntu from one version to the next has always been amazingly reliable; yet, sometimes, things happen. How do you recover when your upgrade is interrupted and you can’t boot the system in the usual way?

First step is to boot the machine to a shell that you can use to recover. This can be done in the standard system by using the Recovery Mode. Press the left shift key as the system boots up and choose a recovery mode kernel from the menu that appears – it will be labeled with “(recovery mode)” on the end. The recovery mode kernel will present you with the option to use a recovery shell.

You could use the rescue mode by booting with an Ubuntu install CDROM and entering “rescue” at the boot prompt, but then you must remember to mount all of your partitions and to perform a chroot to /target where the root partition is mounted.

After the system has booted to the menu, select the item that says “Drop to root shell prompt with networking”. This provides you with the ability to update the software packages from the Internet.

When the prompt appears, start by making sure that all currently installed programs are configured:

dpkg --configure -a

This may take a long time, depending on how many programs were installed before the system stopped. Then go ahead and update the current list of packages:

apt-get update

Following this, upgrade the software on the system:

apt-get upgrade

This last step, again, may take a long time to complete. You should pay attention to the list of packages held back (listed at the top of the output from this command); these packages will have to be requested specifically. Usually, packages are held back because they require new software or other drastic changes: the Linux kernel is always held back. Use a command like this one to install these packages:

apt-get install some-package some-other-package

You may have to repeat this more than once until all of the packages have been installed and none are held back.

Then, you should repeat the update and upgrade in order to completely verify that the system is as updated as it can be:

apt-get update
apt-get upgrade

These last commands should execute quickly, as everything is probably complete – however, it is not impossible that the upgrades would have affected something that requires another upgrade. Not doing this probably won’t matter, but why not do it anyway?

Once done, a reboot is required to make sure that the old software is no longer being used and all that is used is the new upgraded software. Don’t just continue the boot: reboot.

One more thing can be done to clean up afterwards – possibly after the reboot. At a command shell, enter this command to remove unneeded software:

apt-get autoremove

Some general tips on making distribution upgrades as easy as possible: first, upgrade the system to the latest set of patches and software available. If there are bugs in the ugprading software, you want to fix it first.

Once upgraded to the most recent patch set (or the most recent minor level), then upgrade to the next major level (and no more!). Upgrades are normally tested only to the next major release. Any other upgrade is certainly not tested and not guaranteed to work.

If you want to upgrade beyond just the next major release, repeat the above steps for each major release: first upgrade to the most recent software, then upgrade to the next major release.

Make sure that you have reliable power throughout: if a system loses power in the middle of an upgrade process (or otherwise stops cold) then you will have to perform a recovery and hope that nothing was lost.

Lessons in Communications and Reliability Learned from Egypt

You may have already heard about what has been happening in Egypt. If not, the Arab media source Al Jazeera has a dedicated page on the topic. You can watch a live Twitter stream from Twitterfall.

As a part of what is happening in Egypt, Internet access in the country was disrupted and blocked, and cell phone service was halted. In particular, DNS servers were shut down or blocked, and web sites such as Facebook and Twitter were blocked completely. The majority of Egyptian ISPs shut down as well, effectively removing all access to the Internet to Egyptian citizens.

Here are some of the resolutions to these problems that Egyptians found:

  • Broadcast access to Internet via open wireless access points. With wireless access points set up, only one person needs to set up the Internet in order for dozens to receive it. At least one ISP in Egypt remained up (due to government and bank usage) – providing an open hot spot expanded
  • Use gateways and proxies to reach forbidden web sites. Routing traffic to other sites – sites that aren’t blocked – permits access to the blocked sites. This is a form of “forced routing” that goes around censorship.
  • Use alternate DNS servers or IP addresses or both. There are public DNS servers available to all such as Google and OpenDNS; if DNS is down one can switch to these if you know how – and use IP addresses if you don’t.
  • Use out-of-country dialup services. Several ISPs gave out public access to their dialup services for Egyptian citizens to reach the outside.
  • Use non-internet-based methods of communication. In Egypt, there were printed leaflets, as well as amateur radio. When communication via the Internet is out, there are alternatives.

There is a web page that details all the possibilities for getting communication out of Egypt.

If you can handle a man-made disaster – such as the cutoff of Egypt from the Internet, or the dismantling of the Wikileaks technical structure – then natural disasters seem pale by comparison.

We’ll pray for safety and recovery in Egypt.