, , , ,

The blogging site JournalSpace has been shut down after there was significant data loss without backups. The entrepeneur’s blog has more information – apparently, the most likely cause seems to be sabotage by a former IT staff person, combined with the lack of working backups.

What can we learn from this unfortunate incident? There are a number of things to note here:

  • Remove all access for former staff in its entirety – don’t skimp! All access, passwords, server access, everything. Lock it down. If you have only one IT staffer, you are also at risk: you need to be able to call on someone who can lock out your fired (or laid off) employee completely.
  • Disk RAID is not a backup solution. RAID protects you from disk failure, not from “data failure” or operator mistakes. Do not forget to have a complete backup solution in place. It also pays to enable a “hot spare” so that if one of the disks fail, that there is still protection from disk loss.
  • Have a backup solution. You must have a comprehensive backup plan working, tested, and implemented.
  • Have a working backup solution. This point cannot be stressed enough: Test your backup solution before you need to use it! When the data is gone is no time to realize that the backups are useless. Test your backups in real-world scenarios as well: one story described a backup solution that was well-tested, then the tapes went off-site in the operator’s car. Unfortunately, sitting in the car caused the tapes to be demagnetized and this was realized only after the data was gone. Test those backups!

The dreadful story of JournalSpace might have had a different ending if they had only tested their backups: that alone would have saved them. However, solutions (like security) should be in depth: working backups might not be enough next time.