Open Advice/Backups to Maintain Sanity

Backups to Maintain Sanity

by Austin Appel

in: Open Advice

This text is available under the CC-BY-SA license. (see also: Open Advice/Info)

Contents 1 info about the author

Backups are good. Backups are great. A competent admin always keeps current backups. This much can be gathered from any manual on server administration. The problem is that backups are only really used when absolutely necessary. If something drastic happens to the server or its data and one is forced to fall back on something, the backups will come to the rescue in the moment of most dire need. However, this should never happen, right? At any other time, what does having backups do for you and your server environment?

Before going further, it is important to note that the advice espoused is for the smaller open source project server administrators out there – the silent majority. If you maintain services that would cause a large amount of frustration, and even perhaps harm if they experienced any downtime, please take this with a very small grain of salt.

For the rest of us who work with countless smaller projects with limited resources, we rarely have a separate test and production server. In fact, with all of the many services that an open source project typically needs to maintain (version control, web services, mailing lists, forums, build bots, databases, bug/feature trackers, etc.), separate testing environments are often the stuff we can only dream about. Unfortunately, the typical approach to system administration is to tread lightly and only upgrade the systems when absolutely necessary, to avoid risking dependency issues, broken code, or any of the other million things that could go wrong. The reason you are nervous is not because you may be inexperienced. It is important to know that you are not alone in this. Whether or not we admit it to others, many of us have been (and likely still are) in this position. The sad fact is that this inaction – stemming from the fear of breaking a “working” system – often leads to running services which are often several versions behind the curve, and come with a host of potentially serious security vulnerabilities. Rest assured that this is not the only way to play the game though.

People tend to play a game differently when they have infinite lives as compared to needing to start over from the start as soon as one mistake is made. Why should server administration be any different? Approaching the concept of backups with an offensive mindset can change your whole view of administrating systems. Instead of living in fear from a complete dist-upgrade (or equivalent command for yum, pacman, etc.), when armed with backups, one is free to update the packages on a server secure in the knowledge that the changes can always be rolled back if things turn sour. The key to getting over this is all about a state-of-mind. There is no reason to fear as long as you have your safety net of backed-up files beneath you as you jump. After all, system administration is constantly a learning experience.

Of course, if you do not validate your backups, relying on backups in this way becomes a very dangerous game. Fortunately, experienced system administrators know that the commandment “keep current backups” is always followed by “validate your backups.” Again, this is another mantra that people like to recite. What does not fit as elegantly into a catchy mantra is how quickly and easily validating backups can be accomplished. The best way to tell that a backup works is, of course, to restore it (preferably on an identical system not currently active). But again, in the absence of such luxuries, a bit more creativity is required. This is where (at least for files) checksums can help you determine the integrity of your backed-up files. In rsync, for example, the default method it uses to determine which files have been modified is to check the time of last modification and size of the file. However, by using the “-c” option, rsync will use a 128-bit MD4 checksum to determine whether files have changed or not. While this may not always be the best idea to do every time in all situations due to likely taking much longer than a normal rsync and increased io usage, this ensures that the files are intact.

The role of system administrator can be a stressful one at times. However, there is no need to make it more so than it needs to be. With the proper frame of mind, some ostensibly single-purpose defense-seeming tasks can be used as valuable tools to allow you to nimbly move forward with your sanity intact with the speed appreciated by all open source projects.

info about the author

Austin “scorche” Appel is an information security professional who spends his time breaking into things previously thought secure (with permission, of course). He is often seen around security/hacker conferences teaching people how to pick locks. In the open source world, he does a host of things for the Rockbox project and previously volunteered for the One Laptop Per Child project.