backing up the modern house….

One of the things I’ve been trying to get a handle on are the home backups. I’ve been using Retrospect since, well, basically forever, most recently using it to back up the home machines to a firewire drive on my mini. I started out using two 100 gig drives in rotation, and when they filled up, added a pair of 200 gig (god bless digital photography). It’s to the point, though, where a full backup of the house (two laptops, two minis) is now > 200 gigs, and if you rotate your backup sets when the second disk fills up, you’re rotating them fairly often — and think about how long it takes to restart a backup with a full set.

You end up with too many windows of opportunity for things to go wrong, which made me increasingly uncomfortable. Add to that having all of the files stored in a proprietary format by Retrospect, and Retrospect’s long history of breaking every time a new release of Mac OS X coming out, with a delay before they fix it, and then a few patches to get it really right — not my idea of fun for a backup tool. And then there are various features of modern Mac OS X that retrospect simply punts on.

So retropect has increasingly been a tool I’ve been looking to retire.

At the same time, I’ve been working more and more with a tool called Superduper!, which is (to some degree) rsync with a GUI, although it’s not really quite that. It’s allowed me to set up backups of the laptops to portable (now power brick) firewire drives, so the laptops can be backed up even on the road, Just In Case). Superduper also will back up over a network, but does it to a spare disk image — which, of course, Retrospect won’t back up. Another reason to retire Retrospect.

None of this really handles the offsite backup problem to my satisfaction, either, not even close. At one point, I kept three sets of backups via retrospect, rotating one offsite, but as the size of the backup grew, I let that lapse. Shouldn’t have, but I did. Besides, is an offside backup that’s four months old REALLY useful?

So, what do I really want?

1) No more full backups.

2) full automation.

3) No special tools to access files.

4) bootable backups (or backups that can easily be turned into bootable disks again);

5) off-site storage.

6) off-site storage WITHOUT physically moving stuff off-site, or having to make special off-site disks.

7) failure resistance. A failed disk should at worst be inconvenient. Ditto failed backup media.

After spending time researching tools and what other people are doing, I came fairly close, and down the road, I’ll have it all (I think).

The first decision: stop creating new backup sets, and dump retrospect. Instead, use RAID 1 to create a redundant mirror of the data. That way, if any one drive fails, there’s a usable copy that can keep the backup going and you can clone it to rebuild the RAID. RAID 1 also allows you to, if you want, add and remove drives, which gives you the option to create copies to go offsite.

You can do this in a few ways:

1) dedicated network storage device, that hooks up to your network and acts like a file server. they’re called various things like “network hard drives”, and come from any number of companies including Lacie or Infrant.

2) add a RAID system to the mini, using swappable bays to allow you to replace drives as you want. This would make creating off-site copies easy, as well as failure recovery as simple as possible. Wiebetech is a company with a line of products that does this.

3) Software RAID on the mini, and firewire drives.

The third approach is the one I decided on. I did so for a few reasons. First, I didn’t want to add another computer to the house, dedicated or not, so that let out the network disks. I also moved away from the network disks because most of them don’t do RAID. Most of the RAID 1 options require SATA boards, not firewire, although a few connect w/ Firewire 800. Neither is an option on a mini. If you look carefully, you’ll see most of the Firewire 400 RAID units tend to be raid 0 (striping), not Raid 1 (mirror), so they don’t really solve my problem well. The ones that do, along with the RAID systems with removable bays, tend to be significantly more expensive, than the third option — ultimately, I decided that extra expense wasn’t worth it.

My choice: Other World Computing has their Mercury Elite line of firewire drives that support drive sizes up to 750G, and have either one or two drives in them. The two-drive units use Software raid (softraid) to implement the RAID 1, and are pre-configured for RAID 1, so it’s plug and play. They come bundled with Softraid, so you can do other things as you want to.

I ended up buying the OWC Mercury Elite 500×2 with RAID1, plus a 500X1 unit to use as my off-site storage. I installed Softraid onto the mini, set it up to share the drive, and I’m currently using SuperDuper to back up the laptop to a sparse disk image over the net. Once that’s done, I’ll automate updating it nightly, and do the same to the other machines here at home.

The cost of the 500X2 & %00X1 with bundled raid software: ~$900. To create an equivalent with Wiebetech’s RAID systems (with hot swap and etc) would have run closer to $1600. network appliances that would support a 500G network drive in RAID 1 with the ability to roll a third unit for offsite start about $1600 as well, and keep going up from there. That $700 would pay for more drives, if I wanted to keep rotating units offsite.

My long-term goal here, though, is to roll off-site backups over the net, to S3 or some other network storage service. The initial 400Gig upload might be painful (or very painful, or extremely painful), but after that, it wouldn’t necessarily be so bad; You’d want to mount it as a file partition and update the sparseimages via Superduper, not update the backed-up RAID drive. I’m just not sure the technology is quite ready for that, and I’m still investigating what the real costs are in terms of storage charges and network upload charges — but my chickenscratch numbers indicate hauling physical disks offsite wins as far as costs go, even though a bit less convenient.

But I expect that to change, and that’s another reason not to invest in hot-swap RAID bays and stuff; I’m not too far from where that dual drive firewire unit will be my backup drive, and only touched for restores, or to replace a failed drive.And with RAID 1, that’s merely annoying, not a serious problem.

And it’s a setup that’s very resistance to problems caused by, say, upgrading to Leopard. All I need do is hold off upgrading the mini until softraid is updated and stable — no worry about the retrospect client or server software compatibility.

Given the sheer amount of data in a house these days, the only practical backup is to another disk. The only practical way to back up a backup on disk is via RAID 1. And ultimately, the way to protect the entire house is to copy that backup somewhere else.

This new backup setup, once I have the machines configured to do nightly backups, does all of that but the offsite component. I expect to do that manually, but it’s firewire plugs and quick configurations via the softraid gui, so it’s simple and fast. So it’s more likely to actually BE DONE. and everything is stored in ways that can be accessed by Mac OS X without special programs or tools.

What’s lost by removing Retrospect from the mix? The only significant thing are snapshots over time: Laurie and I talked that over, and the answer to the question “when was the last time we actually had to go find a copy of that Word file from last tuesday?” was “I don’t remember”, so in reality, it’s a minor thing we can easily live without. So we will. That’s probably a job for Subversion, if you really care…

One other nice thing about this setup: it scales. If I fill up this 500g, I can add another. and another, and re-arrange what machines backup where without having to completely redo the backups. With the RAID bay units, network disks, you’re scaling options are more limited, and generally limited to “add another unit”, which given the costs, adds up over time.

All in all, I think I met most of my requirements pretty well — and more importantly, set things up to not need any significant work for a few years moving forward. I’m very satisfied with the design. Now we’ll see how it plays in real life….

You might also want to read:

1. Back in the house! I’m happy to say that my lens is back from summer camp with the nice people at Canon’s repair depot. I haven’t had a chance...
2. Backing up your data (the 2011 edition) Back in 2009, I wrote a series of articles on why it’s important to back up your disks, and my strategy for doing so. It’s...
3. As Photographers, Are We Backing Up Too Much? Some nice and sane words about backups and photography. I think one reason we tend to save everything is because we’re still new to this,...
4. Another sad chapter in the storage woe chronicles dispatches: Another sad chapter in the storage woe chronicles: You’d think there’d be an easy solution to this by now. I haven’t found it. There...
5. So the house is painted… So the house is painted, and it looks really good. The crew finished at the early end of estimates, so today was fairly free. I...