Another sad chapter in the storage woe chronicles

dispatches: Another sad chapter in the storage woe chronicles:

You’d think there’d be an easy solution to this by now. I haven’t found it. There must be thousands of us with terabyte-sized storage woes. The latest effort is turning into another debacle.

I hired CreativeTechs to advise me on a storage solution. I don’t have the resources to spring for their first choice (an Apple Xserve RAID ($8 to 12K), so I settled on their pick of a RAID box, a Sans Digital 5 bay. I had the Seattle Mac store install an eSata card when I bought the MacPro, so I was good to go.

Nothing in storage, or anything complex with Macs, is ever good to go. First

I have to disagree here. Part of the trick is to not over-think the problem.

One reason I haven’t been blogging much recently is that I’ve been working to solve this personally. here’s my solution:

I’ve just finished upgrading the laptop with a 250Gb drive, because I was tired of carrying along a 2nd bus-powered firewire drive to carry my “other” files. Now, everything fits on the laptop again. I

I’ve set up the bus-powered drive to carry files I don’t need on a regular basis: old email archives, secondary master images (more on how I have reorganized Aperture later… that was ALSO part of all of this) — things I don’t need regularly. I use a bus-powered firewire drive so when I’m on the road I don’t need power cords. Very nice.

For my bus-powered drives, I use OtherWorld Computing’s “on the go” drives, and have been quite happy with them.

My main backup is via Time Machine to a single housing, dual drive firewire unit (also from OtherWorld computing); which mirrors to both drives via SoftRaid in RAID 1. This unit is 2x 280gb.

For offfsite backups (in case of extreme emergency), I have another Firewire 400 drive that I copy the main backup to once a month and store somewhere “away”.

I have also started sending key files (image masters, my itunes library, documents, etc) to Amazon S3, using Transmit. That’s going to take a while to fully fill out, but once it does, I’ll have my key data in a safe place and independently managed. (I experimented with JungleDisk, but when Transmit came out with S3 support, I decided it was a better option).

This implies that every bit of my data will exist in AT LEAST three places: on my laptop drive, and on each of the two RAID 1 drives. For serious emergencies, I’ll have an offsite backup that’s no more than a month old, and once I get S3 synced up, I’ll be able to store key files for recovery nightly, saving me from some kind of catastrophic issue like the house burning down.

The one thing I still plan to add: one more bus-powered drive w/ 240gb. I’ll use SuperDuper to clone the laptop drive to that. That will give me a bootable backup that I can carry and use on the road, and if the laptop or laptop drive fails, I’m still able to work with minimal data losses — and I can simply plug it into any Mac and boot my disk. I’ll ahve that up and running by next weekend — until then, I’m doing superduper to a free firewire drive, but not a portable/bus-powered one. That’ll mean my “core” data (aka, what lives on the laptop) will live in four places at once (plus the up-to-a-month-old fifth copy offsite), and crucial files will end up on S3 as well for a fifth (possibly sixth) copy. That seems — sufficient, and 90% of the process is automated already, and I’ll likely be able to automate all but the offsite copy of the RAID drive.

My total disk usage right now? about 100G on the laptop, another 80gig on the “2ndary disk”. Of that, 21 gigs is iTunes (16 active, 5 archived), and my Aperture library is 14 giges, my primary set of master images is 56 gigs, and my secondary/archived master images is another 37 gigs.

As someone who remembers when Macs booted off floppies, having 180 gigs of “must have this” data kinda floors me. How far we’ve come.

None of this is rocket science. Firewire simplifies this a lot, but then, I don’t see speed as a primary need for backups over simplicity and reliability.

Honestly, I looked at other options — network servers, RAID systems with removable bays, etc, etc, ad nauseum, and I kept coming back to buying good quality firewire drives and keeping it simple. Laurie’s systems are backed up via superduper, but now that mine are done, moving her to a 240Gb internal laptop drive and Leopard will start, and she’ll end up with essentially the same setup, including S3.

It’s a variation of what I’ve been using for a couple of years now — with Leopard, mostly I’ve swapped out Superduper backups to RAID 1 on a server for Time Machine on a locally attached RAID 1 drive; and I’ve found it pretty reliable. When I’ve needed to fix something, I can. Time Machine improves this, because it gives me incremental backups, where SuperDuper is a clone, so fi you have automated backups that overwrite something that you dn’t realize you need for a few days, you can be screwed. In practice, that’s not a huge problem, but even so, Time Machine fixes that for the most part.

Related posts:

1. Super Secure, Super Easy, Super Cheap On-Line Storage Digital ProTalk: Business Day Thursday; Super Secure, Super Easy,...
2. SezWho Comes to Venture Chronicles : Venture Chronicles SezWho Comes to Venture Chronicles : Venture Chronicles: I...
3. Backup Sunday Backup Sunday: Did you back up your data this...
4. Venture Chronicles: Why I’m done with Digg Venture Chronicles: Sites like Digg were at one time...
5. Wikis and Spammers : Venture Chronicles Wikis and Spammers : Venture Chronicles: Coincidentally, I have...