One of the things I’ve been trying to get a handle on are the home backups. I’ve been using Retrospect since, well, basically forever, most recently using it to back up the home machines to a firewire drive on my mini. I started out using two 100 gig drives in rotation, and when they filled up, added a pair of 200 gig (god bless digital photography). It’s to the point, though, where a full backup of the house (two laptops, two minis) is now > 200 gigs, and if you rotate your backup sets when the second disk fills up, you’re rotating them fairly often — and think about how long it takes to restart a backup with a full set.

You end up with too many windows of opportunity for things to go wrong, which made me increasingly uncomfortable. Add to that having all of the files stored in a proprietary format by Retrospect, and Retrospect’s long history of breaking every time a new release of Mac OS X coming out, with a delay before they fix it, and then a few patches to get it really right — not my idea of fun for a backup tool. And then there are various features of modern Mac OS X that retrospect simply punts on.

So retropect has increasingly been a tool I’ve been looking to retire.

At the same time, I’ve been working more and more with a tool called Superduper!, which is (to some degree) rsync with a GUI, although it’s not really quite that. It’s allowed me to set up backups of the laptops to portable (now power brick) firewire drives, so the laptops can be backed up even on the road, Just In Case). Superduper also will back up over a network, but does it to a spare disk image — which, of course, Retrospect won’t back up. Another reason to retire Retrospect.

None of this really handles the offsite backup problem to my satisfaction, either, not even close. At one point, I kept three sets of backups via retrospect, rotating one offsite, but as the size of the backup grew, I let that lapse. Shouldn’t have, but I did. Besides, is an offside backup that’s four months old REALLY useful?

So, what do I really want?

1) No more full backups.

2) full automation.

3) No special tools to access files.

4) bootable backups (or backups that can easily be turned into bootable disks again);

5) off-site storage.

6) off-site storage WITHOUT physically moving stuff off-site, or having to make special off-site disks.

7) failure resistance. A failed disk should at worst be inconvenient. Ditto failed backup media.

After spending time researching tools and what other people are doing, I came fairly close, and down the road, I’ll have it all (I think).

The first decision: stop creating new backup sets, and dump retrospect. Instead, use RAID 1 to create a redundant mirror of the data. That way, if any one drive fails, there’s a usable copy that can keep the backup going and you can clone it to rebuild the RAID. RAID 1 also allows you to, if you want, add and remove drives, which gives you the option to create copies to go offsite.

You can do this in a few ways:

1) dedicated network storage device, that hooks up to your network and acts like a file server. they’re called various things like “network hard drives”, and come from any number of companies including Lacie or Infrant.

2) add a RAID system to the mini, using swappable bays to allow you to replace drives as you want. This would make creating off-site copies easy, as well as failure recovery as simple as possible. Wiebetech is a company with a line of products that does this.

3) Software RAID on the mini, and firewire drives.

The third approach is the one I decided on. I did so for a few reasons. First, I didn’t want to add another computer to the house, dedicated or not, so that let out the network disks. I also moved away from the network disks because most of them don’t do RAID. Most of the RAID 1 options require SATA boards, not firewire, although a few connect w/ Firewire 800. Neither is an option on a mini. If you look carefully, you’ll see most of the Firewire 400 RAID units tend to be raid 0 (striping), not Raid 1 (mirror), so they don’t really solve my problem well. The ones that do, along with the RAID systems with removable bays, tend to be significantly more expensive, than the third option — ultimately, I decided that extra expense wasn’t worth it.

My choice: Other World Computing has their Mercury Elite line of firewire drives that support drive sizes up to 750G, and have either one or two drives in them. The two-drive units use Software raid (softraid) to implement the RAID 1, and are pre-configured for RAID 1, so it’s plug and play. They come bundled with Softraid, so you can do other things as you want to.

I ended up buying the OWC Mercury Elite 500×2 with RAID1, plus a 500X1 unit to use as my off-site storage. I installed Softraid onto the mini, set it up to share the drive, and I’m currently using SuperDuper to back up the laptop to a sparse disk image over the net. Once that’s done, I’ll automate updating it nightly, and do the same to the other machines here at home.

The cost of the 500X2 & %00X1 with bundled raid software: ~$900. To create an equivalent with Wiebetech’s RAID systems (with hot swap and etc) would have run closer to $1600. network appliances that would support a 500G network drive in RAID 1 with the ability to roll a third unit for offsite start about $1600 as well, and keep going up from there. That $700 would pay for more drives, if I wanted to keep rotating units offsite.

My long-term goal here, though, is to roll off-site backups over the net, to S3 or some other network storage service. The initial 400Gig upload might be painful (or very painful, or extremely painful), but after that, it wouldn’t necessarily be so bad; You’d want to mount it as a file partition and update the sparseimages via Superduper, not update the backed-up RAID drive. I’m just not sure the technology is quite ready for that, and I’m still investigating what the real costs are in terms of storage charges and network upload charges — but my chickenscratch numbers indicate hauling physical disks offsite wins as far as costs go, even though a bit less convenient.

But I expect that to change, and that’s another reason not to invest in hot-swap RAID bays and stuff; I’m not too far from where that dual drive firewire unit will be my backup drive, and only touched for restores, or to replace a failed drive.And with RAID 1, that’s merely annoying, not a serious problem.

And it’s a setup that’s very resistance to problems caused by, say, upgrading to Leopard. All I need do is hold off upgrading the mini until softraid is updated and stable — no worry about the retrospect client or server software compatibility.

Given the sheer amount of data in a house these days, the only practical backup is to another disk. The only practical way to back up a backup on disk is via RAID 1. And ultimately, the way to protect the entire house is to copy that backup somewhere else.

This new backup setup, once I have the machines configured to do nightly backups, does all of that but the offsite component. I expect to do that manually, but it’s firewire plugs and quick configurations via the softraid gui, so it’s simple and fast. So it’s more likely to actually BE DONE. and everything is stored in ways that can be accessed by Mac OS X without special programs or tools.

What’s lost by removing Retrospect from the mix? The only significant thing are snapshots over time: Laurie and I talked that over, and the answer to the question “when was the last time we actually had to go find a copy of that Word file from last tuesday?” was “I don’t remember”, so in reality, it’s a minor thing we can easily live without. So we will. That’s probably a job for Subversion, if you really care…

One other nice thing about this setup: it scales. If I fill up this 500g, I can add another. and another, and re-arrange what machines backup where without having to completely redo the backups. With the RAID bay units, network disks, you’re scaling options are more limited, and generally limited to “add another unit”, which given the costs, adds up over time.

All in all, I think I met most of my requirements pretty well — and more importantly, set things up to not need any significant work for a few years moving forward. I’m very satisfied with the design. Now we’ll see how it plays in real life….

When I left Apple, I started a series of articles as kind of a post-mortem and view on things that I really shouldn’t have gotten into when I was an employee. As those things go, I ran out of interest in writing it before I ran out of things to write.

Left unsaid are three more topics:

Part 6: Should Apple blog?

Part 7: The Marketshare “problem”, (aka, damn you, Mike spindler, or why Apple’s Marketshare only matters to analysts you shouldn’t listen to anyway)

Part 8: Where Apple fits into the big picture.

And at this point, I’m frankly more interested in looking forward instead of looking backward, so I felt it made sense to just close the loop and mark this series done (but not complete) and move on. If you disagree — well, let me know and convince me to carry on. Me, I’m looking into 2007 for the launch (finally) of the Outsider’s Guide, and spending more time on my hockey writing and my photography, and Apple is probably happier if I just stop digging up skeletons, no matter how minor and trivial.

And while we had a big mosh pit over my comments on Apple’s blogging policy (and lack of one), it just reinforced to me that so much of the blogosphere is a self-reinforcing echo chamber, with people not really interested in hearing or learning, but merely making sure everything that gets said gets interpreted to reinforce what they already feel like thinking, or gets ridiculed and ignored. In that way, Blogging has turned very much into USENET of old, only with CSS formatting and moving pictures; the technology changes, the human interactions don’t. Sometimes, it seems blogging isn’t so much the conversation pro-blogging advocates want to promote it as as it is a bunch of people lecturing, all in the same lecture room with microphones. Everyone talking, nobody listening.

Of course, that’s unfair. There’s also a strong group of folks who ARE actually interested in discussing and thinking — but sometimes, it’s hard to hear them through the noise of the “loudest blogger wins” group. If people care about those topics, I can be convinced.

The executive summary of part 6, though, is worth a couple of paragraphs:

Should Apple have a blogging policy? In my mind, definitely, even though I could never convince the folks who needed to agree of that. One reason is simple: a standardized blogging policy would put all employees on the same footing, and it’d be understood what was acceptable and what wasn’t. As it stands, this decision often is made by a direct manager, or perhaps one or two levels up, and different parts of the company create restrictions (some of them very strict) well beyond the intent of the existing policies, and in some areas, in ways that significantly poach into Apple employee’s personal lives and personal time, which I feel is inappropriate (many areas of Apple retail, the brick and mortar part, simply outlaw employee blogs in any way — not just talking company stuff, but talking about anything, including Aunt Jenny’s wedding. That, I think, is excessive, and the reason I felt a blogging policy was needed in the first place. At the same time, however, I think that blogging policy, while making personal blogs acceptable, should clearly put “shop talk” out of bounds, unless blogging about Apple is part of your formal job description. This would (more or less) keep Apple’s blogging reality to the status quo, while making it explicit that non-work blogging is okay — and that’s the balance I think is needed and appropriate at Apple. As it is, Apple employees that want to blog personally simply hide their affiliation, which I think is silly, but in the current environment, necessary.

And should Apple blog? Absolutely, but not in a way that Scoble would promote or consider acceptable. I certainly wouldn’t create blogs.apple.com and open it to all employees the way Sun has — the situations are different — but I’d want to have a blogging system that execs and product managers and people who ARE allowed to be company spokesbeasts in a formal way could use as a communication channel. this is the path I think Dave Hyatt was trying to blaze with his safari stuff, but I don’t think the Apple culture was really capable of embracing it, and Apple management just doesn’t seem to understand how this can be used to advantage — or if they do, didn’t make it any priority to get done.

ohwell. At this point, it’s probably opportunity lost for Apple. I’d still argue “better late than never”, but Apple is definitely missing out on some significant and substantial changes in how people communicate and how companies interact with their customers; here’s hoping they don’t guess wrong and open a market opportunity to a competitor by not doing this.

I will also, just to close this all out, talk about one time when I brought all this up — waiting for a meeting on some subject I don’t recall with some apple managers from various parts of the company, and a couple of Apple’s finest legal beagles, we were basically shooting the breeze waiting for a couple of others who live in the Apple “chronically late” time distortion field (the one in which, despite best of intentions and hard work, all meetings start at 10 after the hour, because everyone is so chronically over-scheduled that they simply can’t get from meeting to meeting in time, because every meeting ends up running the full hour and so many people are booked back to back to back) — and blogging came up as a discussion point.

And I suggested that we have Steve blog. Silence in the room, followed by a few muffled giggles.

But think about it. Is there one person in the universe, who, if they blogged, every person on the internet would read? Imagine the ability of Steve to create a buzz, push a product, set up a marketing program, create an agenda. He could, merely by saying “hello”, give half the internet the vapors, and the other half heartburn.

you’ve seen Steve with the keynote bully pulpit he uses a few times a year. Imagine Steve with the bully pulpit of “Steve’s blog”, available any time he felt like talking about something — Apple or no.

The folks I tossed it at agreed it was a powerful idea, but couldn’t decide if it was one of powerful genius or merely an insane one. The one thing everyone was unanimous about was that they’d die before suggesting it to him.

And they’re probably right — but man, I always felt that would have been such a fun hack. Steve unplugged. Or maybe Steve unfiltered. Is the world ready for it? (is Steve?)

but I guess we’ll never know. As the folks in that meeting who worked directly with Steve all agreed: “Never happen”. But they said so with that look on their faces that indicated they saw the possibilities too — and the risks.