oops!

If you follow me on twitter the other day, you probably saw that I had an “oops!” moment.  It’s a great example of “best laid plans….” situation.

I was doing some work on the server we host our sites on, and I ended up in a different location than the one I thought I was in, and did a “rm -rf *” in it. And as soon as I did, I went — “oops!” because I realized I wasn’t in the right place.

The NEXT thing I realized was I hadn’t synced up my backup before starting work. So that’s “oops!^^2″, I guess.

And yes, the folder I deleted was the one with all of the web site files in it.

Now, the good news. The blogs are all database driven and the database wasn’t touched. the backup existed, it just wasn’t 100% up to date. So it was a minor annoyance instead of a major catastrophe. I had pretty much everything restored and the sites back online in about 20 minutes, and I spent the evening looking at things and cleaning up. My blog lost some patches (which I put back in tonight) and as it turns out, the graphics for two of my postings went poof, and I’m going to have to recreate them.

So all things considered, it becomes a teachable moment instead of a gut-wrenching disaster. And I love teachable moments.

This is the classic reason why people use say things like “I don’t need to wear seat belts, I’m a careful driver” are fooling themselves. “careful driver” doesn’t save you from being rear-ended by the guy on a conference call with the sales team in Cleveland, “careful driver” doesn’t save you from the bee that flies in through your open window and stings you at a stop light, and “careful driver” doesn’t prevent that carefully timed sneeze just as you’re reaching for the brake pedal.

In other words — “careful” is no protection from Lord Murphy, and Murphy’s Law will win, sooner or later. In this case, I made two mistakes that cascaded. One was I skipped a step in my safety process for working on the hosted server (“step 1: back it up. Step 2: BE VERY CAREFUL WHEN YOU HAVE NO NET”), and then I made a simple mistake, which I realized just after doing it. Nobody’s perfect. I’ve spent more years dealing with servers as a root-capable geek than many of you have been programming — and I still make mistakes. Rarely, but I make them.

Which is why I love backups. And because I wasn’t pristine in doing them, this went from being a 20 minute pain in the neck to being about a six hour cleanup, and I have about two more hours rebuilding the missing graphics. That’s a hell of a lot better than “oh my god, it’s gone”, so even though the backup wasn’t perfect, it saved me major troubles.

How are your backups?

Well, mine are now pristine. Both copies (because I immediately made the old one read-only and set it aside, just in case i need a file in a week or a month. Don’t you?)

Ooops.

No related posts.