Search This Site
Silicon Valley veteran doing Technical Community Management. Photographer with a strong interest in birds, wildlife and nature who is exploring the Western states and working to tell you the stories of the special places I've found.
Author and Blogger. They are not the same thing. Sports occasionally spoken here, especially hockey. Veteran of Sun, Apple, Palm, HP and now Infoblox, plus some you've never heard of. They didn't kill me, they made me better.
Person with opinions, and not afraid to share them. Debate team in high school and college; bet that's a surprise.
Support This Site
If you found this page interesting, please consider clicking through this ad and buying something.
If you do, Amazon will pay me a small percentage of the price. You don't spend any more on the item, and the money helps pay for the site and the more people who do this the more time I'll be able to spend on the site improving it and adding content.
More to Read
- Some Thoughts on Lightroom Keywords
- How not to be a doofus with a camera
- Beyond 'Vacation Snaps'
- A teachable moment (or why I love birding, even when I make a fool of myself)
- Sherman, set the wayback machine to…
- An audience of one....
- Talking about 'Stuff'
- What I do for a living…
- 50 reasons Why I Haven’t Been Blogging
Want more? Try this list...
New on the Blog
- The Raffi Torres Hit
- Back from Yosemite
- 2013 playoffs, round 2
- Fuji X100s Review – Fallin’in Love All Over Again
- If you give them an easy out, they’ll take it.
- Another reason Don Cherry should retire (or be retired…)
- Yosemite Bird Photography Workshop openings
- 30 Days Of Sexism
- 2013 playoff predictions
- Calaveras Eagles Nest 2013
Rent Gear at Borrowlenses
Don't buy that gear before trying it out! Renting a lens you're considering buying is a great investment in saving yourself from buyer's remorse!
And if it's a piece or gear you aren't going to use constantly, renting it when you need it is a great way to save money, and I highly recommend Borrowlenses as a place to rent high quality, well-maintained gear.
Monthly Archives: March 2010
Welcome to issue #122 of I and the Bird. Today we’re going to visit with a number of birders and get a glimpse into the birds that they are keeping an eye on, from yellow-billed loons that make you twitch to sandhill cranes in your backyard to the birds that remind us that spring is here and that nesting season (and the joyous cacaphony of life that brings) is firing up.
Don’t forget to look out for the next issue of I and the Bird, coming to you April 15 thanks to the kind endeavours of the Idaho Birding Blog. If you didn’t contribute to this edition, you really should write up one of your birding adventures and share it with us through them!
But first, a quick editorial…. If it’s spring, it’s nesting season. If it’s nesting season, it’s a good time for all of us to remember the potential impact our hobby can have on the birds and that we should be very sensitive to approaching a nest. Birds can abandon a nest if they feel threatened and we can significantly hinder their ability to successfully hatch and raise their young if we aren’t very careful about how we interact with the birds, we can cause the nest to fail.
Keep your distance, and don’t push it just to “get that shot”. If the bird has acknowledged you, you’re too close. If you flush a bird, you’re way too damn close and you really should just get out of there and leave them alone. Their successful nesting is more important than that photograph, and we as birders and bird photographers need to take our stewardship of the birds seriously. If you aren’t absolutely sure you aren’t too close, move back and give them more room. (no nests were annoyed in the creation of these photos…)
Now, onward and forward to I and the Bird!
My entry for I and the Bird is The Bird(ing) and Me. I never intended to become a birder. It just happened. You don’t need to be a birder to look at birds. You aren’t a birder because you carry binoculars. Birding is — ultimately — all about birds, but has nothing to do with birds. Birding is the community that surrounds looking at birds, not the activity of watching them.
Neil Gilbert at OCBirding.com talks about one of the classic challenges of the birder: To Twitch…Or Not? Neil got his bird, a Yellow-Billed Loon. I normally don’t twitch — but I considered going after the same bird, but didn’t, so I’m still waiting for my time with that loon.
Corey at 10,000 birds goes birding at Jamaica Bay with a few of his friends — and takes us along to enjoy it with him.
Andy Gibb attalks about the recent changes to the IOC world list, and notes that his life list grew without his ever leaving his chair.
Speaking of lists, Nate at the Drinking Bird Blog does a nice piece in defense of the lister.
Melissa Cooper at Out Walking the Dog has been thinking about the impact and implications of feeding our wild (and urban “wild”) animals, and some of the issues it raises. Very interesting thinking and something to consider.
Dale Forbes, who happens to work for Swarovski in Austria, talks about the technical details of digiscoping and his digiscoping adventures in a trip to Africa and some of the birds and animals he saw there.
Rebecca of Rebecca in the Woods has a bit of an challenge. The good news is spring is back and the birds are nesting. The bad news is — the Carolina Wrens are nesting THERE?
Dave Alcock of DaveA’s Birding Blog brings us some gorgeous photos he took during an unexpected meeting with a Merlin.
Amber Coakley at Birder’s Lounge writes about the Great-Tailed Grackle in A Little Respect. I agree with her, they’re pretty birds that I enjoy watching — but bring earplugs.
Speaking of bringing earplugs, Puca at Anyone Seen My Focus brings us some nice images of one of my favorite birds, the Northern Mockingbird. And it must be spring, because the neighborhood mockingbirds have returned for another breeding season and kicked all of the Scrub Jays out of the area — that’s a turf way that’s been going on as long as we’ve lived here, and the Mockingbirds always seem to win. Our favorite mockingbird is back for another summer as well, the one we lovingly refer to as “Car Alarm”. Laurie says she’s heard one she swears is trying for “Anna’s Hummingbird”, but it’s not working. We’re worried it’s going to sprain it’s throat trying…
And it must be spring, when the birder’s thoughts turn to — the American Robin. Moe at Iowavoice.com brings us some nice images of this harbinger of spring for so many of us.
Tai Haku at Earth Wind and Water has a different sort of bird — some really amazing photos of a Snowy Owl, taken near him home — on an island in the Caribbean. When I saw “snowy”, for some reason I was thinking egret…
Jill Wussow at Count Your Chicken! We’re Taking Over! has some fun shots of what they think is a Glaucous-Winged x Herring gull hybrid who’s appetite is larger than it’s mouth as it attempts to swallow a starfish that doesn’t seem to want to be swallowed. Yes, it looks about as funny as you might expect. (and thank you, Jill, for admitting that I’m not the only person in the universe who looks at a flock of gulls and thinks to himself “I really should check it for rarities” but just can’t find huge amounts of enthusiasm over the idea….)
Joy at The Little House in the Not-So-Big Woods brings us this former-city-dwellers first vist by a barred own in Surrender Dorothy! There are Flying Monkeys Out There! I bet most of us have one of these WHAT THE HECK WHAT THAT? moments in our background…
Kay Baughman of the Arroyo Colorado Riverblog goes out birding and tells us to Go Fly a Kite. Or watch them…
Wren at Wrenaissance Reflections brings us a few pictures of some of her backyard birds, which just happen to be cranes. I’d kill for that view.
Larry Jordan at The Birder’s Report brings us close and personal with a pair of ospreys and their nest.
John Beetham at DC Birding Blog does a very nice review of the book Birds of Europe, 2nd Edition. I need to put that one on my wish list…
Don’t forget, when you bird, you can help the scientists studying birds to help us all understand them better. If you run into a banded bird, your sighting can help those studying those birds, so please consider reporting it. There’s a centralized banded bird reporting site available to make this easier, hosted by the USGS.
And finally, a free plug: when I visit Southern Cal to see my family, one of my favorite birding places is Bolsa Chica. they’ve just released the latest issue of their newsletter, the Tern Tide, which among other things talks about their new access bridge and urban coyotes. Well worth a read (and a visit!) (pointer via Amy at Wildbird on the Fly)
I never intended to become a birder. It just happened.
You don’t need to be a birder to look at birds. You aren’t a birder because you carry binoculars.
Birding is — ultimately — all about birds, but has nothing to do with birds. Birding is the community that surrounds looking at birds, not the activity of watching them.
It’s the people, and it’s the people involved in birding that made me a birder.
I felt it was time to acknowledge that, and say thanks.
My roots in watching birds go back a long way. I remember standing on a sand dune in Arcata, California, watching the brown pelicans dive fishing through a bait school and standing in awe and staring at these stunning birds in action. I have never tired of watching pelicans.
I’ve always been attuned to water — I find being near the water, especially near the ocean to be calming and regenerative. When I’m tired, when I’m stressed, getting out near the water helps recharge the batteries, release the tension. Living here in the Bay Area I’m blessed. The ocean is a short drive away, and special places like Fitzgerald Marine Preserve in Moss Beach and Pigeon Point and Pescadero exist away from the crowds so even on fairly busy days you can get to places that aren’t exceptionally crowded.
Better yet is to discover places along the Bay itself; Much has been developed, but we’re making progress at pulling some of it back and making it more accessible, less urban. Areas around Alviso, around Mountain View, around Palo Alto, around Redwood Shores.
You see your first black skimmer, and you think to yourself How the hell does a bird like that fly? What committee designed that?
Curiousity wins. You buy binoculars to see better. You buy a guide. You buy another. You figure out that the brash little bird out in the wetlands is a Song Sparrow, and you feel that little tingle of pride. You don’t realize it yet, but you’re hooked.
By the mid 1990′s, a camera, a guide and binocs were standard equipment. I’m primarily photographing shorebirds (and pelicans) and trying to make sense of sandpipers (yes, I hear you all out there quietly laughing. I was naive. They STILL don’t make sense). Laurie and I are in northern Oregon on a trip, at a breakfast place near astoria. We have our binocs on the table, which attracts another couple into discussion. They’re heading off to Fort Stevens after black rails. I’m off looking for sandpipers. “Shorebirds are easy. You should be photographing warblers!” she tells me.
But I like shorebirds! and it’s an excuse to get near the ocean. But she piques my interest.
Little did I know the heaven and hell that conversation was going to open up.
Over time you see people out in the places you out in. Some of those people see the binocs and come up and introduce themselves. They point things out. They answer questions. You start recognizing faces and names. They start recognizing you, and wave and point at things. You discover the mailing lists and find out there are people running around pointing out lots of places and birds and things going on you never realized.
And somewhere along the way, you’ve turned into a birder. For me, it was May, 2006. I was getting more and more serious about my photography and more and more serious about my birdwatching. I decided to get out and trip away from the familiar places — go on a birding trip — and see how much I liked it. I did some research, and did a long weekend in Morro Bay.
I ended up in Sweet Springs in Los Osos. Walking through the wetlands and into the trees. It was migration, and there were warblers all over the place — damned if I had a clue which ones. Well, I do now; Townsend’s and Yellow-Rumps primarily. Suddenly a flash of orange and yellow and a bird sits up in the shadows. Madly thumbing through the guide, I realized it was a Western Tanager, a gorgeous male.
At that moment I started Keeping Lists and stopped birdwatching and started birding. I’ve never looked back. My god, she was right. Warblers are fun to chase and photograph. So are the birds that skulk in bushes and flit in the canopy. They are also an endless source of wonderful frustration. I am not someone who tolerates adequacy in myself. If I choose to do something, I have to do it well. Birding is something you don’t get good at quickly, but you can always see the progress — and the next challenge. I have spent hours practicing with white-crowned sparrows how to see birds in bushes and get in position to get a real look at them. I’ve done the same with yellow-rumps in the canopies.
It’s only been the last six months or so where I really have started to feel like I’m good at this. I’ve been enthusiastic for a while — but enthusiasm isn’t a substitute for skill or knowledge. The hardest lesson to learn, and I think one of the most important, is when to back off and not force an ID into a situation where it’s not appropriate; when to just leave it at “a bird I wish I’d seen better”.
As a new birder, it’s all new, it’s all exotic, and your skills and knowledge are far outstripped by your naivete and enthusiasm. My work situation precluded doing many group outing where I could study from other birders, my nature tended to nudge me towards solo birding since I still saw the outings as a way to get away from everything and recharge and reflect. As such, the mailing lists and the online birding communities became my primary contact, my mentors, and as I made my inevitable mistakes they were my occasional audience for some rather enthusiastic pratfalls.
I didn’t become a birder because of the birds. I could have spent the rest of my life happily watching birds and taking photos of them without “birding”. I became a birder because of the people I found who were birders, and the community I found in birding. The last few years have been somewhat of an interesting time, in a chinese sort of way, and I found that birding because my retreat and sanctuary, and occasionally a thing I rallied my sanity around.
So I thought that, when the opportunity to host I and the Bird came up, that it was a great opportunity to talk not about birds, but about the birding community, and what it meant to me — and if you’ll indulge me a bit, to say thank you to a few people who deserve to be recognized.
If I mention nobody else, I have to mention Kris Olson. We lost Kris this year and that leaves a gaping hole in the local birding community. Kris was some of the glue that binded us together — she seemed everywhere. If a strange bird showed up somewhere, she’d appear to help chase it down and confirm it. She was a key driver in Sequoia audubon rebooting the county sighting lists. She was always there to answer a question or offer advice or suggest a place to visit, and with a smile and some gentle encouragement.
Then there are the senior birders, as I call them. Been doing this a long time, really know the region well, and are out there more or less every day surveying. They are the ones that know where to look and point out what’s there to be found, so the rest of us benefit by knowing where to look. In this area, people like Ron Thorn, Bill Bousman, Al Eisner, Bob Reiling, and Mike Mammoser not only help set the pace for most of the birders, I’ve found them all very willing to answer questions and offer advice (and occasionally kick my butt when my enthusiasm overreaches my skill), and if a report seems strange, they’re folks who’ll confirm it or suggest alternatives and help you get it right. I have been amazed on more than one occasion by Ron’s ability to ID a bird remotely via email better than I could seeing the bird in person.
You can’t pay these people back; they’d be insulted if you tried. So one of the things I’ve been thinking through is how I can contribute back into the community — to pay forward as a way of recognizing what I’ve gotten out of it.
So I’m going to close with a small call to action. Every one of us is an advocate and an ambassador to birding. Whenever we’re out there with binoculars around our necks or a scope on a tripod, our behavior reflects on the entire birdwatching community. If we act like jerks, birders will be seen as jerks — even if we merely act uninterested and aloof, we don’t make our community look good in the eyes of potential birders.
Every time we’re out on the trails or in the marshes there are opportunities for outreach. I’ve found there’s a lot of curious people out there, but many times they’re uncomfortable initiating discussion. When I notice that, I make a point of reaching out and trying to make them comfortable and see if I can answer questions. I’m not the worlds best birder — but I can show off a snowy egret or get a scope on a bluebird and show it off.
I’ve seriously considered having badges created that say “Yes, you can look!” and sew one onto my birding hat, so if I’m glued to the eyepiece, someone can see it and know it’s okay to ask. Anything to get them over that hump and get them engaged. I carry a “business” card with my personal info oriented towards my birding and my photos, and I hand it out a LOT and encourage people to email me if they have questions or want more info. A previous version of that card included a pointer to the signup page for the local birding list; the next version is going to point to a page on my web site that’ll include signup links for all of the regional birding lists as well as the regional Audubon event pages, to help encourage them to seek out a beginners walk or some other kind of activity.
It’s not about recruitment, it’s really just making a good impression and fostering curiousity. If we do that, then we’ll see people recruit themselves into the hobby, and the more people we get involved, the more good we can do for the birds, and isn’t that the point? All it costs is a smile and a hello and a willingness to spend a minute or two explaining or just letting someone look — and you get the benefit of rediscovering the joy of someone going OOH! when the see a great egret in breeding plumage show off. Me, I’m still someone who goos OOH! at that, may I never hit the point that becomes common to me…
And if you’re interested in a “Yes, please take a look!” badge, let me know. If there’s enough interest I’ll make them happen and get one to you…
So it’s now Monday, and about 24 hours after I posted my note on my new backups and disk scheme. And I wrote that after I was mostly done setting things up and the backups were set up and running and etc.
And here we are, and I’m still trying to get Time Machine to finish the damn first backup of my disks. the data is all in there (working set size 300+ gigs), but for reasons it won’t tell me, it hasn’t decided to actually finish. It was busy purring to itself when I left for work, and here it is, busily purring to itself still.
And while I appreciate why Apple designs its stuff to not be scary to non-geeks, when things go sideways, it can be amazingly frustrating, because I have no real status info or way to figure out what it’s actually doing (or trying to do), other than watching the flashing lights on the disk and trying to decipher the insides of the .inProgress package, And that’s the occasional challenge with Apple stuff: when it works, it just works. When it breaks, it sees no purpose in helping you fix the problem. So now I’m in a quandary, do I leave it alone and see if it’ll finish? do I throw out 300 gigs of backed up (and useless) data and let it start fresh?
I compromised. I stopped it and rebooted (which I needed to do for other reasons) and restarted it. and it spent 10 minutes in “indexing backup” and is now in “backing up”, but not actually doing so and not telling me how much it thinks needs backing up. But the disk is really busy…
On the other hand, Superduper finished pretty quickly and so I have good backups, I just don’t have my versioned backups, so I’m not worried. This is the suspenders, not the belt.
John Gruber of Daring Fireball also happened to weigh in on this. He’s right, DiskWarrior is definitely something you want to have. Highly recommended. And one of the things “on my list”. He’s also right about “more copies” — you’ll notice I try to keep 3 or four copies of my key data in various places at all times, and I’m paranoid enough that I prefer some that do NOT update in real time but wait a week or so, in case there’s unfound errors that creep in. but you can buy terabyte disks for not much — $100 or less now. There’s really no excuse not to replace your drive mechanisms (there I go again!) every year or so on your high usage drives and to keep spare copies of everything. call it SneakerRAID if you want, mirroring by making copies and stuffing them in drawers and things.
He’s also right about Dropbox. It’s a nice alternative to MobileMe (and faster, and cheaper, and has VERSIONING). I have been using it for other things, but now that he mentions it, I can consolidate some stuff on a DropBox rather nicely and simplify my life in other places and save a couple of bucks I’m spending on a cloud storage thingie here and an online service there. That rocks. Not dropping MobileME, though. I like having multiple redundant email accounts in case one of them goes spung. Funny how when MobileMe came out everyone was all over it for its problems (justifiably so), but I’ve seen more gmail outages, and geeks seem to give Google a break on those. I’m Google-centric because their stuff works better with webOS for me, but it’s nice having a place I can jump to if for some reason I need to, so I don’t mind keeping two environments around, just in case…
Merlin Mann also chimes in, and he’s right on, too. Notice how Superduper keeps coming up? Because it works. and you can trust it. Trust is probably THE KEY metric for backups, folks. Not features, trust. (via ).
You’ll notice there are a few of us really, hopelessly, anal about backups. That’s because we’ve all been burnt by problems that happen when you don’t, or when you think you are and they aren’t working, or when they’re happening but corrupted. And we’re really, hopelessly, anal about it because we know YOU FOLKS aren’t doing it.
Bless Apple for making backups simple with Time Machine. So many fewer excuses to not do them for people now. If I were to recommend one thing to Apple now, it’s this. Disks are really cheap. Build a mirrored RAID into every computer, so a drive failure no longer screws someone’s data. Make TIME MACHINE less necessary through data redundancy. Even your laptops. hell, especially your laptops. It’s the next step here, we should take it.
One of the things that became painfully obvious during my trip to Yosemite was that I was rapidly running out of hard disk. Being out on the road is not a good time to realize you need a bigger disk, s when I came back, I decided to fix things before it became a real problem. Here’s what my overall “bits on things” setup looked like:
Now, there’s one obvious problem there that I hadn’t thought about — the backup disk is smaller than the main disk. I knew about that, knew I needed to fix it, and forgot. Not a huge problem, but one of those details you need to keep an eye on or they’ll bite you at an inconvenience moment. Even though I had 3/4 of a terabyte for my backup disk, Time Machine was only storing backups for about 3 weeks, which means it was no longer large enough. It was time to update and grow and upgrade.
The biggest problem — the new Canon 7D creates much larger images. That’s good, but creates ripples. It also does video, which I’m starting to experiment with. By the time I convert the 7D RAw image to DNG and store it on disk, it grows to about 49 megabytes in size. Pile up a few hundred of those, and “Hell, disk is cheap” starts ringing a little hollow. To give an idea of the change going from the 30D to the 7D, on the 30D I use a 4Gb memory card and get 400+ images on it. On the 7D, I upgraded to 16Gb cards, and I get 500 images on one. Moderate upgrade in number of images, big upgrade in amount of disk taken. Also, since the 7D shoots 8 frames a second sustained where the 30D shot 4FPS with limited bursts, the opportunity to generate LOTS MORE images quickly exists. And it definitely happens, so at the end of the day, I have more, larger images to store. This is, as they say, a good problem to have.
The easy answer — upgrade the laptop to a bigger disk — won’t work here. The biggest laptop disks now available are 500 Gigabytes. Larger than my 320Gb, but not by that much. Upgrading delays the problem by a period of time, but it doesn’t solve it. I considered doing that, then decided to bite the bullet and shift into the “it no longer fits on the laptop” universe.
I mumbled about this on Twitter, and immediately got back the “install a NAS!” response. NAS (or Drobo, or RAID, or name your favorite disk packaging setup) isn’t a solution — it’s a technology. You don’t start by choosing a technology, you start by figuring out the solution and then choosing things that implement them well.
I’ve written about backups and my philosophy on how to do them before, check out this piece as well as this followup, as well as this piece where I talk about why I stopped using an online backup solution in favor of sneakernetting an offisite backup somewhere. I am, for the record, looking forward to when the price/performance and the network broadband make this worth doing again, but not right now…)
So for me it’s time to shift my data into a multi-disk environment. I live on a laptop, which gets carried around. If your data no longer all lives on the laptop disk, then when you need that data, you have a problem. It behooves you to then think about your data and how you use it, and figure out how to store your data across your disks so that you have access to what you want when you want it.
For my purposes, “data” can be defined as “everything on your disk”, but in practice, I see no reason to think about shifting apps out of the Application folder or similar “optimizations”. You might be able to free up a gig or two of space, but why? That’s not significant, and it can lead to potential complications later, especially if you start mucking in your Libraries, preferences, caches, etc. The savings aren’t significant — or worth the future hassles or possible compatibility issues. So for me, unless you’re a font geek with 50 gigs of fonts or something like that, just worry about the data folders: Documents, Pictures, Music, Movies. (in case it’s not painfully obvious: this info is Mac specific. General concepts work for Windows as well — the nutty details are your problem on that platform).
A few key goals
Here are a few key goals of all of this:
- Scales infinitely. Or close enough I don’t have to go through this again for a while
- My data is available when I need it, wherever I am
- Easy and intuitive. I don’t want something that’s difficult to do, or I won’t.
- Reliable and easy backups: if your backups are difficult, you won’t. Keep it simple. Make it reliable.
- Fast catastrophic recovery. I don’t want to spend days getting my data usable again
- Recover a file or a disk. Some backup schemes work best for a crashed disk, others for a lost file. you really need both.
- Backups on the road are even more important, not less. So make sure you can do them. And do.
Here’s what I ended up with. It’s not hugely different than before, but the changes create significant challenges to understand:
I took the bus-powered disk and upgraded it with a 500 gig drive. This means that instead of having 320Gb available, I now have 3/4 of a terabyte I can carry around and use without needing an electrical outlet. This is a significant detail: you really mess up the concept of a “laptop” if you have to plug it in to use it… Or worse, can’t because the data you need is inaccessible because you didn’t bring it.
Digression: for those of you about to tell me “just live in the cloud”, plesae don’t. The dataset we’re talking about is measured in gigabytes trending to terabytes, and it’s not practical. In reality I am using Google Docs and Dropbox more for some things, but for the set of things “the cloud” solves for me, they also live happily on my internal laptop disk. This is about figuring out now how to scale from having 1,000 photos in my portfolio and 10,000 in my collection to having 20,000 photos in my portfolio and 100,000 in my collection without everything collapsing in a heap, and those kind of data sets aren’t going to live online any time soon, nor do I particularly want them to.
So anyway, I now have three drives going. The internal laptop drive (320Gb) is where everything I need 100% of the time has to live. The external bus powered drive can store other files that I need access to on the road — but which I probably can live without for more casual usage. And my desktop drive (AC powered) stays at home and holds the data that I need easily accessible but don’t need to travel with.
I went through all of my data and figured out where it needed to live. There’s also an unlisted “fourth category”, which is data that lives offline, or on a disk that I maybe need access to once in a while but not keep plugged in, and I spent some time pulling all of that data off my disks and sticking it in a corner to archive into a drawer. (one could also say there’s a fifth category, the “why the hell am I hanging on to THIS?” category of things that ended up in the trash. Things like the Parallel’s virtual image of Ubuntu I haven’t booted since I installed it five months ago, which deleting freed up multiple gigabytes. And why did I feel the need for an Ubuntu disto in Parallels on MacOS, which is just a different flavor of the same thing? I don’t remember, but it seemed a good idea at the time…)
I can hear some of you groaning at the thought of sorting through all of your data. I sympathize. If you don’t want to commit the time to that, I understand — but — putting some time and energy into it now helps you understand what you have and how to organize it. It also means that moving forward you’l have a good sense on where stuff belongs, meaning you’ll spend less time thinking it through and organizing on the fly. And if you do it now, you probably won’t need to do it again for a few years. It’s little more than virtually filing everything in your office, and it never hurts to do that every so often.
It shouldn’t be assumed that you need to turn “Save File” into a “Getting Things Done” adventure — I’m definitely not interested in being that anal about all of this, but it is important to understand how you want to manage your data well enough to know if it’ll do what you need it to do and how well it scales. Scaling was the big issue for me. If I’m seriously having to worry about data in terms of terabytes, I’d just as soon not have to architect this all out again in six months. Once it’s settled down, it’s back to the “that pile on the desk is in the way, let’s put it in the files” mode again…
So here’s how I finally settled on filing things. My internal laptop disk:
And here’s what my secondary disk looks like. Note that it only has Music and Pictures folders.
The Music folder is where I’m storing the video files in my iTunes library. The audio (aka “music”) lives on the main laptop disk. As my creation of video grows, I’ll add a “Movies” folder and split it up the way I do photos, but right now, there’s not much there.
And finally, my third disk, the one that stays at home:
The blue highlighted folders are folders on that disk that I exclude from the Time Machine backup:
which is an option more people should think about if they use Time Machine (or other backups) — some stuff you can live without if you need to, so why back it up? All it does is make it harder to do backups reliably. I flag them with color labels so I don’t forget which ones were excluded — I did that once and had to restore a disk, and spent half a day freaking over “missing data” until I remembered I’d excluded that data from the backups. Oops. It goes without saying, of course, that you should only exclude stuff you really don’t need back if there’s a failure, don’t exclude it because it’s large…
A big part of how this works (or won’t) is splitting up the photo library. In general, I split up my photos into four big piles:
- flickr or better: images I liked enough to post to my account
- 2nd tier: photos which are technically fine, but which aren’t something I think should be posted on flickr. Most of these are effectively duplicates of ones that go on Flickr (think “eight frames per second burst rate”); you want them around in case you want to use them; you stick them somewhere out of the way because you have no plans to actually do so. In theory, these photos are all good enough to publish, except I have some other photo I think is better — but yo never know when you might want some specific expression or a left profile instead of a right profile, and so they’re here if you need it.
- archive and forget: photos that are clearly not as good as the candidates I’d publish, but not bad enough to throw away. To be honest, as I’m getting more comfortable about my abilities as a photographer, I’m doing less keeping photos around that “someday I might try to fix this”. Instead, I ding them and throw them out. These are flagged to be taken offline and stored, and I fully believe I’ll never look at them again and some day throw them out. More and more, I’m comfortable with my choices and simply throwing them out and saving a step…
- dings: And finally, the dings. As I do edits, the ones that are clearly flawed get thrown out and deleted. There are people who tell you to keep everything. I’m not one of those people. Disk is cheap, but it’s not free. Maybe some day those images will be usable (or fixable in photoshop, or whatever), but the reality is I have thousands of BETTER images I could spend that time on, so why bother? So count me in the camp of tossing the crap, especially when it quickly starts turning into gigabytes and terabytes of crap. Why make it harder to find the good images by having to wade through crap, or worse, create a filing system for offline images to keep around stuff you know in your heart you’ll never use? Let it go. Just because you CAN keep everything doesn’t mean it’s a good idea. It’s not.
This setup looks like it’ll scale for a good long time; I can, if I need to, move some flickr or better onto the 2nd disk and prioritize the internal drive to active projects; 2nd tier data easily moves to the “live at home” disk when I need to. I can subset my itunes library the same way if I want to, and the rest of my data isn’t going to grow faster than disk technology seems to be progressing, and as long as I keep my folder structure sane, I can tell at a glance what’s going on, both within the Finder and Lightroom. I can use Lightroom and Spotlight searching to find things if I need to, but with a bit of care the naming structure will let me browse into it quickly as well. It looks pretty solid.
I’ve spent the last couple of days migrating the data to this new setup and I’m now happy with it, at least for now. As I’ve settled in, I’ve made some changes – originally all three disks had Documents folders, I finally realized that either a document lived on the internal laptop or it lived on the “stay at home” drive; no need for a middle phase, it just complicated things. You’ll notice there are folders on the travelling disks to act as placeholders for the stay at home disk. This makes staging stuff to sweep over there easy, so I can stuff files places on the road and then go home and move them off of the travel disks. It may seem unnecessary or trivial, but I’ve found lots of peopple don’t think about that kind of detail, and when I explain it, they love the idea — it lets me make a filing decision at the time I’m using the data, and merely shove it into the file when I get home and not have to “remember” what needs to be filed days later. Make those decisions while you’re using something and then forget it — it’s a great hint for simplifying things.
And once my backups finally sync up and my data is fully redundant again, I’ll be happier. Currently, I have my superduper backups in place, I’m letting Time Machine sync up now. It can be butt slow at times…
Some technical details on implementing this
The drive I bought for the bus powered disk was the Hitachi Traveler 500G. I’ve been using Hitachi drives for my laptop drives for a while and find them pretty reliable. That doesn’t mean others aren’t, it means thse have worked well for me, so I continue to use them. The bus-powered enclosure I use is the Mercury On-The-Go Pro from Other World Computing. I’ve bought RAM and disk from OWC for years and have been very happy with their price, quality and service. I’ve used that enclosure for a long time with never a failure. Their stuff is well-engineered and solid and I feel it’s well priced, and I haven’t been in the mood to explore other vendors because this one works for me.
For my external drives, I use the OWC Mercury Elite-Pro housing. it’s solid, it’s build like a rock, it works reliably. As part of this rework, I’ve retired the last of my IDE systems and I only buy/use drives that have SATA interfaces.
Digression: Every so often, it makes sense to see how technology is moving and migrate away from stuff that’s aging and heading towards end of life — if you refresh your data onto modern storage, you won’t go looking for it some day and find out you no longer have a way to access it. I’m a big fan of refreshing all of my offline storage every couple of years so the chances of having a stored drive failed is minimized. I’m also a fan of keeping two copies of all offline data, preferably one offsite — just in case. Since I’m also a fan of refreshing my active drives on a regular basis (because the best way to never need your backups is to never run your disks until they die!), a nice way to do this is to replace your active drives every 18 months or so, then use the retired drives and copy all of your archived data onto them, and then take the oldest drives and stick them in your files somewhere.
Digression on the digression: I see no reason to ever give a used drive to someone else, either by selling, giving, or donating. I pull the drives out of computers and housings and file them with my tax papers and other files. Once in a while, I pull the really old mechanisms and “retire” them with a big hammer. That way, there’s absolutely no way someone can recover files off of a drive they bought in Goodwill and end up with your data — because it never leaves your hands. If you trust seven-way zeroing and are willing to spend the time to do so, bless you. I jut don’t think a used disk drive is worth the time and hassle to recycle for re-use…
The drive I’m using as my backup drive now is the 2Tb Western Digital “greenpower” Caviar Green with 64 Mb cache. There are cheaper drives out there, but this one has good reviews and is built for server service. In all honesty, there’s nothing quite so painful as finding out your backup drive has failed, especially if you find out while trying to restore something. I don’t want to overpay for this stuff, but cheaping out bites you down the road.
My backup drive is living in a NewerTech Voyager Hard Drive Dock, which allows you to insert and eject SATA drives easily. This means if I want to I can easily pull this mechanism and replace it with another if I need to “do something” with another disk. I’m just starting to use it so I don’t have reliability data on it, but so far, I like it. It’s solid and well-built at first use. I plan on using it for managing my offline archives as well, saving me paying for multiple enclosures down the road.
Geeky details on backups
The 2Tb disk is split into two partitions, one 500Gb and one 1.5Gb. I use two backup technologies, SuperDuper! and Time Machine. I love Superduper for system backups because it makes bootable clones. That makes catastrophic recovery a lot simpler: take your backup drive, plug it into a Mac, and boot from it (then make a backup of it before something bad happens!). Superduper runs nightly and refreshes copies of my two travel disks, which is why the 2Tb is split into two partitions. The 500Gb syncs up the 500Gb external disk, and the 1.5Tb is the clone of the internal boot disk and also is where my Time Machine backups live.
Superduper doesn’t do versioning or archival over time, it makes a snapshot of now. For the “I need that file I threw out two weeks ago” problem, I use Time Machine. It backs up all three disks (minus the exclusions I mention above) to the 1.5 Terabyte partition of the backup disk. Time Machine is useful for casual backups (it’s better than nothing and pretty good for get-single-file recoveries) but I don’t like it for complete disk recovery and after working with a Time Capsule for a while, I really don’t like Time Machine over a network. If anyone really cares why, that’s a whole different blog posting.
The good news is that SuperDuper and Time Machine co-exist nicely on one disk (thank you, Dave!) so I can do both easily, so I’m set up to clone my two key disks onto the backup disk, and then do a time machine backup onto it for incremental backups as well. If my boot disk crashes, recovery is (almost) as simple as booting the backup disk. Wonderful, since crashes almost always happen on deadline…
What this doesn’t cover yet…
There are a few details this new setup doesn’t cover yet. None of them are time critical, but all of them need to be considered and solved, and it’s important you know how to solve them before you implment (lest they blow up your work when you go “oh, damn, didn’t think of that” later). Fortunately, they all are solvable…
- The new setup doesn’t include “on the road” backups. Since I no longer can carry a bus-power drive big enough to back up my systems, the answer is to carry a bigger, plug-in drive. I’m not worried about Time Machine backups on the road, so the easiest solution is a 1Gb external drive in one of my Elite-Pro housings. Even better, that’s cheap, and if I set it up, gives me an easy “spare backup” setup, because I love having a set of backups I only update every week or so, just in case something corrupts that I don’t recognize right away. So that’s probably what I’ll do. The other option would be to carry the 2Tb backup disk with me in the Elite-Pro housing, which also works, but which limits the number of redundant copies I end up having. I don’t like carrying my backup on the road if I can help it, I’d rather carry a “road” backup and leave the main backup at home. But both are options.
- The new setup doesn’t make explicit the off-site backup storage. What I’m doing in the short term is taking my old backup disk offsite. In 4-6 weeks, I’ll buy a 2nd 2Tb disk, plug it into a dock, build it the same as my new backup disk, and run backups onto it, and then swap between the two (the other going offsite) every 4-6 weeks. That’ll fix this for a good while at reasonable cost.
- The setup for moving files onto offline disks (aka “in the drawer”) isn’t spelled out, but is pretty simple: buy a pair of 500Gb SATA drives, plug them into the dock, copy the files to each, carry one offsite. Iterate until full, and then either start another set or decide some of the files can be deleted (or both). Every couple of years, take all of your offline disks, copy them to new (fewer, bigger) disks, and store them again.
But what about “install a NAS?”
I have to admit I’m not a huge fan of NAS in my environment, but I also realize that over time, the amount of data I’m storing on my “stay at home” disk is going to grow without bounds. My plan at this time is to convert that into a Drobo at some point, but not until I need to, so I’ll hold that off until later this year. I realize that at some point the percentage of data I can keep local to the laptop, even with 1 terabyte (500gig internal + 500gig bus powered) is finite, but I’m only using about 275Gb on those two combined right now, so I have some time before I have to worry about that…
Things like Drobo and a NAS add some capabilities, but they also add complexity, cost and new ways for interesting failures, which always seem to happen on deadline when you least can afford the issues. A NAS works best if you’re sharing data among multiple machines, since I’m not, it adds more complexity than it solves problems. Drobo is different being locally hooked up (and there’s a NAS enabler you can buy for it), but adds its own set of complexities and administration — so as long as (a) a single disk works and (b) I can back it up reliably, I’ll stick with a good single disk. Once you start getting into multiple disk environments and/or your backups start being tougher to keep reliable, the addition of mirrored RAID and some of the other features of NAS or Drobo become good to have, but again, I’m not at that point yet.
Finally — speaking of Terabytes
I’ve been around long enough that the thought of buying disk in terabyte sizes amuses me. My first hard drive was ten megabytes — MEGAbytes, not GIGAbytes — and I remember a time when a terabyte would probably store all of the data at Apple, and perhaps all of the data in the state of California. Today, I’m using it for backups of my personal data set. That amount of scaling in the last 30 years or so amazes me when I step back and consider it. But then, my phone has a lot more processing power and memory and disk than my first Mac did. I think my KEYBOARD has a more powerful CPU than my first home computer did….