Back from vacation…

Back from my vacation and Yosemite trip. It was, in a word, awesome (except the food at the Yosemite Lodge food court is still very expensive and kinda sucks, and you don’t have many options, but that’s nothing new….)

As usually happens, I came back to a huge block of stuff that needed my attention. I’ve gotten my inbox at work down from about 600 emails to around 250, and if you haven’t heard back from me at work, I’m sorry — working on it. At home, it’s down to about 30, and ditto.

The time off gave me a chance to completely get away and shut down and unwind, and do a lot of thinking and planning, so hopefully soon there will be new and interesting things happening. it also really recharge the personal batteries, which I desperately needed, so I feel like doing something more than site and stare at the screen in the evenings. I’ve spent the last few days reworking my disks and backup strategies because it was clear I was going to run out of disk space sooner rather than later (and you can read more about that here); and I’m spending a bunch of time in Lightroom working through photos and cleaning up my library.

I’ll talk more about the trip as I have time to write it up — but until then, I’ve put some of the photos up on my new Smugmug site, which is where I’m going to be building my professional portfolio (which I’ll write more about as I have time to write it up…).

I did what I honestly think is my best photographer EVER. I still have a bunch of photos to process and upload, but the best of the ones I’ve done are now online. Here’s a sample:

and I’ve also set up the best images in a slideshow:

My complete set is (or will be, when I finish catching up) over on Flickr, but I haven’t had time to put them into sets or organize them yet..

Enjoy!

Tags: About Chuq, Photography

What to do when you realize you’re running out of disk…

One of the things that became painfully obvious during my trip to Yosemite was that I was rapidly running out of hard disk.  Being out on the road is not a good time to realize you need  a bigger disk, s when I came back, I decided to fix things before it became a real problem. Here’s what my overall “bits on things” setup looked like:

Now, there’s one obvious problem there that I hadn’t thought about — the backup disk is smaller than the main disk. I knew about that, knew I needed to fix it, and forgot. Not a huge problem, but one of those details you need to keep an eye on or they’ll bite you at an inconvenience moment. Even though I had 3/4 of a terabyte for my backup disk, Time Machine was only storing backups for about 3 weeks, which means it was no longer large enough. It was time to update and grow and upgrade.

The biggest problem — the new Canon 7D creates much larger images. That’s good, but creates ripples. It also does video, which I’m starting to experiment with. By the time I convert the 7D RAw image to DNG and store it on disk, it grows to about 49 megabytes in size. Pile up a few hundred of those, and “Hell, disk is cheap” starts ringing a little hollow. To give an idea of the change going from the 30D to the 7D, on the 30D I use a 4Gb memory card and get 400+ images on it. On the 7D, I upgraded to 16Gb cards, and I get 500 images on one. Moderate upgrade in number of images, big upgrade in amount of disk taken. Also, since the 7D shoots 8 frames a second sustained where the 30D shot 4FPS with limited bursts, the opportunity to generate LOTS MORE images quickly exists. And it definitely happens, so at the end of the day, I have more, larger images to store. This is, as they say, a good problem to have.

The easy answer — upgrade the laptop to a bigger disk — won’t work here. The biggest laptop disks now available are 500 Gigabytes. Larger than my 320Gb, but not by that much. Upgrading delays the problem by a period of time, but it doesn’t solve it. I considered doing that, then decided to bite the bullet and shift into the “it no longer fits on the laptop” universe.

I mumbled about this on Twitter, and immediately got back the “install a NAS!” response. NAS (or Drobo, or RAID, or name your favorite disk packaging setup) isn’t a solution — it’s a technology. You don’t start by choosing a technology, you start by figuring out the solution and then choosing things that implement them well.

I’ve written about backups and my philosophy on how to do them before, check out this piece as well as this followup, as well as this piece where I talk about why I stopped using an online backup solution in favor of sneakernetting an offisite backup somewhere. I am, for the record, looking forward to when the price/performance and the network broadband make this worth doing again, but not right now…)

So for me it’s time to shift my data into a multi-disk environment. I live on a laptop, which gets carried around. If your data no longer all lives on the laptop disk, then when you need that data, you have a problem. It behooves you to then think about your data and how you use it, and figure out how to store your data across your disks so that you have access to what you want when you want it.

For my purposes, “data” can be defined as “everything on your disk”, but in practice, I see no reason to think about shifting apps out of the Application folder or similar “optimizations”. You might be able to free up a gig or two of space, but why? That’s not significant, and it can lead to potential complications later, especially if you start mucking in your Libraries, preferences, caches, etc. The savings aren’t significant — or worth the future hassles or possible compatibility issues. So for me, unless you’re a font geek with 50 gigs of fonts or something like that, just worry about the data folders: Documents, Pictures, Music, Movies. (in case it’s not painfully obvious: this info is Mac specific. General concepts work for Windows as well — the nutty details are your problem on that platform).

A few key goals

Here are a few key goals of all of this:

• Scales infinitely. Or close enough I don’t have to go through this again for a while
• My data is available when I need it, wherever I am
• Easy and intuitive. I don’t want something that’s difficult to do, or I won’t.
• Reliable and easy backups: if your backups are difficult, you won’t. Keep it simple. Make it reliable.
• Fast catastrophic recovery. I don’t want to spend days getting my data usable again
• Recover a file or a disk. Some backup schemes work best for a crashed disk, others for a lost file. you really need both.
• Backups on the road are even more important, not less. So make sure you can do them. And do.

Here’s what I ended up with. It’s not hugely different than before, but the changes create significant challenges to understand:

I took the bus-powered disk and upgraded it with a 500 gig drive. This means that instead of having 320Gb available, I now have 3/4 of a terabyte I can carry around and use without needing an electrical outlet. This is a significant detail: you really mess up the concept of a “laptop” if you have to plug it in to use it… Or worse, can’t because the data you need is inaccessible because you didn’t bring it.

Digression: for those of you about to tell me “just live in the cloud”, plesae don’t. The dataset we’re talking about is measured in gigabytes trending to terabytes, and it’s not practical. In reality I am using Google Docs and Dropbox more for some things, but for the set of things “the cloud” solves for me, they also live happily on my internal laptop disk. This is about figuring out now how to scale from having 1,000 photos in my portfolio and 10,000 in my collection to having 20,000 photos in my portfolio and 100,000 in my collection without everything collapsing in a heap, and those kind of data sets aren’t going to live online any time soon, nor do I particularly want them to.

So anyway, I now have three drives going. The internal laptop drive (320Gb) is where everything I need 100% of the time has to live. The external bus powered drive can store other files that I need access to on the road — but which I probably can live without for more casual usage. And my desktop drive (AC powered) stays at home and holds the data that I need easily accessible but don’t need to travel with.

I went through all of my data and figured out where it needed to live. There’s also an unlisted “fourth category”, which is data that lives offline, or on a disk that I maybe need access to once in a while but not keep plugged in, and I spent some time pulling all of that data off my disks and sticking it in a corner to archive into a drawer. (one could also say there’s a fifth category, the “why the hell am I hanging on to THIS?” category of things that ended up in the trash. Things like the Parallel’s virtual image of Ubuntu I haven’t booted since I installed it five months ago, which deleting freed up multiple gigabytes. And why did I feel the need for an Ubuntu disto in Parallels on MacOS, which is just a different flavor of the same thing? I don’t remember, but it seemed a good idea at the time…)

I can hear some of you groaning at the thought of sorting through all of your data. I sympathize. If you don’t want to commit the time to that, I understand — but — putting some time and energy into it now helps you understand what you have and how to organize it. It also means that moving forward you’l have a good sense on where stuff belongs, meaning you’ll spend less time thinking it through and organizing on the fly. And if you do it now, you probably won’t need to do it again for a few years. It’s little more than virtually filing everything in your office, and it never hurts to do that every so often.

It shouldn’t be assumed that you need to turn “Save File” into a “Getting Things Done” adventure — I’m definitely not interested in being that anal about all of this, but it is important to understand how you want to manage your data well enough to know if it’ll do what you need it to do and how well it scales. Scaling was the big issue for me. If I’m seriously having to worry about data in terms of terabytes, I’d just as soon not have to architect this all out again in six months. Once it’s settled down, it’s back to the “that pile on the desk is in the way, let’s put it in the files” mode again…

So here’s how I finally settled on filing things. My internal laptop disk:

And here’s what my secondary disk looks like. Note that it only has Music and Pictures folders.

The Music folder is where I’m storing the video files in my iTunes library. The audio (aka “music”) lives on the main laptop disk.  As my creation of video grows, I’ll add a “Movies” folder and split it up the way I do photos, but right now, there’s not much there.

And finally, my third disk, the one that stays at home:

The blue highlighted folders are folders on that disk that I exclude from the Time Machine backup:

which is an option more people should think about if they use Time Machine (or other backups) — some stuff you can live without if you need to, so why back it up? All it does is make it harder to do backups reliably. I flag them with color labels so I don’t forget which ones were excluded — I did that once and had to restore a disk, and spent half a day freaking over “missing data” until I remembered I’d excluded that data from the backups. Oops. It goes without saying, of course, that you should only exclude stuff you really don’t need back if there’s a failure, don’t exclude it because it’s large…

A big part of how this works (or won’t) is splitting up the photo library. In general, I split up my photos into four big piles:

• flickr or better:  images I liked enough to post to my Flickr account (and the subset of those I think are good enough for my portfolio, which I’m starting to build on Smugmug)
• 2nd tier: photos which are technically fine, but which aren’t something I think should be posted on flickr. Most of these are effectively duplicates of ones that go on Flickr (think “eight frames per second burst rate”); you want them around in case you want to use them; you stick them somewhere out of the way because you have no plans to actually do so. In theory, these photos are all good enough to publish, except I have some other photo I think is better — but yo never know when you might want some specific expression or a left profile instead of a right profile, and so they’re here if you need it.
• archive and forget: photos that are clearly not as good as the candidates I’d publish, but not bad enough to throw away. To be honest, as I’m getting more comfortable about my abilities as a photographer, I’m doing less keeping photos around that “someday I might try to fix this”. Instead, I ding them and throw them out. These are flagged to be taken offline and stored, and I fully believe I’ll never look at them again and some day throw them out. More and more, I’m comfortable with my choices and simply throwing them out and saving a step…
• dings: And finally, the dings. As I do edits, the ones that are clearly flawed get thrown out and deleted. There are people who tell you to keep everything. I’m not one of those people. Disk is cheap, but it’s not free. Maybe some day those images will be usable (or fixable in photoshop, or whatever), but the reality is I have thousands of BETTER images I could spend that time on, so why bother? So count me in the camp of tossing the crap, especially when it quickly starts turning into gigabytes and terabytes of crap. Why make it harder to find the good images by having to wade through crap, or worse, create a filing system for offline images to keep around stuff you know in your heart you’ll never use? Let it go. Just because you CAN keep everything doesn’t mean it’s a good idea. It’s not.

This setup looks like it’ll scale for a good long time; I can, if I need to, move some flickr or better onto the 2nd disk and prioritize the internal drive to active projects; 2nd tier data easily moves to the “live at home” disk when I need to. I can subset my itunes library the same way if I want to, and the rest of my data isn’t going to grow faster than disk technology seems to be progressing, and as long as I keep my folder structure sane, I can tell at a glance what’s going on, both within the Finder and Lightroom. I can use Lightroom and Spotlight searching to find things if I need to, but with a bit of care the naming structure will let me browse into it quickly as well. It looks pretty solid.

I’ve spent the last couple of days migrating the data to this new setup and I’m now happy with it, at least for now. As I’ve settled in, I’ve made some changes – originally all three disks had Documents folders, I finally realized that either a document lived on the internal laptop or it lived on the “stay at home” drive; no need for a middle phase, it just complicated things. You’ll notice there are folders on the travelling disks to act as placeholders for the stay at home disk. This makes staging stuff to sweep over there easy, so I can stuff files places on the road and then go home and move them off of the travel disks. It may seem unnecessary or trivial, but I’ve found lots of peopple don’t think about that kind of detail, and when I explain it, they love the idea — it lets me make a filing decision at the time I’m using the data, and merely shove it into the file when I get home and not have to “remember” what needs to be filed days later. Make those decisions while you’re using something and then forget it — it’s a great hint for simplifying things.

And once my backups finally sync up and my data is fully redundant again, I’ll be happier. Currently, I have my superduper backups in place, I’m letting Time Machine sync up now. It can be butt slow at times…

Some technical details on implementing this

The drive I bought for the bus powered disk was the Hitachi Traveler 500G. I’ve been using Hitachi drives for my laptop drives for a while and find them pretty reliable. That doesn’t mean others aren’t, it means thse have worked well for me, so I continue to use them. The bus-powered enclosure I use is the Mercury On-The-Go Pro from Other World Computing. I’ve bought RAM and disk from OWC for years and have been very happy with their price, quality and service. I’ve used that enclosure for a long time with never a failure. Their stuff is well-engineered and solid and I feel it’s well priced, and I haven’t been in the mood to explore other vendors because this one works for me.

For my external drives, I use the OWC Mercury Elite-Pro housing. it’s solid, it’s build like a rock, it works reliably. As part of this rework, I’ve retired the last of my IDE systems and I only buy/use drives that have SATA interfaces.

Digression: Every so often, it makes sense to see how technology is moving and migrate away from stuff that’s aging and heading towards end of life — if you refresh your data onto modern storage, you won’t go looking for it some day and find out you no longer have a way to access it. I’m a big fan of refreshing all of my offline storage every couple of years so the chances of having a stored drive failed is minimized. I’m also a fan of keeping two copies of all offline data, preferably one offsite — just in case. Since I’m also a fan of refreshing my active drives on a regular basis (because the best way to never need your backups is to never run your disks until they die!), a nice way to do this is to replace your active drives every 18 months or so, then use the retired drives and copy all of your archived data onto them, and then take the oldest drives and stick them in your files somewhere.

Digression on the digression: I see no reason to ever give a used drive to someone else, either by selling, giving, or donating. I pull the drives out of computers and housings and file them with my tax papers and other files. Once in a while, I pull the really old mechanisms and “retire” them with a big hammer. That way, there’s absolutely no way someone can recover files off of a drive they bought in Goodwill and end up with your data — because it never leaves your hands. If you trust seven-way zeroing and are willing to spend the time to do so, bless you. I jut don’t think a used disk drive is worth the time and hassle to recycle for re-use…

The drive I’m using as my backup drive now is the 2Tb Western Digital “greenpower” Caviar Green with 64 Mb cache. There are cheaper drives out there, but this one has good reviews and is built for server service. In all honesty, there’s nothing quite so painful as finding out your backup drive has failed, especially if you find out while trying to restore something. I don’t want to overpay for this stuff, but cheaping out bites you down the road.

My backup drive is living in a NewerTech Voyager Hard Drive Dock, which allows you to insert and eject SATA drives easily. This means if I want to I can easily pull this mechanism and replace it with another if I need to “do something” with another disk. I’m just starting to use it so I don’t have reliability data on it, but so far, I like it. It’s solid and well-built at first use. I plan on using it for managing my offline archives as well, saving me paying for multiple enclosures down the road.

Geeky details on backups

The 2Tb disk is split into two partitions, one 500Gb and one 1.5Gb. I use two backup technologies, SuperDuper! and Time Machine. I love Superduper for system backups because it makes bootable clones. That makes catastrophic recovery a lot simpler: take your backup drive, plug it into a Mac, and boot from it (then make a backup of it before something bad happens!). Superduper runs nightly and refreshes copies of my two travel disks, which is why the 2Tb is split into two partitions. The 500Gb syncs up the 500Gb external disk, and the 1.5Tb is the clone of the internal boot disk and also is where my Time Machine backups live.

Superduper doesn’t do versioning or archival over time, it makes a snapshot of now. For the “I need that file I threw out two weeks ago” problem, I use Time Machine. It backs up all three disks (minus the exclusions I mention above) to the 1.5 Terabyte partition of the backup disk. Time Machine is useful for casual backups (it’s better than nothing and pretty good for get-single-file recoveries) but I don’t like it for complete disk recovery and after working with a Time Capsule for a while, I really don’t like Time Machine over a network. If anyone really cares why, that’s a whole different blog posting.

The good news is that SuperDuper and Time Machine co-exist nicely on one disk (thank you, Dave!) so I can do both easily, so I’m set up to clone my two key disks onto the backup disk, and then do a time machine backup onto it for incremental backups as well. If my boot disk crashes, recovery is (almost) as simple as booting the backup disk. Wonderful, since crashes almost always happen on deadline…

What this doesn’t cover yet…

There are a few details this new setup doesn’t cover yet. None of them are time critical, but all of them need to be considered and solved, and it’s important you know how to solve them before you implment (lest they blow up your work when you go “oh, damn, didn’t think of that” later). Fortunately, they all are solvable…

• The new setup doesn’t include “on the road” backups. Since I no longer can carry a bus-power drive big enough to back up my systems, the answer is to carry a bigger, plug-in drive. I’m not worried about Time Machine backups on the road, so the easiest solution is a 1Gb external drive in one of my Elite-Pro housings. Even better, that’s cheap, and if I set it up, gives me an easy “spare backup” setup, because I love having a set of backups I only update every week or so, just in case something corrupts that I don’t recognize right away. So that’s probably what I’ll do. The other option would be to carry the 2Tb backup disk with me in the Elite-Pro housing, which also works, but which limits the number of redundant copies I end up having. I don’t like carrying my backup on the road if I can help it, I’d rather carry a “road” backup and leave the main backup at home. But both are options.
• The new setup doesn’t make explicit the off-site backup storage. What I’m doing in the short term is taking my old backup disk offsite. In 4-6 weeks, I’ll buy a 2nd 2Tb disk, plug it into a dock, build it the same as my new backup disk, and run backups onto it, and then swap between the two (the other going offsite) every 4-6 weeks. That’ll fix this for a good while at reasonable cost.
• The setup for moving files onto offline disks (aka “in the drawer”) isn’t spelled out, but is pretty simple: buy a pair of 500Gb SATA drives, plug them into the dock, copy the files to each, carry one offsite. Iterate until full, and then either start another set or decide some of the files can be deleted (or both). Every couple of years, take all of your offline disks, copy them to new (fewer, bigger) disks, and store them again.

But what about “install a NAS?”

I have to admit I’m not a huge fan of NAS in my environment, but I also realize that over time, the amount of data I’m storing on my “stay at home” disk is going to grow without bounds. My plan at this time is to convert that into a Drobo at some point, but not until I need to, so I’ll hold that off until later this year. I realize that at some point the percentage of data I can keep local to the laptop, even with 1 terabyte (500gig internal + 500gig bus powered) is finite, but I’m only using about 275Gb on those two combined right now, so I have some time before I have to worry about that…

Things like Drobo and a NAS add some capabilities, but they also add complexity, cost and new ways for interesting failures, which always seem to happen on deadline when you least can afford the issues. A NAS works best if you’re sharing data among multiple machines, since I’m not, it adds more complexity than it solves problems. Drobo is different being locally hooked up (and there’s a NAS enabler you can buy for it), but adds its own set of complexities and administration — so as long as (a) a single disk works and (b) I can back it up reliably, I’ll stick with a good single disk. Once you start getting into multiple disk environments and/or your backups start being tougher to keep reliable, the addition of mirrored RAID and some of the other features of NAS or Drobo become good to have, but again, I’m not at that point yet.

Finally — speaking of Terabytes

I’ve been around long enough that the thought of buying disk in terabyte sizes amuses me. My first hard drive was ten megabytes — MEGAbytes, not GIGAbytes — and I remember a time when a terabyte would probably store all of the data at Apple, and perhaps all of the data in the state of California. Today, I’m using it for backups of my personal data set. That amount of scaling in the last 30 years or so amazes me when I step back and consider it. But then, my phone has a lot more processing power and memory and disk than my first Mac did. I think my KEYBOARD has a more powerful CPU than my first home computer did….

Tags: About Chuq, Photography, The Online Life

Supporting Doctors without Borders through Images without Borders

A project I’ve been working on in the background for a while has finally all come together and I’m thrilled to be able to talk about it. After the Haiti earthquake I donated some funds to support the rescue efforts there but I realized there was going to be a long-term need there and started looking at the right ways to get involved. One of the organizations that I’ve considered donating to for a couple of years is Doctors without Borders, which fits the kind of organization I look to put my donation money into (low bureaucracy overhead, low marketing expenses, high percentage of revenues “on the ground” and not in the home office, etc..).

Then I ran into another organization trying to create a place where photographers could donate images for sale to generate revenue for Doctors Without Borders. Images Without Borders seemed like a cause I wanted to get behind, so I did some research, decided it was doing the Right Things, and contacted them to see if I could donate some images.

I’m happy to announce that five of my images are now available for purchase through Images Without Borders, and all profits for their sale will go to Doctors Without Borders. Each image is limited to ten prints and will then be retired. I want to encourage everyone who reads my blog to support Doctors without Borders, either by going to Images Without Borders and buying a print (mine or one of the other donating photographers), or by donating to the organization directly.

I will sweeten the pot further — if you buy one of my prints via Images Without Borders, I will send you a free 11×14 signed print of any image in my portfolio as my thank you for doing so. Simply email me a copy of the receipt on the purchase and we’ll work out the details.

Here are the images I’ve made available:

To help spread the word, I’ve created some free mobile phone wallpapers of these images. You are welcome to make copies of these and pass them around or install them on your phone. I will also be doing desktop wallpapers of some of my images, including these, to help support this cause — stay tuned for that.

I hope you all will consider supporting this organization and cause, either by buying a print, by donating directly, or by publicizing this and spreading the word to others. Haiti needs our help, and this organization is there on the ground trying to make a difference, and it deserves our help.

Thanks,

Chuq

(p.s. observant geeks will probably notice that my photos are being hosted on Smugmug and not flickr. I’ve been working towards creating a portfolio site where I can start selling prints and licensing images, and Smugmug was the site I decided to use for this (Photoshelter, the site hosting and donating its services to Images without Borders, came in a close second). I’ll be using Smugmug as the site for my professional portfolio the way I use (and will continue to use) Flickr to distributethings more casually and socially. I’ll talk more later about my plans for Smugmug and how this all ties together, but this situation was a great opportunity to fire up the new site and get this next phase of my photography going….)

Tags: About Chuq, Photography

Yosemite on the horizon!

So having mentioned it was time for vacation a couple of weeks ago, I’ve actually done something about it. I was able to grab three nights at the Lodge on the valley floor, so I’m taking the week of March 1 off and heading off to Yosemite in about a week to get away from email and everything else, and just unplug and unwind and focus on recharging the batteries and taking some photos.

It’s still a ways out, but the ten day forecast for that period (3/1-3/4) is encouraging: maybe some overnight snow but temperatures above freezing during the day, and it looks like a storm will roll through during the trip, which I’m hoping for. Nothing scary in the forecast, and encouraging for landscape potential. So we’ll see.

What’s not settled is what to do around those days. I sat down and wrote up all of the things I might want to do and then started pruning out stuff that didn’t seem to make the cut from a time/energy/interest level:

• Grand Canyon (too much driving, too little time actually there)
• Bryce/Zion (ditt0)
• Vegas for birding and hiking in Red Rock (intriguin, but.. not this trip)
• Disneyland (I don’t want to “go urban”)
• San Diego Zoo (ditto)
• Salton Sea and wandering the deserts for wildflowers (tempting on any number of levels; ultimately more driving than I wanted to do)
• Carrizo Plain wildflowers and then SLO/Santa Barbara/Morro Bay (too early for Carrizo by a couple of weeks, and a bit of been-there-done-that on the central coast. It’s been where I’ve gone to hide a lot the last couple of years, so time to do something different…)
• Out to Tahoe/Reno, then down the 395 to Morro lake and Bishop and out the other side (ultimately, it’s too early in the season for some things I want to do in the Eastern Sierra like Bodie, and just way too much driving, much as I’d like some winter time with Mono Lake; Devil’s Postpile is also buried right now…)

As it turns out, I got invited to go birding with a group down in Monterey on the 7th, so that was encouragment to stay a bit more local. Much as I’ve wanted to get down to Salton Sea to bird for a while (it was the trip that got canceled by my dad’s final hospital visit), I find myself interested in driving less and “doing” more. I’m not making final decisions about what I’ll do until I leave Yosemite, but I’ve got a few informal plans I’ve framed out depending on what I feel like and what the weather dictates.

• Plan A: overnight near Galt so I can spend more time with the cranes and geese at Woodbridge and some of the northern Sacramento flyway stops and do some birding photography and try to observe one or two fly-ins and a dawn up there, then home.
• Plan B: go home (but don’t tell anyone), and daytrip. Lots of things I want to do, from the SF Zoo and the aquarium to exploring the marin headlands and presidio or even play tourist from Pier 39 to Ghirardelli… tempting.
• Plan C: change my mind and head down to Morro Bay (because ultimately I really love it down there…)
• Plan D: who knows? I’m open to suggestions…

In any event, I’m really ready and looking forward to this. Definitely planning a busy week, just a week doing stuff I want to do… Blogging may continue to be light as I get things prepped at work and here at home for the trip; then again, maybe I’ll just start a cube sabbatical and get my blogging caught up…

Tags: About Chuq

What’s on my Pre?

The fun guys at Precentral have done an article where they all talk about the apps that live on their Pre or Pixi, so I figured now was a good time to chip in and do the same.  Do I need to attach a disclaimer here? Nah. you all know the drill — and I pay for all my apps, no freebies out the side door on weekends…

Here are the apps I keep close and handy:

• To Do Classic — relatively new to the catalog, but I like it because it’s simple. I’ve used a number of other to do apps but this one is my current favorite because I don’t need the extra power some of the other apps have. I also like GroceryList.
• gDial Pro and Visual Voicemail — connects me to google voice, which connects me to the thirty-seven-gazillion phone numbers I seem to own now.
• TVMCalc — my current favorite calculator
• Twee — my current Twitter client. Because I know you folks can’t survive without constantly hearing what I have to say, even when I’m at a hockey game.
• FourSquare — who knew? people volunteer to be stalked! (but it’s fun!); I’m also starting to experiment with FourSquare as a location tool for birding, and it looks promising.
• Where I’m At — I’m experimenting with location based services and the GPS as tools for my birdwatching. This one is my current favorite tool for grabbing location data and archiving it.
• MediaClock — my travel alarm. One less gadget I have to carry in my bag!
• StopWatch — which I use primarily for timing captures for night photography.

Below the fold, where you can’t see them:

• textPress, because a way to quickly take free form notes is amazingly useful
• Evernote, because you need a place to store your stuff. Having your stuff available on all of your devices rocks.
• The Weather Channel, because when it rains, you get wet. So go inside.
• TealTime, because when you work with people in five different timezones on a regular basis, knowing when they’re at lunch or asleep rocks.
• Parking Place, my current “where in the HELL did I leave my car this time?” app. Now, if I could just teach it to automatically figure out it needs to take a location, I’d be really happy.
• Preware and AppScoop, because we aren’t afraid of homebrew here.
• SuperSudoku, Free Klondike, and Mine Search because, well, sometimes those important staff meetings run a bit long. I’m glad my boss doesn’t read my blog.
• OpenTable, because dinner isn’t just a good idea, it’s the law.
• Flashlight, because I don’t see those funky menus as well as I used to. Isn’t middle age fun?
• DOF Calculator, because I’m a photo geek.
• Yelp!, because I need to know where to get a good plumber who makes lattes.
• Tip Em!, because the pre can never have enough Tip Calculators and this is my current favorite.
• Backgrounds, because plain and grey is boring.
• Sunrise Sunset, because knowing when it’s dark is useful for a photographer for some reason….
• Friendsbook, my current favorite Facebook app
• DirecTV, so when I remember I forgot to schedule that show, I can fix it.

What’s NOT on my Pre? As most of you know, before I came to Palm I used “that other phone”, but I believe in eating my own dog food, and I’ve been chowing away. But there still are a few things on that other phone I can’t do yet. Here are things I really wish I could do on my Pre, but nobody’s written them yet:

• BirdsEye and iBird — my birding field guides. We just don’t have these resources on the pre (yet).
• Best Camera, Focalware, Magic Hour – Photography apps I use once in a while.
• Darkslide — there just isn’t a Flickr browser as good as Darkslide yet. Ditto one for Smugmug.
• TideApp — another useful thing for a nature photographer and birder, because it sucks to get your feet wet for the wrong reason…
• Sirius/XM Radio – 24 x 7 hockey sports talk, baby!

Given I own about 100 apps for “that other phone” (yes, I’m an app slut), that’s actually not bad for a new platform out less than a year. I figure it won’t take too long because I’ll have alternatives. Right, developers? RIGHT? Don’t make me hurt your dog… (and if you think about it, all but Darkslide are what you can charitably call “niche” apps for a specialty audience. they normally trail in availability while we build the audience for them…. ). My “other phone” now lives in my bag, relegated mostly to iPod duty and when I need one of the specialty apps, and if I wasn’t so lazy about it, I could move the music onto the Pre, but I just haven’t bothered…

Of course, I’m constantly trying out new apps and shifting stuff around, but isn’t that half the fun? These, however, are the ones I lean on and use on a regular basis these days.

Tags: About Chuq

A great way to start the new year…

The best laid plans… I have a bunch of blogging stacked up, none of which you’ve seen yet. Just as the New Year kicked in, so did a bug, which struck both myself and Laurie, and after a few rather grumpy days as a head cold, it headed to laurie’s chest and off camping in my ears, so I started off the new year under the weather and on deadline with both the CES announcements and our newly refreshed developer portal and blog.

Thank god for Sudafed, that’s all I can say, even though they make you sign 37 forms to get the damn pills now. I do not, for the record, recommend the sneezing, Sudafed and Starbucks Diet, but it does seem to work. After one last “battle of the bulge” over the weekend, I seem to have fought the bug off for the most part and the energy levels are returning, so the ability to string words together and have them make sense seems to be back. you really didn’t miss anything — insightful — the last week or so, anyway. Trust me.

But if you’re wondering why I’m just getting to looking back and setting goals as we roll into February, that’s why. So 2010 is off to a rousing start…

But it’s time to get back on the horse and start riding again, and I’m thinking through the next couple of months and one thing I’ve decided is it’s time for a damn vacation. I went looking, and I’ve suddenly realized that in the last

• 2009: 2 days (an extended weekend in Morro Bay for Photo Morro Expo
• 2008: 5 days for the trip to Yellowstone
• 2007: 3 days for my aborted research trip for Dare2Thrive after leaving strongmail, 5 days into the Northwest after leaving Laszlo, and 2 days for a spring trip into Yosemite
• 2006: 2 days for a christmas jaunt into Yosemite, and the 8 day summer celebration into the Northwest celebrating leaving Apple and moving on to whatever was going to be next…

The trip to Yellowstone (after spending most of the year dealing with Dad’s illness, death and the estate with my mom) seems like forever ago. Because to some degree it was. My moving to Palm was on a tight schedule so no time off, and this last year has been  an amazing year that I’ve loved just about every minute of (the minutes I didn’t love were the ones I was considering throwing myself, or someone else, off a roof…) — but it’s time for a break, so I’m starting to plan out some time off. Not sure what, or where yet, but I know I need to get in the car and take the camera and mostly unplug for a bit.

I’m guessing late february or early march. have to figure out what the work and hockey schedules are, and of course get Laurie’s thoughts and permission (shh.. I haven’t mentioned this to her yet… literally just thinking this through tonight after she’s gone to bed). The obvious ideas come to mind, which include Yosemite (too late for serious winter work(?), too early for waterfalls and WAY too early for spring and dogwood), but also to finally get to Salton sea and maybe spend time in Joshua tree and Anza/Borrego and the deserts — I had a trip planned for Salton Sea when dad got sick, and it got blown up and I’ve never gotten it rescheduled. But I’m hearing other things whispering also, whether it’s Grand Canyon or Bryce and Zion, or even shooting up the coast into the Northwest (but I’m likely to hold that off for a summer trip with Laurie…); some other venues come to mind like an extended visit to the San Diego zoo (I haven’t shot at a zoo in a while) or Disneyland or Vegas for the kitsch.

Dunno. Have to think. Have to make sure I don’t overschedule and spend too much time travelling and not enough time visiting. Maybe define a starting point and then see what happens. Right now I can definitely feel a tug between revisiting comfort zones (disneyland, yosemite) and pushing into fresh territories. I think I need to lean myself towards the latter, this feels like it’s time for some exploring.

I’m definitely open to suggestion. Feel free….

(and I think I’m going to try for a long weekend or a mid-week jaunt to Yosemite for the dogwood this year, if I can. But I’m always up for more than one trip to that place, especially in times when it’s relatively quiet. It’s been probably 15 years since I’ve visited at a time when Tioga was open…)

Tags: About Chuq

Lists for 2010

Ten Photographers that made me a better photographer in 2009

1. Mike Baird
2. Ashok Khosla
3. Howard Ignatius
4. Hal Schmitt
5. Steve Berardi
6. Rick Sammon
7. James Duncan Davidson
8. Moose Peterson
9. Jerry Ting
10. Bryan Oleson

Photographers that influence my photography

1. Art Wolfe
2. John Shaw
3. George Lepp
4. Galen Rowell
5. Jim Goldstein
6. Michael Frye

Five Photographers that I’m currently studying to become better in 2010

1. David duChemin
2. Harold Davis
3. George Barr
4. David Hobby
5. Joe McNally

Ten most popular postings at chuqui.com in 2009

In 2009, 17,000 people visited chuqui.com about 35,000 times to view 49,000 pages. Overall traffic was up about 20% over the year. The ten most popular pages for 2009 are:

1. And the answer to the question is… — in which I announce I’m taking a job at Palm.
2. More than you wanted to know about backups (also see Some More Thoughts and Following my Own Advice)
3. This iPhone App is truly for the birds
4. Santa  Clara County Salt Ponds and Bay Area Birding
5. Some of my Past Fades to Black
6. Photo Links (and more photo links!) — early versions of what became “stuff you’ll like“
7. MobileMe Renewal? Yes or no?
8. Calaveras Bald Eagles (was out visiting them this weekend!)
9. Apple Drops hints about netbooks (which is only tangentally about an Apple netbook)
10. Some  more thoughts on weight (which is way overdue for some updates…)

My ten favorite posts of 2009

1. Three Rules “they” tell photographers, and why “they” are wrong
2. A teachable moment (or why I love birding even when I make a fool of myself)
3. 50 Reasons why I haven’t been blogging
4. On Paying Forward
5. Understanding the Starting Point
6. More than you wanted to know about backups (also see Some More Thoughts and Following my Own Advice)
7. In Search of Winter Birds
8. The Summer of Hockey’s Discontent
9. Some More Thoughts on Weight
10. 25 Things about me

My ten best photos of 2009

My favorite fiction of 2009

Tags: About Chuq

Final Notes on Obama’s 2009 « Whatever

if Obama were on fire, the GOP would call fire departments a socialist plot. The folks losing it on the left, on the other hand, are being a bit petulant about both the actual human they elected to be president, and the practical constraints on his agenda. The man has monolithic, unified opposition in the Washington GOP, a fractious and fragile base in the diffuse Washington Democrats, and was handed two expensive, unpopular wars, a profoundly degraded political environment at home and abroad, and a national and global economy which were dual scorching pillars of oh shit we’re all going to die. That the man got anything substantive done, much less had what is objectively a politically remarkable first year, is impressive.

via Final Notes on Obama’s 2009 « Whatever.

I normally avoid talking politics here, mostly because it tends to attract the loonies and trolls like the plague (and who really cares what I think, anyway?) — but John Scalzi waded in, and honestly, it deserves a “what he said”, because if I were to write about this stuff, this is what I would have written. Only not as well.

Tags: About Chuq

Proposals For Librelist Moderation Strategies

To understand the feature requirements for moderation we need some goals. Keep in mind that no moderation will be perfect, and you can easily come up with scenarios that will work around anything we come up with.

Therefore, we should focus on just some initial goals that will work right now, and keep in mind that these will need to be constantly tweaked and worked on as the spammers evade the measures.

• If given the choice between restricting free speech and preventing unwanted communication, free speech always wins.
• The system should increase the quality of discourse for any project, regardless of human language used.
• It should never give a small group the ability to hide communications from others.
• It should be implementable and not have high hosting costs.
• It should not rely on a dedicated person’s constant intervention.
• It never gates email through system before sending it, but rather allows initial emails with moderation after.
• It should use information from people’s rating habits to classify them as “ratings trolls” to prevent abuse.

With those goals in mind I’ve teased out two potential list quality strategies that might work.

via Proposals For Librelist Moderation Strategies.

Someone I work with turned me on to Librelist because they knew me interest and history with mailing list systems, and I find it interesting that some folks have decided it’s time to rethink the mailing list again.

They’re right. When I faded to black on the mailman project, it was at least in part because many of us felt that mailing lists were a technological dead end, and that deliverability issues because of anti-spam systems made the “personal mailing list” an increasingly difficult thing to accomplish.

Both are — for the most part — true. I certainly would never run my own mail server again, because the advantages of doing so are far outweighed by the time and hassle of trying to manage deliverability and reputation to make sure mail it sends gets accepted, and the constant onslaught of incoming spam turns them into a permanent infinite time sink. That’s why I either retired our lists or moved them to Yahoogroups (which I personally think is a pretty good system).

But there’s still room here to rethink the concepts and the Librelists seem interested in trying, and I think that’s great. Email and mailing lists are far from dead — but instead of stand alone delivery tools, they really shine as part of an integrated web strategy; Yahoo groups is a nice first generation of that, although there’s a lot more Yahoo could do if they decided to.

Message moderation really breaks down into two big problems:

• “Subscribe spam” where spammers sign up to the list to spam it.
• “Member warfare” where existing, approved members get into fights and they escalate into unacceptable territory.

The first is really simple to solve: new members are moderated, and messages aren’t posted until reviewed by someone to vet their content. Simple implementation; Yahoo Groups does it today, and on the lists I still manage, it works well to keep the spammers at bay. The way I manage it is all members are moderated until their first post. if their first post is acceptable, I turn off the moderation bit. To minimize delays in propogation of new member messages, simply choose a moderator pool large enough to guarantee held messages get reviewed and approved in a timely manner — you could even make that moderator pool all members in good standing if you want, because all you really need is someone you’ve trusted to post vetting that someone new is trusted to post.

Member warfare is trickier. I hesitate to call it trolling because the pure troll is a subset of the larger issue of two (or a small group of) people getting pissed off and going at it. A troll is simply one person going off on the rest of the list.

I’m more and more convinced the answer here are reputation systems, where over time a user’s membership in a group is used to define their abilities and restrictions. The longer a member is in the group in good standing, the more often they contribute material, the higher their reputation goes and the more the can do and the more sway they have on the decisions of the reputation engine. You can tweak the details of the algorithm almost any way you want, but if you define it in terms of “how long they’re a member” and “constructive contribution to the community”, you can come up with a metric on how valuable that member is to the community, and then use that to rank that member’s contributions and recommendations.

Here’s one rough view of how to build this. Please note that I firmly believe karma rankings are private and users have no way to see what their ranking is or compare it to others, except in really broad user categories (“member”, “senior member”, “top contributor”, “advisory board member”). As soon as you create a list of any form, you will attract people who see it as something they can game, and so they will.

User Karma is a value between 0 and 1, which starts at 0.5. Every time a user contributes to the system (a posting, a reply, a moderation recommendation, etc), the number gets bumped by some value. How much the value is incremented or decremented depends on how it’s rated by other users — so if User A posts a message, User B flags it as spam, but 80% of the membership feel that was a bad decision, User B’s karma is reduced in future decisions, they lose influence. Over time, the system self-corrects by giving increased influence of those who’s decisions match the community consensus and reduced influence to those who’s postings and recommendations don’t match up well.

The system can then choose whether to accept or flag for moderation a posting based on a poster’s karma score. You could potentially reject outright users that have karma scores below some value, or allow other members to choose not to see messages by users with karma scores below some value. Over time, users who are disruptive to the community will get karma’ed into the moderation queue (or out the door), and users who are seen as top contributors will have stronger influence.

My goals:

• A system like this can be built nicely with a good SQL backend and a bit of horsepower. I’ve actually done a detailed design and schema on this before, and it’s a fascinating thing I’ve always wanted to implement.
• It enables the power of individuals to police themselves.
• It limits the ability of an individual to harass or cause problems.
• It doesn’t lend itself to people playing the game of gaming the system by not exposing the details of the system (slashdot karma whores need not apply).
• Trolls get edited out of the system because the community will quickly recognize them for what they are and trash their karma, causing their postings to disappear to the bottom of the list.
• Cliques and Mafias have to be large to influence the results significantly. You don’t completely avoid the clique/mafia problem, but you can severely limit it’s ability to wreak havoc.
• It doesn’t require a lot of manual handholding or babysitting. Admins end up stepping in only in extreme cases.
• Because trolls tend to get edited out of the system quickly and automatically, they tend to go elsewhere because without feedback and controversy, they wither and die. And by editing them out of the system quickly, you avoid the whiplash and fighting that happens when people start fighting with the trolls and the wars break out.

Weaknesses:

• Any community tends to turn into an echo chamber. Automated systems like this encourage this because “different thinking” tends to get rated down.
• That’s usually a lesser evil to letting the trolls run wild.
• To my knowledge, nobody’s ever solved the problem of the conflict between the group-mind reinforcing the echo chamber and allowing the free thinkers to poke at the community’s comfort level by pushing them to think about things that make them uncomfortable. One person’s rebel is another person’s troll, and that’s not solvable in real life, much less in automated life like this…

These techniques are all based on (or stolen from) things that are in use around the net, with Amazon’s review feedback being one I really respect; while trying to avoid the pitfalls I’ve seen around the net (yes, I’m going to keep bashing on Slashdot’s karma system, it’s way too easy to game and always has been). It also (I believe) avoids the nasty politics that have made Digg a bit of a pesthole. And it’s also pretty lightweight and low-key, or at least it should be. The implementation details will be crucial, as will be tuning how the karma values adapt…

Tags: About Chuq, The Online Life

Know when to hold it, know when to fold it

Sometimes you have to know when to hold it. Sometimes you have to know when to fold it.

One of the hardest life lessons I’ve learned is this: if the only person who knows (or cares) about a deadline is yourself, it’s not really a deadline — and don’t kill yourself trying to make it. I used to really stress out when I’d commit to something then not make the date because other things came up (even though I felt those things were more important, or out of my control). Even though the only person it mattered to was me.

I finally figured this out, and started teaching myself not to sweat over details that didn’t really matter.  I decided back in 2006 I wanted to try to organize a second career around a camera; I never worked with Bill Atkinson, but you can’t be an Apple geek (or like me, an Apple alum) and not know about Hypercard, and in fact, I did a lot of work in Hypercard over the years. I did work with Dave Cardinal at Apple, and when I started dabbling with digital cameras and then run into their work, it was that seminal eye-opening moment that made me realize that I could make that shift as well, and that was the starting point for this second career planning.

That was in 2005.

In 2007, I thought I’d become a pretty good photographer and started seriously looking to fire up my second career plans. And then I blew out my knee by stepping in a gopher hole, then dad got sick (and then he died) and by the time I surfaced from helping mom through the estate, 2008 was almost over and it was time to try again. Then I landed the gig at Palm, and I knew it was going to an insanely fun few months and it went on hold again. And now it’s almost 2010, and I feel like it’s time to try again. So we will.

That doesn’t mean nothing happened in the meantime. In fact, the delays have been a blessing in disguise. I’ve spent a ot of time and energy in battling bits in photoshop and lightroom, in studying my work and really being honest about my strengths (and flaws) and working to fix the flaws; in studying other photographers and understanding their strengths and how to adapt them into my own work.

In 2007, I thought I was a good photographer. Today, I’m a much better photographer. Sometimes delays can be frustrating, but you turn them to your advantage.

There are always reasons to say “not yet” to your plans; planning is easy — and safe. Doing is hard, involves risk, and may fail. But sometimes, “not yet” is the right decision, even if you don’t like making it at the time. You might miss out on an opportunity, but if your plans are sound and your planning is done well, you’ll run into later opportunities later. (hint: if your plans revolve around a “now or never” situation, it’s probably a badly thought out plan. If you don’t have follow-on opportunities to build your business with, how are you going to grow your business?)

So knowing when to hold it — and using that hold time to improve your chances of later success as you can — can be a positive. You can’t be afraid to say “it’s time” and push the button and make it happen.

At the same time, you have to be realistic, and you can’t force yourself to push the button simply because it’s the date you said you were going to do it on — going in for the wrong reasons and at the wrong time makes the chances of failure skyrocket, and if it fails, it’s going to be harder to generate future opportunities or feel confident about going out and grabbing them when they happen.

There are also times when you simply need to fold the hand and try again. When I left StrongMail, I’d been working on a project called Dare2Thrive, and decided to take some time off and push that project into production. Once I left on my own, some outside factors kicked my motivation in the crotch, and I realized there were some fundamental flaws in my approach that I couldn’t easily solve.

The big one: I was putting myself into primarily an editorial role, not a creative role, and on a long term basis, I decided that was unacceptable; I wanted a situation that focused on my own material rather than creating an environment where I promoted the work of others.

That’s not something you fix by tweaking the CSS.  I realized I needed to start over, tear it down to the bare assumptions, and starting over. I hated the decision at the time; at the time I felt it was fixable, but the external issues I was dealing with made it necessary. I realize now that not only was it the right decision — it saved me from almost certain failure if I’d pushed forward, and that was without taking into consideration what 2008 was going to surprise me with…

Holding it and Folding it. Never fun. To take something you really care about and want to do and stuff it back in the closet is hard. To take it out behind the shed and “Old Yeller” it is traumatic. If it’s the right thing to do, though, it has to be done, and many times, it creates new, better opportunities later if you let it.

And for the record, the core or Dare2Thrive — the piece that first made me want to do the project — is alive and well as part of this new project. And dammit if it isn’t even a better project now than it ever would have been in its original form.

And if anyone wants to buy them, I have the original dare2thrive domains available and parked, at least until I figure out a use for them….

Tags: About Chuq, Thoughts on "The Second Career"

Next Page »