What to do when you realize you’re running out of disk…

One of the things that became painfully obvious during my trip to Yosemite was that I was rapidly running out of hard disk.  Being out on the road is not a good time to realize you need  a bigger disk, s when I came back, I decided to fix things before it became a real problem. Here’s what my overall “bits on things” setup looked like:

Now, there’s one obvious problem there that I hadn’t thought about — the backup disk is smaller than the main disk. I knew about that, knew I needed to fix it, and forgot. Not a huge problem, but one of those details you need to keep an eye on or they’ll bite you at an inconvenience moment. Even though I had 3/4 of a terabyte for my backup disk, Time Machine was only storing backups for about 3 weeks, which means it was no longer large enough. It was time to update and grow and upgrade.

The biggest problem — the new Canon 7D creates much larger images. That’s good, but creates ripples. It also does video, which I’m starting to experiment with. By the time I convert the 7D RAw image to DNG and store it on disk, it grows to about 49 megabytes in size. Pile up a few hundred of those, and “Hell, disk is cheap” starts ringing a little hollow. To give an idea of the change going from the 30D to the 7D, on the 30D I use a 4Gb memory card and get 400+ images on it. On the 7D, I upgraded to 16Gb cards, and I get 500 images on one. Moderate upgrade in number of images, big upgrade in amount of disk taken. Also, since the 7D shoots 8 frames a second sustained where the 30D shot 4FPS with limited bursts, the opportunity to generate LOTS MORE images quickly exists. And it definitely happens, so at the end of the day, I have more, larger images to store. This is, as they say, a good problem to have.

The easy answer — upgrade the laptop to a bigger disk — won’t work here. The biggest laptop disks now available are 500 Gigabytes. Larger than my 320Gb, but not by that much. Upgrading delays the problem by a period of time, but it doesn’t solve it. I considered doing that, then decided to bite the bullet and shift into the “it no longer fits on the laptop” universe.

I mumbled about this on Twitter, and immediately got back the “install a NAS!” response. NAS (or Drobo, or RAID, or name your favorite disk packaging setup) isn’t a solution — it’s a technology. You don’t start by choosing a technology, you start by figuring out the solution and then choosing things that implement them well.

I’ve written about backups and my philosophy on how to do them before, check out this piece as well as this followup, as well as this piece where I talk about why I stopped using an online backup solution in favor of sneakernetting an offisite backup somewhere. I am, for the record, looking forward to when the price/performance and the network broadband make this worth doing again, but not right now…)

So for me it’s time to shift my data into a multi-disk environment. I live on a laptop, which gets carried around. If your data no longer all lives on the laptop disk, then when you need that data, you have a problem. It behooves you to then think about your data and how you use it, and figure out how to store your data across your disks so that you have access to what you want when you want it.

For my purposes, “data” can be defined as “everything on your disk”, but in practice, I see no reason to think about shifting apps out of the Application folder or similar “optimizations”. You might be able to free up a gig or two of space, but why? That’s not significant, and it can lead to potential complications later, especially if you start mucking in your Libraries, preferences, caches, etc. The savings aren’t significant — or worth the future hassles or possible compatibility issues. So for me, unless you’re a font geek with 50 gigs of fonts or something like that, just worry about the data folders: Documents, Pictures, Music, Movies. (in case it’s not painfully obvious: this info is Mac specific. General concepts work for Windows as well — the nutty details are your problem on that platform).

A few key goals

Here are a few key goals of all of this:

• Scales infinitely. Or close enough I don’t have to go through this again for a while
• My data is available when I need it, wherever I am
• Easy and intuitive. I don’t want something that’s difficult to do, or I won’t.
• Reliable and easy backups: if your backups are difficult, you won’t. Keep it simple. Make it reliable.
• Fast catastrophic recovery. I don’t want to spend days getting my data usable again
• Recover a file or a disk. Some backup schemes work best for a crashed disk, others for a lost file. you really need both.
• Backups on the road are even more important, not less. So make sure you can do them. And do.

Here’s what I ended up with. It’s not hugely different than before, but the changes create significant challenges to understand:

I took the bus-powered disk and upgraded it with a 500 gig drive. This means that instead of having 320Gb available, I now have 3/4 of a terabyte I can carry around and use without needing an electrical outlet. This is a significant detail: you really mess up the concept of a “laptop” if you have to plug it in to use it… Or worse, can’t because the data you need is inaccessible because you didn’t bring it.

Digression: for those of you about to tell me “just live in the cloud”, plesae don’t. The dataset we’re talking about is measured in gigabytes trending to terabytes, and it’s not practical. In reality I am using Google Docs and Dropbox more for some things, but for the set of things “the cloud” solves for me, they also live happily on my internal laptop disk. This is about figuring out now how to scale from having 1,000 photos in my portfolio and 10,000 in my collection to having 20,000 photos in my portfolio and 100,000 in my collection without everything collapsing in a heap, and those kind of data sets aren’t going to live online any time soon, nor do I particularly want them to.

So anyway, I now have three drives going. The internal laptop drive (320Gb) is where everything I need 100% of the time has to live. The external bus powered drive can store other files that I need access to on the road — but which I probably can live without for more casual usage. And my desktop drive (AC powered) stays at home and holds the data that I need easily accessible but don’t need to travel with.

I went through all of my data and figured out where it needed to live. There’s also an unlisted “fourth category”, which is data that lives offline, or on a disk that I maybe need access to once in a while but not keep plugged in, and I spent some time pulling all of that data off my disks and sticking it in a corner to archive into a drawer. (one could also say there’s a fifth category, the “why the hell am I hanging on to THIS?” category of things that ended up in the trash. Things like the Parallel’s virtual image of Ubuntu I haven’t booted since I installed it five months ago, which deleting freed up multiple gigabytes. And why did I feel the need for an Ubuntu disto in Parallels on MacOS, which is just a different flavor of the same thing? I don’t remember, but it seemed a good idea at the time…)

I can hear some of you groaning at the thought of sorting through all of your data. I sympathize. If you don’t want to commit the time to that, I understand — but — putting some time and energy into it now helps you understand what you have and how to organize it. It also means that moving forward you’l have a good sense on where stuff belongs, meaning you’ll spend less time thinking it through and organizing on the fly. And if you do it now, you probably won’t need to do it again for a few years. It’s little more than virtually filing everything in your office, and it never hurts to do that every so often.

It shouldn’t be assumed that you need to turn “Save File” into a “Getting Things Done” adventure — I’m definitely not interested in being that anal about all of this, but it is important to understand how you want to manage your data well enough to know if it’ll do what you need it to do and how well it scales. Scaling was the big issue for me. If I’m seriously having to worry about data in terms of terabytes, I’d just as soon not have to architect this all out again in six months. Once it’s settled down, it’s back to the “that pile on the desk is in the way, let’s put it in the files” mode again…

So here’s how I finally settled on filing things. My internal laptop disk:

And here’s what my secondary disk looks like. Note that it only has Music and Pictures folders.

The Music folder is where I’m storing the video files in my iTunes library. The audio (aka “music”) lives on the main laptop disk.  As my creation of video grows, I’ll add a “Movies” folder and split it up the way I do photos, but right now, there’s not much there.

And finally, my third disk, the one that stays at home:

The blue highlighted folders are folders on that disk that I exclude from the Time Machine backup:

which is an option more people should think about if they use Time Machine (or other backups) — some stuff you can live without if you need to, so why back it up? All it does is make it harder to do backups reliably. I flag them with color labels so I don’t forget which ones were excluded — I did that once and had to restore a disk, and spent half a day freaking over “missing data” until I remembered I’d excluded that data from the backups. Oops. It goes without saying, of course, that you should only exclude stuff you really don’t need back if there’s a failure, don’t exclude it because it’s large…

A big part of how this works (or won’t) is splitting up the photo library. In general, I split up my photos into four big piles:

• flickr or better:  images I liked enough to post to my Flickr account (and the subset of those I think are good enough for my portfolio, which I’m starting to build on Smugmug)
• 2nd tier: photos which are technically fine, but which aren’t something I think should be posted on flickr. Most of these are effectively duplicates of ones that go on Flickr (think “eight frames per second burst rate”); you want them around in case you want to use them; you stick them somewhere out of the way because you have no plans to actually do so. In theory, these photos are all good enough to publish, except I have some other photo I think is better — but yo never know when you might want some specific expression or a left profile instead of a right profile, and so they’re here if you need it.
• archive and forget: photos that are clearly not as good as the candidates I’d publish, but not bad enough to throw away. To be honest, as I’m getting more comfortable about my abilities as a photographer, I’m doing less keeping photos around that “someday I might try to fix this”. Instead, I ding them and throw them out. These are flagged to be taken offline and stored, and I fully believe I’ll never look at them again and some day throw them out. More and more, I’m comfortable with my choices and simply throwing them out and saving a step…
• dings: And finally, the dings. As I do edits, the ones that are clearly flawed get thrown out and deleted. There are people who tell you to keep everything. I’m not one of those people. Disk is cheap, but it’s not free. Maybe some day those images will be usable (or fixable in photoshop, or whatever), but the reality is I have thousands of BETTER images I could spend that time on, so why bother? So count me in the camp of tossing the crap, especially when it quickly starts turning into gigabytes and terabytes of crap. Why make it harder to find the good images by having to wade through crap, or worse, create a filing system for offline images to keep around stuff you know in your heart you’ll never use? Let it go. Just because you CAN keep everything doesn’t mean it’s a good idea. It’s not.

This setup looks like it’ll scale for a good long time; I can, if I need to, move some flickr or better onto the 2nd disk and prioritize the internal drive to active projects; 2nd tier data easily moves to the “live at home” disk when I need to. I can subset my itunes library the same way if I want to, and the rest of my data isn’t going to grow faster than disk technology seems to be progressing, and as long as I keep my folder structure sane, I can tell at a glance what’s going on, both within the Finder and Lightroom. I can use Lightroom and Spotlight searching to find things if I need to, but with a bit of care the naming structure will let me browse into it quickly as well. It looks pretty solid.

I’ve spent the last couple of days migrating the data to this new setup and I’m now happy with it, at least for now. As I’ve settled in, I’ve made some changes – originally all three disks had Documents folders, I finally realized that either a document lived on the internal laptop or it lived on the “stay at home” drive; no need for a middle phase, it just complicated things. You’ll notice there are folders on the travelling disks to act as placeholders for the stay at home disk. This makes staging stuff to sweep over there easy, so I can stuff files places on the road and then go home and move them off of the travel disks. It may seem unnecessary or trivial, but I’ve found lots of peopple don’t think about that kind of detail, and when I explain it, they love the idea — it lets me make a filing decision at the time I’m using the data, and merely shove it into the file when I get home and not have to “remember” what needs to be filed days later. Make those decisions while you’re using something and then forget it — it’s a great hint for simplifying things.

And once my backups finally sync up and my data is fully redundant again, I’ll be happier. Currently, I have my superduper backups in place, I’m letting Time Machine sync up now. It can be butt slow at times…

Some technical details on implementing this

The drive I bought for the bus powered disk was the Hitachi Traveler 500G. I’ve been using Hitachi drives for my laptop drives for a while and find them pretty reliable. That doesn’t mean others aren’t, it means thse have worked well for me, so I continue to use them. The bus-powered enclosure I use is the Mercury On-The-Go Pro from Other World Computing. I’ve bought RAM and disk from OWC for years and have been very happy with their price, quality and service. I’ve used that enclosure for a long time with never a failure. Their stuff is well-engineered and solid and I feel it’s well priced, and I haven’t been in the mood to explore other vendors because this one works for me.

For my external drives, I use the OWC Mercury Elite-Pro housing. it’s solid, it’s build like a rock, it works reliably. As part of this rework, I’ve retired the last of my IDE systems and I only buy/use drives that have SATA interfaces.

Digression: Every so often, it makes sense to see how technology is moving and migrate away from stuff that’s aging and heading towards end of life — if you refresh your data onto modern storage, you won’t go looking for it some day and find out you no longer have a way to access it. I’m a big fan of refreshing all of my offline storage every couple of years so the chances of having a stored drive failed is minimized. I’m also a fan of keeping two copies of all offline data, preferably one offsite — just in case. Since I’m also a fan of refreshing my active drives on a regular basis (because the best way to never need your backups is to never run your disks until they die!), a nice way to do this is to replace your active drives every 18 months or so, then use the retired drives and copy all of your archived data onto them, and then take the oldest drives and stick them in your files somewhere.

Digression on the digression: I see no reason to ever give a used drive to someone else, either by selling, giving, or donating. I pull the drives out of computers and housings and file them with my tax papers and other files. Once in a while, I pull the really old mechanisms and “retire” them with a big hammer. That way, there’s absolutely no way someone can recover files off of a drive they bought in Goodwill and end up with your data — because it never leaves your hands. If you trust seven-way zeroing and are willing to spend the time to do so, bless you. I jut don’t think a used disk drive is worth the time and hassle to recycle for re-use…

The drive I’m using as my backup drive now is the 2Tb Western Digital “greenpower” Caviar Green with 64 Mb cache. There are cheaper drives out there, but this one has good reviews and is built for server service. In all honesty, there’s nothing quite so painful as finding out your backup drive has failed, especially if you find out while trying to restore something. I don’t want to overpay for this stuff, but cheaping out bites you down the road.

My backup drive is living in a NewerTech Voyager Hard Drive Dock, which allows you to insert and eject SATA drives easily. This means if I want to I can easily pull this mechanism and replace it with another if I need to “do something” with another disk. I’m just starting to use it so I don’t have reliability data on it, but so far, I like it. It’s solid and well-built at first use. I plan on using it for managing my offline archives as well, saving me paying for multiple enclosures down the road.

Geeky details on backups

The 2Tb disk is split into two partitions, one 500Gb and one 1.5Gb. I use two backup technologies, SuperDuper! and Time Machine. I love Superduper for system backups because it makes bootable clones. That makes catastrophic recovery a lot simpler: take your backup drive, plug it into a Mac, and boot from it (then make a backup of it before something bad happens!). Superduper runs nightly and refreshes copies of my two travel disks, which is why the 2Tb is split into two partitions. The 500Gb syncs up the 500Gb external disk, and the 1.5Tb is the clone of the internal boot disk and also is where my Time Machine backups live.

Superduper doesn’t do versioning or archival over time, it makes a snapshot of now. For the “I need that file I threw out two weeks ago” problem, I use Time Machine. It backs up all three disks (minus the exclusions I mention above) to the 1.5 Terabyte partition of the backup disk. Time Machine is useful for casual backups (it’s better than nothing and pretty good for get-single-file recoveries) but I don’t like it for complete disk recovery and after working with a Time Capsule for a while, I really don’t like Time Machine over a network. If anyone really cares why, that’s a whole different blog posting.

The good news is that SuperDuper and Time Machine co-exist nicely on one disk (thank you, Dave!) so I can do both easily, so I’m set up to clone my two key disks onto the backup disk, and then do a time machine backup onto it for incremental backups as well. If my boot disk crashes, recovery is (almost) as simple as booting the backup disk. Wonderful, since crashes almost always happen on deadline…

What this doesn’t cover yet…

There are a few details this new setup doesn’t cover yet. None of them are time critical, but all of them need to be considered and solved, and it’s important you know how to solve them before you implment (lest they blow up your work when you go “oh, damn, didn’t think of that” later). Fortunately, they all are solvable…

• The new setup doesn’t include “on the road” backups. Since I no longer can carry a bus-power drive big enough to back up my systems, the answer is to carry a bigger, plug-in drive. I’m not worried about Time Machine backups on the road, so the easiest solution is a 1Gb external drive in one of my Elite-Pro housings. Even better, that’s cheap, and if I set it up, gives me an easy “spare backup” setup, because I love having a set of backups I only update every week or so, just in case something corrupts that I don’t recognize right away. So that’s probably what I’ll do. The other option would be to carry the 2Tb backup disk with me in the Elite-Pro housing, which also works, but which limits the number of redundant copies I end up having. I don’t like carrying my backup on the road if I can help it, I’d rather carry a “road” backup and leave the main backup at home. But both are options.
• The new setup doesn’t make explicit the off-site backup storage. What I’m doing in the short term is taking my old backup disk offsite. In 4-6 weeks, I’ll buy a 2nd 2Tb disk, plug it into a dock, build it the same as my new backup disk, and run backups onto it, and then swap between the two (the other going offsite) every 4-6 weeks. That’ll fix this for a good while at reasonable cost.
• The setup for moving files onto offline disks (aka “in the drawer”) isn’t spelled out, but is pretty simple: buy a pair of 500Gb SATA drives, plug them into the dock, copy the files to each, carry one offsite. Iterate until full, and then either start another set or decide some of the files can be deleted (or both). Every couple of years, take all of your offline disks, copy them to new (fewer, bigger) disks, and store them again.

But what about “install a NAS?”

I have to admit I’m not a huge fan of NAS in my environment, but I also realize that over time, the amount of data I’m storing on my “stay at home” disk is going to grow without bounds. My plan at this time is to convert that into a Drobo at some point, but not until I need to, so I’ll hold that off until later this year. I realize that at some point the percentage of data I can keep local to the laptop, even with 1 terabyte (500gig internal + 500gig bus powered) is finite, but I’m only using about 275Gb on those two combined right now, so I have some time before I have to worry about that…

Things like Drobo and a NAS add some capabilities, but they also add complexity, cost and new ways for interesting failures, which always seem to happen on deadline when you least can afford the issues. A NAS works best if you’re sharing data among multiple machines, since I’m not, it adds more complexity than it solves problems. Drobo is different being locally hooked up (and there’s a NAS enabler you can buy for it), but adds its own set of complexities and administration — so as long as (a) a single disk works and (b) I can back it up reliably, I’ll stick with a good single disk. Once you start getting into multiple disk environments and/or your backups start being tougher to keep reliable, the addition of mirrored RAID and some of the other features of NAS or Drobo become good to have, but again, I’m not at that point yet.

Finally — speaking of Terabytes

I’ve been around long enough that the thought of buying disk in terabyte sizes amuses me. My first hard drive was ten megabytes — MEGAbytes, not GIGAbytes — and I remember a time when a terabyte would probably store all of the data at Apple, and perhaps all of the data in the state of California. Today, I’m using it for backups of my personal data set. That amount of scaling in the last 30 years or so amazes me when I step back and consider it. But then, my phone has a lot more processing power and memory and disk than my first Mac did. I think my KEYBOARD has a more powerful CPU than my first home computer did….

Tags: About Chuq, Photography, The Online Life

Mac Netbooks

Ben Long talks about using a Hackintosh (a netbook hacked to run Mac OS X).

For the last year, I’ve been using a hacked MSI Wind as a netbook, but its keyboard played havoc with my repetitive stress injuries. Something about it made me hold my hands in a way that ultimately caused pain. I recently had the chance to type for a while on a Dell Mini 10v and found that I had no pain issues at all, so I sold the Wind and picked up a Mini 10v on sale for only $275.
Compared to my 13″ Macbook, the Mini 10 is considerably smaller and lighter, making it very usable for backcountry trips – something I would never do with my Macbook. With it, I no longer need to carry my Digital Focii FotoSafe for offloading, and I’m not stuck trying to type emails on my iPhone keyboard.

Obviously, if you’re a Windows user, you can use the Mini 10v right out of the box. If you want to use the Mac OS, though, you’ll need to perform a quick and simple hack.
NetbookInstaller is an application that will take care of the hack for you, and using it is very simple. You’ll need a copy of Snow Leopard, and a USB stick with at least 8 gb of capacity. Detailed instructions on the NetbookInstaller site will guide you through the installation. You’ll image your Snow Leopard disk onto the USB stick. and then boot off of that. The NetbookInstaller application will modify the installation to allow it to work on the Netbook.
When you’re all finished, you should have a Mini 10v running the latest Mac OS (at the time of this writing, I’m running 10.6.2). The trackpad supports tapping and two-fingered scrolling, and sleep, restart, shutdown, the web camera, and SD card reader all work fine. The model I got has a gigabyte or RAM and a 160gb drive, though both of these are upgradable. The computer weighs in at 2.6 pounds.

It’s definitely a viable option if you want to depend on an unsupported computer environment, but he neglected to mention a couple of important points:

1. If you  don’t buy a copy of Mac OS X or have a family pack, you’re pirating the software. Photographers need to be really sensitive about violating the licenses of others, or else we should shut up when people ignore our copyrights and rip off our photos. Can’t have it both ways, folks, although I know a lot of people who try.
2. Even if you do buy a copy of Mac OS X to run on your Hackintosh, you’re putting it on hardware that isn’t allowed by Apple’s EULA for Mac OS, so you’re violating their T&Cs, which depending on how you want to rationalize it means you’re pirating the software whether or not you have a paid license for it.
3. If neither of those keeps you up and night sleepless over the moral quagmire of violating Apple’s legal agreements while being hard-ass about protecting your own, it’s still an unsupported and mostly untested hardware/software configuration which may break at any moment (or which at any moment Apple might choose to “make no longer compatible” with a software update, and no matter what breaks — you have no tech support except your own sweat equity and whatever friends you can buy pizza for. And you’re using this computer in a production environment on deadline?

Wherever your choose to draw the lines in the sand in the great “How dare you do that with my photos; but I”ll do what I want with this software!” moral quagmire, you should at least stop long enough to think about it so you know how to explain it if it gets brought up by a client — or by the other party if you happen to end up in court fighting a copyright and this is mentioned to the judge. Whatever you think of them, these EULAs have been mostly upheld by courts. How are you going to react if someone uses the same rationalization for using your photos that you used for choosing to build a Hackintosh?

But I’m not judging. I have enough challenge manging my personal ethical compass, I don’t need the karma of managing yours. But I felt it was important to point these issues out so that photographers understand that this is more complications than “this is unsupported hardware”.

I, personally, would hate to be in a conference room negotiating licensing terms with a client and taking notes no a machine that has unlicensed software on it, or is running software that I knowingly installed in violation of the licensing terms. That to me seems like I’m tempting the karma gods, and they already have me on speed dial, they don’t need excuses to ring me up. You know?

Tags: Photography, The Online Life

The Apple TV has not failed…

One of the memes I’m seeing in the discussion of the iPad is that the Apple TV is one of Apple’s failures. It seems to be a common idea and an easy target, but I think that idea is dead wrong. Yes, it hasn’t sold 800 billion units like the iPod and the iPhone, but that it hasn’t been an insanely successful product doesn’t make it a failure.

(quick digression; when people decide to go talking about “apple’s failures”, the common commentary is something like ‘Apple’s failed products like the Apple TV and the Cube’ — when you realize that the Cube was released in 2000 — that’s ten years ago — and people looking for anything to criticize in Apple’s product line can really only come up with two examples in a decade, well, that says a whole lot about Apple’s success, no? )

I’ve got two arguments why the Apple TV isn’t a failure. It’s subjective and certainly open to discussion, but hopefully this will cause you to stop and consider…

First: Apple doesn’t consider it a failure. If it did, Apple would have dropped the product and moved on by now. They’re still selling it, supporting it and enhancing it — so Apple clearly sees a future to it. Otherwise, it wouldn’t be available to buy.

Second: I’d argue that for a product to lose, some other product has to win. The product that beat the Apple TV is…. It is… Um…

See? The answer is “nobody”.

And that’s why Apple still has the Apple TV and what gives the people who like making immediate judgements on things the quivers. The market the Apple TV is in is still forming. Nobody has won. Nobody has lost. The fight is still in the early stages.

I’m frankly a little surprised at this, I thought the market would mature and mainstream faster than it did. One component that has slowed it down is the lack of standardized interconnects — i.e. the failure to launch of the Cablecard. It’s not clear to me if Apple was ever really looking at the Cablecard as a solution here, but if they were, it didn’t happen and it never really became an option.

So Apple’s real solution here is downloadable content, and that’s dragged by the relatively slow adoption of fast/cheap broadband in the states. It’s also dragged by the owners of the video content being in no super hurry to hand over control of the market to Apple the way music was handed over to it. the music studios hate how Apple can dictate business to the studios instead of the other way around, and so there’s been this deadly and slow dance for control between the studios and Apple, and since the volume (i.e. “money”) isn’t there, the studios can take their time and push for better deals and hope for alternatives so they can play someone off against apple for leverage. So far, nobody’s really come up with something that remotely competes with iTunes in numbers and scale, though.

What can push this market forward is a change in the dynamic. Apple TV wasn’t the product to drive adoption of iTunes for video on a mass scale, so there’s no strong incentive for studios to buy in and get on board. Because of that, it’s been a slow and steady grind to get content into itunes, so things move in slow motion.

But just suppose Apple were to come out with another product, one that hooked up to iTunes, was a good experience to watch video on, was priced in a way that the general consumer would buy it — and sold a zillion units? Suddenly the studios are going to hear cash registers, and more importantly consumers complaining loudly about the things they can’t watch because they’re not in iTunes. And that creates incentive to cut deals to make it available, because now there’s demand (and revenue). And that demand (and revenue) puts titles in iTunes, so suddenly the iTunes/AppleTV option is a viable alternative to Netflix or pay per view.

So my argument isn’t that Apple TV had failed, but it was waiting. Waiting for something to come along and do what Apple TV alone couldn’t do, which was drive demand and sales and rentals via iTunes to generate revenue which attracts the studios which brings in the titles which generates more sales of units which an Apple TV can leverage because the consumer wants ot be able to watch their movie both on their — device — as well as their TV without buying it twice.

And it seems to me Apple just announced that device. And that device has the potential to create the environment where Apple and the studios can sit down and work out getting all of the content into iTunes for consumers to consume. And when they do, suddenly people will realize Apple has this device they sell where that content also will end up on their TV’s!

And gee, Apple just happens to have it sitting there, waiting for consumers to discover it. And because Apple, unlike the pundits, realized the market was still creating itself and was willing to be patient, it has a product there and ready to succeed when the market matures enough to allow it to. That’s a LOT easier than trying to create a product to catch a market as it explodes any day…

So if you ask me, Apple’s stupid like a fox here. It knew that sooner or later, it’d need the Apple TV. It put it out there, it learned from it, it let it help Apple figure out how to create and own the market and bootstrap the functionality they needed to do so (like video rentals, which now exist and are sitting there waiting for the tablet. That wouldn’t have happened without Apple TV being there to implement it for). And when the market starts to grow because tablet sales drive content sales whichget the studios on board which drives consumer interest (and tablet sales which drive content sales which….), Apple can introduce an updated Apple TV to take advantage of it and start the buzz and hype to push it into the success curve — and because they started the process years ago and were patient and didn’t cut off support of the device when it wasn’t an immediate insane success, they’ve made these next steps a whole lot easier for themselves…

Apple TV isn’t dead. It’s in make up waiting for the second act to begin.

Tags: The Online Life

Some thoughts on the iPad…

Or the night after the Apple tablet…

I thought my view of what was coming that I posted last night was pretty darn close, if I do say so myself. With great amusement I’ve been watching the usual suspects say the usual things; the people who live inside the geekdom echo chamber forgetting there’s a real world out there, and Apple tends to build products for the real world, not the self-appointed geek universe.

A few themes within the critics caught my eye, all of them (I think) incorrect. One are the people who really want the tablet to be a replacement for a laptop, and because this isn’t that, it sucks.

This device is a new category, aimed not at the people who spend their life madly typing in their blog while watching a video AND listening to Pandora and madly checking ot see whether their deathless prose is being appropriately retweeted by their adoring followers. It’s aimed at people who — believe it or not — actually want to sit down on the couch or in their hotel room after a long day at work and…

gasp.

RELAX. They want to read their email. They want to browse a few web sites, check the scores of their hockey team, maybe read a book, maybe watch a movie. Take it easy and — do what people did 20 years ago before the velocity of life ratcheted up to the point where some people think that if you aren’t doing 30 things at once you’re lazy.

Well, hint: in the real world, where most people actually live, people do still sit down in the evening, unplug, and read a book or watch a movie. And actually feel guilty doing both at once. There’s a whole bunch of folks within the geekdom echo chamber who’d be a whole lot happier and less stressed if they figured this out, too. But they’re too busy blogging while watching a movie they’ll only half remember a month from now.

This is a device not for geeks, but for consumers. It’s for people who use devices, not hack them. It’s for people who consume content, which is actually most people, as opposed to geeks who want it to be something it wasn’t designed to be. So lots of geeks are disappointed and blogging about it, while I expect this thing will sell many, many copies, mostly to people who won’t blog about it, but merely use it.

Another theme I’m seeing tonight is the lack of flash on the device. No surprise. If you really want to know why, think back a few years when Apple was trying to get back on its feet, and Adobe made a decision not to support its video products on the Mac, and instead tried to convince its mac customers to switch to PCs. Apple’s response then was ultimately to bring out its own video products — final cut — and ultimately ate the market out from Adobe. Later, when Apple was making the conversion to the intel platform, Adobe’s enthusiasm for bringing out Photoshop and its flagship products was most noticable — by how late they were and how uninterested Adobe seemed in actually trying to help Apple succeed. So now, when Apple has these really successful platforms and Adobes wants a piece of them, and yet Apple shows no real enthusiasm or hurry to cooperate? Well, folks, payback’s a bitch, and if you only see your partners for what they can do for you today, well, don’t whine when they choose to return the favor when the shoe is on the other foot. Burn your bridges with thought, folks, because you never know when you might want them back. And they’ll remember.  Apple sure does. And wouldn’t it be great irony if Apple uses its platforms to turn Flash from a success to an also-ran by supporting HTML5 on platforms that are in enough demand that people who currently are building flash-based things end up recoding those things away from flash to support the platforms people are demanding? Just like — oh, say — Youtube just did? Hmm.

A final theme I’m seeing is the geeks defining products as successful or failure. The Apple TV is being tossed about as a failure, even though, every time I look at estimates on unit sales, it’s still outselling Tivo and has been almost since launch. Yet it failed, Tivo is what the geeks keep saying the Apple TV ought to be. Hmm. Apple could use a few more failures like that. Especially given that I agree with most of the geeks that much of the potential of the Apple TV line of products is still ahead of it. Maybe the Apple geniuses were busy on some other product line. Like, oh, maybe a tablet…

Finally is a recurring theme with some that Apple didn’t “blow them away” (and therefore, I guess, this sucks). Folks, you all need to reset your internal adrenalin meter back from 11. Some of you would take anything less than being personally tasered by Steve himself as “boring”. One word: decaf. Not all products and not all announcements have to be over the top. There merely have to be damn good products.

This one is. To me, it’s a perfect device for my mom, who lives and dies by email, yahoo, access to recipes on Food TV, wants her audiobooks and to read Stephen King and Jean Auel novels and watch the occasional movie (and Emeril). THERE is your target audience.

Me? I like the idea of having one. It won’t replace my carrying my laptop on the road, but it’ll give me something I can use while my laptop is processing photos in Lightroom or crunching away at some compile for a program I’m writing. I doubt I’d write a novel on an iPad, but I’d sure write a blog entry and catch up on email. It supplements why I need a laptop wonderfully, and means I won’t need to worry so much about bad cable TV in a hotel room or hauling books around when I travel. It’s a nice supplemental device for my life. For a traveller who’s content creation issues aren’t so — intense — this very well could replace carrying a laptop. If your job is about writing email, memos and presentations instead of Ruby, HTML and Photoshop, you’re probably already ragging your boss to get approval to get one. Or should.

Nope. This isn’t a sexy repackaging of a laptop. It isn’t a tablet-PC. it isn’t a “netbook done right”. It’s an entirely new type of device, and I think it’s going to be rather successful. now, two or three generations down, it well COULD become those things; I could see down the road these things having the potential to make Mac OS X obsolete and running whatever Lightroom becomes and doing the heavy hitting, but right now — it is what it is, and what it is is very good if that’s what you need.

I think it blows away the Kindle, and I wouldn’t be suprised if Amazon doesn’t quietly breathe a sigh of relief that this lets them get away from building devices and go back to what it’s really good at, which is distribution. And I think it effectively kills “unitaskers” like the Epson P-4000 and digital wallets. Why buy that when you can buy an iTab that does it ALSO? Maybe not for the high end user, but for most of the market, definitely.

All in all, I’m impressed. and looking forward to getting my hands on one. One thing I’m going to be curious to see is whether this thing is going to be allowed to take on the Mifi. If I could use it to wire up a wireless network to 3G in a hotel room (even if I can’t use the iTab for other things!) to handle the work to the office, that’s gravy. Then unplug the laptop for the night and use this beast for recreation (and to prepare tomorrow’s presentation for the sales meeting!)…

All you folks dissing the device, I think you’re looking at it wrong. Here’s a hint: Steve’s not stupid, and knows what real people want. And isn’t afraid to offer it. And this is, I think, it.

Tags: The Online Life

The night before the Apple Tablet…

I tend to shy away from talking about Apple much these days because of the possible conflict of interest issues, but I wanted to say a couple of things about the announcement tomorrow.

Derek Powazek sums up my — anticipations — of the product very well.

The typical run-up to the announcement, with the leakers and the guessers hyping each other into a frenzy until people start trashing the product before it’s even announced (because they’re basically tired of hearing it) is in full force. I’m glad I don’t have to deal with all of this stuff any more, except as an amused outsider. But as an amused outsider, I count myself amused, but frankly, I tuned it out days ago because it’s so over the top and silly, after a while, it just stops being interesting.

The tablet looks to be the creation of a new market segment, and I’m going to be fascinated to see how it’s positioned and how well it does — not to mention how well Apple does it. Right now, we have these broad usage capabilities:

Life in your pocket — your phone, which increasingly allows you to carry your essential stuff around without hauling a huge beast to try to manage it. That this data syncs up to other places where it’s available on your other electronic environments is great, but ultimately, this is about managing who you are and what you do in a portable format. We’ve made great strides at turning these pocket devices into information consumers as well, but the small screen makes that a set of compromises. They are also — bluntly — pretty crappy at content creation because of the compromises needed to fit in your pocket. the thought of blogging via my phone doesn’t intrigue me. The thought of writing a novel on my phone scares me.

Portable content creation — your laptop. Carry your office with you. I remember long ago when I got my first PowerMac Duo, which in many ways was a netbook 15 years before anyone thought to invent a netbook. Loved that machine, and the docks, because for the first time I could carry my life with me and turn it back into a desktop when I wasn’t mobile. I still strive for that model today, rather than keeping multiple computers and trying to keep the data in sync. Today’s laptops have enough power and a good enough screen than you really CAN turn one into a portable office with few (if any) compromises.

Desktop content creation — the iMac, the mini, the mac pros. We’ve seen this class of machine shrink out of prominence over the last few years because, frankly, laptops have replaced them for most people. Plug a laptop into a monitor and you have the best of both worlds, large screens AND portability. iMacs continue to have some popularity because there are times and situations where portability isn’t a feature (kids being a big part of that). Mac Pros exist, but are clearly a power user (or ego user) niche now; few of us really need that kind of oomph.

What’s missing here? There’s a huge class of user that’s never had a product designed specifically for them. Tomorrow we’ll see the first one.

That’s the content consumer.

Laptops are aimed at creation, they  carry a lot of “stuff” with them that are underused by a lot of people, starting with the keyboard. If your primary use of a keyboard is typing in URLs or emails, you don’t need all of the bulk and mechanics a laptop keyboard bring along (not to mention weight and power consumption and…). The iPhone model (software keyboard, etc) work fine here, but the iPhone form factor for the screen creates other compromises that make the phone tough for these people (but great for the “life in the pocket”).

The thing that kept me from buying a Kindle was simple — it’s a unitasker, and while it does it quite well (and I have the kindle software on my iPhone), I don’t want multiple devices to do the different things I want in this usage space. The reason I think the earlier attempts at PC-based tablets didn’t take off was because they were really “laptops in a tablet”, not tablets designed for content consumption — and just created a new set of compromises that most of us realized made them — compromised.

That’s where I think this tablet lies: content isn’t “books” or “newspapers”, it’s web, it’s video, it’s audio, it’s games, it’s text and content. And this device is going to be all about consuming content, and all of it in a single device.

If it is, it’s going to sell zillions. It’s going to cannibalize laptop sales to some degree, but that’s a good thing. It’s the kind of device that I have wanted for my mom, who’s primarily a consumer of infomration and doesn’t need the complexity of a Mac (much less a windows computer).

Would I buy one? Depends. I’m a content creator, and I’m rarely without my laptop. A device like this isn’t going to be optimized for the kind of things I’m doing (especially my photography, I don’t see this as a device particularly interesting for serious photography geeking), so it’s interesting only to the degree I can’t also do these things on my laptop; it’s not a replacement device, but a supplementary device. But then, I bought an Xbox 360 for gaming as a supplement to my Mac for computing, so who knows….

To the degree that this device makes using your content as painless as your phone makes managing your email/contacts/calendar, it’ll be a huge success.

One thing I’ll be fascinated to see, perhaps not tomorrow: how much of what they do on this device also ends up on my Mac. the closer they come to a “virtual tablet” on the mac (via iTunes?) the less I need one, but I can believe ultimately I’ll have one because I do like to sit down on the couch with a good book, and I’ve never really found a way to do that comfortably with a laptop.

I don’t think tomorrow’s device will solve the final problem — taking a good book with me into the bathtub for a soak. But who knows? maybe that’s a third party opportunity.

What my gut tells me: tomorrow’s announcement is going to change things significantly, is going to be hugely successful, and many people are going to trash it because they don’t get it.  This may turn out to be the biggest thing yet. And given the things that have come from Apple (and Steve) over the years, that’s saying something.

Tags: The Online Life

Proposals For Librelist Moderation Strategies

To understand the feature requirements for moderation we need some goals. Keep in mind that no moderation will be perfect, and you can easily come up with scenarios that will work around anything we come up with.

Therefore, we should focus on just some initial goals that will work right now, and keep in mind that these will need to be constantly tweaked and worked on as the spammers evade the measures.

• If given the choice between restricting free speech and preventing unwanted communication, free speech always wins.
• The system should increase the quality of discourse for any project, regardless of human language used.
• It should never give a small group the ability to hide communications from others.
• It should be implementable and not have high hosting costs.
• It should not rely on a dedicated person’s constant intervention.
• It never gates email through system before sending it, but rather allows initial emails with moderation after.
• It should use information from people’s rating habits to classify them as “ratings trolls” to prevent abuse.

With those goals in mind I’ve teased out two potential list quality strategies that might work.

via Proposals For Librelist Moderation Strategies.

Someone I work with turned me on to Librelist because they knew me interest and history with mailing list systems, and I find it interesting that some folks have decided it’s time to rethink the mailing list again.

They’re right. When I faded to black on the mailman project, it was at least in part because many of us felt that mailing lists were a technological dead end, and that deliverability issues because of anti-spam systems made the “personal mailing list” an increasingly difficult thing to accomplish.

Both are — for the most part — true. I certainly would never run my own mail server again, because the advantages of doing so are far outweighed by the time and hassle of trying to manage deliverability and reputation to make sure mail it sends gets accepted, and the constant onslaught of incoming spam turns them into a permanent infinite time sink. That’s why I either retired our lists or moved them to Yahoogroups (which I personally think is a pretty good system).

But there’s still room here to rethink the concepts and the Librelists seem interested in trying, and I think that’s great. Email and mailing lists are far from dead — but instead of stand alone delivery tools, they really shine as part of an integrated web strategy; Yahoo groups is a nice first generation of that, although there’s a lot more Yahoo could do if they decided to.

Message moderation really breaks down into two big problems:

• “Subscribe spam” where spammers sign up to the list to spam it.
• “Member warfare” where existing, approved members get into fights and they escalate into unacceptable territory.

The first is really simple to solve: new members are moderated, and messages aren’t posted until reviewed by someone to vet their content. Simple implementation; Yahoo Groups does it today, and on the lists I still manage, it works well to keep the spammers at bay. The way I manage it is all members are moderated until their first post. if their first post is acceptable, I turn off the moderation bit. To minimize delays in propogation of new member messages, simply choose a moderator pool large enough to guarantee held messages get reviewed and approved in a timely manner — you could even make that moderator pool all members in good standing if you want, because all you really need is someone you’ve trusted to post vetting that someone new is trusted to post.

Member warfare is trickier. I hesitate to call it trolling because the pure troll is a subset of the larger issue of two (or a small group of) people getting pissed off and going at it. A troll is simply one person going off on the rest of the list.

I’m more and more convinced the answer here are reputation systems, where over time a user’s membership in a group is used to define their abilities and restrictions. The longer a member is in the group in good standing, the more often they contribute material, the higher their reputation goes and the more the can do and the more sway they have on the decisions of the reputation engine. You can tweak the details of the algorithm almost any way you want, but if you define it in terms of “how long they’re a member” and “constructive contribution to the community”, you can come up with a metric on how valuable that member is to the community, and then use that to rank that member’s contributions and recommendations.

Here’s one rough view of how to build this. Please note that I firmly believe karma rankings are private and users have no way to see what their ranking is or compare it to others, except in really broad user categories (“member”, “senior member”, “top contributor”, “advisory board member”). As soon as you create a list of any form, you will attract people who see it as something they can game, and so they will.

User Karma is a value between 0 and 1, which starts at 0.5. Every time a user contributes to the system (a posting, a reply, a moderation recommendation, etc), the number gets bumped by some value. How much the value is incremented or decremented depends on how it’s rated by other users — so if User A posts a message, User B flags it as spam, but 80% of the membership feel that was a bad decision, User B’s karma is reduced in future decisions, they lose influence. Over time, the system self-corrects by giving increased influence of those who’s decisions match the community consensus and reduced influence to those who’s postings and recommendations don’t match up well.

The system can then choose whether to accept or flag for moderation a posting based on a poster’s karma score. You could potentially reject outright users that have karma scores below some value, or allow other members to choose not to see messages by users with karma scores below some value. Over time, users who are disruptive to the community will get karma’ed into the moderation queue (or out the door), and users who are seen as top contributors will have stronger influence.

My goals:

• A system like this can be built nicely with a good SQL backend and a bit of horsepower. I’ve actually done a detailed design and schema on this before, and it’s a fascinating thing I’ve always wanted to implement.
• It enables the power of individuals to police themselves.
• It limits the ability of an individual to harass or cause problems.
• It doesn’t lend itself to people playing the game of gaming the system by not exposing the details of the system (slashdot karma whores need not apply).
• Trolls get edited out of the system because the community will quickly recognize them for what they are and trash their karma, causing their postings to disappear to the bottom of the list.
• Cliques and Mafias have to be large to influence the results significantly. You don’t completely avoid the clique/mafia problem, but you can severely limit it’s ability to wreak havoc.
• It doesn’t require a lot of manual handholding or babysitting. Admins end up stepping in only in extreme cases.
• Because trolls tend to get edited out of the system quickly and automatically, they tend to go elsewhere because without feedback and controversy, they wither and die. And by editing them out of the system quickly, you avoid the whiplash and fighting that happens when people start fighting with the trolls and the wars break out.

Weaknesses:

• Any community tends to turn into an echo chamber. Automated systems like this encourage this because “different thinking” tends to get rated down.
• That’s usually a lesser evil to letting the trolls run wild.
• To my knowledge, nobody’s ever solved the problem of the conflict between the group-mind reinforcing the echo chamber and allowing the free thinkers to poke at the community’s comfort level by pushing them to think about things that make them uncomfortable. One person’s rebel is another person’s troll, and that’s not solvable in real life, much less in automated life like this…

These techniques are all based on (or stolen from) things that are in use around the net, with Amazon’s review feedback being one I really respect; while trying to avoid the pitfalls I’ve seen around the net (yes, I’m going to keep bashing on Slashdot’s karma system, it’s way too easy to game and always has been). It also (I believe) avoids the nasty politics that have made Digg a bit of a pesthole. And it’s also pretty lightweight and low-key, or at least it should be. The implementation details will be crucial, as will be tuning how the karma values adapt…

Tags: About Chuq, The Online Life

Protecting mailto links (my advice: don’t)

Got this in email the other day, decided the answer might interest some of you.

I actually just had a quick random question about your Contact Us page on chuqui.com

I agree about not putting a phone number on a personal or small business site unless you are prepared for the idiot factor.

Since yourself and of course myself too are all too familiar with the world of spammers I was wondering why you don’t obfuscate or somehow protect your mailto: link?

It’s a serious question, as I am actually wondering if you do want to see how much spam will come to it and which types of spam?

good question, complicated answer… Part of it is that my email addresses have been “out there” for so long — I’ve owned plaidworks.com since 1995, for instance — that I assume I’m on every spam list in the universe, because, from what I can tell, I am. So why hide when it’s too late already?

I also think those obfuscators are fake-security. Anything you can build programmatically, they can unbuild programmatically. All they have to do is care enough to try. They really don’t fix things, but they make you feel better, and over time, they get compromised — so you add complexity to things and in the long run, it doesn’t really solve the problem. Or it does, for a while, but how do you know when it stops working?

I don’t see any purpose in having an arms war with someone who can out-gun you from day one. I’d rather put my time into useful things.

So here’s what I do:

I hire someone else to worry about it.

I don’t believe it’s possible for an individual to “win” a way with the spammers. Or even “break even”, or even stick with “moral victories” for long. Even if I could, I’d much rather put my time and energy into other things.

So that means having your email hosted by someone who does have the resources to fight spam. I currently have three email hosts: gmail/google, mobileme/Apple, and my personal ISP (plaidworks.com/chuqui.com). Of the three, the personal ISP has the most leak-through, but they honestly do a good job and I have no complaints, given the complexity of the task.

Apple/Mobileme uses Brightmail for filtering (unless things have changed), and Google uses Postini, which they bought a few months after I turned down a job at Postini to work for Strongmail instead. Both groups have organizations individuals can’t hope to do better than (IMHO), no matter how much the geeks think they can “better mousetrap” the problem. My experience shows it to be a situation with rapidly diminishing returns for constantly increasing resource commitments.

So let the experts handle it. Then, realize it’s never going to be 100% perfect, and don’t get your knickers in a knot when it really IS imperfect. A few pieces of spam sneaking through won’t kill anyone; the stress you get spazzing out over the spam just might.

Right now my final mailbox lives on gmail, because it works best with my webos/Pre phone. When I was living on an iPhone, I used MobileME’s mail server. Depending on where I live, I have the other servers set to auto-forward to the final repository, and everything works pretty well.

In reality, the anti-spam aspects of email work pretty well now if you’re involved with a mail host that has their act together. Many corporate environments don’t. Most geeks fighting this battle on their own don’t (and complain about it loudly, so I think the general view is it’s a lot LESS solved than it is). Living on a mail host run by pros costs a few bucks (well, it doesn’t on gmail, but you get ads. I would happily pay a few bucks to do away with them..) but I’m a lot more worried about spending time than money in most cases.

Things like mail obfuscators never really worked well; they might have been ignored by spammers, but if the spammers decided they were worth investing in cracking, they got cracked. Very few geeks who installed them actually did any kind of scientific testing on how well they worked, they noticed no spam in their boxes for a few days and declared victory. A month later? three? six? Compared to non-obfuscated control addresses?

shrug. very little science here. Including myself. What science I do have is a couple of years old and pretty thin as well, so I don’t declare myself an expert, but when I did experiment, I just didn’t see anything worth the time investment, not compared to just putting my email on a server where a staff was in charge of solving the problem for me.

The proper place to solve the spam problem is on the incoming connection; even if you do obfuscate, all it takes is one mistake to leak, or someone else to leak it FOR you (and I found those leaks everywhere when I was tracking this stuff; painfully sad) to require having to do the incoming filtering as well. If you have to do that anyway, isn’t the proper answer to focus on doing that better and not do things that ultimately don’t really help solve the problem?

My bottom line: you aren’t going to keep email addresses away from the spammers. Trying to do so is a false security solution, and ultimately a waste of time and energy. Instead, it’s keeping spam out of the incoming email stream, and if you do that well, you don’t need to worry about the addresses leaking. So I don’t.

Tags: The Online Life

Following my own advice on backups….

While writing my article on backups (and it’s followup) I decided some of my practices weren’t what I wanted them to be. The primary issue was the online catastrophic backups, which used Jungle Disk as a front end to Amazon S3 for storage. I really like the setup — Jungle disk was almsot flawless in doing what I needed the way I wanted it done, and S3 was reliable and backed by Amazon, so I didn’t need to worry about the “not here tomorrow” problem you sometimes have with startups.

But there were a few negatives: the cost — I was spending about $17/mo on storage costs with S3, plus about $1.50 a month to Jungle disk for their advanced features. I also worried about the occasional delays (I was only uploading about a gigabyte of fresh data a day, so a few days of heavy photo shooting could cause backlogs before the data was stored offsite; uploading the network is an option, but costs. Everything costs…) and finally the time it would take to recover via online recovery if I ever needed the backed up data bothered me.

So I decided to move to a simpler strategy: clone my disks and keep them offsite. Over the weekend I disabled and deleted the S3 store and turned off my Jungle Disk setup, and ordered a new drive.

Not an offsite drive, which might surprise you, but a new internal drive for the Macbook, a 340 Gig Hitachi 7200RPM drive. Why, might you ask?

Well, let me tell you: if you read the previous pieces, you saw where I noted that one of the best ways to never NEED your backups (always my preferred policy!) is to replace my primary disks on a regular basis; my laptop drive was (over)due, so it made sense to replace it before it failed. By upgrading to a larger internal disk, I could take the files I’m currently storing on an external firewire drive and put them back on the laptop drive.

That simplifies my computing universe — fewer spinning things hooked up to computers, fewer places to misplace data and one less “i have that data, but it’s not with me” opportunity. And it means I can repurpose that firewire drive for my offsite backup.

Amazon had a nice deal (about $75) on a Western Digital 320gig 7200RPM drive with a 16 meg cache (hmm. old phart warning: the first hard drive I ever bought — for a Mac 512K that plugged into the floppy port!) was a ten Megabyte drive that was wonderful and had more disk than I could ever think of using, especially compared to floppies… Now that drive is smaller than the performance cache on a hard drive… wow).

I could have gone as far as 500 gig, but that changes the price/performance,a nd even with copying all of the files off my secondary drive, I still have 150 gig free. By the time I start worrying about the disk filling up, the bigger drives will get cheaper, or I can simply buy a nice cheap external and split it up again. But for now, I’m happy on a single drive, a single backup drive, and a single offline external archive drive. Plus backups of each, of course.

I wired up the raw drive via USB to my laptop and used superduper to clone my primary drive. Then I let everything sit for two days with the new drive spun up, refreshing it daily with superduper — because infant mortality on your laptop drive really sucks, and giving it a couple of days before opening up the guts and swapping things is a bit of insurance. Just saying — nothing like putting your laptop back together and having it fail (or not boot).

A bonus feature of this change: the old drive in the laptop was a 5400 RPM drive. Upgrading to 7200 RPM improves the I/O characteristics and speeds the overall performance, especially if you’re doing things that eat lots of virtual memory (like, oh, photoshop or lightroom, or running both). On a mac, check /var/vm and see how many swapfiles you have and how big they are. The larger your VM set, the more your disk is going to affect overall performance, and in many cases, a slow disk is the real problem to performance, not lack of RAM. hint: people how are proud of NEVER REBOOTING THEIR COMPUTERS are never re-initializing their VM environment. Silly boys. I’ve seen major complaints about slow performance in firefox and photoshop magically disappear on computers that were simply rebooted. Just saying.

So for $75, I remove a $20/mon charge to pay for the online backups, I add enough disk to my primary computer to keep me for at least six months or longer, added a faster disk to speed up overall performance of the computer, and I really didn’t do anything to weaken my backup strategy. All I need to do is remember to bring the catastrophe disk home once a month, refresh it, and get it back offsite again; I can choose to do it more often if I finish a significant project, too. And that’s very manageable for me — just put repeating tasks in your calendar to nag you, right?

There is one other upgrade I’ll need to make, which is my bus-powered backup drive is now too small to fully backup my primary drive, but I don’t need to worry about that for another 50 gigs of data or so; that’ll cost me another $75 (or less by then) down the road.

So I now have my primary disk (320 gigs) with a time machine backup and THREE superduper cloned backups (one offsite, one nightly, one weekly); I also have a pair of 500 gig drives which store all of my long-term offline archival stored data; they’re clones of each other, and one is stored offsite. These hold the files I don’t ever expect to touch again but won’t throw out for a while just in case (the import backups of all of my photo shoots, for instance), so they are in fact more redundancy. Just in case, you know?

The old laptop drive? It goes into the anti-static bag and gets filed with all of my yearly paperwork and taxes; it’ll stay there until I dispose of this year’s paperwork down the road, at which point the drive will have a bad date with a big hammer and some other power tools and get tossed in the trash. I much prefer my hard drives not end up in hands of strangers (if I do give a drive to somene else, it’s generally after a multi-pass, multi-pattern write/erase sequence, but drives are cheap enough, I kinda feel they’re not worth losing track of until you know they’re destroyed, and there’s no sense destroying until later, because you never know when you might need it…. It hurts nothing to wait…)

So I’m following my own advice and fixing my own backups — again. Using simple technology to reduce “failure by complexity”, using as few mechanisms as reasonable, using multiple redundant backups and backing up on different time sequences to avoid the problem of corruption not found until too late — and keeping copies off site, without being anal about it.

It’s probably overkill, but disk is cheap enough now that it’s cheap insurance. And I’d rather have one too many backups than one too few.

Tags: The Online Life

Comment behaviour: How far is too far?

According to Greenbaum’s blog post (which was mirrored on his personal blog), someone posted a comment on a story in which they used a colloquial or slang term for female genitalia. It was deleted, but then was reposted. Greenbaum says he noticed that the comment alert from Wordpress showed that it came from a nearby school. So Greenbaum called the school, and they asked him to send them the email with the comment, which he apparently did. About six hours later, he says, the school called and said that an employee had been confronted and that he had resigned.

Am I the only one who thinks that doing this goes way beyond the normal course of editorial behaviour?

via Comment behaviour: How far is too far?.

There’s been some interesting commentary on this case — but there are some aspects that I think haven’t been addressed very well yet. It’s a more complicated situation than many have considered, and the answers really aren’t clear cut.

Here’s my take:

There are really two separate issues here.

Did Greenbaum over-react by reporting this person to his employer?

Yes — but.

Yes, he did. In the grand scheme of things, reporting a violator back to their host is a serious thing because it can have serious implications — like getting someone fired. Which effectively happened in this case. So it’s a last resort thing. Before you do something like that, I prefer taking many other tactics first:

• Delete the post
• Warn the User
• Ban the User (and ban the IP and/or IP range as necessary)
• Make it clear that if it doesn’t stop, they’re going to be reported

If those all fail, or if for some reason aren’t possible, THEN you start considering going back to the user’s host for support in making the behavior stop. As far as I can tell, only the first was tried, so a number of (to me) necessary steps were skipped. This could have been ended with much less serious ramifications, and wasn’t.

However, here’s the butt:

• The post was deleted, and the user insisted on putting it back. The admins made it clear it wasn’t acceptable, and the user decided to overrule their authority. This user was far from innocent here.
• Once the user is reportd back to their host (and I use that term carefully, because it’s many times unclear if it’s an employer or what, and to some degree it doesn’t matter if it’s an ISP or a boss or whatever), it’s out of Greenbaum’s control. The rest of the escalation to losing the job was the result of actions of the host (i.e the school, or this person’s boss). None of that is caused by Greenbaum (directly) or his fault, beyond that he should have been sensitive to the fact that his action in reporting might have caused other actions to happen.

So, you know what? I think Greenbaum’s transgression is a lot less serious than the user’s transgression in reposting his vulgarity after it was made clear it wasn’t welcome. I would have tried other tactics to cut the abuse, but let’s not forget that it was abuse, and it was repeated abuse after the site made it clear the posting wasn’t welcome. Whether you shoot over someone’s virtual bow one time or three times is a minor thing in the scheme of it.

The user’s fault in this problem was a much bigger problem than Greenbaum’s reaction.

But what about the school? They’re the group that took the complaint and escalated it into a situation where the person lost their job. None of that is Greenbaum’s fault. Was the school wrong for turning this into a termination issue?

I’m not so sure. It’s easy to say they over-reacted, but let’s not forget:

• This person did this using the school’s network
• It looks like he did it while on duty at the school – while he was being paid by them.
• He likely was on a school-owned computer
• He was (I’m sure) under some kind of employment contract with behavior clauses. The school very likely has acceptable use standards for computers and networks, and for all we know, also personal use restrictions (which this would be a violation of).
• So while this cascaded into a situation where someone lost their job, it’s not at all clear that the details of the action were the cause. We also don’t know if this person has a history of previous violations of work rules that might have been part of this. Has this person been warned about this kind of behavior before? We don’t know. It could well be from the school’s view that this was a “last straw”. We don’t know.

And those complications are why I believe reporting back to the host is something not to be taken lightly; once you do, the final outcome is not really under your control. On the other hand, the person who could have prevented this was the user who posted the vulgarity — either by not doing it in the first place, or by stopping after it was deleted the first time, or by being smart enough to not do it from his employer on company time and company equipment. He had plenty of opportunities to not turn this into what it was; Greenbaum had one.

And it’s not as simple as many of the folks commenting on it want to be. Real life never is…

Tags: The Online Life

Some thoughts on Google Chrome OS (epecially for photographers)

Google Chrome OS was announced today and everyone is talking about it. Lots of interesting thoughts going down. Here is mine:

I have never been interested in the “netbook” because I’m MacOS-centric and I saw the netbook as a series of compromises from the functionality I want in a laptop that I found unacceptable — AND I believe that 95% of what most users see as the core functionaliry of a netbook, I see as functionalty that belongs on my phone. the netbook is at best a transitional technology, and smartphonees are quickly taking on most of what netbooks are trying to do today. I don’t like the price/performance of a netbook. Hell, I just don’t like the performance very much, not when I can already do most of it on an iPhone or webOS or Android.

As a photographer, I have also never been interested in buying so-called digital wallets such as the Epson P-7000; too expensive for limited functionalty, and to me, it’s more cost effective to just buy enough cards to handle the sitaution and wait until you can get back to the laptop to offload them and make a real backup.

But I take one look at at the Chrome OS type devices and I see something that can be set up as a rather nice digital wallet — PLUS give you access to some key computing capabilities (email, web, wifi, file uploading) and suddenly, for about the cost of what you pay for a decent digital wallet today, you get something that can hopefully act as a wallet AND access to some key computing features as well? Being able to offload pictures and send a few samples online via gmail or upload to smugmug from the field?

NOW you start having something I’d consider having. Neither device alone is worth the cost of buying it to me. But Chrome OS looks like it’ll have the capability to make these two devices one, at about the same price point of either — and now I’m interested. And it’s going to be a while before phones can take on the capacity we’d be looking for to do the digital wallet capabilities, so this is something a good smartphone can’t yet do.

If I’m a product manager for a digital wallet today, I’m looking at the end of life for this type of product and trying to figure out how to move to this new merged capability before someone eats my lunch. If I’m a developer for these new Chrome OS devices, I’m looking to see how to make it act like a digital wallet, because photographers are going to want to buy this.

And if I’m a photographer, I’m watching and hoping this happens, beacuse this has the potential to be a really nice addition to the photo bag that could allow me to stop carrying my laptop in the field but still be able to do a lot of what I currently want while out shooting.

Count me intrigued and hopeful here.

Tags: The Online Life

Next Page »