Followup on: MobileMe Problems Show Apple Needs an Infrastructure Lesson

Chuqui 3.0: MobileMe Problems Show Apple Needs an Infrastructure Lesson:

That this release was botched isn’t about Apple not having a clue, but about the MobileMe people either blowing it (I can think of any number of scenarios — scaling it hard). The ultimate failure seemed to be more capacity planning mistakes than anything else, if I’m guessing right. but the ultimate failure was not being willing to tell Steve “we aren’t ready” and taking that heat. They thought they could release and make it work, and guessed very wrong (or thought they were in good shape, which is worse).

I want to thank everyone who’s read, linked, emailed or commented on my thoughts on MobileMe the last few days. it’s been really interesting to see the reaction and hear the feedback. There’s a great and fascinating comment thread on the posting that I encourage everyone to read.

I have to admit that the first reaction when I realized that this thing was going to get huge readership was “man, am I going to get in trouble again?” — then I remembered I didn’t work at Apple any more. But old habits die hard, it seems.

The feedback from “the inside” was heartening. thanks, all. And to those of you emailing me from your apple accounts, what were you thinking? (grin). I’ll simply say that the reaction from that quarter indicates to me I was fairly close to the mark, and leave it at that.

And now, to make a few comments on the comments…

I think the credit card transaction delay example is a bad one, as it’s by design. Apple aims to gather up multiple purchases for a single credit card transaction to reduce fees.

There are a couple of reasons for this; three, actually — one, pulling together a bunch of small transactions into one larger one limits the card charges to Apple, so they pay less to the cards to handle charges; second, SAP is, pure and simple, not a real time processing system, so you HAVE to batch stuff into it, you can no scale SAP to handle what iTunes does in real time; third: this allows Apple to moderate the flow of transactions during peak times and spread the load out, it’s a form of scaling that lets you use the quieter times to avoid having to throw ever more hardware at a problem just for the peak loads.

The second reason (SAP is not a real time system) is one not well appreciated. Apple’s IT crews have some some unbelievable work around SAP, and people don’t appreciate just how critical this is to the company’s success. This is one part of it — SAP simply isn’t capable of doing what Apple needs it to do, and Apple’s geeks have found ways to beat it into submission. Mac OS X and Aperture and iLife get the coverage in the press, but down in the trenches are a bunch of people working their butts off doing stuff that’d make most CIOs drool.

By the way, every time I see someone say “Apple doesn’t do Enterprise” or “Apple doesn’t get the Enterprise”, I have to laugh. If you’re a CIO or an IT/Datacenter wonk, you would find a briefing by Apple on Enterprise stuff eye opening. Niall O’connor and his band of gleeful leprechauns are doing stuff (and doing it on Xserves to a huge degree) that’ll fry your brain. I spent my last decade at Apple in IT, and got to see (and work with, and sometimes fight with) those folks a lot, and they’ve really put together a top notch crew and a top flight operation. Apple is the largest, global, single-instance SAP environment in existence, and the entire company is driven by that beast and the tools they’ve built. amazing stuff.

I find it curious that people would hold up the iTunes Store as a good example of an Apple service done well. The release of iPhone OS 2.0 was not too long ago, and the store aspect of that release was a complete debacle. That wasn’t the first time the store went down hard, and I’m sure it won’t be the last.

Okay, here’s a basic reality: sometimes, there’s absolutely nothing you can do. Sometimes, you simply can’t scale to meet the demand of a high-profile, high-demand launch. Everything I’ve seen about the iPhone, iPhone 2.0 and the App store indicates that interest and demand is something like an order of magnitude larger than anyone expected in their wildest dreams. They seemed to have planned for a couple of thousand developers, they got tens of thousands. Entire cities camped out to buy phones on day one (it seems).

So sometimes, you do your best, you know there are going to be problems, you weather them as well as you can, and hope it’s good enough. Eddy’s group has done wonders at hiding launch problems and solving them on the fly — I’ve literally seen flocks of people flying pallets of xserves down hallways and creating a fire-wagon line to get them unboxed, in racks, networked up and in the load balancers just to try to catch up with capacity demand on a launch day. You try that some day….

But sometimes, nothing you do is enough. By the way, have I pointed out that I don’t ever, EVER join you fanboys in those day 1 lines? I mean, seriously — you do this to yourself, and Apple does a great job of protecting you from yourself in this rather silly game. but throw enough drooling geeks at an intro, and stuff’s going to break, because capacity is always finite, it’s merely a matter of where you draw that line.

I don’t remember Aperture ever being anywhere near state-of-the-art. I do remember a rapid fire series of price cuts when it failed to shift enough units, though.

it was first to market by a good measure, and redefined how photos were handled on computer. Unfortunately, there were internal problems in the team (according to rumors) and getting it all straightened out took a while. By the time Aperture 2.0 finally saw the light of day, Lightroom had caught up and passed it, and even Photoshop CS3 and Bridge had made good progress. I was an early adopter of Aperture, and I admit, I finally gave up and moved to CS3/Bridge when I got tired of waiting. And this weekend, I’ll be trying out the Lightroom 2.0 release and seeing if I like it (I was whelmed by LR 1, didn’t bother — right now, I use Bridge and CS3). Aperture 2.0 caught up with, but didn’t really leapfrog, Lightroom 1, and Lightroom 2 really blows it away from what I can see, so Apple’s got a problem in trying to get Aperture back into this game seriously. Too bad. I admit: I really tried to get on with the Aperture team before I left apple, that’s how much I loved the initial product. Today — I own Aperture 2.0, but I’ve moved everything out of it to Bridge.

I’ve talked about Aperture a lot on my blog in the past. You can find most of that here. it’s a great idea that took too long to mature. Ohwell.

I regularly see iTunes Store errors here. Maybe the Canadian one less stable, but I always thought they shared the same back end.

it may have changed by now, but it all used to be one big glop o’ back-end. Which created lots of complications. Like my last big project for Apple, successful beyond it’s architecture, and you end up spending time making things work while you figure out how to rebuild it. I know they had a team involved in re-architecting stuff, I assume some or all of that is in place now (I hope).

And I’ll close with this: I hopped on MobileMe about a week after release; you’ll note that I’ve publicized my email address as chuqui@me.com (shouldn’t surprise anyone, given I used to use me@chuqui.com — which should still forward, fwiw). I also am using it to sync between my Macs, publish calendars, etc. the first week or so after I moved was a bit rocky, but these days, it seems solid and stable. I’m not pushing the envelope, but it’s the environment I was hoping to see; my stuff syncs across machines fine, so I don’t have to think about it. My email is in one place, and it’s someone else’s problem to maintain and support the hardware. I long ago gave up the idea of running all of my own stuff, because I found I was spending all of my time supporting myself and not actually creating new stuff. Now, if you’re a geek who likes to do that, fine. Me? I’d rather be doing photography and writing than installing patches into DNS, at least in my off-hours. go figure.

So MobileMe is worth the cost, hands down. And I expect it’ll only get better. After the rumored september announcements, I’ll make a decision about whether to go with an iPhone G3 or an iPod touch (or both), and until then, my Blackjack does just fine. and by not being in the day 1 line follies, I avoid the day 1 glitches, too. Some of you could learn from that thought, but probably won’t.

Oh, and this may offend some of you, but what the heck. We used to see these lines forming, and watch the blog reactions and the like with both amusement and fear. That people are that involved and dedicated to Apple “stuff” is both great, and a great responsibility. And we recognized that. We also mentioned William Shatner a lot…

Take care.

You might also want to read:

1. MobileMe renewal: Yes or no? It’s a bit pricey. The standard fee is $99US/year. I’ve got one additional email address for my wife, so tack on another ten bucks. Before...
2. MobileMe Problems Show Apple Needs an Infrastructure Lesson August 8 update: I’ve written some followup thoughts on this message here. MobileMe Problems Show Apple Needs an Infrastructure Lesson – GigaOM: Steve Jobs, in...
3. (a quick followup on) Thoughts on Apple’s “Brain Drain” Chuqui 3.0: Thoughts on Apple’s “Brain Drain”: So, Apple loses another long-term senior employee. Actually, more than one, since Red Sweater notes that Peter Bierman...
4. Crazy Apple Rumors Site » Blog Archive » Apple Settle With Apple Corps. Crazy Apple Rumors Site » Blog Archive » Apple Settle With Apple Corps.: Apple Inc. and the Beatle’s Apple Corps announced today that they have...
5. Problems I think Apple should fix and challenges Apple faces (Apple Post-mortem, part 9 of some number…..) Previous episodes: Part 1 Part 2 Part 3 Part 4 Over here, I suggested that my “Apple post-mortem” series was going to be eight parts,...