MobileMe Problems Show Apple Needs an Infrastructure Lesson

August 8 update: I’ve written some followup thoughts on this message here.

MobileMe Problems Show Apple Needs an Infrastructure Lesson – GigaOM:

Steve Jobs, in an internal email seen by Ars Technica, makes clear that he’s upset about the botched launch of MobileMe, Apple’s new online suite of applications that has been plagued with bugs, including being flat-out unavailable to some for days at a time.

Or as I have been saying to folks here at work, “just imagine Steve Jobs wandering the hall with a flame thrower in hand, asking random people ‘do you work on MobileMe?’

I expect a bunch of friends and people I know were involved in that project, and I feel really bad for them. But the reality is, the thing wasn’t ready and the release got botched. And Steve and Apple aren’t terribly tolerant of that kind of major screwup. I expect heads have rolled and there are a few tanned hides waiting for the welts to go away.

“It was a mistake to launch MobileMe at the same time as iPhone 3G, iPhone 2.0 software and the App Store,” he says. “We all had more than enough to do, and MobileMe could have been delayed without consequence.”

There are two aspects to this. Steve is absolutely right — but also remember that ultimately, it was Steve’s call to go live (or not). he’s never been afraid to say “this ain’t ready” and pull something from release; his rehearsals for MacWorld Keynotes are legendary (and sometimes brutal), and stuff literally has disappeared in the last 24 hours, if he wasn’t satisfied with it.

So Steve has some responsibility here a swell, but with a caveat: someone he depended on to tell him what reality was told him it was ready to roll, and Steve believed him. And whoever told him that was wrong, and made everyone (including Steve and Apple) look bad. That’s not a good way to advance your career at Apple.

In his email, Jobs says: “The MobileMe launch clearly demonstrates that we have more to learn about Internet services.” You can say that again. The big question in the wake of the MobileMe debacle is whether or not the company even knows how to plan for heavy load.

Or not. Gruber nails this (see below). MobileMe is a tiny thing compared to iTunes. Apple gets it, and executes it amazingly well.

That this release was botched isn’t about Apple not having a clue, but about the MobileMe people either blowing it (I can think of any number of scenarios — scaling it hard). The ultimate failure seemed to be more capacity planning mistakes than anything else, if I’m guessing right. but the ultimate failure was not being willing to tell Steve “we aren’t ready” and taking that heat. They thought they could release and make it work, and guessed very wrong (or thought they were in good shape, which is worse).

I have picked up some tidbits from my Internet infrastructure sources, who tell me that:

* There is no-unified IT plan vis-a-vis applications; each has their own set of servers, IT practices and release scenarios.
* Developers do testing, load testing and infrastructure planning, all of which is implemented by someone else.
* There’s no unified monitoring system.
* They use Oracle on Sun servers for the databases and everything has its own SAN storage. They do not use active Oracle RAC; it is all single-instance, on one box, with a secondary failover.
* Apparently they are putting web servers and app servers on the same machines, which causes performance problems.

One of my sources opined that Apple clearly wasn’t too savvy about all the progress made in infrastructure over the past few years. If this insinuation is indeed true, then there is no way Apple can get over its current spate of problems. It needs a crash course in infrastructure and Internet services. Apple’s problem is that it doesn’t seem to have recognized the fact that it’s in the business of network-enabled hardware.

Not completely true, not necessarily a bad thing.

Some areas of Apple “run their own show”, effectively using Apple’s IT datacenters as a hosting facilities. Others build and operate within Apple’s IT infrastructure. One of the groups that basically runs its own IT outside of Apple’s core IT group is Eddy Cue’s group — because of the way stuff Eddy is in charge of gets built and managed.

There are unified monitoring services — and each service also tends to run a layer above that to monitor specific details. That’s not a negative.

the Oracle/Sun single instance thing? true to a degree, but I don’t see it as a negative. And don’t forget, Apple runs the largest global single-instance SAP environment on this stuff. it’s not exactly doing things wrong.

The bottom line is — Apple’s got its act together here better than these informants want to imply. The failures aren’t because Apple doesn’t know how to do this — it does — it’s because this project got botched.

And now Eddy has been brought in to fix it, which means it’s going to get fixed.

Eddy’s name isn’t familiar to most apple people, but he’s in his way as important to apple’s success as Jonathan Ives. His specialty: the back-end infrastructures that make Apple’s online universe tick. His groups did the Apple online store, iTools (later .Mac), iTunes store, etc, etc. It’s the not-sexy part of the company, but it’s the guts that make all of the sexy front ends actually work.

I’m actually amazed that Eddy hasn’t been poached by a startup, much as I’m amazed that Tim Cook hasn’t been poached — but the reality is that if you survive and become one of Steve’s inner core of people he trusts (and that ain’t easy) — you tend to stay. Apple doesn’t generally get poached by startups or other places at the exec level often, anyone notice?

A lot of that is because it’s not easy working for Steve, but if you can do it, you get to do really great stuff, and that’s addictive. trust me. you just don’t see people running off from apple to CEO a startup the way you do Yahoo or Google, not out of the top few levels of the company.

Eddy’s real specialty is to be able to take what Steve asks for, implement it, hit the target dates, make it work, and KEEP THE DAMN THING A SECRET UNTIL STEVE ANNOUNCES IT. That’s a big reason why his team is self-contained. It also means his people can do what needs to be done to implement things that never existed before and which don’t fit into normal IT “this is how we do things” standards. he and his teams spends most of his time off in uncharted territory where a need to be innovative and flexible is a must, and yet they have to do it on huge scales.

On the other hand, Eddy’s no easier to work with than Steve is, for obvious reasons. I invariably warned people not to hire into his groups unless they wanted to donate their life to the cause. When I was there, I worked pretty closely with various parts of his world, and it was populated with equal who were just as maniacal about this as Eddy and steve and people who were in process of burning out. Not much middle ground (but it works).

(full disclosure time: Laurie worked with Eddy way back when; me, I once almost got re-orged into his world until management remembered my vow to die before working for him, and re-arranged reality to fit (otherwise, lists.apple.com never would have existed….) — but I had a chance to deal with him while I was there and I’ve got a lot more respect for him now than I used to. I still wouldn’t want to work in the kind of grind his organization demands, though, but it does pretty good work under really scary conditions.

So you can bet, MobileMe will get fixed.

The looks, UI and edge devices are only as good as the networking experience — whether it comes from Apple or from its partners. MobileMe could just be the canary in the coal mine as far as the Cupertino Kingdom is concerned. MobileMe isn’t that big a portion of their revenues right now, but what happens when the problems hit the iTunes store? Imagine the uproar when your 3G connections slow to a crawl because AT&T’s wireless backhaul can’t handle the traffic surge.

It might not be a problem of Apple’s making but the company will face the brunt of the backlash. Remember, most of us instinctively blame the device first, then curse the carrier.

Daring Fireball has the right view here:

Daring Fireball Linked List: Om Malik on MobileMe’s Infrastructure:

But the iTunes Store does gangbuster traffic and has terrific track record for uptime. The message I read from yesterday’s reorg that put MobileMe under Eddy Cue (Apple’s VP for iTunes) is that MobileMe could and should be as responsive and reliable as the iTunes Store.

Om just doesn’t know the Apple internals very well. This wasn’t Apple failing, it was one group within Apple blowing chunks. That happens — remember when Aperture was the state of the art? and now it’s fighting to catch up with Lightroom, and may simply never regain that dominance. ohwell.

Apple has the expertise; this isn’t a case of MobileMe problems crawling out into itunes, but Apple bringing the iTunes expertise into MobileMe. And having thrown Eddy Cue at the problem, that’s exactly what’s going to happen here.

You might also want to read:

1. MobileMe renewal: Yes or no? It’s a bit pricey. The standard fee is $99US/year. I’ve got one additional email address for my wife, so tack on another ten bucks. Before...
2. Followup on: MobileMe Problems Show Apple Needs an Infrastructure Lesson Chuqui 3.0: MobileMe Problems Show Apple Needs an Infrastructure Lesson: That this release was botched isn’t about Apple not having a clue, but about the...
3. .Mac Morphs into MobileMe TidBITS Networking: .Mac Morphs into MobileMe: Although still costing $99 per year (with a free 60-day trial), the idea is that MobileMe is less a...
4. Crazy Apple Rumors Site » Blog Archive » Apple Settle With Apple Corps. Crazy Apple Rumors Site » Blog Archive » Apple Settle With Apple Corps.: Apple Inc. and the Beatle’s Apple Corps announced today that they have...
5. Problems I think Apple should fix and challenges Apple faces (Apple Post-mortem, part 9 of some number…..) Previous episodes: Part 1 Part 2 Part 3 Part 4 Over here, I suggested that my “Apple post-mortem” series was going to be eight parts,...