There’s a new iPhone in town, and with it, a new release of IOS. The new iPhone is selling a zillion units and making Apple a few zillion dollars. Meanwhile, in some alternate universe, Dan Lyon is trashing everything in the hope that someone, somewhere, will link to him and maybe some folks will read his babblings again. Sorry, Dan, not me.
But with every release, it seems, the media and pundits have to find something to turn into a Major Crisis, and for IOS 6, that crisis is the new Maps. Apple’s removed the old Google Maps and replaced it with their own. Horrors, it’s got issues. Horrors, it’s not perfect. Obviously, this is the death of Apple. (But before you declare that, go read Kontra).
The fact is, the only thing more trendy than finding excuses to complain about Apple is buying their products. Which drives the complainers crazy, it seems. So every time Apple releases new products people find reason to complain and declare doom, and every release those declarations of doom are roundly ignored and the product goes on to be a massive success. Fact is, not having a controversy to generate page views over is bad for business for many of these blogs and media sites, so of course they’re going to find a controversy. “Hey, damn, this is great” doesn’t trigger outraged comments and doesn’t drive eyeballs or ad rates or readership. And frankly, there are segments of the geek/tech pundit crew that seem to have hit that point where if we gave them unicorns, they’d complain about them being purple instead of blue. And if we gave them unicorns two years in a row, they’d complain, because unicorns are boring and the industry is really demanding dragons now.
Honestly, it’s pretty sad to watch. But I don’t expect it to change any time soon.
The fact is, the maps we get with IOS 6 are mostly okay and definitely have some bugs in the system. It’s not as good as Google’s maps, especially away from North America. And the vast majority of users won’t ever notice, because their usage of maps is very casual. I’m sure Apple’s also going to be working hard in the background to clean up the issues and fix the problems that get reported to them. In six months, this whole thing will be largely forgotten, except for a couple of tumblr blogs nobody will visit until the next iPhone release when the pundits do their “remember when?” articles…
This all sound familiar? It should. It sounds a lot like Antennagate (iPhone 4). And the MobileMe Launch, which was never a great product, but pretty quickly became a decent and boring one. Remember when Apple revamped iMovie? or Final Cut?
Fact is, Apple isn’t afraid to take apart a product and push it in another direction if they feel it’s the right thing to do over the long term. Today, Antennagate has been proven to be a lot of smoke over very little fire. MobileMe moved forward and became iCloud, which is a huge part of MacOSX and IOS now, and is mostly ignored by the tech press because, well, it’s decent and boring. iMovie is back to being a pretty good consumer product, and Final Cut is actually pretty well liked in the advanced consumer and prosumer segments, which is where Apple wants the product to live. (pretty much everyone I saw that had a cow over the new Final Cut replaced it with a 5 figure Avid system — i.e., people who spend lots of money on their setups. And they had some legitimate issues over being abandoned by Apple, but Apple also has a right to choose that it doesn’t want to support a market segment any more, too)
Maps are headed down that path, too. Apple bought it’s first mapping technology company (Placebase) back in 2009. In fact, do a google search on Apple and Placebase, and you can see all of the articles written at that time about how Apple was replacing Google Maps. Here we are, three years (and at least three mapping technology buys) later, and they actually did it.
There’s been some amount of “we don’t know who pushed who into removing Google Maps” speculation going on. Honestly, given Apple’s been working on this at least four years, it’s hard to think Google pulled the trigger on this. More likely? The last time Apple and Google renewed the agreement on the maps, Apple came away from that thinking that they needed an alternative to Google before the agreement needed to be renewed again. Did Apple see Google boost the cost of the technologies and have no choice? Perhaps. Whatever, I think it’s pretty safe to say that Apple’s been planning this for a while.
The big question that’s been asked is “how could Apple ship something this buggy?” To which my answer is: “Have you ever tried to debug a data set like this?” I’ve seen a lot of writers complaining about the quality of the data. I’ve only seen two offer substantive suggestions on what Apple needs to do to fix it (and both are mapping industry insiders). I haven’t seen anyone offer any real suggestion on what Apple could have done, other than vague “it needed more testing”.
Data sets like this frankly defy testing. The only reason Google Maps is as good as it is today is it has a multi-year head start at fixing reported bugs. Ultimately, the only way you clean up a data set like this is to shove it out into the real world and take your lumps while you and your users beat it into shape.
Let me offer a similar problem I’m intimately familiar with. Someone dumps a list of 50 million email addresses on your desk (or in your Dropbox). Your job is to debug and clean up that data. How do you do that?
It is possible to find and fix/delete syntactically broken email addresses (although honestly, there are a bunch of RFC822-legal email addresses in the edge cases that I promise you will fail or destroy pretty much every production email system out there, which is okay, because nobody uses them. So do you clean up this list to be correct to the standard? Or do you define a subset of the standard and clean up to that? – even answering the question “what is a legal and valid email address?” is subjectively complex)
You can come up with lists of domains or account names you know you want to exclude from the list. Domains like “whitehouse.gov” probably shouldn’t be on there. Role accounts like “abuse@” or “root@” shouldn’t, either. You can also come up with specific email addresses you know you should exclude, like “firstname.lastname@example.org” or “email@example.com”. You can snuff out email addresses for which there’s no valid DNS and/or MX records and are therefore not routable (but there are complexities to that, too. In reality, you probably want to mark them and try at least three times over a two week period to get away from transient issues…)
In the end, what do you have? A list of email addresses that is somewhat smaller than the original list, one that you’ve been able to remove the typos and syntax errors and some percentage of the abuse/attack and other problem accounts. You have a list that you can pretty much guarantee is conformat.
But that’s about it. Do you know if it’s a good list? A bad list? Are those email addresses valid? they’re syntactically correct, but do those addresses exist and are there people reading email sent to them?
You still have no idea. And you won’t, until you actually start sending email out to the list.
Welcome to Apple’s Maps problem, except this list of 50 million email addresses is a LOT simpler and a much smaller data set than the mapping data.
Do more testing? Anyone have an idea how much testing Apple did? (I don’t). But I know if I have a list of 50 million emails, you could hire me a hundred temps for a month to roll through the data and it wouldn’t significantly change the quality of the list. We could identify more problem emails and remove them, but honestly, that’s a lot of work when sending emails and trapping the bounces would do the same for a lot less time and money.
The mapping problem is the same problem (only a lot bigger and harrier). Would hiring a thousand temps for six months solve the problems we see in this release of IOS?
It’s solve some of them. If a tester happened to look at the imagery of the Tacoma Narrows bridge, it’d be obvious there was a problem that needed to be fixed. But what percentage of the maps are we going to cover, and how are we going to cover it? do you think that even with a huge set of testers looking at random data sets out of the mapping data that particular problem would show up in the testing? Could you build a team that could cover, say, 10% of the mapping data in any time worth doing? And at what cost?
But think about some of the other problems we’ve heard about, such as an airport being turned into a farm. Would a random tester viewing that random map page have a chance of finding that problem? It looks just fine: it’s syntactically correct, just like those emails. It’s just wrong.
So if you want to debug something like map data, the only way to do tehat would be to build a testing team that has members with local knowledge of widely distributed locations and have them debug the areas of their expertise. In other words, you’d need to hire a global team and coordinate their testing globally as each group tests their local part of the data. How granular should these global teams be? Would a team based in Cupertino recognize a missing airport outside of Cody Wyoming?
Hopefully you start to see the scale of the challenge. At some point with that list of 50 million emails, you have to say “okay, let’s use it and see what happens” — and then capture the feedback (bounces, complaints, etc) and fix it as you go along.
Same with the maps. At some point “do more testing” doesn’t actually improve the quality of the data, it just costs you time and money.
That’s the point Apple is at now. Where the maps go from here depends on how well Apple is set up to receive, evaluate and repair complaints from users.
This is a new class of problem. The Maps looks like an app, smells like an app, and it walks like an app, but in reality, the vast majority of the guts of it is a huge database that’s living in a server, so you can’t treat it or test it like an app, and the old school testing techniques for an app fail miserably. These “big data” solutions are still an emerging part of the industry, and how to deal with, test and evaluate the data is still something everyone is grappling with — just ask any Hadoop geek.
But I think a basic fact is that in a lot of cases, the only way to validate and fix the quality of these data sets is to push them out the door and see what happens when everyone starts poking and prodding at them, because not only can’t you hire, train and manage a testing team large enough to deal with data sets this large in a rational way, doing the testing almost requires an infinitely large set of testers, each with intimate knowledge of some small subset of the data.
Which, if you think about it, only exists in one place: your user base.
So you can gripe all you want at Apple for the quality of the maps they released. I’m sure they knew about it. Just like we could know that list of 50 million email addresses was “clean”, until you actually start pushing it out the door and having it interact with the real world, you have no clue out “good” it is.
So if you’re one of those people complaining about the quality of the maps — stop and ask yourself how you would have done the testing better to avoid it. Not just for that little part you know, but on a global basis at a local level. Because that’s what testing that data set needs.
Good luck. you’ll need it.