Yearly Archives: 2008

Drop the puck!

It’s hockey season again, so we dusting off the blog and getting back to work…

Hope you all had a great off-season!

With the baseball season ending here in the household a couple of days ago (Cubs lose! Cubs lose!), I’m now officially ready for hockey to start.

We missed the first two pre-season games in San Jose while we were in Yellowstone, but made the third, and to be honest, I like what I see. Aspects are a work in progress, but it’s definitely not Ron Wilson’s team any more. More aggressive, offensively focussed, puck control. Just like (gasp) the Red Wings.

And to be honest, much as I liked Campbell, Boyle was my first choice. Now he’s a shark. So is Rob Blake. Will Jeff Friesen? I have to admit, I’m rooting for him.

I am so very much looking forward to opening night on Thursday, to see how the team handles things when it’s for real.

I have plans to contribute more regularly here now that the season has started, and I have some interesting things (at least, interesting to me, hopefully to you as well!) that I want to do this season, if life cooperates — many of them are things I planned last year, but life simply didn’t allow. This year, we’re thinking, will be less — complicated, so we can move them forward.

The one thing I’ll admit to here at the start of the season — I’ve got a series of postings I’ve been working on under the overall theme of “how to make the NHL better”. Note: not “how to fix the NHL”, because unlike some, I don’t consider it broken — but it can always be better. some of the things I’m going to talk about are pretty straight-forward, some of them are obscure, and some of them are going to be either silly or outright flamebait, but hopefully, it’ll get people thinking and talking.

Stay tuned, and we’ll see if you agree.

How Apple Should Handle the App Store Blacklist

How Apple Should Handle the App Store Blacklist:

I’ve been trying to decide if I wanted to wade in here, lest people thnk this blog’s turning into “all apple, all the time”. but what the heck, why not?

To a good degree, I agree completely with this piece, with a couple of minor caveats.

The question of how Apple should use the App Store blacklist has been bandied about lately and so far, no one really has the answer. Should Apple act unilaterally and remove apps without any warning? Should it ask for user input? The questions are numerous and the answers are in short supply. I think it needs to have a full-fledged plan that’s made available to the public so developers and users alike will know what to expect.

So what exactly should Apple be doing? It should first start out with a real policy. How can it summarily remove applications from the App store without warning the developer or user? It doesn’t make any sense.

Included in that policy, it should develop an understanding between both the user and Apple that makes both entities work together to achieve the lofty goal of making it a better service for all parties involved.

First and foremost, Apple needs to install a “report” button that lets the users alert the company to ridiculous applications like “I am Rich” and helps them sift through the good and the bad.

By doing that, it also helps create a rapport between Apple and users, who have been kept in the dark so far about what’s really going on when it removes applications like NetShare, Box Office, and others. Let’s face it – users are downloading these applications and they have every right in the world to know what’s going on with them. I don’t think that’s asking too much.

Secondly, Apple needs to set parameters for how apps should be priced. I have no problem with developers assigning prices to their work, but $1,000 for an iPhone application that gives you a mantra isn’t worth $1, let alone $1,000.


99% of the current complaints about what’s going on with the store are solved with a simple thing: communication. Whatever the policy for the App store is, it’s a secret. When something is removed from the store, the reason is a secret. what is cause for acceptance and rejection is basically a secret. “Magic happens” and you either appear or disappear from the store.

All that engenders frustration, and the developers and users are right. that has to stop. I don’t think Apple is “being evil” here as much as scrambling with serious overload, but honestly, they’re trying to work TOO fast and creating problems in their wake by what corners they’re cutting.

So my suggestions are:

1) get in touch wtih the developers; there’s an approval policy internally, somewhere. release it. explain it. At least let them know where the lines in the sand are.

2) remember the mobileMe blog? the one that magically went quiet again as soon as the crisis was over? (gee, funny that. nice conversation). How about an App store blog, so when something is removed from the store, users are told about it. AND WHY. Especially if it’s because of some kind of security or data leakage problem, which users deserve to know.

3) create and publish an appeal process for developers. there has to be somewhere for them to get a decision reviewed. right now, that’s a black hole.

do 1 and 3, and life gets MUCH better for Apple, really fast. Do 2 and you get even closer to the ideal state (to quote Bill Cosby: “Parents don’t want justice, they want quiet”. and what we want here, for apple, is quiet; justice would be nice, too)

as to reporting buttons? Not a huge fan, I don’t think they work well in real life, and they’re easy things to create and ignore and let someone think they’re being heard when in reality they’re being ignored.

Adn having Apple tell a company what to charge for their app? Nope. let th free market play out here. you trust users (with the button) to tell Apple about bad apps, but don’t trust those same users to tell developers they mispriced their stupid product? hey, if the users are smart enough for one, they’re smart enough for both. Let the users vote with their wallets.

my view on Apple managing the app store: you want Apple taking care of key issues, and that includes things like interface integrity, security bugs, data leakages and stuff that could really bork over a phone or it’s owner. but beyond that, the fewer things Apple is involved in, the better. I’d rather see 1000 stupid apps in the store die of neglect than one app not make it into the store because Apple thought it’d be stupid but really didn’t understand what it was all about….

Reports of Usenet’s Death Are Greatly Exaggerated

Reports of Usenet’s Death Are Greatly Exaggerated:

Is Usenet dead, as Sascha posits? I don’t think so. As long as there are folks who think a command line is better than a mouse, the original text-only social network will live on. Sure, ISPs will shut down access out of misled kiddie porn fears but the real pros know where to go to get their angst-filled, nit-picking, obsessive fix.


Nope. USENET is dead. Dead, dead. dead. really dead.

The fact that there are small enclaves of civilization hiding in boarded up bars hiding from the zombies that have destroyed the city doesn’t mean the city isn’t destroyed, folks.

This isn’t “yeah, there are some issues, but things are generally okay” out on USENET. This is “hello, I”m charlton heston, and I’m the Omega man” out on USENET. There is some humanity alive out there, but the zombies have won and the city is destroyed.

New and interesting uses for webmail

For the last couple of weeks people at work have heard me muttering in the halls about “those damn geeks”. I’ve been chasing down and cleaning up after a group that’s been using the webmail system as a distribution system for — stuff. Mostly warez cracks and video, from what I can tell.

Since this seems to be fairly widespread and flying under the radar at most sites I’ve talked to about this, I thought I’d give it some wider visibility and go into some of the details.

I want to emphasize this part:

Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.

Nothing was hacked in any way, this is purely (in its way) a social engineering hack taking advantage of free webmail sites all around the internet — I saw at least 15 involved from my investigation.

I’d noticed some changes in network usage on the site the previous couple of months; bandwidth usage had doubled in both May and June, far beyond what I thought normal given the growth in new users we’re seeing. It didn’t seem too serious, though, so I stuffed it in the back of my head to investigate at some point.

Early July hits and I look at the numbers again — and in the first 7 days of July we’ve used 10X the network bandwidth we used in all of June. We’re talking orders of magnitude change, for no good reason.

That’s generally a bad thing. So I went looking….

What I found was both fascinating and a little depressing. It was a group of people based in Poland that have turned public webmail systems into the equivalent of a Bittorrent network.

Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.

Here’s how it seems to work: when they have a package to distribute, it is packaged up into pieces small enough to be attached to and sent as emails. Most webmail systems allow attachments up to about 10 megabytes. Files were split up and encoded in MIME as standard packages, although the details of name and type seemed to be ignored (lots of powerpoint files, in theory).

Then accounts were created on various webmail sites. In my sample of addresses, I see over a dozen different sites being used. The person doing all of this then emails the files to that mailbox, where they sit. Now, anyone who wants that set of files only has to get the access information for one of those accounts, log in via IMAP and let his email system download them. It looks like any given package is stored on between 3 and 8 different webmail accounts.

Account creation seems to be semi-automated. All accounts are of a similar format, a semi-random “word”, followed by a 1-3 digit number. Passwords use the same format (but are never the same), ditto the “from” address and the “return-path” in the headers of the emails. Sometimes the files are stored in more than one account on a single webmail (another reason why I think this is at least semi-automated), but generally, it’s sent to 4-6 webmail accounts on 4-6 different sites.

It looks like the actual account creation is manual, or semi-manual, because some of the sites involved use CAPTCHA on account creation and that isn’t stopping them. I don’t think this setup is sophisticated enough to have cracked CAPTCHA, so there are people involved in the setup. I think the account naming, and packaging is automated, but people are involved in the account creation and uploading. Once someone downloads the emails, there seems to be another script to put it all back together again, because it’s not depending on the MIME data in the message to do naming or decoding — in fact, that stuff is set up to (at least casually) make the content itself look innocent.

There’s obviously a web site somewhere that tells you how to access the mailbox to get the content, but I haven’t gone looking for it.

If you think about it, this is a pretty nice hack. With Bittorrent being scrutinized by many ISPs, they’ve set up a fairly low-tech, under-the-radar way of distributing “stuff” without easy detection. The original distributor only has to upload the files once, and then the rest of the resource costs are borne by the mail systems — the webmail site pays the network to upload the files into the system, pays for the disk to store them, and pays for the network to distribute them back out.

Needless to say, I spent some time shutting all of this down. We ended up with a couple of hundred accounts that I closed out. All told I identified and closed a couple of hundred accounts that accounted for over 200 gigabytes of disk storage, and the network bandwidth they were starting to suck was going to be measured in terabytes, and we’re a fairly small webmail site right now. One can only wonder what they’re doing to some other sites….

The group is based in poland. 99% of the access of these files also came from Polish IP ranges. Fortunately, once you know what to look for, it’s fairly easy to find these accounts, given the standardized naming, the limited IP range they’re coming from, and the exceptionally large average message size. The latter is the easiest way to identify them, no “real” webmail account (at least on our system) has an average message size > 5Meg. Even accounts where users are parking files in their Imap for storage tend to have no more than a 1 meg average storage size.

This group spent some time experimenting with the site, evidently to see if we were paying attention. The earliest record I can find of them accessing the site is in April. In June, they ramped their volume significantly, and in July, they opened the floodgates (and I found it four days later, fortunately). It’s hard to tell from the outside if this was them experimenting to see if we’d catch them and then ramping up when they felt safe or if this is a new network that was finally ramping up as they finished building it. Either way, it’s clear there’s a lot of network being used on a lot of webmail systems globally by these guys.

How to stop this? No easy answers. They aren’t really “doing” anything we don’t allow, it’s more of a Terms of Service on content issue with policing. If the account creation was fully automated we could possibly plug that hole (and probably should on general principles; CAPTCHA might not stop this but it can’t hurt, but some of the webmail sites being used have CAPTCHA enabled and it didn’t stop them). On the other hand, there’s no reason we should feel the need to let them pass around warez on our dime — and they only have to use network to upload it once, and then the webmail sites pay for the bandwidth to accept and then deliver it as often as it gets downloaded, plus disk storage and the typical overhead of backups and etc.

What it really goes to show is that people will find interesting uses for any publicly available technology, whether or not you intended for them to be used that way. It also, I think, means we should be aware of what those possible uses might be and see if we can influence our systems to discourage the ones we don’t like. For instance, a 5 megabyte limit on attachments might have discouraged these guys, but doesn’t seem to significantly impact “normal” users — I found very, very few emails on the system that large.

One of the things I’ve been pondering is ways to automate finding or setting alarms for this kind of “non-standard” behavior; quotas solve some problems, but not this one. I wrote a script that finds these accounts with really large average message sizes. It seems to me something that automates that process, or ways to monitor or rate-limit network usage on a per-account basis would be another way, or simply looking at accounts with the highest network usage.

Things that definitely don’t help this kind of problem: quotas, looking for accounts at or close to quota, accounts with large number of log-ins, or even usage from many different IP addresses. None of those were true. I also didn’t see any significant sign of multiple simultaneous users. The things I think of as “obvious” signs of abuse are missing here, it’s a different set of parameters that become visible once you look.

One option I’m just starting to investigate is coming up with some kind of “typical” network usage per user, sort of a capacity planning number — and then if the system deviates from that significantly it gives you a hint you need to look in more detail. I want to avoid having to monitor at the per-user level to the greatest extent possible, and find metrics at the system-usage level that might tell me if the system is within expected usage ranges or not.

In reality, there’s nothing “wrong” going on here other than the sheer size of the operation and the costs it involves (and the fact that most of the content is likely illegal). technically it’s pretty simple and straightforward — a nice hack — to shift the cost of distribution off to others in a way that’s (in theory) low-key enough to not be noticed, at least until they get greedy in resource consumption. If they hadn’t spiked usage in July like they did, I might not have gotten around to chasing them for a while.

My ultimate take-away, though, is that the users “use cases” for a technology are rarely the same as the developers. Sometimes the users innovate in really interesting and positive ways, sometimes they distribute warez — but either way, people are going to see opportunities in your technology and that should be part of the discussion in designing those technologies.

My suggestion: if you run a webmail site that allows users to create accounts, you might just want to look and see what you find. Might surprise you.

Oh, for what it’s worth, I’ve held off posting on this for a bit because I gave advance warning to the other sites I found involved in this. Of the 15 or so abuse@ accounts I sent the details to (including accounts, IP ranges, Received header data, etc, etc), one responded immediately and started their own search and destroy operation — they happened to be one of the larger “white label” webmail, so that’ll shut down any number of the domains involved.

But three of the webmail sites had their abuse@ addresses bounce as user unknown. One sent me email letting me know he was on holiday for a few weeks (in italian). And from the rest, including the two Polish ISPs where all of the upload activity intiated, total silence. Ohwell. Kinda sad, but hey, it’s their network bill, if they don’t mind paying it, I shouldn’t complain… And I just did a check of our site to see if they took the hint, and I see no sign of them creating new accounts now or doing any kind of activity, so I think they’re gone. Well, for now. I’ll know if they come back…

Looking at Sharks 2009

So the dust is settling, and the new sharks roster is taking shape, and I’m finally back at a point in my life where blogging seems not only possible, but interesting. Been an interesting three months.

So now we can start to look at the Sharks for 2008-2009 and see if this is a better team. Is it?

First, coaching: I like the hiring of McLellan, but it’s not without risks. Sometimes a really top-notch assistant coach is — a top-notch assistant coach. He could be the next Bruce Boudreau or John Anderson, but he could also be Dave Lewis or Wayne Cashman, two guys who tried to make the leap to NHL coach and found out they made damn good assistant coaches. Or he could be Kevin Constantine, who’s a pretty damn good coach, just not at the NHL level.

So the move is not without risk, but the Sharks aren’t afraid of taking risks, and I think this one makes sense. I’m a lot happier with the idea of bringing in a new voice that has some ability to relate to younger kids than to bring in a “safe” retread who overplays veterans and doesn’t grow his players. I think it’s a good hiring, and I’m looking forward to seeing how he fills out his assistants.

A while back, I wrote about what I thought should be (or would be) changed in the roster offseason. A few highlights and lowlights:

Two for Elbowing: Picking up pieces and an update on Michalek – The San Jose Mercury News Sharks Hockey Blog -:

the Sharks are a damn good team, but it’s clear changes need to be made for the team to get better.

I’d like to see Nabokov backed off to 60-65 games next year (his going to the world championships notwithstanding). Rest him a bit more, keep him a bit fresher.

If that means bringing back Boucher, or someone else, so be it.

And so it is.

Core group (do not touch under penalty of death):


Not coming back:

Curtis Brown (Sorry, Brownie, but I think it’s time).

All of which happened. I honestly felt a top six forward would go — I’m happy that McLellan and Wilson think this group can be kept together and improved without being swapped around.

Players I expect back, but which aren’t “no trade under any cirucmstance” types (as part of the right deal? sure):

Tomas Plihal
Patrick Rissmiller
Jody Shelley
Marcel Goc

I want to see come back:

Jeremy Roenick

Rissmiller was allowed to leave, Shelley is back, as far as I can tell, Plihal and Goc are still unsigned. Plihal will be, Goc, not so sure. If of this crew we lose Rissmiller (like him, replaceable) and Goc (like him, somewhat disappointing), I don’t think the sharks miss a beat. No game changers.

So where does this put the Sharks?

Shelley-Plihal (probably)-Mitchell

and two black aces to be named. Goc maybe one of them.

It’s kinda hard to complain about this roster, especially if they play to potential. So I won’t. you can see why Rissmiller wasn’t kept, when guys like Setoguchi and MItchell and Plihal are having to fight for third line time?

Now the fun begins. The Defense was the thin spot on depth last year. I thought going into the season it was good enough. I was wrong. This year, it’s looking a lot different:

Core group (do not touch under penalty of death):

Douglas Murray (what an improvement this year!)

Matt Carle (struggled at times, but seems to be growing into it; I’d hate to give up too soon)

M-E Vlasic (wow; at his age?)

Craig Rivet

I’d like to see back:

Brian Campbell (but not for Phaneuf money; if someone wants to pay him that, be my guest. he’s missing that “punk brat” aspect to his game, which keeps him a rung below Phaneuf on the ladder. But $25m over 5 years? sure. Just not $30 over 5.

hint: I expect Campbell to stay. He seems happy. He likes playing 30 minutes a game. Why screw it up?

Not coming back:

Sandis Ozolinsh: thanks, Sandis. for everything.

Alexei Semenov: ditto. Neither of these are NHL caliber in today’s NHL.

Kyle McLaren: love his guts and drive, but his knees are problematic. I think it’s time to consider an upgrade.

So one of my “untouchables” goes away in Matt Carle, but we’re getting (if rumors are true) Dan Boyle in return. We lose Brian Campbell, but for the money he’s getting, I hope Chicago enjoys his play. Carle was a lot more expendable to me than Vlasic, so I’m happy.

So our D now looks like:

Rob Blake-Dan Boyle
Murray-lukowich (rumored coming from tampa)


Again, not much to complain about here. Blake/Boyle/Lukowich instead of Campbell/Carle and either Semenov or Ozolinsh? It’s a more veteran crew, but I like what Blake brings to the team in intrinsics, even if we’re giving up some youth to get it (indirectly, because losing Campbell makes bringing Blake in and getting Boyle possible — although I get the impression Wilson was going to bring Blake in anyway).

This is a team that’s now completely oriented towards the next two season. Yeah, after that we’ll have to see about bringing youth in and reloading, but that’s wilson’s problem later. This team needed to be about “NOW OR ELSE”, and now it truly is.

In retrospect, two problems last year:

no backup goalie to take the load off of Nabby and limit his playing time a bit. I don’t think this really hurt the sharks, but I don’t want my goalie playing that many games.

The defense was too young and too thin; trying to patch in with Semenov and Ozolinsh was the red flag, and that proved to be true.

One thing I can guarantee: Doug Wilson will do something completely different than this, and when he does, I’ll go “wow, I never would have thought of that” and like it. Whcih is why he’s GM, and I’m a blogger…

Well actually, Wilson’s done pretty much what I expected; couldn’t re-sign Campbell, went and got Boyle. I have to admit that Rob Blake was one of the guys I thought would be great on the Sharks — only I never thought he’d leave the Kings, so I didn’t really consider it an option. Fortunately, Wilson did, and Lombardi (if you ask me) mis-stepped here. but more on that later. But thanks, Dean. I expect Detroit will send you flowers for helping us (not).

I can’t see how Wilson could have handled this better, given things not under his control (campbell couldn’t be forced back without seriously overpaying him, which the Sharks don’t do). I’m really happy they didn’t move Marleau, I’m really happy they didn’t make any “make a splash” moves at the draft and overpaying to do so. It’s all a very solid, methodical, well-thought out strategy.

So far, a great offseason. And what it ends up doing is sending a big message to the players: no excuses. Now, it’s up to the players.

Can’t wait for camp.