Search This Site
Silicon Valley veteran doing Technical Community Management. Photographer with a strong interest in birds, wildlife and nature who is exploring the Western states and working to tell you the stories of the special places I've found.
Author and Blogger. They are not the same thing. Sports occasionally spoken here, especially hockey. Veteran of Sun, Apple, Palm, HP and now Infoblox, plus some you've never heard of. They didn't kill me, they made me better.
Person with opinions, and not afraid to share them. Debate team in high school and college; bet that's a surprise.
Support This Site
If you found this page interesting, please consider clicking through this ad and buying something.
If you do, Amazon will pay me a small percentage of the price. You don't spend any more on the item, and the money helps pay for the site and the more people who do this the more time I'll be able to spend on the site improving it and adding content.
More to Read
- Some Thoughts on Lightroom Keywords
- How not to be a doofus with a camera
- Beyond 'Vacation Snaps'
- A teachable moment (or why I love birding, even when I make a fool of myself)
- Sherman, set the wayback machine to…
- An audience of one....
- Talking about 'Stuff'
- What I do for a living…
- 50 reasons Why I Haven’t Been Blogging
Want more? Try this list...
New on the Blog
- The new flickr design
- Yosemite Road Trip 2013 – Day 1, teaching.
- The Raffi Torres Hit
- Tioga Pass, Yosemite
- Back from Yosemite
- Bobcat before and after
- 2013 playoffs, round 2
- Fuji X100s Review – Fallin’in Love All Over Again
- If you give them an easy out, they’ll take it.
- Another reason Don Cherry should retire (or be retired…)
Rent Gear at Borrowlenses
Don't buy that gear before trying it out! Renting a lens you're considering buying is a great investment in saving yourself from buyer's remorse!
And if it's a piece or gear you aren't going to use constantly, renting it when you need it is a great way to save money, and I highly recommend Borrowlenses as a place to rent high quality, well-maintained gear.
Monthly Archives: July 2008
For the last couple of weeks people at work have heard me muttering in the halls about “those damn geeks”. I’ve been chasing down and cleaning up after a group that’s been using the webmail system as a distribution system for — stuff. Mostly warez cracks and video, from what I can tell.
Since this seems to be fairly widespread and flying under the radar at most sites I’ve talked to about this, I thought I’d give it some wider visibility and go into some of the details.
I want to emphasize this part:
Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.
Nothing was hacked in any way, this is purely (in its way) a social engineering hack taking advantage of free webmail sites all around the internet — I saw at least 15 involved from my investigation.
I’d noticed some changes in network usage on the site the previous couple of months; bandwidth usage had doubled in both May and June, far beyond what I thought normal given the growth in new users we’re seeing. It didn’t seem too serious, though, so I stuffed it in the back of my head to investigate at some point.
Early July hits and I look at the numbers again — and in the first 7 days of July we’ve used 10X the network bandwidth we used in all of June. We’re talking orders of magnitude change, for no good reason.
That’s generally a bad thing. So I went looking….
What I found was both fascinating and a little depressing. It was a group of people based in Poland that have turned public webmail systems into the equivalent of a Bittorrent network.
Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.
Here’s how it seems to work: when they have a package to distribute, it is packaged up into pieces small enough to be attached to and sent as emails. Most webmail systems allow attachments up to about 10 megabytes. Files were split up and encoded in MIME as standard packages, although the details of name and type seemed to be ignored (lots of powerpoint files, in theory).
Then accounts were created on various webmail sites. In my sample of addresses, I see over a dozen different sites being used. The person doing all of this then emails the files to that mailbox, where they sit. Now, anyone who wants that set of files only has to get the access information for one of those accounts, log in via IMAP and let his email system download them. It looks like any given package is stored on between 3 and 8 different webmail accounts.
Account creation seems to be semi-automated. All accounts are of a similar format, a semi-random “word”, followed by a 1-3 digit number. Passwords use the same format (but are never the same), ditto the “from” address and the “return-path” in the headers of the emails. Sometimes the files are stored in more than one account on a single webmail (another reason why I think this is at least semi-automated), but generally, it’s sent to 4-6 webmail accounts on 4-6 different sites.
It looks like the actual account creation is manual, or semi-manual, because some of the sites involved use CAPTCHA on account creation and that isn’t stopping them. I don’t think this setup is sophisticated enough to have cracked CAPTCHA, so there are people involved in the setup. I think the account naming, and packaging is automated, but people are involved in the account creation and uploading. Once someone downloads the emails, there seems to be another script to put it all back together again, because it’s not depending on the MIME data in the message to do naming or decoding — in fact, that stuff is set up to (at least casually) make the content itself look innocent.
There’s obviously a web site somewhere that tells you how to access the mailbox to get the content, but I haven’t gone looking for it.
If you think about it, this is a pretty nice hack. With Bittorrent being scrutinized by many ISPs, they’ve set up a fairly low-tech, under-the-radar way of distributing “stuff” without easy detection. The original distributor only has to upload the files once, and then the rest of the resource costs are borne by the mail systems — the webmail site pays the network to upload the files into the system, pays for the disk to store them, and pays for the network to distribute them back out.
Needless to say, I spent some time shutting all of this down. We ended up with a couple of hundred accounts that I closed out. All told I identified and closed a couple of hundred accounts that accounted for over 200 gigabytes of disk storage, and the network bandwidth they were starting to suck was going to be measured in terabytes, and we’re a fairly small webmail site right now. One can only wonder what they’re doing to some other sites….
The group is based in poland. 99% of the access of these files also came from Polish IP ranges. Fortunately, once you know what to look for, it’s fairly easy to find these accounts, given the standardized naming, the limited IP range they’re coming from, and the exceptionally large average message size. The latter is the easiest way to identify them, no “real” webmail account (at least on our system) has an average message size > 5Meg. Even accounts where users are parking files in their Imap for storage tend to have no more than a 1 meg average storage size.
This group spent some time experimenting with the site, evidently to see if we were paying attention. The earliest record I can find of them accessing the site is in April. In June, they ramped their volume significantly, and in July, they opened the floodgates (and I found it four days later, fortunately). It’s hard to tell from the outside if this was them experimenting to see if we’d catch them and then ramping up when they felt safe or if this is a new network that was finally ramping up as they finished building it. Either way, it’s clear there’s a lot of network being used on a lot of webmail systems globally by these guys.
How to stop this? No easy answers. They aren’t really “doing” anything we don’t allow, it’s more of a Terms of Service on content issue with policing. If the account creation was fully automated we could possibly plug that hole (and probably should on general principles; CAPTCHA might not stop this but it can’t hurt, but some of the webmail sites being used have CAPTCHA enabled and it didn’t stop them). On the other hand, there’s no reason we should feel the need to let them pass around warez on our dime — and they only have to use network to upload it once, and then the webmail sites pay for the bandwidth to accept and then deliver it as often as it gets downloaded, plus disk storage and the typical overhead of backups and etc.
What it really goes to show is that people will find interesting uses for any publicly available technology, whether or not you intended for them to be used that way. It also, I think, means we should be aware of what those possible uses might be and see if we can influence our systems to discourage the ones we don’t like. For instance, a 5 megabyte limit on attachments might have discouraged these guys, but doesn’t seem to significantly impact “normal” users — I found very, very few emails on the system that large.
One of the things I’ve been pondering is ways to automate finding or setting alarms for this kind of “non-standard” behavior; quotas solve some problems, but not this one. I wrote a script that finds these accounts with really large average message sizes. It seems to me something that automates that process, or ways to monitor or rate-limit network usage on a per-account basis would be another way, or simply looking at accounts with the highest network usage.
Things that definitely don’t help this kind of problem: quotas, looking for accounts at or close to quota, accounts with large number of log-ins, or even usage from many different IP addresses. None of those were true. I also didn’t see any significant sign of multiple simultaneous users. The things I think of as “obvious” signs of abuse are missing here, it’s a different set of parameters that become visible once you look.
One option I’m just starting to investigate is coming up with some kind of “typical” network usage per user, sort of a capacity planning number — and then if the system deviates from that significantly it gives you a hint you need to look in more detail. I want to avoid having to monitor at the per-user level to the greatest extent possible, and find metrics at the system-usage level that might tell me if the system is within expected usage ranges or not.
In reality, there’s nothing “wrong” going on here other than the sheer size of the operation and the costs it involves (and the fact that most of the content is likely illegal). technically it’s pretty simple and straightforward — a nice hack — to shift the cost of distribution off to others in a way that’s (in theory) low-key enough to not be noticed, at least until they get greedy in resource consumption. If they hadn’t spiked usage in July like they did, I might not have gotten around to chasing them for a while.
My ultimate take-away, though, is that the users “use cases” for a technology are rarely the same as the developers. Sometimes the users innovate in really interesting and positive ways, sometimes they distribute warez — but either way, people are going to see opportunities in your technology and that should be part of the discussion in designing those technologies.
My suggestion: if you run a webmail site that allows users to create accounts, you might just want to look and see what you find. Might surprise you.
Oh, for what it’s worth, I’ve held off posting on this for a bit because I gave advance warning to the other sites I found involved in this. Of the 15 or so abuse@ accounts I sent the details to (including accounts, IP ranges, Received header data, etc, etc), one responded immediately and started their own search and destroy operation — they happened to be one of the larger “white label” webmail, so that’ll shut down any number of the domains involved.
But three of the webmail sites had their abuse@ addresses bounce as user unknown. One sent me email letting me know he was on holiday for a few weeks (in italian). And from the rest, including the two Polish ISPs where all of the upload activity intiated, total silence. Ohwell. Kinda sad, but hey, it’s their network bill, if they don’t mind paying it, I shouldn’t complain… And I just did a check of our site to see if they took the hint, and I see no sign of them creating new accounts now or doing any kind of activity, so I think they’re gone. Well, for now. I’ll know if they come back…
With everything going on, I was wondering if I’d ever get past the 200th bird on my birdwatching life list. I set myself two goals for birding in 2008: 200 species, and to be the first to discover a notable bird in the area.
The latter is really a function of luck, time spent birding and a bit more luck; and I’ve come close a couple of times in the last year, but it’s never been confirmed. It’ll happen when it happens.
But I’ve finally been able to do a bit of birding again, and I’ve now shot past 200 species. I’m now thinking I might amend the goal to 200 species for the year and see what happens.
Bird # 200 was, of all things, a Barn Owl. There’s a Barn Owl in a box at Don Edwards EEC; I went out there on the 11th to see if I could find the Wilson’s Phalaropes (no luck because I was limited in how far I could walk out after them), and realized I’d never logged the owl onto my list. Looked i the box, it looked back and blinked. Done.
Leading up to 200 included a couple of nice birds: 199 was Snowy Plover, down in Bolsa Chica (yes, I’m spending a LOT of time in SoCal these days, and birding Bolsa Chica on the way out home most trips; it’s a nice place to visit and a good break after the fun of Southern Cal right now). 198 was Blue-Gray Gnatcatcher, a bird I’ve missed multiple times, even when Bob Power has pointed it out — yet when I was reworking my photo library, there was a bird from 2006 labeled “sparrow” that I saw at a glance was wrong; a close look showed it to be a Blue-Gray Gnatcatcher, so I added it to the list retroactively.
Also added to my list via photo evidence were Gila Woodpecker and Northern Cardinal from a trip to Tucson in the 1990s, and Rhinoceros Auklet from 2005 and a trip to Victoria at Odgen Point, those made 201, 202 and 203. Today I added Wilson’s Phalarope and the Ruff out at EEC for 204 and 205, and I could have added a Pacific Golden Plover, but my ankle just wasn’t up to the walk. The walk out to island 4 wasn’t bad, but it stiffened up watching the ruff, and the walk back got pretty brutal; still, it’s slowly improving.
Nice to know that despite everything going on the last few months, at least one goal is accomplished…
On the way back from SoCal yesterday with the first batch of Dad’s “stuff” to be sorted and organized, and with the key estate issues taken care of (at least this round), I hit Bolsa Chica again and got some really nice Snowy Plover photos, as well as some least tern chicks, and got to watch a black skimmer on the hunt again. Fascinating, weird-but-beautiful birds, the black skimmer.
Some quick notes on today’s birding trip:
I headed out this morning to Don Edwards in search of fame and fortune, or at least a Wilson’s Phalarope. Starting out around 10, I walked out to Island 4 and back, running into numerous other birders out searching for same or better.
It was a very successful day. The Ruff continued on Island 4, living most of the time on the far side of the island but popping up into sight every so often; while I was there, it came into full view three times, and popped it’s head up once more, over about 30-40 minutes.
Other birders reported the golden plover continuing on island 5, but my ankle was already complaining, so I gave it a pass (sigh. but right decision for me).
there was a Wilson’s Phalarope at the eastern edge of Island 4, another on island 3, and a third in the shallows on the S side of the berm across from Island 4, but not great numbers. I found two ruddy turnstones on Island 3. Walking back towards the parking lot near island 2, I had a sparrow fly past me. I chased it a bit before it flew off into the brush, and it looks (I think) like a moulting juvenile Savannah
From talking to the other birders, the black tern had been a no-show that morning. I’d stopped to rest the ankle near Island 1 on the way back (about 12:30ish) and noticed a tern out on the algae mat out beyond island one. It was only there for a minute or so, but I got the scope on it and it was a black tern (much darker underwings than forsters, and much different flying habits, it was flying maybe 1-3 feet over the water and dipping in to skim, much like a black skimmer, rather than the plunge dive; very distinctive once you see it). It flew off to island 1 and I thought it landed near the pelicans, but I couldn’t find it, but it was definitely there for a very short period of time.
In the reeds of the marsh between the EEC and the pond I spent some time trying to coerce the marsh wrens to come into view; one finally did, but there were four or five in the reeds. While doing that I had another bird fly through and perch; my initial thought was warbler, when I got my binocs on it, the face seemed more like a kinglet, but it had bright yellow on the chest. Coming home and researching, I realize now it was a female common yellowthroat, so my first guess was pretty close (I was initially thinking yellow-rump but no yellow on top or back).
A couple of birders reported a peregrine playing around near island 1; I didn’t see it, the terns did and weren’t happy.
Other birds seen included canada geese (which seemed to be migrators, not feral, and not terribly friendly), snowy and great egret, black-crowned night heron, one great blue heron, white pelican, a few mallards and a couple of pied-billed grebes, double-crested cormorants (lots of blonde younger ones), turkey vulture, lots of stilts and avocets, two really, really, REALLY cute baby stilts still in down on one of the islands (3, I think), one practicing catching bugs, one practicing swimming, yellowlegs, dowitchers, red-necked phalaropes (50+), western and least sandpipers (my brain cramp of the day: “least sandpiper. that’s a lifer. yeah, right. it’s semi-palmated I need.. gah). swallows, anna’s, and the usual cast of characters.
the golden plover, by reports, has moved onto island 5 and evidently went to sleep there, so it may hang around. the ruff is definitely hanging around, and well worth going and looking for; patience is needed because of its tendency to wander the far side of the island. When I was there, it’d make an appearance every 5-15 minutes for a bit. The black tern is around, look for the tern that isn’t acting like the Forsters — it tends to fly much closer to the water and swoop/skim rather than dive/plunge.
(and Ruff is 204 on the life list, wilson’s phalarope 205, and black tern 174 on the year list…. finally over 200….)
When asked about cut-and-paste support, something that many iPhone users—ourselves included—have clamored for, Joz said that the feature simply didn’t make—if you’ll pardon the expression—the cut on Apple’s priority list for the latest software release. There’s nothing against cut-and-paste, Joz claimed, it’s just that other features were determined to be more in demand.
I think Joswiak and Apple are being a bit disingenuous here, but for all of the right reasons.
There’s a deeper issue that needs to be examined here, but it boils down to a few key points:
1) Once you implement something, it’s really hard to throw it out and replace it with something better; Apple’s more willing to do this than most companies (think, for instance, the “new” iMovie in iLife ’08 — and the whining that happened; personally, Apple was right, IMHO, but that’s a different blog entry — but most of the whining came from folks who honestly should have moved to Final Cut Express long ago and were pissed when Apple took iMovie back into being a “my mother can use this” entry level app)
2) One of Apple’s core values is “do it right”.
3) Something as core as cut and paste isn’t shipping until it passes the “Steve test”; and Steve is not big on “well, it’ll do”.
4) it’s easy to do cut and paste badly on an iPhone. Or even do it in a “hey, this doesn’t suck” way. But doing it the Apple way?
Basically, I think the real reason this doesn’t exist is because Apple knows once they implement it, they’re stuck with it, and so they’d rather not do it at all until they do it right.
And they’re right. It’s a lot easier to fix “we don’t have cut and paste” than “damn, but cut and paste sucks”.
but it’s a lot easier politically to simply say “hey, there are other priorities”. to a degree, he’s right; the priority he’s implying but not explicitly bringing forward is “we want to make sure it works like an Apple product and doesn’t suck” first.
And that’s why Apple sold a million of these buggers already; because they are careful about core functionality and compromises, and the geeks know it. Few companies are willing to play the “better to not do than do badly” game, much less Apple’s “… than do so-so” standard.
As an aside, since the Xbox 360/Netflix agreement has brought it forward again:
this is why Apple hasn’t done a PVR or PVR software for the Apple TV. There are so many factors out of its control — anyone who’s hooked one of these bastards up understands — that building a PVR that “works like an Apple” is somewhere beyond difficult and towards impossible (which is why so many of us would love something like this; it solves a problem nobody’s really solved, even Tivo, where interfacing to random cable boxes in random ways is still a bit of a horror show)
FWIW, I like the Xbox/Netflix deal. It’s impact on Apple and iTunes is less than most people think, because it really comes down to whether you prefer a subscription model (netflix) or a pay per view model (Apple), and neither model really matters for online video until both platforms fix the “there’s no freaking content” problem — the amount of downloadable content on Netflix is still a tiny proportion of it’s library, and bluntly, Netflix’s real value is in its library, not its technology. Which is why, everytime I talked to someone in the iTunes group when I was at Apple, I used to harp on “we have to buy Netflix” until they finally told me to just shut up… But iTunes with a subscription and PPV model and Netflix’ library depth and an Apple TV is one hell of a business proposition… Still is, but Apple never showed any significant interest in it, even though some of us wandering the project and its peripheries thought it was a killer combo.
But that’s ultimately why I haven’t bought an Apple TV (or a Roku) — neither gets me access to much of the content I want, which is the library beyond the last 3 years of hit movies. Talk to me when I can stream, as, Big Chill or Season 5 of M*A*S*H to my Apple TV on PPV prices.
So the dust is settling, and the new sharks roster is taking shape, and I’m finally back at a point in my life where blogging seems not only possible, but interesting. Been an interesting three months.
So now we can start to look at the Sharks for 2008-2009 and see if this is a better team. Is it?
First, coaching: I like the hiring of McLellan, but it’s not without risks. Sometimes a really top-notch assistant coach is — a top-notch assistant coach. He could be the next Bruce Boudreau or John Anderson, but he could also be Dave Lewis or Wayne Cashman, two guys who tried to make the leap to NHL coach and found out they made damn good assistant coaches. Or he could be Kevin Constantine, who’s a pretty damn good coach, just not at the NHL level.
So the move is not without risk, but the Sharks aren’t afraid of taking risks, and I think this one makes sense. I’m a lot happier with the idea of bringing in a new voice that has some ability to relate to younger kids than to bring in a “safe” retread who overplays veterans and doesn’t grow his players. I think it’s a good hiring, and I’m looking forward to seeing how he fills out his assistants.
A while back, I wrote about what I thought should be (or would be) changed in the roster offseason. A few highlights and lowlights:
Two for Elbowing: Picking up pieces and an update on Michalek – The San Jose Mercury News Sharks Hockey Blog -:
the Sharks are a damn good team, but it’s clear changes need to be made for the team to get better.
I’d like to see Nabokov backed off to 60-65 games next year (his going to the world championships notwithstanding). Rest him a bit more, keep him a bit fresher.
If that means bringing back Boucher, or someone else, so be it.
And so it is.
Core group (do not touch under penalty of death):
Not coming back:
Curtis Brown (Sorry, Brownie, but I think it’s time).
All of which happened. I honestly felt a top six forward would go — I’m happy that McLellan and Wilson think this group can be kept together and improved without being swapped around.
Players I expect back, but which aren’t “no trade under any cirucmstance” types (as part of the right deal? sure):
I want to see come back:
Rissmiller was allowed to leave, Shelley is back, as far as I can tell, Plihal and Goc are still unsigned. Plihal will be, Goc, not so sure. If of this crew we lose Rissmiller (like him, replaceable) and Goc (like him, somewhat disappointing), I don’t think the sharks miss a beat. No game changers.
So where does this put the Sharks?
and two black aces to be named. Goc maybe one of them.
It’s kinda hard to complain about this roster, especially if they play to potential. So I won’t. you can see why Rissmiller wasn’t kept, when guys like Setoguchi and MItchell and Plihal are having to fight for third line time?
Now the fun begins. The Defense was the thin spot on depth last year. I thought going into the season it was good enough. I was wrong. This year, it’s looking a lot different:
Core group (do not touch under penalty of death):
Douglas Murray (what an improvement this year!)
Matt Carle (struggled at times, but seems to be growing into it; I’d hate to give up too soon)
M-E Vlasic (wow; at his age?)
I’d like to see back:
Brian Campbell (but not for Phaneuf money; if someone wants to pay him that, be my guest. he’s missing that “punk brat” aspect to his game, which keeps him a rung below Phaneuf on the ladder. But $25m over 5 years? sure. Just not $30 over 5.
hint: I expect Campbell to stay. He seems happy. He likes playing 30 minutes a game. Why screw it up?
Not coming back:
Sandis Ozolinsh: thanks, Sandis. for everything.
Alexei Semenov: ditto. Neither of these are NHL caliber in today’s NHL.
Kyle McLaren: love his guts and drive, but his knees are problematic. I think it’s time to consider an upgrade.
So one of my “untouchables” goes away in Matt Carle, but we’re getting (if rumors are true) Dan Boyle in return. We lose Brian Campbell, but for the money he’s getting, I hope Chicago enjoys his play. Carle was a lot more expendable to me than Vlasic, so I’m happy.
So our D now looks like:
Rob Blake-Dan Boyle
Murray-lukowich (rumored coming from tampa)
Again, not much to complain about here. Blake/Boyle/Lukowich instead of Campbell/Carle and either Semenov or Ozolinsh? It’s a more veteran crew, but I like what Blake brings to the team in intrinsics, even if we’re giving up some youth to get it (indirectly, because losing Campbell makes bringing Blake in and getting Boyle possible — although I get the impression Wilson was going to bring Blake in anyway).
This is a team that’s now completely oriented towards the next two season. Yeah, after that we’ll have to see about bringing youth in and reloading, but that’s wilson’s problem later. This team needed to be about “NOW OR ELSE”, and now it truly is.
In retrospect, two problems last year:
no backup goalie to take the load off of Nabby and limit his playing time a bit. I don’t think this really hurt the sharks, but I don’t want my goalie playing that many games.
The defense was too young and too thin; trying to patch in with Semenov and Ozolinsh was the red flag, and that proved to be true.
One thing I can guarantee: Doug Wilson will do something completely different than this, and when he does, I’ll go “wow, I never would have thought of that” and like it. Whcih is why he’s GM, and I’m a blogger…
Well actually, Wilson’s done pretty much what I expected; couldn’t re-sign Campbell, went and got Boyle. I have to admit that Rob Blake was one of the guys I thought would be great on the Sharks — only I never thought he’d leave the Kings, so I didn’t really consider it an option. Fortunately, Wilson did, and Lombardi (if you ask me) mis-stepped here. but more on that later. But thanks, Dean. I expect Detroit will send you flowers for helping us (not).
I can’t see how Wilson could have handled this better, given things not under his control (campbell couldn’t be forced back without seriously overpaying him, which the Sharks don’t do). I’m really happy they didn’t move Marleau, I’m really happy they didn’t make any “make a splash” moves at the draft and overpaying to do so. It’s all a very solid, methodical, well-thought out strategy.
So far, a great offseason. And what it ends up doing is sending a big message to the players: no excuses. Now, it’s up to the players.
Can’t wait for camp.