A couple of minor “where I am online” updates…

I had someone ask me why I had switched from posting my shared link posts to posting a collection of my twitter feed. The answer is simple: when Google updated Google Reader, they removed the sharing option I was using to queue up posts for the Shared Links, breaking my setup. I played with a couple of options to replace it (funneling them through Instapaper worked, but seemed the wrong answer to the wrong question), and finally decided to just consolidate it into twitter, because (a) I could with minimal work, and (b) I didn’t have a better option I liked.

To be honest, I think it makes the feed too noisy, and I’m not thrilled with it. I keep hoping that Google will release an RSS feed out of Google+, which I think would make a better option for this. Or maybe I’ll go see about creating something with iffft. I’m also thinking that maybe this could be a custom app I build that I feed stuff to, that spits out the articles once in a while. In other words, I’m doing this until I decide what a better solution is and I get that built.

My general view of what I want to post on the blog looks something like this:

  • Photos, with or without some supporting content (generally short, 1-2 paragraphs)
  • Long form writing (> 1,000 words), where I work on a topic in more depth and spend some time putting the article together. In general, when I go beyond 2,000 words, I’ll tend to split it up into multiple parts, because I find really, really long form gets unwieldy and people stop reading.
  • Short form writing (200-500 words), a quick note on something that doesn’t need in-depth analysis.
  • Links, which optimally are in a digest/list format. These are designed to give those links some visibility and Google juice, and are to bring to your attention stuff I find interesting and well-written. Please note “interesting and well-written” may or may not include “agree with”. These are things that I don’t feel warrant my commenting on, though, because I find blog posts that boil down to “hey, this is neat, read it!” ungodly awkward and boring…

The link setup has annoyed me since, well, forever. I mostly like how Daring Fireball and Duncan handle it, but I still think doing links as a once-a-day digest reduces the noise factor and pulls them together better. I’m at the point where I’m ready to build a quick web-app that I can feed it to and a cron job on the back end to suck it out once a day for posting. Hmm. I wonder if I could do something like that with evernote and a special tag? (Hmm. have to go explore that…)

In any event, the Twitter feed stuff is “good enough” for now, but not what I want long term. And I’m open to suggestions on ways to solve this problem, whether it’s something off the shelf, wordpress plugins, or other ideas. But I hadn’t thought about doing it via Evernote until just now; I have to go look into that…

I should also note in passing that I killed my 500px.com account today. I can’t say anything negative about 500px, I simple never figured out what I wanted to do with the site that made it worth investing time into building the site up or putting effort into posting and interacting over there. In my continuing effort to not let “keeping up with my social media stuff” take over my life and lead to information bankruptcy, sites like this have to fit into my long-term ideas for where I want to have my stuff exist. I could never find a way to use 500px that seemed like it added anything, it all seemed to duplicate other things i was already doing.

I like 500px; they do a good job of displaying images and taking care of photos. Their social media aspect is decent. At some point, I may figure out how to leverage the site. When I do, I’ll go back. Until then, I didn’t want something that never got enough time and energy to stay up to date hanging around looking vaguely abandoned (which it mostly was).

It’s not you, it’s me (and in this case, I’m not just being polite…).

 

 

we’re having the comment fight again…

While I wasn’t looking, it looks like the “Comments: good idea, or tool of Satan?” fight has broken out again. Matt Gemmell fired it off:

Comments Still Off – Matt Gemmell:

Just over a month ago, I switched comments off for this blog. I wanted to post a very brief follow-up on that decision.

 

In a nutshell, it was definitely the right move.

 

but a number of people with a clue have chimed in, including:

 

That’s some heavy talent with a lot of experience in dealing with the practical realities of this issue. Who’s right?

They all are. It comes down to what you’re trying to accomplish and what you want for your own blog or publication.

I will note for the record that this discussion happened across the various blogs for the most part, and also note for the record that if it had happened in the comment section of any of the blogs except for Fred Wilson’s, it would have gotten buried and almost nobody would have seen it because comments are notorious for not ending up in RSS feeds, search engines and the like, and most rational people get to about the third troll in a busy comment area and bail out, because they have better things to do than wade into the mosh pit.

Which is my way of noting that while comment sections definitely can work (and do, if you work at them), most comment sections fail the “why am I looking at this?” test pretty quickly, Fred Wilson’s blog being a notable exception. And Youtube being a site that proves the rule beyond any need to argue, because, as usual, absent landlords end up breeding slums.

Now, I use Disqus on my blog, and Akismet, and I have almost no spam problem, because my blog is small and generally ignored by the spammers and trolls. I’m also pretty careful to vet comments and back links and don’t encourage trolls and don’t post trackback links that point to spammy sites, which I think discourages them from trying a bit. And mostly, because I’m small I don’t get lots of comments in the first place. If I got popular (hah! not likely) and started seeing high numbers of comments (I wish!) I might change my mind and go commentless without feeling guilty. I think right now, they’re a net positive to my site, but I long ago stopped seeing them as necessary, required or some kind of freaking inalienable right like some people (mostly trolls, I think) do. Heck, if I were a troll, I’d demand free places to do my trolling and insist on no adult supervision, too. I’d love to spend other people’s nickels to spread my opinions…

So my bottom line is that comments are useful, but are mostly broken. You need to put too much work into them to keep them useful — even disqus, which I think does a better job than the others I’ve looked at. But I’m not sure “nuclear” is the ultimate answer here, either.

Some suggestions:

I’d like a way to configure Disqus to turn off commenting after a period of time (like 30 days, or after 3-4 days of no comments); there is little reason to carry on a conversation after it dies down the first time, and so open comments (and trackbacks on blogs) after a couple of weeks is useful only to spammers; reduce the places they have a chance to lay their stuff by turning comments off on older material.

I’d like a way to feature good comments, give them a visibility that doesn’t exist right now. Great example: The Online Photographer, which as far as I can tell, is manually editing them into the body of the article. It’d be awesome if Disqus supported a way in the admin interface to click a checkbox “feature this” and have them appear “above the fold”, so that we can start curating the good comments into the conversation stream as a way of giving them visibility, instead of only trying to keep the noise down by moderating out the worst stuff.

But really, this is a job for a reputation engine. Disqus is well suited to implement this, and spread a reputation across all sites that use Disqus. Allow a site to define what minimum reputation is needed to display them on a site, and track the +1 and abuse flagging back to the Disqus user to generate their reputation. the trolls will sink, and a site owner can choose just where to draw a line and say “below this, you don’t get on my site”, either by not accepting comments or not displaying them. And then let a disqus user override that on an individual basis if they want. even a decent reputation setup with some minimal metrics would make it a lot easier for a site to choose whether to display or dump the trolls, and if someone does post a troll note, let the other users vote it into oblivion if they want.

I think there’s still a lot of life in comments. Fred’s blog shows the possibility, just as this discussion about comments shows how well the alternate possibility (distributing across many blogs) shows how well it can work as well. But to make the kind of environment Fred’s fostered work without the kind of fostering that someone like Fred (or Teresa Nielsen Hayden does at Making Light) we need better technology underpinnings. Most site owners/admins/moderators don’t have the “touch” to guide a community into becoming what Fred and Teresa have. Or maybe they do, but not the time or will to make it happen.

But isn’t that what all this technology is about? finding ways to enable these things and free humans from having to drudge through the grunt work? And moderating comments is drudge work. serious drudge work. With some thought and some code, we can enable the community to self police itself here. So why not do it?

(and just because I can, here are some previous rants on this topic from previous rounds of this discussion: 2008, 2003, 2011 (think comments as critiques here)

Note 1: my infamous emacs vs. VI joke: What’s the difference between an emacs user and a VI user? Give the Vi user a file and set of changes and they will sit down and edit the changes into the file and then go to lunch. Give it to an Emacs user, they’ll sit down and code a macro that they’ll use to make the changes automatically while they’re at lunch. Afterward, both of them have the changed file, but the Emacs user has a macro that he’ll file in his library of macros and never touch again in his life.

 

Flickr deletes 5 years worth of photos

Flickr deletes 5 years worth of photos | Pixiq:

Imagine, for a second, that you have a Flickr account. Then, imagine you’ve posted more than 4,000 photos to it, and that Flickr deletes it by accident.

 

Next, imagine that you contact Flickr to find out what the hell is going on, only to be told that ‘whoops, yeah, we did that by accident. We’ll re-instate your account, but your photos are gone’.

That’s what happened to Mirco Wilhelm, in an absolutely cruel mix-up. He had, in fact, been in touch with Flickr to report another user who had been posting photos that were infringing on copyright – and he included his own Flickr name in the E-mail as well.

You know, mistakes happen.

And since mistakes happen, we should plan for them.

So if this user didn’t have a backup of those images under his own control and on his own disks, he has some blame here. Nobody should put the only copy of stuff in the hands of a third party; honestly, you should never have a single copy of ANYTHING you care about. So everyone who’s using flickr and not keeping a private copy of that data, you’re being naive in trusting flickr with your data and no alternative.

But having said that, don’t think for a second I’m letting Flickr off the hook here. But also, don’t think for a second that this is only a problem with flickr.

Almost every company, whether it’s yahoo (and flickr), or google, or Facebook, or anyone, does a crap job of account management of social data. This is just the latest in an endless round of “I woke up one morning and my account was gone” — I recently saw one with Facebook nuking some of Microsoft’s pages, and the horror stories of waking up to find your gmail (and everything google tied to it) being gone and the labyrinth you go through just to get a response? this is an endemic problem, not flickr specific).

It is far past time for companies to get serious about account management with social accounts and data. That means there’s a warning process in place, that means there are suspensions before termination, and an appeal process, and ways to contact companies and get answers and be able to discuss the situation.

It means companies have to design their systems to allow for suspensions — that data becomes unavailable, not GONE — for a period of time so that mistakes can be undone, and for affected users to appeal and for that appeal to be considered. It also gives companies another tool in their chest for managing users, the time-out. Trust me, that one can be very effective when used properly. And frankly, I’m amazed corporate lawyers haven’t asked for terminated accounts to be kept around in case there’s some need for legal forensics in case of a lawsuit or police request…

Can you name a major company that does this well? Google is notoriously opaque and hard to deal with when they decide an account needs to be terminated, and they’re notoriously bad about notifying you that they’re upset about things prior to terminating thigns. You wake up, and it’s all poof. And if you have your analytics, your email, your google docs, your domain registrations, etc, etc tied up to taht account, it can be really painful. no wonder people look at google at times and twitch a little.

flickr and facebook are not that bad, but both run into this on a fairly regular basis. It’s NOT a lot of work for flickr to implement a suspension wall around an account, and it’d save them one or two major PR headaches a year if they would do that and use it to suspend an account for 30 days before deletion, just in case they make a mistake (like this) or there’s an administrative judgement that needs to be reconsidered. To me, for all of these companies, it shows a disregard for the value of the data we’re trusting them to hold onto.

Flickr has a second problem — and that’s backups. Since one of the things you need to do is back up your own data, flickr (and all of these companies) need to get serious about making it easy (or easier) to actually do that, to remove some of the onus on the company when these things happen. They really can’t have it both ways. If they don’t want to allow you to easily (and non-geekily back up all your stuff, they have to make a commitment not to delete it and to give you a chance to get it even if they terminate you.

And honestly, if the first time a user hears from the account/community management is the termination notice, the your systems are seriously broken.

That should be really obvious, but nobody seems to do it well yet. Can you think of one?

 

Why they want your email address…

Gawker Hacked: Seeing Past Your Nose | Jeff Nolan – Venture Chronicles:

However, did you ever wonder why media sites force you to register in order to comment? They want your email address and identity information for driving marketing and promotions as well as enabling data services businesses. They provide no real utility in exchange for getting you to hand over a piece of personal information… unless you consider their email products useful.

Two words — trolls and griefers.

I’m actually a little disappointed to see people still don’t understand this — and I’m not pretending that the marketing aspects don’t exist (they do), but if you don’t have some way of managing access, then at some point, the trolls and griefers WILL move in, and you have a problem because you can’t stop them. So you need some form of identification, and the most reliable (but low-friction) way of that is by email address. Part of the reason that succeeds is because the free email services have all had to deal with the spammers and trolls and griefers and are fairly effective and limiting their ability to do high volume email address creation, and you can hide behind their shoulders to a degree by using email as an identifier. For these kinds of situations you don’t need to know who a person is, but you do need to know that this person IS a specific person, so you can block further access by them.

Take a look at just about any site that doesn’t have registration, and you see a commenting/discussion area that’s useless because it’s been taken over by trolls, griefers, porn spammers and people who believe the way to win an argument is to be the last person screaming.

Bottom line, if you run a nice sports bar where people gather to talk about football or hockey, and a band of bikers drive up and wander in and start demanding beers, you have two choices; you either kick the bikers out, or you turn into a biker bar, because they’re going to cause everyone in your primary audience to stop coming and go somewhere else. And today, online, the most effective limiter for these situations is the email address. The occasional really motivated troll that’ll continually reinvent identities to come back and keep abusing your site can usually be handled as a special case, and msot of the bikers can be kept out by choosing which addresses you can block and prevent further abuse of your system. Don’t have that, and you’ve lost.

 

Proposals For Librelist Moderation Strategies

To understand the feature requirements for moderation we need some goals. Keep in mind that no moderation will be perfect, and you can easily come up with scenarios that will work around anything we come up with.

Therefore, we should focus on just some initial goals that will work right now, and keep in mind that these will need to be constantly tweaked and worked on as the spammers evade the measures.

  • If given the choice between restricting free speech and preventing unwanted communication, free speech always wins.
  • The system should increase the quality of discourse for any project, regardless of human language used.
  • It should never give a small group the ability to hide communications from others.
  • It should be implementable and not have high hosting costs.
  • It should not rely on a dedicated person’s constant intervention.
  • It never gates email through system before sending it, but rather allows initial emails with moderation after.
  • It should use information from people’s rating habits to classify them as “ratings trolls” to prevent abuse.

With those goals in mind I’ve teased out two potential list quality strategies that might work.

via Proposals For Librelist Moderation Strategies.

Someone I work with turned me on to Librelist because they knew me interest and history with mailing list systems, and I find it interesting that some folks have decided it’s time to rethink the mailing list again.

They’re right. When I faded to black on the mailman project, it was at least in part because many of us felt that mailing lists were a technological dead end, and that deliverability issues because of anti-spam systems made the “personal mailing list” an increasingly difficult thing to accomplish.

Both are — for the most part — true. I certainly would never run my own mail server again, because the advantages of doing so are far outweighed by the time and hassle of trying to manage deliverability and reputation to make sure mail it sends gets accepted, and the constant onslaught of incoming spam turns them into a permanent infinite time sink. That’s why I either retired our lists or moved them to Yahoogroups (which I personally think is a pretty good system).

But there’s still room here to rethink the concepts and the Librelists seem interested in trying, and I think that’s great. Email and mailing lists are far from dead — but instead of stand alone delivery tools, they really shine as part of an integrated web strategy; Yahoo groups is a nice first generation of that, although there’s a lot more Yahoo could do if they decided to.

Message moderation really breaks down into two big problems:

  • “Subscribe spam” where spammers sign up to the list to spam it.
  • “Member warfare” where existing, approved members get into fights and they escalate into unacceptable territory.

The first is really simple to solve: new members are moderated, and messages aren’t posted until reviewed by someone to vet their content. Simple implementation; Yahoo Groups does it today, and on the lists I still manage, it works well to keep the spammers at bay. The way I manage it is all members are moderated until their first post. if their first post is acceptable, I turn off the moderation bit. To minimize delays in propogation of new member messages, simply choose a moderator pool large enough to guarantee held messages get reviewed and approved in a timely manner — you could even make that moderator pool all members in good standing if you want, because all you really need is someone you’ve trusted to post vetting that someone new is trusted to post.

Member warfare is trickier. I hesitate to call it trolling because the pure troll is a subset of the larger issue of two (or a small group of) people getting pissed off and going at it. A troll is simply one person going off on the rest of the list.

I’m more and more convinced the answer here are reputation systems, where over time a user’s membership in a group is used to define their abilities and restrictions. The longer a member is in the group in good standing, the more often they contribute material, the higher their reputation goes and the more the can do and the more sway they have on the decisions of the reputation engine. You can tweak the details of the algorithm almost any way you want, but if you define it in terms of “how long they’re a member” and “constructive contribution to the community”, you can come up with a metric on how valuable that member is to the community, and then use that to rank that member’s contributions and recommendations.

Here’s one rough view of how to build this. Please note that I firmly believe karma rankings are private and users have no way to see what their ranking is or compare it to others, except in really broad user categories (“member”, “senior member”, “top contributor”, “advisory board member”). As soon as you create a list of any form, you will attract people who see it as something they can game, and so they will.

User Karma is a value between 0 and 1, which starts at 0.5. Every time a user contributes to the system (a posting, a reply, a moderation recommendation, etc), the number gets bumped by some value. How much the value is incremented or decremented depends on how it’s rated by other users — so if User A posts a message, User B flags it as spam, but 80% of the membership feel that was a bad decision, User B’s karma is reduced in future decisions, they lose influence. Over time, the system self-corrects by giving increased influence of those who’s decisions match the community consensus and reduced influence to those who’s postings and recommendations don’t match up well.

The system can then choose whether to accept or flag for moderation a posting based on a poster’s karma score. You could potentially reject outright users that have karma scores below some value, or allow other members to choose not to see messages by users with karma scores below some value. Over time, users who are disruptive to the community will get karma’ed into the moderation queue (or out the door), and users who are seen as top contributors will have stronger influence.

My goals:

  • A system like this can be built nicely with a good SQL backend and a bit of horsepower. I’ve actually done a detailed design and schema on this before, and it’s a fascinating thing I’ve always wanted to implement.
  • It enables the power of individuals to police themselves.
  • It limits the ability of an individual to harass or cause problems.
  • It doesn’t lend itself to people playing the game of gaming the system by not exposing the details of the system (slashdot karma whores need not apply).
  • Trolls get edited out of the system because the community will quickly recognize them for what they are and trash their karma, causing their postings to disappear to the bottom of the list.
  • Cliques and Mafias have to be large to influence the results significantly. You don’t completely avoid the clique/mafia problem, but you can severely limit it’s ability to wreak havoc.
  • It doesn’t require a lot of manual handholding or babysitting. Admins end up stepping in only in extreme cases.
  • Because trolls tend to get edited out of the system quickly and automatically, they tend to go elsewhere because without feedback and controversy, they wither and die. And by editing them out of the system quickly, you avoid the whiplash and fighting that happens when people start fighting with the trolls and the wars break out.

Weaknesses:

  • Any community tends to turn into an echo chamber. Automated systems like this encourage this because “different thinking” tends to get rated down.
  • That’s usually a lesser evil to letting the trolls run wild.
  • To my knowledge, nobody’s ever solved the problem of the conflict between the group-mind reinforcing the echo chamber and allowing the free thinkers to poke at the community’s comfort level by pushing them to think about things that make them uncomfortable. One person’s rebel is another person’s troll, and that’s not solvable in real life, much less in automated life like this…

These techniques are all based on (or stolen from) things that are in use around the net, with Amazon’s review feedback being one I really respect; while trying to avoid the pitfalls I’ve seen around the net (yes, I’m going to keep bashing on Slashdot’s karma system, it’s way too easy to game and always has been). It also (I believe) avoids the nasty politics that have made Digg a bit of a pesthole. And it’s also pretty lightweight and low-key, or at least it should be. The implementation details will be crucial, as will be tuning how the karma values adapt…

Comment behaviour: How far is too far?

According to Greenbaum’s blog post (which was mirrored on his personal blog), someone posted a comment on a story in which they used a colloquial or slang term for female genitalia. It was deleted, but then was reposted. Greenbaum says he noticed that the comment alert from WordPress showed that it came from a nearby school. So Greenbaum called the school, and they asked him to send them the email with the comment, which he apparently did. About six hours later, he says, the school called and said that an employee had been confronted and that he had resigned.

Am I the only one who thinks that doing this goes way beyond the normal course of editorial behaviour?

via Comment behaviour: How far is too far?.

There’s been some interesting commentary on this case — but there are some aspects that I think haven’t been addressed very well yet. It’s a more complicated situation than many have considered, and the answers really aren’t clear cut.

Here’s my take:

There are really two separate issues here.

Did Greenbaum over-react by reporting this person to his employer?

Yes — but.

Yes, he did. In the grand scheme of things, reporting a violator back to their host is a serious thing because it can have serious implications — like getting someone fired. Which effectively happened in this case. So it’s a last resort thing. Before you do something like that, I prefer taking many other tactics first:

  • Delete the post
  • Warn the User
  • Ban the User (and ban the IP and/or IP range as necessary)
  • Make it clear that if it doesn’t stop, they’re going to be reported

If those all fail, or if for some reason aren’t possible, THEN you start considering going back to the user’s host for support in making the behavior stop. As far as I can tell, only the first was tried, so a number of (to me) necessary steps were skipped. This could have been ended with much less serious ramifications, and wasn’t.

However, here’s the butt:

  • The post was deleted, and the user insisted on putting it back. The admins made it clear it wasn’t acceptable, and the user decided to overrule their authority. This user was far from innocent here.
  • Once the user is reportd back to their host (and I use that term carefully, because it’s many times unclear if it’s an employer or what, and to some degree it doesn’t matter if it’s an ISP or a boss or whatever), it’s out of Greenbaum’s control. The rest of the escalation to losing the job was the result of actions of the host (i.e the school, or this person’s boss). None of that is caused by Greenbaum (directly) or his fault, beyond that he should have been sensitive to the fact that his action in reporting might have caused other actions to happen.

So, you know what? I think Greenbaum’s transgression is a lot less serious than the user’s transgression in reposting his vulgarity after it was made clear it wasn’t welcome. I would have tried other tactics to cut the abuse, but let’s not forget that it was abuse, and it was repeated abuse after the site made it clear the posting wasn’t welcome. Whether you shoot over someone’s virtual bow one time or three times is a minor thing in the scheme of it.

The user’s fault in this problem was a much bigger problem than Greenbaum’s reaction.

But what about the school? They’re the group that took the complaint and escalated it into a situation where the person lost their job. None of that is Greenbaum’s fault. Was the school wrong for turning this into a termination issue?

I’m not so sure. It’s easy to say they over-reacted, but let’s not forget:

  • This person did this using the school’s network
  • It looks like he did it while on duty at the school – while he was being paid by them.
  • He likely was on a school-owned computer
  • He was (I’m sure) under some kind of employment contract with behavior clauses. The school very likely has acceptable use standards for computers and networks, and for all we know, also personal use restrictions (which this would be a violation of).
  • So while this cascaded into a situation where someone lost their job, it’s not at all clear that the details of the action were the cause. We also don’t know if this person has a history of previous violations of work rules that might have been part of this. Has this person been warned about this kind of behavior before? We don’t know. It could well be from the school’s view that this was a “last straw”. We don’t know.

And those complications are why I believe reporting back to the host is something not to be taken lightly; once you do, the final outcome is not really under your control. On the other hand, the person who could have prevented this was the user who posted the vulgarity — either by not doing it in the first place, or by stopping after it was deleted the first time, or by being smart enough to not do it from his employer on company time and company equipment. He had plenty of opportunities to not turn this into what it was; Greenbaum had one.

And it’s not as simple as many of the folks commenting on it want to be. Real life never is…

Fighting sockpuppet reviews on the App Store

App Store reviews have been controversial from the beginning — while they can be helpful for buyers, you often have no idea just who’s leaving comments or what their real agenda is. Njection, the makers of Nmobile (which we played with a while ago) are having a huge problem with what they’re calling “sockpuppet” reviews on the App Store.

Someone (they believe this person is in cahoots with their competitor) is posting bad reviews on their app and trying to trash them and their product elsewhere (including in a comment here on TUAW). And unfortunately, as they say, they don’t really have much recourse against this behavior — they’ve appealed to Apple, who’ve replied that they’ll leave comments up, unless they’re offensive or extremely false. Apple’s own guidelines for reviewing apps asks that the reviewers deal with apps on their own merit rather than attacking competitors, but that seems to be more of a recommendation than a firm rule.

Njection says the comments have kept consumers from trying out their apps, though it seems difficult to actually track how many people haven’t tried your app (and why). It’ll be interesting to see if Apple makes other changes to the review system if this sort of thing rears its ugly head more often. At this point, it seems devs just have to deal with it by doing damage control when necessary and making their app good enough that “sockpuppeting” doesn’t strongly affect public opinion.

via Fighting sockpuppet reviews on the App Store – The Unofficial Apple Weblog (TUAW).

I guess I’m not convinced. Looking at the app in question, there are a total of twelve reviews, seven of them 4 or 5 stars, only 3 gave it one star. Since others have the ability to rate the usefulness of each review, there’s some feedback going on with the reviews themselves, and so it’s not until the ninth review that you get a rating of less than 3 stars when sorted by “most helpful”. that seems like some fairly positive reviews overall.

Given that Apple only allows one review per store account and that account has to have bought the product, it’s rather hard for me to see a significantly organized turfing attack here. I don’t know which is reality, but my gut feel is that the turfing worries are overblown.

You could also think maybe it’s a developer looking for a way to explain away bad reviews. And it presumes that the developer didn’t have their friends all log in and report the five star reviews, too. Turfing can go both ways, of course. Not that we’d ever do that — and not that I’m implying the developer did. Definitely saying they shouldn’t, FWIW.

Having said that — there are some ways to limit the impact of turfing if it exists.

First: free limited versions. If users are hesitant to pay for the App because of some bad reviews, then give them a way to trial the app before paying. That’s been very successful with me trying out various free versions of apps on the store and then buying the full version. there’s really little reason to NOT do this, and yes, Apple really needs to formalize support for this in the store in some way, but until then, Lite versions rock, and remove the worry of buyer remorse.

Second: Yelp has this same problem. One way its gets solved is via high numbers of reviews. The larger the set of comments on something, the less impact any individual or turfing campaign can have. So a simple way for developers to limit the impact of turfing attacks is to encourage the users to submit their own reviews. Something as simple, perhaps, as when they fire up the app after having used the app for some period of time, putting up an alert encouraging them to review the app and explain how. add in a couple of buttons (“take me there”, “not yet”, “stop annoying me”), and make it as easy as possible for them to put the review in.

If you think about it, if your users are happy with you, a percentage of them will go and say so. And that stream of reviews will blow out any impact of a turfing attack.Of course, if the users aren’t thrilled, you might get buried, but you wrote a great app, right? aren’t afraid of some criticism, right?

There are other things you can do — a lot of it boils down to giving users information about the person writing the review so they can evaluate the reviewer and decide how much to trust them — and I went into some details on my ideas on that a couple of months ago. Most of that would be relevant to upgrading the App Store reviewing system. Honestly, though, I don’t think it’s all that bad these days. Could be better, but the big missing piece is the ability to do free demos. I expect Apple to solve that at some point, but developers can do something about it on their own.

I can’t think of an app I’ve used that suggested I go to the store and review it, though. Why the heck not? Free advertising, folks. Do it in a tactful manner, and I’ll bet a good chunk of the users will cooperate. Seems to me the BEST advertisement for an app isn’t a five star rating, but that 500 or 1000 users reviewed and recommended it. That’s what you want to aim for.

Open Source Communities – Push cx

Open source projects should be judged as much by their community as by their technological achievements. The code tells you what it’s good for, but the community tells you what its future is.

Communities need to be active to continue improving the project, deal with bugs and changes to their ecosystem. If no one is interested enough to talk about the project, none of that will happen. Newcomers need to meet experienced users to be sold on why to use the software, to get help as they learn their way around, to maybe be drawn into contributing to the project itself.

via Open Source Communities – Push cx.

I nice view of the dynamics of communities by Peter Harkins. One of the aspects of this, I think, is that from the communities I’ve been involved in over the years, the smaller the set of people actively involved in the decision process, design and implementation, the more sensitive that project is to fading or falling apart if the life or motivation of a key member changes. For that reason alone, communities really need to foster new members into the project and ways to recognize and enable the most effective and capable into the “inner circle” where they’re ready and able to step up and move a project forward. If you don’t do this kind of “succesion planning” from the start, when you need it, it won’t be there.

Geeks tend to think you don’t need marketing, but they’re wrong. Marketing, even of an open source project, is key to enable adoption and convince people to evaluate it and join the project. projects really should consider community growth as a key metric in he success of a community, and communities really need to look at outreach, evangelism, and recruitment to be tasked out the same way bugs, features and documentation are, and those members should be part of the “core team” whether or not they actually code.

One reason it looks to me that Rails has taken off faster than django is simple: the rails guys did a lot of talking and promoting and evangelizing of rails, where the django folks have been quieter and less self-promoting of themselves and the technology.

A technology nobody knowss about may be great, but it won’t change the world.

Social Media is No Place for Robot Behavior | chrisbrogan.com

Social Media is No Place for Robot Behavior | chrisbrogan.com:


I’m sick of robots. Truly. Your automated direct message back thanking me for following you does three things exceptionally well:

1. Irks me because it’s a robot.
2. Annoys me because you ask me to click your junk.
3. Tempts me to go back and unfollow you on principle.

tweetdeck You don’t need to use robots to thank me and click on your stupid website. If you’re too busy to be an actual human on a social network, don’t join another social network. It’s okay. We understand.

I’ve run into a couple of these recently, and it took me a bit to realize what it was. Given how bad a reputation this kind of auto-responder has in the e-mail marketing world, I’m amazed people are adopting them into Twitter.

Please stop.

I’ll go a step further than Chris. If I get auto-DMed, I will immediately unfollow you. Period. I will also follow you if it turns out your twitter feed is nothing more than self-marketing, nothing more than an alternative to the RSS feed, or contains nothing interesting to read. But if you start our “relationship” on twitter with a sales pitch, I’ll treat you like any other telemarketer and not bother giving you the time to prove to me you’re worth my attention.

Don’t be lazy. That’s what these auto-DMs are. You’re better off with absolute silence than this stuff. Really.

Social media at its worst – post-mortem cyber-bullying | Measuring Social Media

Social media at its worst – post-mortem cyber-bullying | Measuring Social Media:


The problem of cyber-bullying hit me hard last week. As I wrote last week, I volunteer as a critical incident stress management debriefer. Most of the debriefings I do are with first responders – fire, EMS, police, dispatchers and similar workers. But our team also reaches out to the community; when you hear on the news that grief counselors are available to an organization after an incident, that’s us. The reason cyber-bullying is on my mind is that over the last few weeks, I have spent quite a bit of time with teenagers who are trying to cope with the suicides of friends.

Confidentiality is paramount, so I cannot offer any details of any incident I’ve been involved in. But imagine a middle school or high school student who learns that a friend has committed or attempted suicide, who goes to that friend’s MySpace, Facebook or other social media home page and finds mean and horrible things written about them. What’s worse, imagine if those things were written after their friend took that awful step.

I came away from one recent set of debriefings absolutely convinced that if there is any possible way to do it, the industry should figure out a rapid way to disable, freeze or at least moderate the pages of any minor who has been a victim of violence. I emphasize “rapidly” because word gets around fast (a whole separate problem; texting is not a good way to find out your friend is dead) and cyber-bullies can post unbelievably nasty messages in no time at all.

One of those important things we tend to forget in designing sites; social sites need some way for the authorities to contact them quickly and reliably and get things frozen while real life gets sorted out. Sad that it’s needed, but it is.

Comments: Messy and flawed, but valuable

Comments: Messy and flawed, but valuable — mathewingram.com/work:


In my new role as the Globe’s “communities editor” (you can find more details on that in this post), I’ve been spending a lot of time thinking about comments — that is, reader comments on news stories, columns, blog posts, etc. The Globe and Mail was the first major newspaper in North America to allow comments on every news story when it launched the feature in 2005, and judging by the ever-increasing numbers of people who use them, they are hugely popular. On some major news stories, we can sometimes get as many as 500 comments.

Comments aren’t popular with everyone, however. Some readers (and even some Globe and Mail staffers, to be honest) complain that too often our comment threads are filled with what might charitably be called “noise” — everything from bad spelling and grammar all the way up to partisan political in-fighting, ad hominem attacks and all-around rude and boorish behaviour. Some say they don’t really care what most people think about a topic, and don’t see the value in having public comments on stories at all.

The big problem is that comments are currently unfiltered; ultimately it’s still part of the wild-wild-west of the internet, and so the people who get filtered out in other areas of the net still show up in comments. Ultimately, reputations seem to be taming the trolls and the flamers, but haven’t really migrated to comments yet. It’s a reason why I’ve been watching things like Disqus — but I keep wondering if distributed reputations for comments is really a positive. We’ll find out.

Think about a typical comment: a site may require some ID/registration, but in many cases, it’s faux-authentication, where you can more or less make it up as you go along. That kills accountability, so users can play whatever games they want without much worry about policing or future impact to their ability to comment; at best, a post gets deleted. Bans are, well, pretty trivial to circumvent if you’re motivated and don’t mouth-breathe.

So where this is all headed, and to some degree has to go, is reputation.

A while back I started a project (which I ended up abandoning unbuilt) that had a lot of the same feel as what Yelp now does. A big part of the design was how to create a reputation system that is:


  1. Primarily or completely automated (or it doesn’t scale)

  2. Limits users ability to “game” the ratings

  3. Doesn’t turn the reputation system into something to be gamed

  4. Actually helps someone decide whether or not to read (or trust) a piece of content

  5. Non-intrusive

Easier said than done. A first approximation are the karma systems of places like Slashdot, but it fails for me on (2) and (3), and is really of limited utility for the key issue, which is (4). It’s more of a chainsaw to help a user hide the worst.

So back to the yelp-like example. You look up a restaurant. There are four reviews of the restaurant, two good, one so-so, one hate. there are a few comments on some of the reviews, mostly people disagreeing with various points.

How the hell is a reader supposed to figure out what this all means? That’s the crux of the comment problem; how to put a COMMENTER in context. First, there has to be a context — and that’s missing in commenting systems today. this kind of harps back to my belief that anonymity on the net is bad, but the net mixes up anonymity with pseudonymity – i.e., I don’t need to know who you are, but I sure need to know that you are you (but I digress; see, if you care, identity proxies, 2004, anonymity destroys transparency, 2007, A group is its own worst enemy, 2008, SezWho, 2007, (who seem to have disappeared behind Disqus), A history lesson from usenet, 2007. That’s a hell of a digression…)

The idea is the basis of reputation systems — that over time, the “real you” comes out, and other users can use that information to judge whether or not to value your contribution — or perhaps tell the system to not even show it to you.

In the Yelp-like system, here’s what I came up with as a first cut. If I’m a J Random User looking at those reviews, what information would be useful for the user to determine what reviews and comments are useful? Try this:

First review: five stars. Best Restaurant Ever. the submitter created his account 2 hours before posting the review, hasn’t posted any content since. Easy guess: it’s the owner, or his spouse, astroturfing. Even if it’s not, you ought to assume it is.

Second Review: 1.5 stars. hate the service. rant. rave. grump. Again, account created an hour before posting, never used since. Obviously someone with an axe to grind. or maybe the waitress broke up with him.

Third Review: 3.5 stars. good food, uneven service, dirty fork. yada. The poster’s been a member for seven months, posted 25 items, average rating 3.8.

Fourth Review: 4 stars. Great food, good service, owner came out and talked. Went back and enjoyed it. Member for 3 months, posted 5 items, average rating 2.8.

Suddenly, with just a few bits of information, things clear up significantly. Astroturfing issues become visible quickly if you simply make it easy to see how active a member is in the larger community — if they’re a hit-and-run commenter, you can bet there’s some ulterior motive (positive or negative). This actually creates a fairly complex web of interactions, it encourages users to contribute to the site to build a reputation, for instance, and that’s good for the community.

Once users have been on the site for a while, they’ll get rated by other users. In my system, I used the rating of the user doing the rating to weight how strongly to count a rating, something I haven’t seen sites try yet, but that is a way to discount the idiots and encourage the strong contributors in a quiet but important way — the less others think of what you say, the less power it’ll have to affect other users on the site. In theory, below a certain number we’d likely just throw your opinion on the trash. Quietly, of course.

Quiet is a big aspect of this; to me, the second you start publishing these “reputation” numbers, it becomes a game of trying to “win” the reputation game. So simply don’t go there. I planned on sticking to the more general five star rating as part of the user profile, but no comparative public stats. Instead, users would be honored with “senior member” type labels based on longevity, activity and rating. Make up half a dozen titles, and allow them to be earned over time as a way to reward your best members. Just make sure that how you determine “best member” actually causes them to contribute and improve the community”. Bad metrics kill.

the final piece, of course, is making this information easy for someone browsing the site to find and use; something like showing the posting account name and rating (chuqui: 1.7 stars), and popping up more detailed info if they mouse over it (3 postings, member for 8 months, this was their 2nd posting and they were a member for a month at the time, last activity a month ago….); for users who want them, you could create slashdot-like filters that would automatically exclude, say, material posted by people with ratings < some number, or with fewer than N postings, or whatever.

The system is still open to gaming — but it’s a lot harder to hide from it, I think. Never got around to implementing it, but maybe one of these days. I’m still mulling bringing it back to life, but not i the original form.

Similar things could be done on a news site, or pretty much any community site. It’s a combination of


  • making people create an identity

  • tracking that identity’s actions

  • allowing other identities to rank those actions

  • allowing access to those rankings in rational ways

The combination of an identity, ranking/tracking and weighting things to discourage the one-post wonders can really put a dent into the sock puppets and trolls. sock puppets get marginalized by not building a track record to base a reputation on, trolls get marginalized because, well, as soon as you start building a reputation on a troll, it becomes self-evident. And if all of this encourages more contributions to a site and more community activity as a way to build that reputation, so that people will want to hear what you say, how is that bad for the community?

And done right, it’d be 99% self-policing and automated. I think.


This view isn’t confined to Globe readers, by any means: in a column in the National Post, author George Jonas said that the Web is like “an elegant restaurant with garbage on the menu,” and that “a huge blackboard on which anyone can write anything doesn’t mean much for those with nothing to say, i.e., most people.” Similar feelings have been expressed by various writers about comments on blogs, and some prominent Web writers have turned theirs off completely. Even the director of BBC News said in a recent speech that while she values comments, they are the work of a “vocal minority” and therefore shouldn’t carry too much weight.

It’s not an elegant restaurant with garbage on the menu; it’s a large, vibrant city where you aren’t even noticing that you’ve self-selected into that elegant restaurant. but otherwise, they’re all right. And the way to fix that?

Build accountability into the system. How do you do that? well, what’s worked so far online are reputation systems. Simply requiring a name and email isn’t going to be enough. And yet, that’s basically what we do today in comments. We focus on identifying someone, but forget that it doesn’t matter if we know WHO you are — it matters that we know whether you are worth reading. A simple identify doesn’t do that. A reputation does.

So the future for “fixing” comments has to be a reputation system of some sort. It’s not (just) about better identification systems, or about giving up. This is an area we’ve just started to explore and innovate.

A Group Is Its Own Worst Enemy

Clay Shirky is at it again. Go read the whole thing, it’s awesome.

Shirky: A Group Is Its Own Worst Enemy:


n the Seventies — this is a pattern that’s shown up on the network over and over again — in the Seventies, a BBS called Communitree launched, one of the very early dial-up BBSes. This was launched when people didn’t own computers, institutions owned computers.

Communitree was founded on the principles of open access and free dialogue. “Communitree” — the name just says “California in the Seventies.” And the notion was, effectively, throw off structure and new and beautiful patterns will arise.

And, indeed, as anyone who has put discussion software into groups that were previously disconnected has seen, that does happen. Incredible things happen. The early days of Echo, the early days of usenet, the early days of Lucasfilms Habitat, over and over again, you see all this incredible upwelling of people who suddenly are connected in ways they weren’t before.

And then, as time sets in, difficulties emerge. In this case, one of the difficulties was occasioned by the fact that one of the institutions that got hold of some modems was a high school. And who, in 1978, was hanging out in the room with the computer and the modems in it, but the boys of that high school. And the boys weren’t terribly interested in sophisticated adult conversation. They were interested in fart jokes. They were interested in salacious talk. They were interested in running amok and posting four-letter words and nyah-nyah-nyah, all over the bulletin board.

And the adults who had set up Communitree were horrified, and overrun by these students. The place that was founded on open access had too much open access, too much openness. They couldn’t defend themselves against their own users. The place that was founded on free speech had too much freedom. They had no way of saying “No, that’s not the kind of free speech we meant.”

But that was a requirement. In order to defend themselves against being overrun, that was something that they needed to have that they didn’t have, and as a result, they simply shut the site down.

Now you could ask whether or not the founders’ inability to defend themselves from this onslaught, from being overrun, was a technical or a social problem. Did the software not allow the problem to be solved? Or was it the social configuration of the group that founded it, where they simply couldn’t stomach the idea of adding censorship to protect their system. But in a way, it doesn’t matter, because technical and social issues are deeply intertwined. There’s no way to completely separate them.

What matters is, a group designed this and then was unable, in the context they’d set up, partly a technical and partly a social context, to save it from this attack from within. And attack from within is what matters. Communitree wasn’t shut down by people trying to crash or syn-flood the server. It was shut down by people logging in and posting, which is what the system was designed to allow

This is a classic pattern that we re-invent on the internet time and time again:

First, we think of whatever we build as something new and revolutionary (which it may well be), and therefore we can ignore past history because it’s not relevant to what we’re doing (which is invariably wrong).

Second, we start using it with a small set of people who generally have a common set of goals and ambitions, so “can’t we all just get along” works. For a while.

Then, if the technology is useful and proves out, the user base expands. the more it expands, the more it gets used by people who’s goals and ambitions are different from the original core group, and the conflicts of “what’s appropriate” starts. How well a system scales is more dependent on how well it allows for these divergent uses than how well the technology can handle the load. the more these conflicts stand in the faces of the users, the more likely the system will fall over and die.

This is really the ultimate failing of mailing lists, because the only way to scale mailing lists across these conflicts of “what this list is about” is to create more mailing lists for each diverging sub-population, and once you split the group up enough ways into enough shards, it loses all context among shards and it’s no longer a community (if it ever way). the only way a mailing list scales and survives is to get seriously anal about focussing content on the tightest definition of “acceptable” for the community and limiting side chat, which is a serious limiter of building community among the users.

At some point, the freakers and trolls move in, because there’s a section of society that gets off on destroy stuff other people build. If you don’t plan for this, when they move in, you die.

And yet, with decades of repeating these mistakes under our belts, we keep re-inventing systems that don’t deal with these problems up front. USENET’s lack of any authority system. non-verified SMTP. Wiki’s without authentication. Anonymous blog comments. non-validated trackbacks. The list goes on. We end up wasting huge numbers of resources trying to backpatch solutions instead of designing them in up front.

And we probably will continue to. sigh.

To me, it ends up to a few simple rules:

Anonymity bad. The net mixes these things up pretty badly. Anonymity implies there’s no way to know who you are, so there’s no way to police or manage your actions. Reality: for every person with a legitimate need for Anonymity, there’s 99 hackers, trolls and freakers taking advantage of the system to frack things up.

Pseudonymity good. No Anonymity doesn’t imply full disclosure. It’s not about knowing who you are, it’s about being able to know that YOU ARE YOU, so that if what you do is unacceptable, it can be policed. And yes, it means that some people (site admins) need to have some identifying info about you, but it can be implied identification, maybe as little as an email address or an IP address. Enough to give them a handle to enforce rules, although obviously, a really motivated troll will make any admin grumpy in any online system, if they want to. Fortunately, those types are fairly rare.

Always authenticate. Every village needs walls and gates, because if you don’t have them, when the vikings arrive up the river, they WILL burn the village. Even with walls and gates, they may still burn the village, but if you don’t do the basics, you dn’t have a chance. And they will arrive, someday. In my experience, usually at 2AM when you’re on deadline before vacation…

You need authority. Anarchy is a nice theory, but if you don’t set rules, when people push beyond what’s acceptable for the group, you have problems putting the genie back in the bottle. The middle of a crisis is a lousy time to try to build consensus on where to draw the lines.

You need police. Even if they spend 99.9% of the time in the donut shop drinking coffee (and in a good community, they will, because it self-polices well) that over .1% of the time, they can mean the difference between losing the community over a conflict.

But beware of self-appointed police. There will be people who will want to define things in terms of what they want instead of what the community at large wants, and will enforce their personal rules on the community if you let them. Don’t let them.

Enable the quiet voices. Most of the material created within a community is from a very small percentage of the user base. Look for ways to find those “quiet voices” that get crowded out of the mosh pit and enable them to contribute. It’s well worth it. Not everyone wants to be part of the loud and noisy group that loves the fight to be heard — and many times, those quieter voices will be your most interesting contributors, if you get them involved.

Beware the squeaky wheel. Just because there are some folks loudly complaining about something doesn’t mean they speak for the community in general. It’s key to understand these complaints in the larger context of the entire group. For me, a classic example of this is “reply-to” on mail lists. If you didn’t set it, there were always a couple of people loudly whining that it was the One True Way of setting up mailing lists, and they hated taking no for an answer. In reality, every time I did a survey of the ENTIRE mailing list population, I found — invariably — that the vast majority (80% or so) simply didn’t care either way, and of the ones that did, the “pro reply-to” group was the minority. I did this survey maybe a dozen times over the years, and got the same result on every list despite wildly divergent populations (from geek to sports fan to skiffy fan). the pro reply-to people hated this, because, of course, they knew better than the entire list population what was good for them….

I always saw the communities I built as community bars where people of similar interests congregate, in fact, I liked to promote the mailing lists not as “a place to talk about the San Jose Sharks” as much as “A place for Sharks fans to talk about stuff”. you’d never walk into a sports bar and get told to shut up if you tried to talk about something other than sports, for instance, but online, that’s fairly common — but in the side chatter is where the friendships and community building happen.

As an admin, don’t be afraid to let a group self-police. Probably the hardest lesson I ever learned. But having said that, the key to being a successful admin is knowing when to step in, and doing so decisively when necessary. Lots of admins (and the most active members of the mosh pits of the community) like to think they can just let the group figure it out; the reality there is that the loud and noisy and the trolls and freakers will drive out everyone else, and then all you have left are noisy freakers and trolls.

If you don’t police it, you’ll end up with that friendly community sports bar being turned into a biker bar by the bikers — at which time the people you built the thing for will all run off and find some other place to watch sports and chatter. Is that really what you wanted to run? a biker bar? Maybe, but not me.

Of course, the bikers always hated that… funny, that.

CAPTCHA Cracked…

Interesting piece by Ars Technica on CAPTCHA which more or less puts the wooden stake into the heart of the technology.

http://arstechnica.com/news.ars/post/20081002-right-back-at-ya-captcha-bad-guys-crack-gmail-hotmail.html

In some cases, the black hats have cracked them via technology, when the CAPTCHA has been badly designed. For the rest, they’re simply building farms of people and paying them piece rate to solve them.

Once again proving what I’ve said all along, these things are like “the club” — they only work as long as it’s not worth the other person’s time to deal with them. The Club doesn’t actually prevent car thefts, it just convinces the thief to steal an easier car…

How do you sidestep technologies designed to prevent bots from whacking your system? By hiring people to do it. Now, how do you prevent that?

Three Tales of Trolls

Derek Powazek – Three Tales of Trolls:


Sometimes things happen in threes. I recently read these stories and, maybe it’s just me, but I think they share a common thread.

In the first story, Mattathias Schwartz goes deep into the troll subculture.

[...]

Finally, in the third, Duncan Riley reports on the latest incident of Thomas Hawk getting thrown out of somewhere for taking photos.

In all three cases, consider how the outcome would have been different had the people involved followed the old net axiom: Don’t feed the trolls. Online or off, the best solution is often to ignore the guy who’s out to fuck with you.

Or more. Humans have a tendency to find clumps within things that aren’t really related — it’s how our brains are wired. But there’s also a fourth recent one that ties in here that Derek missed.

The first item can be defined as “to better defeat your enemy, understand them.

And in the third? In some ways, it’s too bad this happened to Thomas Hawk; as Derek noted, it’s not the first time, and Hawk has a tendency to be — strident? assertive? a jerk? — about these situations. I’ve jumped on him a bit for this in the past. He tends to forget that we all have rights, and those rights are many times in conflict and his don’t “win” just because he wants them to. While my gut tells me the SFMOMA guy blew this one royally (I’ve seen that “I’m in charge” ego play too often), since Hawk is involved I really wish I knew the parts of the story that haven’t come out yet. I just have to assume it’s more complicated than it seems on the surface, because he has a past as an instigator.

But let’s not forget the fourth, which really ties back into the first. And that’s that William Patry shut down his blog, in large part because he got sick and tired of fighting the trolls.

The Patry Copyright Blog: End of the Blog:


When other blogs or news stories refer to the blog, the inevitable opening sentence now is: “William Patry, Google’s Senior Copyright Counsel said,” or “Google’s top copyright lawyer said… .” There is nothing I can do to stop this false implication that I am speaking on Google’s behalf.

Yeah, just like I never got away from “the Apple blogger” attribution, when in fact I was never more than a blogger who happened to work for Apple. And then people wonder why more people from places like Apple start blogs (and admit they work for Apple….); the reality is, both Apple and Google have enemies; they are looking for any excuse to put some pain on those companies, and they aren’t afraid to spin something however they think it gives them advantage to do so. Reality just isn’t high on the list.

And you can’t fight it, and you can’t win, because you’re trying to play fair, and they take advantage of that. Or perhaps they’re simply naive and don’t understand the implications of the power of words. Ultimately, it doesn’t matter.

I’ve been involved with and running online communities for decades. Whether it’s “don’t let the turkeys get you down” or “don’t feed the energy monsters” or “ignore him” — it’s advice easily given and difficult to follow. The reality is, trolls and instigators are few in number, but it doesn’t take too many to completely take the fun out of it or destroy a community. We had a notably persistent one in the Maple Leafs mailing list, and he ultimately won — a good chunk of the group moved elsewhere when they got tired of him, and I finally gave up trying to keep him out. Eventually, it stops being worth it — and you hate letting them win, but you end up no longer caring. That’s the ultimate sadness.

So I have a lot of sympathy for what Patry went through; it sucks. And because of the actions of a few, the greater population loses a great resource. And you know what? there’s not a whole lot we can do.

My one big suggestion to Patry is this, though: take some time off, relax, get away from it all, and see what happens. Sometimes the distance and time gives you a new enthusiasm or takes you in a different direction. Sometimes it gives you perspective to see how you can better cope or ignore the negatives and celebrate the positives of contributing. And sometimes you end up saying the hell with it and go play video games. All of them are good options, if they work for you — just don’t be afraid to say “hey, let’s try it this way…” if you feel it’s worth a shot.

Time off, I’ve found, really helps.

Which is, amusingly enough, another meme wandering the net right now:

louisgray.com: Relax, Bloggers: Nobody Is Keeping Score, and There’s No Quota.:


With the dog days of summer upon us (in the Northern hemisphere), I’m seeing the issue crop up again, as peers are talking about taking time off from blogging or social media, explaining holes in their publishing schedule, or openly questioning their enthusiasm.

Bloggers are finding out they aren’t immune to the realities of the grind — I sit back somewhat amused that they thought they were. Too many folks decided this blogging thing was really neat, and they could even make some money at it, and set themselves up into a situation where they could never turn off, unplug, relax. Instead of asking another “web 2.0 worker” what to do, maybe ask someone off in the real world — there’s a long history of people running one-person businesses, and the successful ones learn early on that weekends matter, and vacations matter, and evenings matter. Having a life matters. Forget that, and having a life will at some point force itself on you, probably at 3AM, and probably on deadline when you can least afford it.

You need to schedule it in and plan for it, or it’ll simply be another crisis, when you least can afford one. And burnout is one of those things that really puts trolls in perspective, because in effect, you end up trolling yourself.

But isn’t this whole “web 2.0″ thing different? it’s new! it’s online! it’s in a coffee shop and a laptop!

Well, no. Ultimately, it’s still just a job, no matter what the tools are and what your pay scales are (if you have any). Just because it’s a laptop in a starbucks doesn’t change the basics of real life, any more than being online stores made pets.com or webcan invulnerable to the economic realities of the real world. hint: it’s all the real world, folks….

New and interesting uses for webmail…

For the last couple of weeks people at work have heard me muttering in the halls about “those damn geeks”. I’ve been chasing down and cleaning up after a group that’s been using the webmail system as a distribution system for — stuff. Mostly warez cracks and video, from what I can tell.

Since this seems to be fairly widespread and flying under the radar at most sites I’ve talked to about this, I thought I’d give it some wider visibility and go into some of the details.

I want to emphasize this part:

Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.

Nothing was hacked in any way, this is purely (in its way) a social engineering hack taking advantage of free webmail sites all around the internet — I saw at least 15 involved from my investigation.

I’d noticed some changes in network usage on the site the previous couple of months; bandwidth usage had doubled in both May and June, far beyond what I thought normal given the growth in new users we’re seeing. It didn’t seem too serious, though, so I stuffed it in the back of my head to investigate at some point.

Early July hits and I look at the numbers again — and in the first 7 days of July we’ve used 10X the network bandwidth we used in all of June. We’re talking orders of magnitude change, for no good reason.

That’s generally a bad thing. So I went looking….

What I found was both fascinating and a little depressing. It was a group of people based in Poland that have turned public webmail systems into the equivalent of a Bittorrent network.

Let me say right up front: no system cracking involved here, no security issues, no hacks, no cracks, no leaks, no bugs. They are simply using these systems as designed, not doing anything to penetrate or compromise the system.

Here’s how it seems to work: when they have a package to distribute, it is packaged up into pieces small enough to be attached to and sent as emails. Most webmail systems allow attachments up to about 10 megabytes. Files were split up and encoded in MIME as standard packages, although the details of name and type seemed to be ignored (lots of powerpoint files, in theory).

Then accounts were created on various webmail sites. In my sample of addresses, I see over a dozen different sites being used. The person doing all of this then emails the files to that mailbox, where they sit. Now, anyone who wants that set of files only has to get the access information for one of those accounts, log in via IMAP and let his email system download them. It looks like any given package is stored on between 3 and 8 different webmail accounts.

Account creation seems to be semi-automated. All accounts are of a similar format, a semi-random “word”, followed by a 1-3 digit number. Passwords use the same format (but are never the same), ditto the “from” address and the “return-path” in the headers of the emails. Sometimes the files are stored in more than one account on a single webmail (another reason why I think this is at least semi-automated), but generally, it’s sent to 4-6 webmail accounts on 4-6 different sites.

It looks like the actual account creation is manual, or semi-manual, because some of the sites involved use CAPTCHA on account creation and that isn’t stopping them. I don’t think this setup is sophisticated enough to have cracked CAPTCHA, so there are people involved in the setup. I think the account naming, and packaging is automated, but people are involved in the account creation and uploading. Once someone downloads the emails, there seems to be another script to put it all back together again, because it’s not depending on the MIME data in the message to do naming or decoding — in fact, that stuff is set up to (at least casually) make the content itself look innocent.

There’s obviously a web site somewhere that tells you how to access the mailbox to get the content, but I haven’t gone looking for it.

If you think about it, this is a pretty nice hack. With Bittorrent being scrutinized by many ISPs, they’ve set up a fairly low-tech, under-the-radar way of distributing “stuff” without easy detection. The original distributor only has to upload the files once, and then the rest of the resource costs are borne by the mail systems — the webmail site pays the network to upload the files into the system, pays for the disk to store them, and pays for the network to distribute them back out.

Needless to say, I spent some time shutting all of this down. We ended up with a couple of hundred accounts that I closed out. All told I identified and closed a couple of hundred accounts that accounted for over 200 gigabytes of disk storage, and the network bandwidth they were starting to suck was going to be measured in terabytes, and we’re a fairly small webmail site right now. One can only wonder what they’re doing to some other sites….

The group is based in poland. 99% of the access of these files also came from Polish IP ranges. Fortunately, once you know what to look for, it’s fairly easy to find these accounts, given the standardized naming, the limited IP range they’re coming from, and the exceptionally large average message size. The latter is the easiest way to identify them, no “real” webmail account (at least on our system) has an average message size > 5Meg. Even accounts where users are parking files in their Imap for storage tend to have no more than a 1 meg average storage size.

This group spent some time experimenting with the site, evidently to see if we were paying attention. The earliest record I can find of them accessing the site is in April. In June, they ramped their volume significantly, and in July, they opened the floodgates (and I found it four days later, fortunately). It’s hard to tell from the outside if this was them experimenting to see if we’d catch them and then ramping up when they felt safe or if this is a new network that was finally ramping up as they finished building it. Either way, it’s clear there’s a lot of network being used on a lot of webmail systems globally by these guys.

How to stop this? No easy answers. They aren’t really “doing” anything we don’t allow, it’s more of a Terms of Service on content issue with policing. If the account creation was fully automated we could possibly plug that hole (and probably should on general principles; CAPTCHA might not stop this but it can’t hurt, but some of the webmail sites being used have CAPTCHA enabled and it didn’t stop them). On the other hand, there’s no reason we should feel the need to let them pass around warez on our dime — and they only have to use network to upload it once, and then the webmail sites pay for the bandwidth to accept and then deliver it as often as it gets downloaded, plus disk storage and the typical overhead of backups and etc.

What it really goes to show is that people will find interesting uses for any publicly available technology, whether or not you intended for them to be used that way. It also, I think, means we should be aware of what those possible uses might be and see if we can influence our systems to discourage the ones we don’t like. For instance, a 5 megabyte limit on attachments might have discouraged these guys, but doesn’t seem to significantly impact “normal” users — I found very, very few emails on the system that large.

One of the things I’ve been pondering is ways to automate finding or setting alarms for this kind of “non-standard” behavior; quotas solve some problems, but not this one. I wrote a script that finds these accounts with really large average message sizes. It seems to me something that automates that process, or ways to monitor or rate-limit network usage on a per-account basis would be another way, or simply looking at accounts with the highest network usage.

Things that definitely don’t help this kind of problem: quotas, looking for accounts at or close to quota, accounts with large number of log-ins, or even usage from many different IP addresses. None of those were true. I also didn’t see any significant sign of multiple simultaneous users. The things I think of as “obvious” signs of abuse are missing here, it’s a different set of parameters that become visible once you look.

One option I’m just starting to investigate is coming up with some kind of “typical” network usage per user, sort of a capacity planning number — and then if the system deviates from that significantly it gives you a hint you need to look in more detail. I want to avoid having to monitor at the per-user level to the greatest extent possible, and find metrics at the system-usage level that might tell me if the system is within expected usage ranges or not.

In reality, there’s nothing “wrong” going on here other than the sheer size of the operation and the costs it involves (and the fact that most of the content is likely illegal). technically it’s pretty simple and straightforward — a nice hack — to shift the cost of distribution off to others in a way that’s (in theory) low-key enough to not be noticed, at least until they get greedy in resource consumption. If they hadn’t spiked usage in July like they did, I might not have gotten around to chasing them for a while.

My ultimate take-away, though, is that the users “use cases” for a technology are rarely the same as the developers. Sometimes the users innovate in really interesting and positive ways, sometimes they distribute warez — but either way, people are going to see opportunities in your technology and that should be part of the discussion in designing those technologies.

My suggestion: if you run a webmail site that allows users to create accounts, you might just want to look and see what you find. Might surprise you.

Oh, for what it’s worth, I’ve held off posting on this for a bit because I gave advance warning to the other sites I found involved in this. Of the 15 or so abuse@ accounts I sent the details to (including accounts, IP ranges, Received header data, etc, etc), one responded immediately and started their own search and destroy operation — they happened to be one of the larger “white label” webmail, so that’ll shut down any number of the domains involved.

But three of the webmail sites had their abuse@ addresses bounce as user unknown. One sent me email letting me know he was on holiday for a few weeks (in italian). And from the rest, including the two Polish ISPs where all of the upload activity intiated, total silence. Ohwell. Kinda sad, but hey, it’s their network bill, if they don’t mind paying it, I shouldn’t complain… And I just did a check of our site to see if they took the hint, and I see no sign of them creating new accounts now or doing any kind of activity, so I think they’re gone. Well, for now. I’ll know if they come back…

“If the news is important, it will find me”

“If the news is important, it will find me” – - mathewingram.com/work:


Think about that for a second — or longer, if necessary. I think that sums up, in ten simple words, what has happened to the way that many people (and not just young people, but those who use RSS readers and blogs and social networks as well) consume the news. Not only is there just so much of it out there that it’s virtually impossible to consume it all, but the very fact that someone you know — or trust — has passed on or blogged or Twittered or posted a link makes it more likely that you will read it.

And here is one of the great differences — and changes for the worse — of the move to online and social network news distribution. It’s news by echo chamber.

If you get news from the circle of friends (or “friends”), you’re being fed information from a group of people that at a first degree of interpretation have the same, or similar, interests as you do. So the news that finds you is most likely news that reinforces your existing interest areas and knowledge set.

One aspect of the traditional media that’s being lost is the ability to tell people things they need to know but don’t know they need to know.

Where does “The Jungle” or “Silent Spring” or “The Pentagon Papers” — or hell, Watergate in its totality — exist in the new reality? If you’re not already interested in the environment, will your RSS feed push at you until you become interested in global warming? Or will you unsubscribe to the feeds that keep annoying you wtih stuff you don’t think is relevant?

How will those issues that are important but not already on your radar GET on your radar?

My worry is we’re creating an environment that doesn’t inform as much as reinforce.

Is the best we can do in the future Michael Moore? Or will even that fade as people lock themselves further and further into the echo chambers they choose to be a part of?

Evaporative cooling of group beliefs

Evaporative cooling of group beliefs:


Over on Overcoming Bias there was a great post called “Evaporative cooling of group beliefs” where the author talks about how ejecting outliers moves the group’s average position towards the other extreme.

[...]

My own theory of Internet moderation is that you have to be willing to exclude trolls and spam to get a conversation going. You must even be willing to exclude kindly but technically uninformed folks from technical mailing lists if you want to get any work done. A genuinely open conversation on the Internet degenerates fast.

[...]

It’s interesting to compare this to the techniques Theresa Nielsen Hayden uses on her “Making Light” blog and on Boing Boing comments. There’s an art to building online communities that nobody has yet well documented.

Does it move it towards the other extreme, or merely back away from the far edges of the bell curve? A move towards the other extreme can also be a move towards the middle, depending on where the starting point is.

To me, the reality of community management is that early on in the process, a moderator’s policies, style and attitude shapes the community, because it’ll attract certain people and attitudes and discourage (or ban) others. Once a community is established, it really needs to dictate policy and the administrator’s job is to steer the group in the direction it wants to be steered, and where necessary protect it from those who put personal interests above the needs and wants of the group (trolls are, ultimately, people who are more interested in garnering attention than contributing to the group, or in some cases, simply people who insist on “winning”, which excludes others from the option of “winning”, however the community defines that)

I admit to not being a big fan of how Teresa manages her communities, but I’ll be the first to say that it works for her and her groups. On the other hand, I find those groups very “echo chamber”-ish and narrow in viewpoint and not terribly tolerant of views that don’t fit the common worldview. Well, most communities are like that, but those are even more so. But they’re happy with it, and that’s what matters. What I think doesn’t, since I’m not involved in them.

At one point I thought it might be fun to write “the book” on community management, whatever that is. Later on, I thought maybe building a wiki or some kind of community for community managers to write an online guide might work — then I realize what it would take to be community manager for a community of community managers, and I went and put a washcloth over my eyes on the couch until the thought passed…

I still don’t think I’ve seen “the book” on all of this, or even if we’re at the point where it can be rationally written. There are so many right answers — each depending on a situation and the personalities involved; and for every situation where a community policy works — it’ll fail miserably in a dozen other situations with different people and different needs. So maybe it’s best not to put all this in stone. Or paper. Or whatever…

Wikis and Spammers : Venture Chronicles

Wikis and Spammers : Venture Chronicles:


Coincidentally, I have been battling spammers for over a month on another wiki I have, the Web 2.0 in the Enterprise wiki. In the interests of ensuring as much public and unfettered access as possible, this wiki required no registration for updating and spammers were hitting it with startling efficiency and replacing all of the content with links for fake Rolex watches.

Okay, we have 20 years of experience proving that any time you don’t put some kind of protection up on a writable computer, that the hackers and spammers will wander in and destroy it.

So why the hell are people still trying to set up public systems without any kind of authentication?

This isn’t just the same as going on vacation without locking the front door, this is going on vacation and taking the front door off the house before you leave. TV gone when you get back? Gee, what a surprise.

I thought we’d learned this lesson by now? Well, I guess not. Oh well.

Messaging, not email

A VC: Messaging, not email:


Coming out of our ‘smtp is dead, long live smtp’ brainstorming session I am thinking that we need to be talking about messaging, not email.

Email is a subset of a much larger messaging market. What we’ve seen over the past 10 years is that internet messaging (primarily text but let’s not make that distinction) has evolved from predominantly email to a host of other systems

We seem to be having yet another round of “email is dead” going on.

Nope. But Fred Wilson has the gist of it here. Email is a protocol, a specific way to communicate. Good communication transcends a protocol (or should), because there are many different needs and priorities to communication.

The reason email is so endemic is because — for about 20 years — it was basically all we had. And so it was made to fit all of those needs, even when it wasn’t really good at doing it. That’s why email on most mobile devices up until the last year or so really sucks — emails a bad model for most of that, and so comes the rise of SMS. On the other side, using email for group discussions REALLY sucks, because the typical email-based mailing list gets way too chatty and email is inherently an interruptive protocol (it comes when IT wants, and you get interrupted and you have to go and decide whether the incoming message warranted being interrupted for; by the time you figure out it’s not, it’s too late). The technologies invented for limiting those interruptions, from digest-formatted mailing lists to mail filters, are all band-aids on the larger problem. That’s why web forums have really supplanted lists, and RSS and other pull technologies are increasingly key in distributing these messages.

And that’s one reason why blogging has succeeded. Can you really imagine being subscribed to, say, 300-400 mailing lists where every time a blogger posts it ends up in your email? Of course not, you’d go crazy. you might want 5 or so KEY blogs in your email, and the rest out of the way until you decide to go visit them.

Which is the key. Email is still a core communications technology. Will be forever, I think. It just won’t be the ONLY one, and “death of email” is a misinterpretation of the fact that communications that aren’t well suited to email are finally moving to other services better designed to distribute them appropriately.

(I’ve talked about this stuff more than once; here are some previous rants: 2003 when RSS was replacing email (hah!) (also here), and 2006 on revamping Yahoogroups)

Update: Mathew Ingram chimed in:

Is email dead? No, but it’s not well – - mathewingram.com/work:


Email may not be dead, but it certainly isn’t looking too healthy, and hasn’t for years. As Zoli points out, the best approach is not to replace email with other things like IM or Facebook messages — which have their own flaws — but to make use of as many different methods as possible, depending on the situation. In some cases a wiki makes more sense, or a Google document, or a live chat, or (God forbid) even a phone call.

That may seem so, but in reality, email isn’t any more challenged than any other tool on the internet, except for the ones so small or so niche that the spammers and crackers don’t bother. The problems with email aren’t email’s problems, but simply the way email makes visible the problem that the internet has in general.

Redefining Friendship

Redefining Friendship : Venture Chronicles:


“But really, these sites aren’t about connecting and reconnecting. They’re a platform for self-branding.”

I really don’t agree with, or even understand, the point Stein is trying to make in this article in Time, because even in non-technology enabled social networks there is an element of branding. Stein’s critique of social networks primarily reflects a generational gap (and he’s 5 years younger than me), which he gives himself away on with his repeated lament on the loss of privacy. Stein seems to lose sight of the fact that privacy is not being taken away from me but rather that I am opting-out of a private exchange format.

His assertion that these sites are not about connecting and reconnecting is absurd on it’s face, contrary to how millions of people are using these web services everyday.

[....]

Stein’s writing about the dishonesty of self-branding is all the more ironic by the fact that he has a wikipedia entry. Now there’s self branding for you!

I think the real problem here is that everyone is trying to define social networking sites (and “friends”, and “connections” and whatever else they get called) as if they are some single thing, as if they are (and can only be) used in a single specific way.

That’s what’s false about all of this. These are all very generalized ideas and concepts that everyone uses in the way they find most comfortable and/or useful.

And blaming Stein for a wikipedia entry is also bogus, given that Wikipedia doesn’t allow a user to create or edit their own entry. So defining it as self-branding is sort of by definition incorrect.

Page 1 of 41234