Comments: Messy and flawed, but valuable

Comments: Messy and flawed, but valuable — mathewingram.com/work:

In my new role as the Globe’s “communities editor” (you can find more details on that in this post), I’ve been spending a lot of time thinking about comments — that is, reader comments on news stories, columns, blog posts, etc. The Globe and Mail was the first major newspaper in North America to allow comments on every news story when it launched the feature in 2005, and judging by the ever-increasing numbers of people who use them, they are hugely popular. On some major news stories, we can sometimes get as many as 500 comments.

Comments aren’t popular with everyone, however. Some readers (and even some Globe and Mail staffers, to be honest) complain that too often our comment threads are filled with what might charitably be called “noise” — everything from bad spelling and grammar all the way up to partisan political in-fighting, ad hominem attacks and all-around rude and boorish behaviour. Some say they don’t really care what most people think about a topic, and don’t see the value in having public comments on stories at all.

The big problem is that comments are currently unfiltered; ultimately it’s still part of the wild-wild-west of the internet, and so the people who get filtered out in other areas of the net still show up in comments. Ultimately, reputations seem to be taming the trolls and the flamers, but haven’t really migrated to comments yet. It’s a reason why I’ve been watching things like Disqus — but I keep wondering if distributed reputations for comments is really a positive. We’ll find out.

Think about a typical comment: a site may require some ID/registration, but in many cases, it’s faux-authentication, where you can more or less make it up as you go along. That kills accountability, so users can play whatever games they want without much worry about policing or future impact to their ability to comment; at best, a post gets deleted. Bans are, well, pretty trivial to circumvent if you’re motivated and don’t mouth-breathe.

So where this is all headed, and to some degree has to go, is reputation.

A while back I started a project (which I ended up abandoning unbuilt) that had a lot of the same feel as what Yelp now does. A big part of the design was how to create a reputation system that is:

1. Primarily or completely automated (or it doesn’t scale)


2. Limits users ability to “game” the ratings


3. Doesn’t turn the reputation system into something to be gamed


4. Actually helps someone decide whether or not to read (or trust) a piece of content


5. Non-intrusive

Easier said than done. A first approximation are the karma systems of places like Slashdot, but it fails for me on (2) and (3), and is really of limited utility for the key issue, which is (4). It’s more of a chainsaw to help a user hide the worst.

So back to the yelp-like example. You look up a restaurant. There are four reviews of the restaurant, two good, one so-so, one hate. there are a few comments on some of the reviews, mostly people disagreeing with various points.

How the hell is a reader supposed to figure out what this all means? That’s the crux of the comment problem; how to put a COMMENTER in context. First, there has to be a context — and that’s missing in commenting systems today. this kind of harps back to my belief that anonymity on the net is bad, but the net mixes up anonymity with pseudonymity – i.e., I don’t need to know who you are, but I sure need to know that you are you (but I digress; see, if you care, identity proxies, 2004, anonymity destroys transparency, 2007, A group is its own worst enemy, 2008, SezWho, 2007, (who seem to have disappeared behind Disqus), A history lesson from usenet, 2007. That’s a hell of a digression…)

The idea is the basis of reputation systems — that over time, the “real you” comes out, and other users can use that information to judge whether or not to value your contribution — or perhaps tell the system to not even show it to you.

In the Yelp-like system, here’s what I came up with as a first cut. If I’m a J Random User looking at those reviews, what information would be useful for the user to determine what reviews and comments are useful? Try this:

First review: five stars. Best Restaurant Ever. the submitter created his account 2 hours before posting the review, hasn’t posted any content since. Easy guess: it’s the owner, or his spouse, astroturfing. Even if it’s not, you ought to assume it is.

Second Review: 1.5 stars. hate the service. rant. rave. grump. Again, account created an hour before posting, never used since. Obviously someone with an axe to grind. or maybe the waitress broke up with him.

Third Review: 3.5 stars. good food, uneven service, dirty fork. yada. The poster’s been a member for seven months, posted 25 items, average rating 3.8.

Fourth Review: 4 stars. Great food, good service, owner came out and talked. Went back and enjoyed it. Member for 3 months, posted 5 items, average rating 2.8.

Suddenly, with just a few bits of information, things clear up significantly. Astroturfing issues become visible quickly if you simply make it easy to see how active a member is in the larger community — if they’re a hit-and-run commenter, you can bet there’s some ulterior motive (positive or negative). This actually creates a fairly complex web of interactions, it encourages users to contribute to the site to build a reputation, for instance, and that’s good for the community.

Once users have been on the site for a while, they’ll get rated by other users. In my system, I used the rating of the user doing the rating to weight how strongly to count a rating, something I haven’t seen sites try yet, but that is a way to discount the idiots and encourage the strong contributors in a quiet but important way — the less others think of what you say, the less power it’ll have to affect other users on the site. In theory, below a certain number we’d likely just throw your opinion on the trash. Quietly, of course.

Quiet is a big aspect of this; to me, the second you start publishing these “reputation” numbers, it becomes a game of trying to “win” the reputation game. So simply don’t go there. I planned on sticking to the more general five star rating as part of the user profile, but no comparative public stats. Instead, users would be honored with “senior member” type labels based on longevity, activity and rating. Make up half a dozen titles, and allow them to be earned over time as a way to reward your best members. Just make sure that how you determine “best member” actually causes them to contribute and improve the community”. Bad metrics kill.

the final piece, of course, is making this information easy for someone browsing the site to find and use; something like showing the posting account name and rating (chuqui: 1.7 stars), and popping up more detailed info if they mouse over it (3 postings, member for 8 months, this was their 2nd posting and they were a member for a month at the time, last activity a month ago….); for users who want them, you could create slashdot-like filters that would automatically exclude, say, material posted by people with ratings < some number, or with fewer than N postings, or whatever.

The system is still open to gaming — but it’s a lot harder to hide from it, I think. Never got around to implementing it, but maybe one of these days. I’m still mulling bringing it back to life, but not i the original form.

Similar things could be done on a news site, or pretty much any community site. It’s a combination of

• making people create an identity


• tracking that identity’s actions


• allowing other identities to rank those actions


• allowing access to those rankings in rational ways

The combination of an identity, ranking/tracking and weighting things to discourage the one-post wonders can really put a dent into the sock puppets and trolls. sock puppets get marginalized by not building a track record to base a reputation on, trolls get marginalized because, well, as soon as you start building a reputation on a troll, it becomes self-evident. And if all of this encourages more contributions to a site and more community activity as a way to build that reputation, so that people will want to hear what you say, how is that bad for the community?

And done right, it’d be 99% self-policing and automated. I think.

This view isn’t confined to Globe readers, by any means: in a column in the National Post, author George Jonas said that the Web is like “an elegant restaurant with garbage on the menu,” and that “a huge blackboard on which anyone can write anything doesn’t mean much for those with nothing to say, i.e., most people.” Similar feelings have been expressed by various writers about comments on blogs, and some prominent Web writers have turned theirs off completely. Even the director of BBC News said in a recent speech that while she values comments, they are the work of a “vocal minority” and therefore shouldn’t carry too much weight.

It’s not an elegant restaurant with garbage on the menu; it’s a large, vibrant city where you aren’t even noticing that you’ve self-selected into that elegant restaurant. but otherwise, they’re all right. And the way to fix that?

Build accountability into the system. How do you do that? well, what’s worked so far online are reputation systems. Simply requiring a name and email isn’t going to be enough. And yet, that’s basically what we do today in comments. We focus on identifying someone, but forget that it doesn’t matter if we know WHO you are — it matters that we know whether you are worth reading. A simple identify doesn’t do that. A reputation does.

So the future for “fixing” comments has to be a reputation system of some sort. It’s not (just) about better identification systems, or about giving up. This is an area we’ve just started to explore and innovate.

You might also want to read:

1. 2011 Playoff Picks, round 2 edition. And other comments on the playoffs so far. Round one is done, and how did I do on picks? In the west, I picked Vancouver in 5, San Jose in 6, Detroit in...
2. My policy on interviews/comments about Apple and Steve It looks like everyone has decided to write books about Apple and/or Steve Jobs, and I seem to be getting about one request a month...
3. A Better Way to Manage News Comments: Technology Evangelist A Better Way to Manage News Comments: Technology Evangelist: For those who don’t want to invest in proper moderation of things like discussion boards or...
4. The Future of Communities Blog – Your comments are mostly a waste of time :) The Future of Communities Blog » Blog Archive » Your comments are mostly a waste of time : Most people have been told that in...
5. A few comments about my first wife… as long as I’m being so vocally introspective, let me say a few words about my first wife. We were married for four years, mostly...