O’Reilly Radar > Rewarding Users for Contributing Data

O’Reilly Radar > Rewarding Users for Contributing Data:

Users should contribute data for a reason other than “Nat’s business model is predicated on collecting user-generated data”. The best reasons give a reward that’s related to the data. BitTorrent gives you fast downloads if you are in turn offering fast uploads; you’re rewarded in bandwidth for offering bandwidth.

Community is a good reason: my theory is that Amazon reviewers and list makers are passionate book-lovers contributing data because it builds their place in this community. I’m not particularly enthusiastic about Yahoo! Local Reviews because it’s not a site that has a community and so I have trouble finding incentives, rewards, or reasons for contributing reviews other than you feel particularly passionate about a particular restaurant. End result: I predict Yahoo! will get only extreme reviews from a minority of the population. A quick check of restaurants in my area show extremes in ratings, despite there being a lot of mediocre restaurants around.

Self-interest is a good motivator: eBay feedback began as a way to weed out the fraudsters making life miserable for marketplace participants, and by contributing information on whether they were good traders you were helping to promote good trading. Now the feedback’s more of a ritual (“AAAA+++++!!!!! BEst sellerr evaH! !!!”).

on the other hand, that feedback mechanism has been corrupted. It’s gotten to the point where ANY negative feedback on a seller’s or buyer’s record can be death on eBay — and buyers and sellers have both used that as a threat. the feedback mechanism seems to be missing an important piece: data on the reliability of the feedback. And increasingly, feedback issues are ending up in court.

The worst kind of reward is money, I think. There was talk of MSN search paying people to use them. That’s completely disconnecting the reward from the behaviour. The best reward for searching is relevant results. And guess what? The site with the most relevant results is also the top search site.

Money, and any other type of reward that substitutes for money such as vouchers or access to powerful site features (e.g., features that would otherwise be subscription-limited) encourages deceit and gaming.

I disagree. Look at one of the most successful implementations of this: Amazon Affiliate program. It was, in many ways, the first serious implementation of what became Web 2.0 data sharing and may have in practice created the concept of the sharing API.

If you pay me to search, I’ll search when I don’t need to (just to make money), advertisers will be getting impressions that aren’t useful to them, and it’ll drive the price of impressions down. Because you can’t reward people for the quality of the data they contribute, you have to settle for the act of contributing something. And once the reward is worth money, you’ll get people contributing crap just to get the rewards. Your reward system is now paying people to piss in your data pool. That hasn’t happened yet for Chris Sells’s offer to share royalties in return for Amazon reviews, but that’s only because of the scale he’s operating at. Needless to say, the rules are also different for intranet applications rather than public Internet applications.

And that’s because in these cases, you’re paying people based on the wrong metrics. Don’t pay them for CREATING CONTENT. Pay them based on the positive revenue impact that content causes. you want to review your GOOD reviewers, not your prolific ones. Unless they happpen to be both.

So anyone can write reviews — but the payouts go to people based on the advertising revenue from the page views of reviews; why NOT give them a cut of that money for the ads on the pages showing their reviews? You can also use the “usefulness” feedback aspects of the site to determine the best reviewers adn best reviews, and give them featured placement. (you still have possible fraud problems here, similar to click-fraud issues, but they’re more manageable). You now have a system that allows anyone to contribute, new, good contributors to be discovered, and the best contributors to be rewarded and given special status, while all good contributions have some ability to reward their creator.

This is the financial model of the new reality: it’s not the creation of content that rewards someone, it’s the value of that content in the market. In the old traditional ways, an editor had limited space to publish stuff in, and therefore culled the less interesting material to make the best stuff available (based on how much room they had to print it); in the new online world, we can (in theory) publish EVERYTHING, but it doesn’t really imply that everything deserves to be rewarded the way published material now is.

Instead, what this new publishing model does is allow the ability for the good stuff to be found and rise to the top; instead of one editor making publishing decisions, you have infinite editors doing so. The new systems not only have to figure out how to publish all of this — it needs to figure out how to enable that infinite row of editors to raise the cream into visibility, and then reward the creators of that cream appropriately. The new publishing model won’t turn all that previously unpublished crap into great material. It’ll still be crap, and it doesn’t deserve to be rewarded. What it DOES do is change the model from the fail/success decision being made by one person, or a tiny pool or people into one being made by the collective decisions of the pool of self-defined editors.

Leaderboards are similarly problematic. Competitions as motivation for contributing data are workable, but only if you can validate the data cheaply. Otherwise you’ll have people submitting bogus data just to get a higher position on the leaderboard. See any Orkut profile with >300 “friends” for an example of this in action.

A big problem I always had with slashdot was that their karma system became a game in itself. Gaming the system became a goal unto itself, leaving the underlying purpose of karma pretty meaningless. But karma-like systems are still necessary. I think the trick is to build it such that you don’t turn the act of dealing with karma into it’s own system to manipulate (that probably means USING karma information in ranking and rating, but not makgin those rankings public or explicit. But it’s still important to identify the “good stuff” and give it priority in publishing and placement. Newer systems, like Digg, are doing a better job of this, but there’s still work to do.

You might also want to read:

1. Backing up your data (the 2011 edition) Back in 2009, I wrote a series of articles on why it’s important to back up your disks, and my strategy for doing so. It’s...
2. I have committed NAS; updating my data storage and backup strategy One of the projects I’ve been working on the last few days is cleaning up and organizing my files ands refreshing my offiline archives. I...
3. How to Help Mailing Lists Help Readers (Results of Recent Data Analysis) How to Help Mailing Lists Help Readers (Results of Recent Data Analysis): This second, independent experiment reinforced my conviction that: Although mailing lists and newsgroups...
4. O’Reilly Radar > The Medium Changes the Preferred Format O’Reilly Radar > The Medium Changes the Preferred Format: I think we’ll see this phenomenon all over publishing: the rebirth of short-form content and collections,...
5. O’Reilly Radar > On Failing to Think Long Term O’Reilly Radar > On Failing to Think Long Term: Diamond reported that his students at UCLA tried to imagine how the guy who cut down...