Using AMT to run web-based experiments allows researchers to obtain large quantities of data in a cheap and efficient way. However, there is lack of evidence regarding the actual reliability of AMT’s workforce as a subject pool; as any new instrument, AMT requires to be tested thoroughly in order to be used confidently. This blog aims at collecting any individual effort made in order to validate AMT as a research tool. It reports results from experiments that increase or decrease the reliability of AMT as an online subject pool, as well as general guidance for running experiments through this service.

The blog is administered by Gabriele Paolacci and Massimo Warglien. As a social scientist, you are strongly encouraged to participate to this project. We are welcoming any effort helping to cumulate knowledge about the reliability of AMT as a tool for experimental research. In particular, we hope to receive both new contributions, along the lines suggested in this blog, and notices of any existing evidence that may enhance our understanding of the validity of AMT as a source of experimental subjects and data. Comments are also welcome.

Experimental Turk

Sensationalistic Newsweek?

You would think that something like Amazon Mechanical Turk would be a big, big thing in a recession. I mean, it’s a little money, but in times like these, a little money means so much to so many people, you know? I don’t work for Amazon, but I think they have something that can be really big for Turkers all over the world.

At the bottom of the pyramid are mindless tasks that are still too tricky for a computer to handle. Amazon’s little-known but game-changing Mechanical Turk offers tasks for as little as penny a pop. One task displays pictures and asks “Turkers” to label them with keywords. Another seeks people who know coffee-shop owners—and then asks them to send the owners e-mails, $1 per recipient. Workers discover the Web site themselves or hear about it from friends.

There’s no perfect system, but Amazon Mechanical Turk has a chance of becoming a game changer – especially if they listen to my kiosk idea! Hah. Seriously, though, they’re doing big things, I believe. To be fair, Newsweek’s piece isn’t on Amazon’s Mechanical Turk alone, but I’m tired of constantly seeing it in media next to the words sweatshop, etc. That’s forced labor. No one’s forcing people to Turk.

Am I right? Leave a comment and let me know what you think about this article. Heck, I’m lonely – let me know what you think about the color green if you want. I’ll listen and maybe even respond!

Until next time, keep turking!

I wonder if Amazon has put anything in place to stop this from their end? Apple sure is clamping down because they don’t want their ratings system messed up by “money votes.”

More than one developer has been publicly blasted for using Mechanical Turk to try to get 5-Star app reviews. The idea being that if something can make it into the “highest-rated” category for that particular application, that exposure will lead to sales that will offset any bribery costs.

iPhone Music Apps

To be fair, I haven’t started digging around at Amazon to see if they have anything written down that would stop Tasks like this from getting through the system. Does anyone out there have any information?

What about thoughts about it? Should Amazon control more tightly what tasks are allowed to be used on the Mech Turk System?

Leave your comments below. I know it’s not another forum, but maybe we’ll get some good discussion about Amazon’s Mechanical Turk going anyway. If not, I’ll keep giving information and thoughts on Mech Turk…