Using AMT to run web-based experiments allows researchers to obtain large quantities of data in a cheap and efficient way. However, there is lack of evidence regarding the actual reliability of AMT’s workforce as a subject pool; as any new instrument, AMT requires to be tested thoroughly in order to be used confidently. This blog aims at collecting any individual effort made in order to validate AMT as a research tool. It reports results from experiments that increase or decrease the reliability of AMT as an online subject pool, as well as general guidance for running experiments through this service.

The blog is administered by Gabriele Paolacci and Massimo Warglien. As a social scientist, you are strongly encouraged to participate to this project. We are welcoming any effort helping to cumulate knowledge about the reliability of AMT as a tool for experimental research. In particular, we hope to receive both new contributions, along the lines suggested in this blog, and notices of any existing evidence that may enhance our understanding of the validity of AMT as a source of experimental subjects and data. Comments are also welcome.

Experimental Turk

Ten Thousand Cents

So does mechanical Turk work well? Yes it does! A lot of people have made a lot of nice write-ups about artistic experiments they made using Turk. They primarily tested Turk’s reliability for simple art experiments and they got quite good results. Another great analysis they got show that Turk is cheap, fast, and as good as machine learning data.

If you really want to have the best mechanical Turk experiments, then it’s good if you could post your questions to Turk to find out or interact with other researchers doing the same thing. You can find a lot of information online as to how you can entice people to participate in your experiment.

Amazon mechanical Turk was actually not designed with researchers, scientists, or artists in mind, nor does it have the rights of human subjects on their mind in looking for participants. There are times when this system can be frustrating when doing your experiment especially more complex studies; however, it can be one great tool for a lot of things. This means that best mechanical Turk art experiments are really possible.

Mechanical Turk Art Experiment Examples? Here you go.

• Ten Thousand Cents
• The Sheep Market
• Faces of Mechanical Turk

Red Box Video

So I was thinking the other day (I know, I know, dangerous…) and it struck me that Amazon could take the red box video idea and merge it with Mechanical Turk for something really revolutionary. If you live in North America, I’m sure you’re familiar with the red box video service that you can find at McDonald’s, Walmart, etc. They’re all over now. Basically, it’s a vending machine for DVDs.

The idea of kiosks got me thinking, though. What if, instead of a retail machine, it was something entirely different from Amazon – a Mechanical Turk kiosk – a Green Box. Picture this. Worker comes to McDonald’s, answers a few simple questions and gets to choose money or menu. If menu is picked, depending on how many tasks he completed, pictures of food and drink comes up. Maybe 5 tasks for a Big Mac. The details would have to be worked out.

Amazon Mechanical Turk Kiosk?

Then, after choosing. Maybe McDonald’s asks him a bonus question in exchange for a small fry or a free super-size of the item picked. This would be a great way for McDonalds to get customer feedback at a low cost. Economies of scale. That’s what Mechanical Turk is all about, in my humble opinion. And this could kick Mechanical Turk to the next level.

I know some of you are probably moaning – there are already not enough tasks, you say. Well, I imagine if the numbers were huge (and this could scale really, really large on Amazon’s cloud, I imagine) then large corporations would probably be very, very interested at the discounted information/opinions/work. So, I say if Amazon Green Boxes (or whatever they’re called – Mechanical Turk Kiosks maybe) do come about, they’ll actually be beneficial to Turkers everywhere.

There are social benefits too. Think of a single mother who needs diapers and just doesn’t have the money. Instead of doing something illegal, it could be as simple as going down to Walmart and answering some tough art history questions in exchange for the diapers. Maybe they’re set-up at libraries so people can learn and be able to do more and more difficult and specialized tasks.

Of course it would still be available online, but this would really take Amazon’s Mechanical Turk to the next level, which could be a good thing for many people – Amazon included.

So, what do you think? Is there merit to the idea? Can you envision it? Should a letter writing campaign be started so Amazon implements the idea?

Talk to me, world.