Crowdsourcing: Increased pay gets more time for the same work

For those of you not familiar with Amazon Mechanical Turk, it’s a platform that allows people to assign micro-tasks to a crowd for small sums of money. The sweet spot for these tasks lie between what one person does not want to do at scale, and what machines are not yet smart enough to do accurately. For example, it’s often used for tagging images, consumer research, writing content, and transcribing audio. Therefore, it’s often dubbed artificial-artificial intelligence. The workers are dubbed “turks” and complete assigned tasks coming from requesters, who pay for work.

We at PMC are leveraging the power of Mechanical Turk to improve SEO for some of our clients. The relevant tasks include image tagging and writing content. During the crowdsourcing process, we’ve learned quite a few lessons that we’ll share over the next few months. To start, lets talk pricing.

The more you pay the longer people spend on a task, even if it is the exact same task.
We tested this out with two sets of writing assignments, one which asked for a 50-100 word description and the other which was 300-500 words. For our tasks, when given an extra 25 cents, workers spent an additional 5% of their time for the shorter descriptions but over 15% more time when writing longer descriptions even though the tasks were the same. Of course the marginal time spent is not as high as the additional cost.

Workers also tried to game the system a little more more and submit false or incorrect descriptions when the possible gains were higher. The increase in percent of people who tried to game the system was 2% and not statistically significant.

The major downside of lower payments is that it takes longer for workers accepts the task since the price is not as attractive. The supply for lower wage work is infrequent. An additional 33% bump in pay can decrease the amount of time it takes to attract a worker by 35% for the shorter 50-100 word task, and 20% less time for the 300-500 word task.

The two biggest lessons we’ve learned when adjusting payments is that 1) higher pay immediately gets more eyes on the task, and 2) higher pay does not necessarily translate to the same improvement in quality (at least with respect to time). The biggest benefit with a higher pay so far seems to be the tasks are completed more quickly. We’ll take a closer look at other measures of quality with respect to pay. Stay tuned as we bring you more of what we’ve learned from Mechanical Turk over the next few weeks.