Tehila Minkus

PhD Candidate
Department of Computer Science and Engineering
New York University

About - News - Publications

Recruiting Survey Subjects from Mechanical Turk: FAQs

Below, I have compiled some tips that I learned through my experiences in conducting online surveys with Mechanical Turk (occasionally abbreviated to mTurk below) as a recruitment tool.

What is Amazon Mechanical Turk? I refer you to their webpage or Wikipedia entry for this answer.

What are some advantages of using mTurk to recruit subjects? Mechanical Turk gives you fast, easy, and inexpensive access to many workers from all over the world. As compared to traditional sampling methods (e.g. canvassing undergraduate students, finding people through classified ads, or calling people on the phone), you may be able to reach a more diverse set of users with a much smaller investment of time and money.

What are some disadvantages of using mTurk to recruit subjects? As the famous New Yorker cartoon pointed out, "On the internet, nobody knows you're a dog." Since your survey is being conducted entirely online with no human contact, you cannot know definitively if respondents are actually being honest.

Additionally, in contrast to some online survey systems (such as SurveyMonkey) where you can buy a specific audience, you have limited control in dictating which specific types of users can answer your survey. This means that it may be difficult to obtain a sample with the exact characteristics of your target population; however, see the questions below for some tips on limiting your target subjects to relevant people.

Are there ethical or moral issues related to using Mechanical Turk? As with all research, you need to consider all stakeholders when you conduct research using Mechanical Turk. This means considering the interests and welfare of the workers whom you've hired. Check with your institution's IRB to make sure that you are following best ethical practices.

The Mechanical Turk marketplace is largely unregulated, which gives the requester a lot of power. You can set your own prices, and then market forces will determine how fast you get your tasks accomplished. Keep in mind that different workers on Mechanical Turk have different goals while working. For some workers, it's a way to earn extra cash. For others, though, it is their main source of income. Panos Ipeirotis, from NYU's business school, has done some research determining who is working on Turk and what their goals are; I recommend checking it out (as well as his general work on leveraging mTurk to get good and reliable results in other tasks, such as labelling data). You will want to keep this in mind when you establish the payments for your tasks to avoid taking advantage of the workers on Turk.

How can I make sure that a subject actually filled out the survey before submitting the task? You can accomplish this in two easy ways: a completion code at the end of your survey, and a field in your survey that requires them to enter their mTurk ID. For a completion code, add a random string on the final page of your survey. Include a field in the mTurk task where they can enter this string to show that they got to the end of your survey.

You can also cross-reference a specific user to a specific survey entry by requiring them to enter their Mechanical Turk ID at the beginning of your survey. This will be useful if you later want to reject "bad" entries to avoid paying for bogus work.

How long will it take to gather my subjects? This depends on how much you pay, how many workers you request, and how restrictive your requirements are. If you only request Masters workers, then you will get slower responses, since they are few among the general Turk population. If you overpay (or pay a lot) for a task, then it will probably go faster. If you need a lot of workers, then again it will take longer. But in general, tasks on Mturk in my experience are finished rather quickly; it takes a matter of days for hundreds of responses.

What is the basic method to recruit survey respondents via Mechanical Turk? When building your task after registering at Mechanical Turk as a requester, they offer a few templates. One of these is a survey template. Use the template and replace any links or text with your links and text. This template already has a box for your completion code.

Generally, you will want to host your survey elsewhere and provide a link on Mechanical Turk for your workers to follow. Some options for hosting surveys are on Google Forms, Survey Monkey, or Qualtrics. Each of these has different feature sets and different price points. Check with your university or institution to see if they have any subscriptions with survey service providers.

If I only want specific types of respondents, how can I accomplish that on Mechanical Turk? There are specific filters that you can choose when creating a task on Mechanical Turk. For example, you can choose to restrict the locations of users to a specific location (probably implemented via IP address). So if your only requirement is that users are in the USA, this will be enough for your purposes.

However, what if you want to ensure that all your subjects are parents - or Facebook users - or owners of multiple cats - then you can implement an extra step in your survey. I call this a qualication test.

In a nutshell, a qualification test is a set of survey questions that you are using simply to screen your applicants. So if you only want users in their 30s who like to watch Disney movies, you might implement a qualification test as follows: Before your actual survey questions, you add two questions. One question is used to identify their age, and the second question is used to infer their Disney-preference status.

However, you want to be careful that you don't make it clear to the potential subject what demographic you are looking for. So instead of asking, "Are you in your 30s?" you might ask "Which of the following age groups includes you?" As answers, you can offer buckets such as 18-24, 25-34, 35-44, and so on. And to assess their Disney predilections, you might ask them a multiple-selection question such as "Which of the following movies have you watched:" and then give choices such as "Titanic", "Lord of the Rings", "Bambi", "Weatherman", "Hobbit", "Avatar", and "Lion King", and then use those as clues to their Disney preferences.

Once you've determined if the user fits your target demographic, you can then direct them to the proper destination. If they fit, you can send the on to the real survey; if they don't fit, you can send them to a disqualification page (without a completion code) and direct them to return the hit without submitting it.

How much should I pay the turks? Due to the free-market nature of Mechanical Turk, this is a decision that you may want to make on an individual basis. However, keep in mind that on average, Turk workers make between 4 and 5 USD per hour; you can use that as a lower bound. To assess how long a task will take a Turk, you can try it out yourself and have some friends also tell you how long it took them on a trial. Then you can multiply that by your desired hourly rate to obtain a fair payment.

When estimating your costs, keep in mind that Amazon assesses a commission-based fee for each completed task. This is higher if you only use the experienced workers called Masters, and lower if you include less stringent requirements.

How do I ensure that the turks are actually paying attention during my survey so that I can conduct quality control for relevant results? You can accomplish this by inserting some trick questions. Design these questions to look like the rest of your questions, but with one twist: in the question text, tell them which options to select. Insert a few of these in your survey, and then you can tell which users were paying attention by searching for the respondents who answered these questions correctly.

Should I only use Masters, or can I use regular turks? For most surveys, I do not believe that you would need Masters. Masters are credentialed Turk users who have demonstrated excellence. This sounds good, but has a few problems: firstly, they expect relatively high payment levels for their work. Secondly, there are few of them, so responses are sparser and slower. Thirdly, Amazon assesses a higher commission for using Masters. Therefore, you may want to uncheck the box which selects "Only Masters" by default (you may need to view Advanced Options for this).


I hope my research experiences will enrich yours! Do you have questions, comments, or something to add? Let me know at tehila[at]nyu[dot]edu.