Mechanical Turk has a built in HIT builder which is basic at best. In particular, it doesn’t allow for between-subject design. The only tools available are HTML and JavaScript, which is not enough for feature rich experiments.
However, it is possible to program your experiment on your own server using any technology you wish.
Quick overview:
- Once you have used the command line tools to load your HIT on MT, it will load on the Mechanical Turk website, in an iframe (which shows the content of your experiment on your own server).
- The Iframe passes certain variables you need to keep hold of: workerid, assignmentid, and turkSubmitTo as an HTTP GET request to the address of your experiment.
- The worker can now interact with your own experiment, which runs on your own server, as normal through the iframe. Store all the data you need from your own experiment as usual.
- At the end of the experiment, you need to submit a form back to Mechanical Turk, with the variables given by the iframe earlier, so that you can pay the worker.
Step 1: Install and fix command line tools
What you need to use is the command line tools.
Download here.
Windows users, I have written a detailed guide on getting these working here
tl;dr instructions (all platforms) are
- Extract the files somewhere sensible
- Change the mturk.properties file in the bin folder to have the correct access/secret keys for your aws account, and edit both service_url lines to use https:// rather than http://
- Set an environment variable that is called MTURK_CMD_HOME which is set to the path of the tools folder (yes the folder above bin)
- Set an environment variable called JAVA_HOME which points to the path of your JRE (yes, the folder above bin).
Step 2: Code your experiment
So – now you need to code your experiment. You can use any language you like, but PHP is convenient as the University web server has PHP installed. If you want to use PHP v5 – you need to do some tinkering (the Uni web server defaults to v4).
Accessing the University Filespace off campus is easy using SFTP (on Windows, WinSCP is great). Address: ftp.abdn.ac.uk Protocol: SFTP Port: 22
If you use Netbeans with the PHP addon installed (I recommend that you do!), you can set this up as a remote project by selecting “PHP application on Remote Server”. Then every time you save a file, it is updated in your filespace automatically.
2a) Set the university web server to use a newer version of PHP5 with register_globals off (for sanity) :
2b) Set up your database
The university has a MySQL server that you can use to store data. If you haven’t used this before, you need to fill out the form here: https://www.abdn.ac.uk/local/mysql/rf21/rf21.php for instant access.
After you have done that, you can create your database here: https://www.abdn.ac.uk/local/mysql/mysqladminhomepages/
Then you can use PHPMyAdmin to create your tables. Full userguide for using MySQL on the uni server is here: http://www.abdn.ac.uk/local/mysql/
2c) Creating the start page
Create a folder INSIDE public_html to hold your experiment – eg exp1
In this folder create a file called index.php – this is what your workers will see in the preview AND when they first accept the HIT
2d) Grabbing the workerId and assignmentId, and turkSubmitTo
MT shows workers a preview of your HIT, before they accept it. So you need to handle this.
On the first line-
<?php session_start();
//and you should probably connect to your db and make sure it’s working
//see PHP manuals for how to do that
?>
Then add the standard html doctype etc
<!DOCTYPE html>
<html>
<head>
<title>My Hit</title>
</head>
<body>
<h1>My HIT</h1>
<p>This hit will ask you to do this that or the other and will store this stuff about you: </p>
Then you need to grab the workerid (if this is a worker PREVIEWING the HIT, this will be blank) and the assignment ID so MT knows who to pay.
In PHP $_SESSION is an array where you can store values from one page to the next so:
<?php if (!empty($_GET['workerId']) && !(empty($_GET['assignmentID'])) {
//if this webpage was passed these two variables, the worker has accepted the HIT
//store them in the session for when we submit later
$_SESSION['workerId'] = $_GET['workerId'];
$_SESSION['assignmentId']= $_GET['assignmentId'];
$_SESSION['submitURL']= $_GET['turkSubmitTo'];
//make the start button appear, but you’ll need to comment this out until you’re finished programming and ready for sandbox testing.
//echo ‘ /*html to make your start button*/’;
}
?>
Test that you can see something. Save the file. Then go to https://homepages.abdn.ac.uk/local/publish and follow the instructions
(note, you MUST use HTTPS)
Then your page will be visible at https://homepages.abdn.ac.uk/my.name/pages/exp1
where my.name is the first part of your email address (before the @).
2e) Program your experiment. Can’t help you here 😀
If you’re using the uni filespace, you need to run the publish script each time you create a file, otherwise you will see 403 forbidden errors.
2f) Submit your data to MT
After Workers finish your task, they need to submit to MTurk for payment. So the final step is a form like this:
<form id ="mturk_form" method ="POST" action = "<?php echo urldecode($_SESSION['submitURL']).'/mturk/externalSubmit'; ?>"
<input type="hidden" id="assignmentId" name="assignmentId" value="<?php echo $_SESSION['assignmentId'] ?>" />
<input type="hidden" id="workerId" name="assignmentId" value="<?php echo $_SESSION['workerId'] ?>" />
<!-- any other data you want to send to MT send in a similar way, using hidden form fields. These appear on your requester interface on the MT website for when you approve/reject responses. I usually send the time taken, to reject those who did it too quickly etc.-->
<input id="submitButton" type="submit" name="Submit" value="Submit to MTurk"/>
</form>
You can add more variables using hidden input fields as above, which show on the MT interface when you are approving or rejecting results. This makes life easier as you don’t need to check each one in the db to see if their response is valid. I usually include the time taken to do the experiment.
3. Set up the HIT files for Mechanical Turk
The command line tools are responsible for telling mech turk about the HIT, and making it live on the system.
Inside the command line tools folder (eg C:\mech-turk-tools-1.3.1) there is a folder called Samples. In there are some basic samples, and the files you need to edit are in the external_hit folder.
Basically you edit the files:
external_hit.properties: How much you are paying, and how long you want your experiment to run for (the file is commented).
I usually set the location to US only and the acceptance rate of workers to be >=90% to work on the HIT, to do this, paste:
#qualifications
qualification.1:00000000000000000071
qualification.comparator.1:EqualTo
qualification.locale.1:US
qualification.private.1:1
qualification.2:000000000000000000L0
qualification.comparator.2:GreaterThanOrEqualTo
qualification.value.2:90
qualification.private.2:0
To the bottom of the properties file.
external_hit.question: Put your URL of where your experiment is inbetween the <ExternalURL> tags. Replace whats there already eg https://homepages.abdn.ac.uk/my.name/pages/exp1
(note: make sure that you use https:// otherwise browsers will not display your HIT).
external_hit.input: If you want a between subject design (one worker, one variant, one response), add two lines:
null
noinputdata
If you have many variants of your study, and you want to allow workers to take your HIT many times and see each variant (and be paid many times), put the names of the variables to change in this file, along with their values. (Within Subject Design). This is documented on the MT website.
4. Testing
As you should have coded in that the 3 GET variables (workerId, assignmentId and turkSubmitTo) are needed for the HIT to start, this makes testing a pain. However, you can use my MT Simulator to get round this 🙂
When you are ready for a final test on the Mech Turk website, ensure that you have the sandbox selected in the command line tools:
Go into the bin folder in the mech-turk-tools-1.3.1 folder, and open the mturk.properties file. Comment out the production URL and uncomment the Sandbox URL (the file shows where these are).
Open the command line tools. Enter:
cd ..\samples\external_hit
run
(on mac/linux use a forward slash)
Then go to workersandbox.mturk.com and sign in to take your own experiment to check it works. The command line also outputs the direct URL of your HIT.
If you have included qualifications in the properties file – comment these out (put a # in front of all the lines in the qualifications section) before deploying to the sandbox or you may find that you aren’t qualified to take your own HIT!
5. Deployment to Live System
- Expire the HIT on the sandbox. You can do this at https://requestersandbox.mturk.com/mturk/manageHITs
- Clear your db of testing data
- Uncomment the qualifications in the properties file
- Change the mturk.properties file (in the bin folder) to use the production site instead of the sandbox
- Then run the same command as in section 3 to make your HIT live.
6. Approving and Rejecting Results
I find that the easiest way to approve/reject results is to not use the tools. Use the web interface here. Your external hit will be there and you can accept/reject the results as they come in.
