Turk workflow and tools . mitcho + Hadas mitcho,hkotek@mit.edu - - PowerPoint PPT Presentation

. .

Turk workflow and tools

mitcho + Hadas

mitcho,hkotek@mit.edu

Hackl Lab Turkshop March 2013

. .

Review from week 1

How do you run something on Turk?

. . 1 Log in to Turk . . 2 Create a new experiment (probably duplicate a previous one) . . 3 Change attributes (title, description, payment, location, etc.) . . 4 Upload and verify template

today!

. . 5 Upload items (batch)

today!

. . 6 Confirm and publish . . 7 Approve and reject subjects

today!

. . 8 Download and analyze results

next session!

. . 9 Success!

. . 2

Today

. . 1 The relationship between templates and Turk item files . . 2 Examples of different kinds of experiments . . 3 Preparing for Turk (technical workflow) . . 4 Approving and rejecting subjects

. .

3

Some terminology

• An item is a set of sentences/stimuli that vary the factors of interest

in a systematic way, and hold constant everything else.

• A condition is a label for a particular setting of all of the factors of

interest.

• Individual stimuli within an item are called trials.
• Items are grouped into sections. Normally, “target” and “filler.”

# blocking 1 inanimate-make-v That’s the ball that the coach bounced on the floor. # blocking 1 inanimate-v That’s the gymnast that the coach bounced on the floor. # blocking 1 animate-make-v That’s the ball that the coach made bounce on the floor. # blocking 1 animate-v That’s the gymnast that the coach made bounce on the floor.

. .

4

First things first

Technical prerequisite: ☞ Python 2.7.3: http://www.python.org/getit/ (choose 2.7.3, not 3.x) If you had trouble, um, maybe share your neighbor’s computer for today? Now download this: ☞ Tools and sample files (secret URL) (Just the sample files are up at http://web.mit.edu/hackl/www/ lab/turkshop/examples/week3.zip)

. .

5

Your template and items

What people see:

. .

6

Your template and items

What you give Turk: Template file Turk items file .html .turk.csv ☞ Open up example1-template.html in your browser. Probably double-clicking on it will work.

• What kind of experimental paradigm is this template for?
• How many items is this template file expecting?
• How is it different than what the subject sees?

. .

7

Your template and items

What you give Turk: Template file Turk items file .html .turk.csv ☞ Open up example1-items.turk.csv in Excel. (CSV files are a kind of plain-text spreadsheet.)

• How many separate lists does this include?
• How many items do these lists have?
• What’s the relationship between list 0 and 1? 2 and 3?

. .

8

Your template and items

When you upload a template and item file to Turk, each one of these lists will be turned into a HIT (“Human Intelligence Task”). Each string in that row in the Turk items file will be plugged into “fields” in the template. list trial_1_1 trial_2_1 · · · That’s the rabbit that the magician made vanish into thin air. That’s the boy that the instruc- tor made float in the pool. · · · . . . . . . . . . ... In list #0, the text ${trial_2_1} in the template file will be replaced with “That’s the boy that the instructor made float in the pool.”

. .

9

simulator.py

We have a tool to simulate this, so that you can see how your data will look in your template without uploading to Turk. + — → template Turk items simulator.py simulation ☞ Run simulator.py:

• Mac: Open with “Python Launcher.” Maybe right-click, “Open With >

Python Launcher.” Double-clicking might work.

• Windows: Double-clicking on the script probably works.
• Linux: You know what you’re doing.

At the prompts, enter example1-template.html, example1-items.turk.csv, and 0. Now you (should) have a example1-template-simulation.html file in your folder. Look at it in your browser. Note that Turk will add a submit button, so our template doesn’t need to include that.

. .

10

Experiment types and skeletons

We create templates from “skeleton” files. Think “template recipes.” Each skeleton corresponds to a different kind of experiment. Let’s open some skeletons in our browser.

• binary.skeleton.html
• binary-image.skeleton.html
• completion.skeleton.html
• image-choice.skeleton.html

Audio and sentence choice coming soon...

. .

11

The Turkolizer

We use a (largely rewritten) version of Gibson, Piantadosi, and Fedorenko (2011)’s Turkolizer to take items and turn them into a Turk items file. The Turkolizer does the following:

• Create Latin Square lists from the items, so that each participant sees
• nly one condition per item.
• Randomize the lists and mix the targets and fillers together, following

certain constraints.

• Print the resulting lists into a Turk items file (.turk.csv).
• Print information about which list contains which trials into a

.decode.csv file. This will be used later to analyze the results.

. .12

The Turkolizer input file

☞ Open example2-items.txt, which is a Turkolizer input file. Each trial has a header beginning with #. The header includes a section name, item number, and condition name. # target 1 most Most of the dots are blue. http://web.mit.edu/hkotek/www/experimentpictures/ gradientYN/2C-1.png . . . Each line of the trial corresponds to a different field in the template. For this experiment, each trial has a sentence and a URL of an image. At least

• ne blank line is required between trials.

☞ How many sections are there? How many items do they have? How many different conditions are there? ☞ How many items will an individual participant be asked to answer?

. .

13

Let’s Turkolize it!

— → & Turkolizer input

turkolizer.py

Turk items decode file

.txt .turk.csv .decode.csv

☞ Run turkolizer.py. Enter example2-items.txt and request a multiple of 2 lists. It will ask you about filler placement constraints. It will then print a nice summary of how many sections, items, conditions,

• etc. were found and (hopefully) tell you that the CSV files were

successfully created.

NB: Double the number of lists you request will be created. That’s

because the reverse order of each list is also always included, to attempt to counter ordering effects. ☞ How many items does it say are in each list?

. .

14

templater.py

We need a template of the right length. We have a tool to create templates

• f arbitrary length from skeletons:

— → template skeleton templater.py template ☞ Run templater.py. Create a template based on binary- image.skeleton.html for 108 trials. Pick a code, any code. Open the resulting template file and verify that it’s built for 108 trials.

. .

15

Putting it together

☞ Use simulator.py to put your new template and our new Turk items file together. Make sure the resulting simulation looks good. At this point you would be ready to upload your template and Turk items (.turk.csv) file to Turk!

. .

16

Serving files

For images or audio files, we recommend using your Athena account.

• Put it in your home directory’s public subdirectory and it will show

up in http://web.mit.edu/username/www/...

• How to access your Athena web space via FTP:

http://kb.mit.edu/confluence/display/istcontrib/ Transferring+files+to+and+from+Athena

. .

17

Completions exercise (time permitting)

. . 1 Open example3-items.txt. Here are a few sentences Hadas

wrote for a completion study, where participants will choose between two quantifiers in a sentence.

• Targets will have two conditions, did and was, based on the verb at the

end of the sentence.

. . 2 Open completion.skeleton.html. Each trial must be split up

into four strings:

• field 1: text before the gap
• field 2: text afuer the gap
• field 3: gap option one
• field 4: gap option two

. . 3 Prepare these sentences for the Turkolizer.

Hint: regular expressions will save you time!

. . 4 Create Turk items with turkolizer.py. . . 5 Create template from completion.skeleton.html with the

appropriate number of items.

. . 6 Simulate! Win all the prizes!

. .

18

Turk technical workflow overview

aka “Turkflow” → — → &

raw

(RegEx)

Turkolizable

turkolizer.py

Turk items decode file

.txt . .turk.csv .decode.csv

— →

template skeleton

templater.py

. template

.

simulation

.

simulator.py

. Turk!

. .

19

Approving subjects

. . 1 Download the results file (CSV) and open in Excel.

☞ Open example4-results.csv and take a look through it.

. . 2 Sort by Worker ID and check for duplicates. . . 3 Decide which HITs should be approved and which should be rejected.

• For participants who had JavaScript on, the count from the counter is

in Answer.numanswered.

• Find unanswered questions visually with conditional formatting:

Mac: Select all cells, go to Formatting > Contextual Formatting, choose “Classic” style, “format all cells that contain”, “blanks”, specify color. Windows: Select all cells, go to Contextual formatting, New Rule, “Format only cells that contain” + “Blanks”. Click on “format”, choose “fill” and choose your favorite color.

. . 4 Go back to the site and reject/approve. You can reject multiple

people at the same time and enter a single rejection message.

. .

20

Benediction

You have $20. Spend it wisely. Go forth! Run something over break!

. .21