Intro to Conjoint Experiments | Lab - 3
1 Why Qualtrics
- Quite flexible
- Does not require (much) programming
- Excellent support
- Widely used in Academia
- Widely used by survey companies
- Other options are not there yet
- Check out Lime Survey if are willing to learn JS.
2 Getting started: Create a sruvey
- Create an account
- Create a new survey
- Create new Project -> Survey
- Project Name -> Import OR Create a blank Project
3 Adding Questions
- Organize the survey in blocks
- Changing block means changing page
- Each block should correspond to a part of the survey (Demographics, Controls, Experiments…)
- Insert questions in each blocks
- Click on each block to add text and change options
- Use the right panel to modify the type of questions and their behaviour
- Use the left panel for the randomization of the questions’ options
4 CJ Block: HTML Table
- Open the .html generated using Conjointdt file using a text editor
- textEdit on Mac
- notepad on Windows
- Sublime
- Modify according to your needs
4.1 CJ Block: Qualtrics
- On Qualtrics: Create a new multiple choice question
- On the text area select “HTML View”
- Copy the content of each .html file generated by Conjointsdt
- Modify the text if necessary
- Add a binary Multiple Choice Options (e.g. Candidate 1, Candidate 2)
- Validation Options -> Request Choice
- Position -> Horizontal
- Answer -> Single Answer
- Insert a Page Break
- Repeat for all the CJ tasks
4.2 CJ Blocks: Image
5 Add Java Script
- Start by choosing a question that appears somewhere before the conjoint in the survey flow.
- It can be any question that appears to all respondents (e.g., the consent form, a demographic variable that is asked of all respondents).
- Find the JavaScript console.
- By default, Qualtrics pre-populates some functions
- All of the JavaScript for the randomization goes inside the Qualtrics.SurveyEngine.addOnload() function, as indicated below:
.SurveyEngine.addOnload(function()
Qualtrics
{// Code to randomly generate conjoint profiles in a Qualtrics survey
// Terminology clarification:
// Task = Set of choices presented to respondent in a single screen (i.e. pair of candidates)
// Profile = Single list of attributes in a given task (i.e. candidate)
// Attribute = Category characterized by a set of levels (i.e. education level)
// Level = Value that an attribute can take in a particular choice task (i.e. "no formal education")
// Attributes and Levels stored in a 2-dimensional Array
/* Randomize array in-place using Durstenfeld shuffle algorithm */
function shuffleArray(array) {
for (var i = array.length - 1; i > 0; i--) {
var j = Math.floor(Math.random() * (i + 1));
var temp = array[i];
= array[j];
array[i] = temp;
array[j]
}return(array);
}
// Function to generate weighted random numbers
function weighted_randomize(prob_array, at_key)
{var prob_list = prob_array[at_key];
// Create an array containing cutpoints for randomization
var cumul_prob = new Array(prob_list.length);
var cumulative = 0.0;
for (var i=0; i < prob_list.length; i++){
= cumulative;
cumul_prob[i] = cumulative + parseFloat(prob_list[i]);
cumulative
}
// Generate a uniform random floating point value between 0.0 and 1.0
var unif_rand = Math.random();
// Figure out which integer should be returned
var outInt = 0;
for (var k = 0; k < cumul_prob.length; k++){
if (cumul_prob[k] <= unif_rand){
= k + 1;
outInt
}
}
return(outInt);
}
var featurearray = {"Campaign statement" : ["The political power should return to the people","Competent party leaders make the best decisions"],"Proposal" : ["Ban extreme groups rallies in the state capital","Rule by executive order if the opposite party does not collaborate","Ban intollerant speech against political minorities"],"Language Spoken" : ["French","Turkish","Italian","Portuguese","Dutch"],"Party" : ["PVDA (Radical-Left)","ECOLO (Greens)","PS (Left)","CDH (Liberals)","MR-VLD (Right)","N-VA (Radical Right)"]};
var restrictionarray = [[["Age","30s"],["Years in Politics","20"]]];
var probabilityarray = {};
// Indicator for whether weighted randomization should be enabled or not
var weighted = 0;
// K = Number of tasks displayed to the respondent
var K = 5;
// N = Number of profiles displayed in each task
var N = 2;
// num_attributes = Number of Attributes in the Array
var num_attributes = featurearray.length;
// Should duplicate profiles be rejected?
var noDuplicateProfiles = false;
var attrconstraintarray = [];
// Re-randomize the featurearray
// Place the $featurearray keys into a new array
var featureArrayKeys = Object.keys(featurearray);
// If order randomization constraints exist, drop all of the non-free attributes
if (attrconstraintarray.length != 0){
for (const constraints of attrconstraintarray){
if (constraints.length > 1){
for (var p = 1; p < constraints.length; p++){
if (featureArrayKeys.includes(constraints[p])){
var remkey = featureArrayKeys.indexOf(constraints[p]);
.splice(remkey, 1);
featureArrayKeys
}
}
}
}
}
// Re-randomize the featurearray keys
= shuffleArray(featureArrayKeys);
featureArrayKeys
// Re-insert the non-free attributes constrained by $attrconstraintarray
if (attrconstraintarray.length != 0){
for (const constraints of attrconstraintarray){
if (constraints.length > 1){
var insertloc = constraints[0];
if (featureArrayKeys.includes(insertloc)){
var insert_block = [];
for (var p = 1; p < constraints.length; p++){
.push(constraints[p]);
insert_block
}var begin_index = featureArrayKeys.indexOf(insertloc);
.splice(begin_index+1, 0, ...insert_block);
featureArrayKeys
}
}
}
}
// Re-generate the new $featurearray - label it $featureArrayNew
var featureArrayNew = {};
for (var h = 0; h < featureArrayKeys.length; h++){
= featurearray[featureArrayKeys[h]];
featureArrayNew[featureArrayKeys[h]]
}
// Initialize the array returned to the user
// Naming Convention
// Level Name: F-[task number]-[profile number]-[attribute number]
// Attribute Name: F-[task number]-[attribute number]
// Example: F-1-3-2, Returns the level corresponding to Task 1, Profile 3, Attribute 2
// F-3-3, Returns the attribute name corresponding to Task 3, Attribute 3
var returnarray = {};
// For each task $p
for(var p = 1; p <= K; p++){
// For each profile $i
for(var i = 1; i <= N; i++){
// Repeat until non-restricted profile generated
var complete = false;
while (complete == false){
// Create a count for $attributes to be incremented in the next loop
var attr = 0;
// Create a dictionary to hold profile's attributes
var profile_dict = {};
// For each attribute $attribute and level array $levels in task $p
for(var q = 0; q < featureArrayKeys.length; q++){
// Get Attribute name
var attr_name = featureArrayKeys[q];
// Increment attribute count
= attr + 1;
attr
// Create key for attribute name
var attr_key = "F-" + p + "-" + attr;
// Store attribute name in returnarray
= attr_name;
returnarray[attr_key]
// Get length of levels array
var num_levels = featureArrayNew[attr_name].length;
// Randomly select one of the level indices
if (weighted == 1){
var level_index = weighted_randomize(probabilityarray, attr_name) - 1;
else{
}var level_index = Math.floor(Math.random() * num_levels);
}
// Pull out the selected level
var chosen_level = featureArrayNew[attr_name][level_index];
// Store selected level in profileDict
= chosen_level;
profile_dict[attr_name]
// Create key for level in $returnarray
var level_key = "F-" + p + "-" + i + "-" + attr;
// Store selected level in $returnarray
= chosen_level;
returnarray[level_key]
}
var clear = true;
// Cycle through restrictions to confirm/reject profile
if (restrictionarray.length != 0){
for (var v = 0; v < restrictionarray.length; v++){
var falsevar = 1;
for (var mp = 0; mp < restrictionarray[v].length; mp++){
if (profile_dict[restrictionarray[v][mp][0]] == restrictionarray[v][mp][1]){
= falsevar*1;
falsevar else{
}= falsevar*0;
falsevar
}
}if (falsevar == 1){
= false;
clear
}
}
}
// If we're throwing out duplicates
if (noDuplicateProfiles == true){
// Cycle through all previous profiles to confirm no identical profiles
if (i > 1){
// For each previous profile
for(var z = 1; z < i; z++){
// Start by assuming it's the same
var identical = true;
// Create a count for $attributes to be incremented in the next loop
var attrTemp = 0;
// For each attribute $attribute and level array $levels in task $p
for(var qz = 0; qz < featureArrayKeys.length; qz++){
// Increment attribute count
= attrTemp + 1;
attrTemp
// Create keys
var level_key_profile = "F-" + p + "-" + i + "-" + attrTemp;
var level_key_check = "F-" + p + "-" + z + "-" + attrTemp;
// If attributes are different, declare not identical
if (returnarray[level_key_profile] != returnarray[level_key_check]){
= false;
identical
}
}// If we detect an identical profile, reject
if (identical == true){
= false;
clear
}
}
}
}= clear;
complete
}
}
}
// Write returnarray to Qualtrics
var returnarrayKeys = Object.keys(returnarray);
for (var pr = 0; pr < returnarrayKeys.length; pr++){
.SurveyEngine.setEmbeddedData(returnarrayKeys[pr], returnarray[returnarrayKeys[pr]]);
Qualtrics
}
;
})
.SurveyEngine.addOnReady(function()
Qualtrics
{/*Place your JavaScript here to run when the page is fully displayed*/
;
})
.SurveyEngine.addOnUnload(function()
Qualtrics
{/*Place your JavaScript here to run when the page is unloaded*/
; })
6 Add Embedded content
- This is needed to store the results of your randomization.
- If you forget to do this, everything will still seem fine in Qualtrics but you won’t be able to analyse the data.
- Click the ”survey flow” link in the upper-left corner of your Qualtrics screen.
- Add a new embedded data element to it by clicking “add below” or “add a new element here”.
- Click and drag this element so that it is placed near the beginning of your survey.
- It must be above the question block that includes your randomization JavaScript.
- I usually put these at the very top of my survey flow.
- Enter the same names from the conjoint tables
- Leave the values blank.
- This must be repeated for every conjoint task.
7 Attention check
7.1 Instructional Manipulation Checks (IMCs)
- They should check whether a participant has paid attention to the question, not so much to the instructions above it
- Participants must be explicitly instructed to complete a task in a certain way (e.g. ‘click ’Strongly disagree’ for this question’)
- No room for misinterpretation (e.g. ‘Bird is a clothing brand. Do you agree?’)
- They must be easy to read (i.e., should not use small font, or have reduced visibility)
7.2 A good example of an IMC
7.3 A bad example of an IMC
8 Nonsensical Items
- Use a scale response
- No prior knowledge should be required
- Participants should not be given a ‘neutral’ response option (i.e., no middle scale value)
- Correct responses should either be extreme agreement or extreme disagreement, however any response indicating disagreement or agreement should be accepted
- Participants who answer in the opposite manner to the objectively correct answer have failed
8.1 A good example of a nonsensical item
8.2 A bad example of a nonsensical item
9 Comprehension checks
- Participants must be free to re-read the key information at the time the comprehension check is presented
- These checks cannot involve free-text responses
- Comprehension checks should be given at the start of the study/experiment
- If a participant fails a comprehension check twice then they should be excluded
9.1 A good example of a nonsensical item
From @gordon_keeping_2019
10 Fraud Detection
10.1 Prevent Multiple Submissions
- Go to Survey options.
- Go to Security.
- Enable Prevent multiple submissions.
- End survey with a message: The respondent is sent out of the survey and we display an end of survey message.
- Redirect to URL: Redirect respondent out of the survey to a URL.
- Flag responses: Duplicate respondents will not be sent out of the survey, and will be allowed to finish their response. However, their responses will be assigned a value under the field Q_BallotBoxStuffing.
10.2 Bot Detection
- Go to Survey options.
- Go to Security.
- Enable Bot Detection.
- This activates the field Q_RecaptchaScore, which uses Google’s invisible reCaptcha technology.
10.3 Scraper Detection
- Go to Survey options.
- Go to Security.
- Enable Security Scan Monitor.
- Prevent email scanning software from inadvertently starting a survey session when a survey link is included in the email.