Survey Design

(1) are relevant 

(2) ask a single question 

(3) are clear and concise

(4) use appropriate language 

(5) have mutually exclusive and exhaustive response options 

Each of these are outlined in more detail below. 

• Each indicator should have a purpose in the study 
• Each indicator should be directly measuring one of your operational definitions OR a directly related to the analysis of the study 
• “It would be interesting” does not make something relevant 

Double barreled questions should not be used in a survey. Since double barreled questions ask multiple questions, when respondents answer we do not know if they are responding to (a) the first question, (b) the second question, or (c) both questions. Thus, make sure you ask a single question at a time. 

Example: Do you agree or disagree that guns and prayer should be allowed in schools?

• This question is asking two questions and thus should be asked as two single questions. 
• Question 1: Do you agree or disagree that guns should be allowed in schools?
• Question 2: Do you agree or disagree that prayer should be allowed in schools?

Terms: 

Double Barreled questions: A question that actually asks two (or more) questions. 

• Clear and concise questions reduce the amount of cognitive burden on respondents.
• Higher cognitive burden is associated with higher item non-response and drop offs. 

Example: 

High cognitive burden question:

• We are interested in understanding the amount of time you spend each week on various tasks outside of work. In the last month how many minutes have you spent exercising?
• This question has a high level of cognitive burden because it is asking people to (a) recall data over an entire month and (b) convert hours (how people often measure exercise) into minutes  

Low cognitive burden question: 

• In the last week how many hours have you spent exercising?

Terms: 

Cognitive burden: effort being used in the working memory

Item non-response: when respondents do not answer a given question (i.e. do not answer question 12)

Drop offs: when respondents stop the survey (i.e. do not answer the last 20 questions) 

• Keep your study population in mind
• Languages offered: If your study is about migrant farm work you will likely need multiple languages
• Reading level: General rule- no higher than 5th grade
  • Flesch Reading Ease: ideally 81 or higher 
  • Flesch-Kincaid Grade Level: ideally no higher than 5th grade

How to set up the Flesch Reading Ease and Flesch-Kincaid Grade Level measures

1. Open Microsoft Word
2. Go to File
3. Go to Options
4. Go to Proofing
5. Click Show Readability Statistics
6. Click OK

Response options should be both mutually exclusive and exhaustive

Example 1: 

Please select your age range 

• 18 or younger 
• 19 - 30 
• 31- 50 
• 51- 70 
• 70 or older 

These response options are exhaustive but not mutually exclusive. If someone is 70 years old they would need to select two age ranges. To fix this problem either (a) "51-70" should be "51-69" or (b) "70 or older" should be "71 or older" 

Example 2: 

How many books have you read in the last month?

• 1
• 2 
• 3 or more  

These response options are mutually exclusive but not exhaustive. If someone has not read a book in the last month they are not able to answer this question. So the response option of "0" or "I have not read any books in the last month" should be added. 

Terms: 

Mutually exclusive: response options do not overlap

Exhaustive: all responses are included