5 Judgment Rule 3B for Surveys
Judgment Rule: Throw out any survey questions that are biased, misleading, or too hard to answer.
Key Takeaways
The reader needs to judge more than whether the survey questions will answer the research question; specifically, the reader needs to examine the quality of each survey question. Each question should be one that the respondents can answer, that they would be willing to answer, and that does not bias or mislead the respondent.
Put more positively, the questions should be clear, simple to understand, and not too demanding.
What should the reader do when the questions are biased or misleading? The reader should discard any answers to unclear, biased, or misleading questions, but throwing out one question does not affect the survey’s overall reliability. However, if there is a consistent pattern of poorly worded questions, then that pattern indicates that the researcher either lacks skill or is willing to mislead the respondent. The reader should use a pattern of biased or misleading questions to judge the entire survey. Questionnaires from special interest groups, for example, are particularly likely to be filled with questions where the “correct” answer is the answer that is closest to the positions that the special interest groups take. These surveys are literally worthless in terms of understanding the respondent’s actual positions. (See the section on social desirability bias for more information.)
As a part of judging the research, the reader should carefully examine each of the sample questions that the researcher provides in the methods or results sections of their research report, in order to check for a pattern of hard-to-answer or biased questions. Example 5.1 provides a sample of common hard-to-answer questions.
Example 5.1
Categories of hard-to-answer questions
Too vague: Had any problems with your computer lately?
Not mutually exclusive response categories: What is your current age?
- Below 18
- Between 18—30
- Between 30—50
- Above 50
Unbalanced scales: How much did you like the movie Fifty Shades of Grey?
- Extremely
- Very much
- Mostly enjoyed
Not covering all answer categories: What electronic media do you own?
- Phone
- Computer
- Television
Using jargon and acronyms: Which processor are you more likely to purchase?
- AMD A10-6700
- AMD A8-7600
- Intel Core i7-3960
- Intel Core i7-4790K
- Some other processor
Double-barreled question: Do you have a cell phone or a landline?
- Yes
- No
Too demanding on memory: What movies did you watch when you were six?
Hard-to-Answer Questions
Vague Questions
Questions should be clear, as short as possible, and easy to understand—both the question and the answer categories. The question “How good is your television?” has the advantages of being short, but is actually vague and therefore confusing. Is the survey asking the respondents to talk about the quality of their reception, or the quality of the shows they watch? Either understanding of the question is reasonable, which means that neither the researcher nor the reader knows which interpretation the respondent is making. Basically, then, the question is worthless.
Vague Response Categories
Response categories follow the same general rules that questions do. The response categories should be clear, direct, and not confusing or leading. In addition, all response categories should be mutually exclusive and balanced, and cover all the potential answer categories. For example, in Example 5.1, the question “What is your current age?” has two response categories a 30-year-old respondent could legitimately fill out (response 2: between 18 and 30, and response 3: between 30 and 50). It is not clear which category the respondent should choose. Given that some of the respondents will chose one and some will choose the other, the researcher will not be able to make any clear conclusions about whether 30-year-olds are more like the 18- to 29-year-old group or the 31- to 50-year-old group.
There is a problem of another sort with unbalanced scales where the scale is weighed in one direction (usually favorable). The list of answers to how much the viewer enjoyed the movie Fifty Shades of Grey at worst only allows the respondent to say that he or she “mostly enjoyed” the movie; there is no response category, for example, for “absolutely hated.”[1]
Jargon and Acronyms
Both jargon (specialized words used by a particular profession or group) and acronyms (initial letters of organizations) are confusing and irritating to those who are not familiar with the terms and are likely to be answered with a guess. Of course, the effectiveness of jargon also depends on the specific population studied. If the research population is composed of computer hardware specialists who buy computer equipment for university students and professors, you might expect this audience to be up-to-date with the newest technology and terminology in the field, so referring to computer processers by number (e.g., Intel Core i7-4790K) should be understandable to that population. If the survey respondents, however, are average computer users, they probably would not know these terms.
In addition, questions need to be appropriate for the respondent’s age, culture, and ability to read (literacy level).
Double Negatives
Avoid double negative questions, which are simply difficult for any respondent to understand. For example, instead of asking people to respond to the following double negative statement—“It is not true that video games are not violent”—the researcher should have asked the respondents their reaction to the direct statement “Video games are violent.”
Double-Barreled Questions
Double-barreled questions are two questions in one: Do you like rap and rhythm-and-blues? The answer might be clear if the survey respondent either doesn’t like both or does like both, but if the respondent likes one and doesn’t like the other, answering the question is very difficult. For example, the double-barreled question in Example 5.1—“Do you have a cell phone or a landline?”—assumes that people have either a cell phone or a landline. How should the people who have both answer this question? If the survey respondent doesn’t get angry and quit filling out the survey (which does happen), they will make an executive decision to privilege one answer (either what they have or what they don’t have) and ignore the other. Since no one other than the respondent knows what decision-making rule the respondent has used, the answer is essentially useless to understand what is going on in the population.
Questions That Are Too Demanding on Memory
Again, the purpose of surveys is to find out what the population is thinking, feeling, or doing, which is only a good method when the respondent is capable of answering the question. Questions that are too demanding or require too much effort to read or respond to are likely to get a guess at best or a random response, neither of which will deliver an accurate picture.
In surveys, a respondent’s willingness to please can produce a response that portrays them in a more positive or favorable manner than their actual response would. This phenomenon is called social desirability bias. Even in the fleeting, and often totally anonymous, social interaction of a survey, people want to please—and they can be very skilled at reading what researchers want from the way questions are phrased. Respondents tend to over-report pleasing behavior—such as overestimating how much time they spend in community service or how much money they give to charity. On the other hand, they tend to underreport behaviors that they think others will disapprove of—such as how much pickup sex they have or how racist they are.
Questions to Trigger Social Desirability Bias
Leading Questions
Leading questions, whether asked by survey researchers or the police, are “questions” that tell the respondent what answer the researcher wants. It is very unlikely that a respondent wouldn’t know that the preferred answer to the question, “Do you agree that the natural beauty of the Arctic National Wildlife Refuge should be preserved?” is “Yes, by all means let us save the National Wildlife Refuge.” The people who say they don’t want to preserve the Arctic National Wildlife are fighting against a natural desire to please; therefore, it is quite likely that they are reporting their true feelings. It is also quite likely that some of the respondents who agree with preserving the wilderness are just agreeing with the researchers’ obvious preference, and that actual support for preservation may be lower than the survey responses would indicate.
Halo Effect
Researchers can also inject bias into a survey by tying the questions to an obvious good or an obvious bad.
For example, let’s examine the question:
“Do you agree with the heavy metal guitarist who bites the heads off of bats when he says, “I believe that playing first-person video shooting games decreases violence?”
The question ties together beliefs about the impacts of playing first-person videos to feelings about heavy metal music and feelings about people who bite the heads off of bats.
Respondents who hate heavy metal music or think that heavy metal itself promotes violence and mayhem would be more likely to disagree with the statement, “I believe that playing first-person video games decreases violence,” than they would have if they had been asked with no reference to heavy metal (or bats). (Of course, the people who love heavy metal (or biting bat’s heads off) would also be likely to be influenced in the opposite direction.)
The tendency to react to what is tied to a question, rather than the question itself, is called the halo effect, and as with leading questions, is likely to shift the overall direction of respondents’ answers. Those who like what the question is tied to will be more likely to agree (“Do you agree with Jesus when he said…”), and those who dislike the tie (“Do you agree with Hitler…”) will be more likely to disagree with the statement.
- Given that Fifty Shades of Grey had a 24% rating on the Rotten Tomatoes tomatometer, a lot of people absolutely hated the film. ↵
