4 Judgment Rule 3A for Surveys
Judgment Rule: Determine if the survey questions’ answers will collectively answer the research question.
Key Takeaways
Before the survey is distributed to the sample members of the population, the researchers also need to construct a list of questions to ask the respondents.
These questions, when answered of course, collectively have to provide an answer to the researcher’s initial question.
Definition Box 4.1: Definition of Survey Questionnaire
A survey questionnaire is the list of survey questions the respondents are asked.
If, for example, the researchers are interested in the question, “Does watching Marvel movies decrease concern for environmental sustainability?” there are a number of different ways that the researchers could ask the respondent, and all would be good as long as the questions asked satisfactorily answer the researcher’s research question. The first option, of course, is simply asking directly.
Example 4.1
Questionnaire A: Marvel and environmentalism (short version)
- Do you believe that watching Marvel movies decreases your concern for the environment?
- Yes
- No
For Questionnaire A, the answer to question one (above) will tell what the respondents believe to be the effect of watching Marvel movies (see Example 4.1).[1] However, the question asked is a behavioral question: Does a specific type of movie impact behavior? The question assumes that a respondent can reliably tell that their behavior is/or is not affected, which they most likely would not be able to tell.
Questionnaire B takes a different approach (see Example 4.2 below). First, Questions 1 and 2 determine if the respondents have watched any of the more recent Marvel movies, or if they are more likely to watch serious art films (The Dig and Nomadland) or lighter fare (The Boss Baby and Cruella). As a result, the researcher can develop a more nuanced understanding of whether people watch clusters of film types, and they can identify and contrast people who watch, for example, art films, with people who watch Marvel. In terms of answering the first part of the research question, “identifying who watches and who does not watch Marvel films,” the answers will tell whether or not the respondent has watched the more recent Marvel films, so the reader can say, “Yes, these questions will allow the research to at least partially answer the first part of the research question.”
Example 4.2
Questionnaire B: Marvel and environmentalism
- Have you watched any of the following movies? Please circle all that you have seen.
Black Panther Yes No Spiderman: No Way Home Yes No The Dig Yes No Avengers: Endgame Yes No Stillwater Yes No Cruella Yes No Nomadland Yes No Eternals Yes No Joe Bell Yes No The Boss Baby Yes No - Please circle the number that most closely mirrors how strongly you agree or disagree with the following statements on environmental degradation, with 1 indicating Strongly Agree, 2 = Agree, 3 = Neither Agree nor Disagree, 4 = Disagree and 5 = Strongly Disagree.
Statement SA (1)
A (2)
N (3)
D (4)
SD (5)
It poses a hazard to the whole world. 1 2 3 4 5 It is on the rise. 1 2 3 4 5 It is under control. 1 2 3 4 5 The U.S. has strong environmental pollution laws. 1 2 3 4 5 It is already impacting life as we know it. 1 2 3 4 5 It will impact future generations on the planet. 1 2 3 4 5 It is causing extinction of flora and fauna. 1 2 3 4 5 There’s not much that can be done to stop it. 1 2 3 4 5 It is the government’s job to deal with it. 1 2 3 4 5 All of us have to contribute towards reducing it. 1 2 3 4 5
Please answer the following questions about yourself.
- What is your preferred gender identification?
- Male
- Non-binary
- Female
- What is your age?
- Under 18
- Between 18 and 22
- Over 22
- Are you in college or of out of college?
- Graduated from or in college
- Not a graduate from or in college
- In the next election, are you more likely to vote mostly for Republican or for Democratic candidates?
- Mostly Republican
- Mostly Democratic
The second set of questions gives the researcher a measure of belief in the need for environmental change which is presumable related to environmental behaviors. (Those who believe that action can help and that there is a need for action are more likely to support environmental activity than those who either believe nothing can be done or that there is no need for environmental protection.)
The alternative, Questionnaire C lists all of Marvel movies produced between 2008 and 2021 (the first major arc of the Marvel Multiverse).
Example 4.3
Questionnaire C: Marvel and environmentalism
- I do not watch Marvel Movies. (Please skip to question 3.)
- Have you watched:
Black Widow Yes No Captain America: The First Avenger Yes No Captain Marvel Yes No Iron Man Yes No Iron Man 2 Yes No The Incredible Hulk Yes No Thor Yes No The Avengers Yes No Thor: The Dark World Yes No Iron Man 3 Yes No Guardians of the Galaxy Yes No Guardians of the Galaxy 2 Yes No Avengers: Age of Ultron Yes No Ant-Man Yes No Captain America: Civil War Yes No Spider-Man: Homecoming Yes No Black Panther Yes No Doctor Strange Yes No Thor: Ragnarok Yes No Ant-Man and the Wasp Yes No Avengers: Infinity War Yes No Avengers: Endgame Yes No Spider-Man: Far from Home Yes No - We would like to ask you a few questions about you. Do you:
Recycle paper Yes No Recycle batteries Yes No Recycle smartphones Yes No Recycle computer equipment Yes No Drive a car to work Yes No Shower instead of bathe to save water Yes No Turn the tap water off while brushing teeth Yes No Vote for candidates who support environmental regulation Yes No - What is your preferred gender identification?
- Male
- Non-binary
- Female
- What is your age?
- Under 18
- Between 18 and 22
- Over 22
- Are you in college or of out of college?
- Graduated from or in college
- Not a graduate from or in college
- In the next election, are you more likely to vote mostly for Republican or for Democratic candidates?
- Mostly Republican
- Mostly Democratic
Interestingly enough, although Questionnaires B and C ask very different questions, the two aren’t that different in terms of determining whether or not the respondents watch Marvel Movies. Questionnaire B will give more insight into whether the respondents watch other kinds of films, and Questionnaire C will tell the researcher just how much of a Marvel fanatic the survey respondent is (see Example 4.3), but both can determine whether the respondents are Marvel movie watchers, which is the primary research question.
In terms of whether Questionnaire B or Questionnaire C’s survey questions will elicit more complete or more accurate answers to the second half of the research question (Question 3 on both survey instruments), again, both questionnaires will work. Questionnaire B’s questions are phrased in terms of concern about the environment, while Questionnaire C’s questions will elicit what survey respondents do.
Both questionnaires, then, will answer the research question, because both will elicit some idea on Marvel watching compared to environmental attitudes. The survey will not be able to tell the researcher whether watching Marvel films will decrease concern for the environment—the researcher would need a different method (an experiment) to determine causality. What a survey can do, however, is show whether the survey respondents who watch a lot of Marvel films have less concern for the environment compared to those who primarily watch other kinds of films (Questionnaire B) or fewer Marvel films (Questionnaire C).
- A further problem is that respondents know that the answer they “should” give is that they do care about environmental sustainability, so the respondents are likely to slant their answers to the more socially acceptable answer, but even if they didn’t, all you would know is whether the respondents felt that this particular genre of film did or did not decrease concern about environmental issues. ↵
