2 Judgment Rule 1 for Surveys

Judgment Rule: Check whether a survey method will adequately provide an answer to the research question.

Survey research is one of the most common methods of social science research, primarily because the method is a relatively simple and direct way to find out about people—you ask them.

Survey research is used to gather information when the research subject (the respondent) (1) can and (2) will answer a researcher’s questions. When both of these basic conditions are met, the researcher and the reader can get a pretty good idea about what a person thinks, feels, or does by asking. If the researcher asks a number of people in a group—and those people adequately represent who is in the group—then the researcher can reliably describe what the group as a whole thinks, feels, or does.

To put more formally, social science survey research is a systematic way to ask a selected group of people “a question or a series of questions in order to gather information about what most people do or think about something”, and to do so in a way that “examines and delineates the form” of the group.^[1]

Start with the Research Question

Any research question that can be answered by asking people questions can appropriately be researched with a survey.

Surveys are good for asking questions about:

• Values or beliefs: Do you believe that college athletes should get a share of college television revenue? Do you believe that the college athletes should get a salary? Do you believe that nudity should be allowed on television? Do you believe that privacy is a right?
• Feelings: What feature do you like most about your iPhone? What is the primary satisfaction you get from using Twitter? Have you ever been scared watching a movie?
• Knowledge: Who do you think is the president of the United States? Who do you think is the richest person in the United States? What movie studios can you name? How many symptoms of breast cancer do women know about?
• Actions or behaviors: Do you use Twitter? Do you have a Facebook account? Have you seen a movie in a movie theater in the last week? Have you ever screamed in a movie theater (while watching a movie)? Have you ever screamed during a movie that you were watching at home?
• Classification or demographics: What party did you vote for in the last political election? What is your occupation? What is your income? What is your gender?

To determine whether a research question can be answered with survey data, the reader must first ask what the research question is, and then whether respondents can accurately answer questions about that thing that the researcher is studying. In most cases in social science research, the researcher’s fundamental question is a simple reworking of the article title into a question. From the research question, the reader should be able to get a good sense of who the researcher should be talking to and whether the specific survey questions asked will allow researchers to answer their research question.

To illustrate: For the research article “Psychological well-being and demographic factors can mediate soundscape pleasantness and eventfulness,”^[2] the research question is, “Does psychological well-being and demographic factors affect (mediate) the subject’s perception of soundscape pleasantness and eventfulness?”^[3] The article “Cameras of Merit or Engines of Inequality? College Ranking Systems and the Enrollment of Disadvantaged Students” is answering the question, “Do college ranking systems promote inequality between disadvantaged and advantaged students, or do these ranking systems help the truly meritorious (regardless of the student’s background)?” Only in rare cases in peer reviewed journal articles, and more commonly in books,^[4] will the title not communicate the research question.

The research question also will tell the reader who the researcher should have talked to (the research population) and will establish the basic criteria that the reader needs to know in order to judge whether a specific question will help provide an answer to the more general research question.

The research question, “What is Latino/Latinas’ level of knowledge about diabetes prevention and treatment,”  for example, establishes that the researcher must talk to Latino and Latina participants and must ask questions that determine the survey group’s level of knowledge about diabetes. In interviews, the researcher should ask specific questions about diabetes: “Does diabetes run in families?” (It does), or, “Are shortness of breath and chest pains symptoms of diabetes?” (No, those are the symptoms of a heart attack). Based on how people answer these questions, we can find out how much they know about the disease. We can also ask people where they got their information, which would be useful in answering the question “Where do Latinos get information about diabetes?” The interview questions could be fairly straightforward (“Where do you get this information?”), or the researcher could give people a range of choices (“Where did you learn that diabetes causes shortness of breath and chest pains: Family? Friends? Your doctor? The web? The library?”)

The exact interview question, in a sense, does not matter (but see the section on judgment rule three). There are a lot of potential questions that could be asked that are part of the general universe of what is known about diabetes and a lot of ways to phrase a question about how the respondents got the information they have (or think they have). What is important is that the researcher could answer his/her fundamental research question by asking people questions, and therefore, survey is an appropriate method to use. (For more examples, see below.)

Additional Examples of Research Questions

• What bothers children online? This one is fairly simple.
  • Who to talk to: children
  • Sample question: “What have you seen on the internet last week that made you sad or unhappy?”
• Are depressed women who report that their health providers listen to them more likely to get help than depressed women who say their health providers do not listen to them?
  • Who to talk to: depressed women
  • Sample questions: Do you feel that your health provider listens to you? What help did you receive for your depression from your health care provider?

The answers to these two interview questions should allow the researcher to answer the research question. Survey is therefore an appropriate method.