7 Summary of Judgment Rules for Survey Methods

Decision Rule Set for Research Method

If a survey is a reasonable method for answering the research question, go on to the following decision rules. If NO, throw out the study discussion and conclusions. The answer to question 2 must be YES in order to have any trust in the study at all.

Decision Rule Set for Sampling

1. The survey can only be generalized back to the research population that was sampled.
   1. You need to check each time the researcher narrows the sample. How was the sample selected? How good was the sampling frame?
   2. Did the researcher check for external validity? (Did the researcher check his/her sample against the general population? That is, did the researcher check whether the demographics (age, race, gender) are like the general population?)
   3. Is the response rate respectable (see also section on response rate)?
2. If there are no major flaws with the selection of the sample, then you can trust that the survey results–for good questions–can be generalized back to the research population.
3. If the survey sample is NOT representative of the population, then you will have a range of answers.
   1. The survey results can be generalized back to the population, with the exception of specific groups that were not sampled. THE RESULTS PROBABLY REFLECT THE POPULATION EXCEPT FOR THE GROUPS THAT WERE NOT INCLUDED.
   2. The survey results show that some people have this opinion (feeling, belief, behavior), but the results shouldn’t be extended to the population. THE FINDINGS APPLY TO THE GROUP SAMPLED AND JUST THAT GROUP.
   3. The sampling procedure was so flawed that you have to wonder whether the people who answered the survey might actually be a little weird (if only an unusual group of people might have answered the survey). DO NOT TRUST!

Decision Rule Set for Questions

1. Accept the findings when the questions are clear, easy to answer, and not leading or biased.
2. Findings (or conclusions) that are based on flawed questions should be thrown out.
3. You can also use the questions to judge a researcher’s competence. If there is a consistent pattern of poorly worded or biased questions, you should assume that the researcher was equally sloppy (or biased) in other parts of the survey.

Decision Rule Set for Response Rate

1. Accept when the response rate is over 60 percent.
2. Accept with reservations when the response rate is between 10 and 59 percent. (That is, the findings apply to the group sampled and only that group.)
3. Under 10 percent, highly questionable. (The people who answered may be systematically unusual.) (Can accept if multiple surveys with a 10 percent response rate all give similar results using different sampling techniques.)