6 Judgment Rule 4 for Surveys

Judgment Rule: Hesitate to accept surveys with a response rate lower than 60.

Response rate. The response rate is the number of people who successfully fill out and complete a survey questionnaire out of all people invited to respond to the questionnaire. If you sent out a questionnaire to 1000 people and 100 people answered, you would have a response rate of 10 percent. (It does not matter if the population the sample was drawn from was a billion people or a thousand; the response rate would still be 10 percent.)

The response rate is an indicator that the researchers have gotten most of the variety of responses that are in the population. If the response rate is a large number (over 60 percent is the general rule of thumb), most likely the researcher has gotten a representative range of the potential answers. That is, the 40 percent of sample who did not respond are presumably a lot like those who did respond. The lower the response rate, the less confident the reader can be in assuming the responses represent the population. With a lower response rate, there are a lot of people who are not answering, and many of those people could be quite different from those who do answer.

If you don’t know how many people were asked to respond to the survey, you cannot know what percentage of the people who saw the questionnaire chose to not answer it. By definition, then, if you cannot calculate a response rate, all you can say is that some people feel, think, or do something, but you cannot generalize to the study population as a group.

Over the last several decades, the number of people who are willing to answer surveys has gone down, particularly with the use of online survey research (which commonly has a response rate between 10 to 30 percent). As a result, statisticians have put considerable effort into investigating just what percentage of people need to respond to a survey to guarantee a representative sample. The answer is that there is no clear, absolute cutoff point in terms of the number of responses per 100 people contacted. If significant groups refuse to answer the questions, a response rate above 60 might still give a biased result. Any rule that is developed about response rates is a guide rather than an absolute indication of the external validity of a survey. In general, however, a response rate over 60 percent indicates a fairly good representation, while response rates of 20 to 30 percent are far more likely to misrepresent some groups. There is a caveat, however. Researchers and readers are far more likely to start trusting the results when multiple surveys on the same topic yield highly similar results. Twenty surveys with a 10 percent response rate that give very similar answers to the questions are collectively probably fairly accurate, while one survey with a 10 percent response rate could be accurate or could be wildly off—there just isn’t enough information to tell.