3 Judgment Rule 2 for Surveys

Judgment Rule: Determine how the sample is different from the population.

The second judgment rule is all about the difference between the population that the research is trying to find out about and the (usually) much smaller sampling of people the survey researcher has interviewed. When you are at a high-end ice cream shop and you get a sample of a particular flavor, in general, you assume that the small taste of blueberry walnut ice cream is going to be very much like (that is, the sample will represent) the flavor you would get from anywhere in the bucket. Your assumption is probably correct, but if the ice cream hasn’t been mixed thoroughly, your sample might have a clump of nuts or no nuts at all, leading to an impression that isn’t like the full blueberry-walnut ice cream experience. Unfortunately, in sampling a population, the nuts are often incompletely mixed, and so readers have to make special efforts to understand just how much trust they can have that the sample is just like the population.

Analyzing the research population: The reader’s first task is to identify what population the researcher is interested in studying.

If the researcher is interested in the question, “What percentage of gamers are geeks?” then the population is gamers. If the question is, “What percentage of adults use Twitter?” then the population is adults who may or may not use Twitter (in other words, adults). If the question is, “What percentage of U.S. workers use computers as a part of their daily jobs?” then U.S. workers would be the population.

Of course, defining the population means providing clear definitions of who is a member of the population and who isn’t. For example, does “workers” mean people who labor, or just the people who get paid for their labor? If the researchers are only looking at paid labor, they are leaving out all people who stay at home and do housework/childcare/eldercare and are not compensated for their jobs. That is fine if what the researchers are interested is paid labor, but not if they are interested in all labor. Similarly, is the population that the researchers are actually interested in people who labor within the geographical United States (which would include undocumented immigrants), but not American citizens who are working abroad (which would include a decent chunk of the military)? In other words, who exactly are “U.S. workers”? Essentially, when researchers decide who legitimately is a part of the population, they are also deciding who is important and who is not. By revealed preference, those groups that are important will be included; those groups that are not important should be left out deliberately, not because the researcher is too lazy to be clear about the definition of labor they are using.

The issue of generalization: Why sample a few individuals rather than interviewing everyone?

Usually, different populations are large enough that interviewing every person in a group is too expensive and too time-consuming to do, particularly when there is a cheaper and more effective alternative available—selecting a few people to speak for the whole. If done correctly, the sample will accurately reflect the range and types of responses in the larger population. On the other hand, sampling also is one of two major ways errors can creep into surveys.

To illustrate, let’s go through an example where we know the actual characteristics of the population. The research question is: What uses and types of gratification do Twitter users get from using Twitter? In Table 3.1 (below), we have the results for the full population.

From Table 3.1, we can see that Twitter users tweet to communicate with friends, to get political news, to follow trends for work, and to listen to celebrities. But if you look closely, older users are distinctly different from younger users, and male users are distinctly different from females. Older people are far more likely to get political news or follow trends for work, while younger people are more likely to use Twitter to talk to friends and follow celebrities. Males are more likely to get political news and far less likely to follow celebrity tweets than females.

If the researcher has a representative sample, then (most likely) the sample will show the same, or roughly similar, percentages as shown in Table 3.1 above for the full population. (See Definition Box 3.2 for definition of representative sample.)

On the other hand, a sampling method that doesn’t allow each and every person to have an equal chance of being selected is likely to produce a sample that does not reflect the entire population. Let say that a researcher—a female researcher—used the following method to get her sample. She tweeted to her friends (mostly female and under the age of 30); she put an announcement on her Facebook page (viewed mostly by females under the age of 30); she then asked these people to retweet and repost. By and large, her friends will also be communicating with females who are predominantly under the age of 30. According to Table 3.1, females under age 30 are much more likely to use Twitter to talk to each other and to follow celebrities, so a sample that is accurate for the female-under-30 group is likely to widely overemphasize how much the population as a whole uses Twitter to talk to friends and to follow celebrities, simply because the sample is much more likely to sample young females than males under age 30, and males and females over age 30.

Doing sampling. It is relatively easy to name a research population (geeks, Twitterers, workers), but finding the population can be more difficult. Naming the research population simply means identifying the group of people that the researcher wants to study – whether those people are practicing doctors, farmers, Shetland Sheepdog (Sheltie) owners, breast cancer survivors, or people who have been exposed to mercury.

Finding the survey population depends on being able to locate and identify each person who is in the actual study population and to keep out each person who isn’t. Depending on how you select the sample, you could either get a sample that looks a lot like the population, or a flawed sample that is systematically different in important ways. In terms of the original description of a survey as “delineating the form and the outline” of the population, the flawed survey would miss some of the population’s distinctive features. It would be the same as having a topological map that did not include some of the hills or the valleys in the real region.

For the respondents to reflect the population, the sample respondents need to resemble the range of groups in the population. The best way to get a representative sample is to allow each member of the population to have an equal chance of being selected. The actual mathematics of why random sampling works are fairly complex, but fortunately selecting a random sample is relatively easy. Since humans tend to have built-in biases or systematic ways of simplifying selections, the best ways to select an unbiased sample are (a) to have a computer generate a list of random numbers and use those numbers to select the sample from an already compiled list of the population, or (b) to develop a sampling method that gives everyone in the population an equal chance of being selected.

The sampling frame method: The sampling frame is a list of all people who are known to be in the population. If a researcher was interested in doing a survey of practicing doctors, they could buy a list of all physicians who are licensed to practice. (In fact, there are businesses whose job it is to constantly update and monitor this list.^[1]) Finding Sheltie owners is more problematic. The American Kennel Association keeps a list of all purebred Shetland Sheepdogs registered with the organization, but this list wouldn’t include either mixed breed dog owners or purebred Sheltie owners who haven’t bothered to register. In addition, it is somewhat unlikely that the American Kennel Association constantly updates their list, so the lists they have are likely to include an unknown number of animal owners who have lost or given up their pets.

In the case of the population of “people who have mercury in their blood,” not only is there no available list, but the people in the population might not even know that they are in the population. (That is, they might not know that they have detectable levels of mercury in their bloodstream.)

Generating randomness with screening. When there is no list, researchers cannot use a sampling frame to generate a sample. Instead, researchers need to develop another way to determine the population. Often, the method they use is to generate a large list that is random, but will have both people who are in the survey population and people who are not in the survey population. The researchers will talk to all people on the generated list and remove the people who aren’t actually a part of the research population, commonly by starting the survey with a screening question that asks if the person answering the question is actually a member of the population. The paragraphs below give an example of how survey method and a screening question work.

In a study of cellphone and landline users, PEWS researchers used “a combination of landline and cellular random digit dial (RDD) samples […] to represent all adults in the United States who have access to either a landline or cellular telephone. Both samples were provided by Survey Sampling International. […] Numbers for the landline sample were drawn with equal probabilities from active blocks (area code + exchange + two-digit block number) that contained three or more residential directory listings. The cellular sample was not list-assisted, but was drawn through a systematic sampling from dedicated wireless 100-blocks and shared service 100-blocks with no directory-listed landline numbers.”

Aaron Smith, Pew Research Center^[2]

In other words, landline phone numbers have a pattern—some blocks of numbers are all businesses, some are residential and businesses combined, and some are all residential. Since businesses are not part of the desired population, then the blocks of numbers that were all businesses were deleted. The researchers then generated a random sample from all numbers assigned to the mixed business-residential block and then the residential blocks. Obviously, some of the numbers that the researchers generated from the mixed business-residential block would be businesses, and so the researchers would need to drop these phone numbers from the sample. Since the researchers don’t actually know which numbers are for businesses and which are for homes, they will need to call everyone in the sample and ask if the number that they called is for a business or a home. Only those people who said the phones were their personal phone numbers continued on to answer the rest of the study. The people who said the number was for a business would be thanked (politeness counts), but not asked any survey questions. This process of making sure that the person contacted is truly a member of the survey population is called screening.

Checking randomness: Generally, opinions are linked to life experiences. In this society, as in many societies, our experiences systematically differ by major demographic variables—age, gender, sexual preference, occupation, race, family status (whether we have kids or not). In addition, many of our basic and derived values and attitudes differ by political orientation: Republicans and Democrats differ on lifestyle, energy consumption, and a host of issues outside of pure electoral politics.

Since we know that demographics systematically affect attitudes, one easy method to check the randomness of the sample is to compare the sample with known characteristics of the population the sample was drawn from.

Race, gender, and age are fairly common measures to use because the population distribution of each of these is often well known. In Example 3.1, for example, the researchers were studying adults (above 18 years old) in the United States.^[3] We know, from both the U.S. census and other studies, the ranges in age, gender, race, and income of the U.S. population, and so we can compare known population characteristics with the sample. In this case, the sample clearly oversampled men and under-sampled females, African Americans and people in the highest income bracket.^[4] Comparing what you could expect from the average population figures and what the researchers reported, the researchers did not capture the demographics of the actual population, a point the researchers implicitly acknowledged by characterizing the sample as a convenience sample.

Authors of research studies are obligated by custom to include a description of how they sampled their population. While I’m sure that there is at least one case of a study slipping through the review process that doesn’t include the sampling technique, in forty years of reading social science research, I have not run across an example yet – and I’m not expecting one, either. Commonly, however, researchers assume that readers know both the strengths and the limitations of different major types of sampling. The next section will review basic types of survey sampling.

Different Types of Survey Sampling

Simple random sampling: In this type of sampling, every person or unit in the sample has an equal chance of being selected, either by a list of random numbers or by electronically generated random numbers. For example, let’s say that the University of Illinois housing office wants to find out what freshmen in the dorms think are the top three problems affecting dorm life.^[5] The university housing office has a list of all people living in the dorms, so it would make sense for them to use the sampling frame method. In order to get a random sample, the researchers would give each person on that list a number (from 1 to the last person in the list). The researchers would then decide how many people they want to interview and generate a list of that many random numbers. The list would tell the researchers which people in the population list to interview. (That is, if the list of random numbers was 7, 23, 47, and 56, the researchers would skip the people in the population list who were assigned the numbers 1, 2, 3, 4, 5, and 6, interview the person in the population list who was assigned the number 7, skip the people who were assigned 8 through 22, interview the 23rd person, and so on.)

The quality of a simple random sampling is primarily determined by the quality of the sampling list and the sampling frame. In the example given above, the housing list is likely to be very accurate. Since the university housing office is getting a great deal of money from each student, they are likely to put a lot of effort into making sure that the list is both complete and current. Therefore, a random sample taken from the list is likely to be very good. One major flaw, however, is that the researchers might want to know more detail about some relatively small subpopulation, and a simple random sample might not capture enough people to draw accurate conclusions. In a population of 1001, with 800 reds, 200 yellow, and 1 blue, drawing a sample of 100 from that population would give some 80 reds (plus or minus a little random error), 20 yellows and 1 time out of 100 a blue. Nine-nine times out of 100, however, just by random chance, there would be no blue in the sample, and therefore all of the blue’s opinions would be missed. This might not matter that much if what the housing authority was looking at was color preference for rooms, but if the population was 800 cis women, 200 cis men, and 1 trans woman, the housing authority needs to know the preferences of that one transgender student to make good (that is, inclusive) policy decisions.

Stratified sampling: Stratified sampling is a technique that researchers use when it is important to sample members of subgroups within a population. It’s commonly used during elections when the pollsters want to make sure to include minority populations (such as racial or religious populations) in the same proportions as those groups occur in the actual population. Let’s say that the pollsters wanted to check religious reaction to some proposed legislation. The US population breaks down into the following proportions of known religious preferences: Christian 70.6%, Jewish 1.9%, Buddhist .7%, Muslim .9%, Hindu .7%, Unitarians and liberal faiths 1%, New Age Spirituality 0.4%, and Native American 0.3%.^[6] To take a stratified sample, the researchers would randomly sample the same proportions of respondents as in the general population, or—out of a thousand-person sample—706 Christians, 19 Jews, 7 Buddhists, 9 Muslims, 7 Hindus, 1 liberals/Unitarians, 4 New Age, and 3 Native Americans (religious, not necessarily ethnic, representatives), which in practice means that once they filled their quota for Christians, they would keep sampling for the other categories until they found the prescribed number of Muslims, Hindus, Unitarians, New Age and Native Americans. Again, this is an acceptable method of random sampling, although it means that the researchers will have to adjust statistically for this oversampling. (The statistical adjustments – while interesting — are outside the scope of this book.)

Cluster sampling: This type of sampling is useful when the research population has naturally occurring clusters, such as schools or families. The clusters should be relatively homogenous within each cluster (for example, all grade schools are divided up into grades or levels, families into adults and children, Dungeons and Dragons players into cleric, druid, fighter, and so on). In the cluster sampling method, the procedure is to randomly select among clusters and then exhaustively sample within the cluster. Examining research that uses this technique, you should first look at whether you have naturally occurring homogeneous categories. For example, you would have to determine if all the “groups” in a Dungeons and Dragons sampling were Dungeons and Dragons players, or whether some of the “groups” were actually—for example—teams of paintball players who were wandering through but not actually playing the D&D game.). Once the researcher selected a D&D group, however, he or she should interview all people in the group. The same goes for families; the researcher selects a random sample of families (this is the random selection among clusters), but within each family selected, the researcher should survey each member of that family. Again, this type of sampling is considered to produce representative random samples.

Convenience sampling: Convenience sampling literally means sampling whoever is convenient, including volunteer sampling. This kind of sampling is easy, but seriously biased. In some cases, when obtaining the population is extremely difficult—interviewing lesbian couples in a strongly anti-gay setting, for example—researchers will use convenience samples, but readers generally will have strong reservations about the generalizability of that survey. (The safest assumption to make is that the sample represents the opinions of some people in the population, but you cannot tell the range or distribution of the population.)

Snowball sampling: Snowball sampling is a technique used to reach difficult or very small subgroups of a sample. For example, snowball sampling is useful for reaching people who either:

1. have a rare characteristic (such as an infrequent disease or some other trait of interest), or
2. have reason to hide—either from social shame (such as pedophiles) or legal reasons (such as people who are fencing illegal materials).

Once the researchers gain the trust of one member of the research population, they ask that people to refer them to others in the group. Using the logic that “birds of a feather really do flock together,” the researchers use their interviewers’ contacts to reach other members of the population. By definition, then, a snowball sample is not a random sample. The most that one can take away from this sampling technique is that some people feel, act, and believe what the sample believes, but one cannot generalize the data to the general target population.