10 Judgment Rule 2 for Content Analysis

Judgment Rule: Determine how the sampling procedure changes the sample from the original population and how the changes from population to sample limit what conclusions can be drawn about the population.

Population

The population is the group that the researcher is interested in studying. If the research question is “How violent are video games?” then video games are the study population. If the research question is “Are toy dolls (action figures) for young boys hypersexualized?” then toy action figures for young boys are the population. As with survey methodology, the population of the study includes all those that are the objects of the study and only those.

Figuring out what is a member of the population and what is not a member may not be as easy to break down as it seems. For example, are Batman, Captain America, and the Teenage Mutant Ninja Turtles all examples of action figures for boys? What makes an action figure an action figure and not a doll? Is Queen Elsa from Frozen an “action figure” or a “doll”? Would Queen Elsa be tossed out of the action figure category because she is a girl-doll, and boys do not play with girl-dolls, or would she be tossed out because she is not active enough for “action” figure? If we take the definition of action figure to be “a doll representing a person or fictional character known for vigorous action, such as a soldier or superhero,” are Teenage Mutant Turtles, also known for their association with pizza, active enough to be considered action figures?

In describing a research study, the researcher should provide clear rules that distinguish what is included in the research population and what is not—both to increase clarity (as in the action figure example above) and to reduce an unmanageable question to a more easily researched focus. If the researcher was interested in knowing whether women are hypersexualized in media, then the population is all women, in all media, in all countries, in all time periods and all languages (an overwhelming task). In most cases, the researcher will narrow the study population to a much smaller group—for example, female characters in the top ten blockbuster movies in this century.

In each of the three examples (see Example 10.1), the researcher is narrowing the research population—Disney cartoons (Study 1), prime time dramas (Study 2), magazine ads (Study 3)—down from the theoretically interesting question “Are women hypersexualized in the media?” to a more easily researched question. Since the researcher has narrowed the research population, neither the researcher or the reader (you) can legitimately extend the findings beyond the research population studied. If the researcher did all three studies and all showed that women were hypersexualized, then you still could not answer the theoretically interesting question of whether all media hypersexualize women, but you could have a lot more confidence that large segments of the commercialized mass-produced media do.

Let’s go through one, more complex, example. In the study below, the researcher is interested in issues of race, specifically how many video games allow players to choose the race of their avatar. What the researcher is examining is whether video game players can construct an avatar with African features (head and hair). The researcher is also going to insert these findings into a complex argument based on past research. Research has shown that having a same-sex character in a movie or television show is important to viewers. Films generally have a range of characters, any one of which the viewer can identify with. Video games are different. Most research suggests that game players exclusively identify with their game avatar. A female video game player, for example, will identify with her avatar, even if the avatar is male. If games systematically exclude some characteristic important for self-identification (such as race), then readers would conclusively know that video games were not, as a group, providing adequate positive self-identification for an important group. The overall argument is based on combining the findings from past studies with the specific findings from this study. However, you, the reader, are going to specifically be looking at whether the findings of this particular study are based on sound methods.

In terms of judging soundness, you need to ask specific questions about population. First, what is the population? (Answer: Avatars of role-playing video games.) Next, since the researcher is not going to look at all video games with avatars, how did the researcher select the specific games to look at? To answer these questions, you need to look at how the researcher described the method used to locate the games he studied (see Box 10.2).

Did the researcher use a reasonable method to a) develop the population and b) draw the sample from the original population of “role-playing avatar games?” I would say yes. He did a good job of finding role-playing video games for both online and offline games. There are some holes with the population (games that were no longer in operation, games that had poor graphics), so the research sample did not have a perfect population of all games released during his study period. However, we can be fairly confident that looking at five websites whose purpose is to gather and publish reviews of games will develop a list of most of the widely played and widely known role-playing games. All in all, the research did a great job of finding the population, with the caveat that he did introduce distortion by only using Windows-compatible games.^[1] Since he did not give any information on whether Windows-compatible games are different from other computer games, this is potentially a serious distortion of the overall population of games and the reader needs to keep this distortion in mind when making any claims about games overall.

Readers should always check whether the sample list that the researcher used actually reflects the population. Errors in this step are not uncommon. Sparkman’s (1996) study of 163 U.S.-based advertising content analyses, for example, found that 97 percent of the articles claiming to represent national print ads were in fact samples that also included regional and local ads. Since researchers have also found that regional and local ads systematically differ from national ads on important characteristics of the ad (type of product, localization information, etc.), using generalizations from regional and local ads to make inferences about the character of national ads can be seriously misleading.^[2]

Unfortunately, while researchers are expected to describe how they collected their samples, the conventions of research article writing do not force researchers to describe in detail how their sample doesn’t resemble their research population, which forces the burden on the reader to think through how each step of collecting a sample introduces blind spots that ignore distinct subgroups in the overall population.

Sampling

The same types of sampling—census, random sampling, cluster sampling—used in survey analysis are also used for content analysis. The reader needs to check that the researchers gave each and every member of the population an equal chance to be selected, and, if not, consider how the systematic exclusion misses data that might be vital to fully understanding the research population. Each of the different types of sampling have their own limitations, discussed in the survey chapter, and the same limitations apply when used in selecting a sample for content analysis. However, content analysis has a few unique elements that are important.

Content analysis researchers are generally more likely to use archives, places where content has been stored (see Example 10.3 for a sample listing of available archives).Kimberly A. Neuendorf, author of The Content Analysis Guidebook, includes the name, the website, and a short description of various online archives in the section on Message Units and Sampling: Archives. These archives are generally only as good as the procedures used to collect, index, and store the archived information. The first time you (and any other reader) see an archive mentioned, you should look up and understand how that archive was developed. The degree to which you can generalize the findings from using an archive is limited to how systematically that archive gathered its material. Vanderbilt television news archives and LEXUS-NEXUS, a news archive, are regularly used, and their limitations are well known. (When developing your basic research reading skills, you should read up on these archives the first time that you see them, but you will find many of these data sets, including LEXUS-NEXUS, are used time and time again.)

At the low end of generalizability, you will find personal collections such as the Gish Film Theater Collection at Bowling Green State University, which has donations of private papers from several Hollywood actors, including collections from two major silent film stars, sisters Dorothy and Lillian Gish. Private collections of papers are most likely to be collections of documents that the donors happened to keep (including correspondences). Embarrassing papers or the papers that the collectors considered “trivial” are likely to have been thrown out, which means that the archive likely presents a far more positive view of the object of the collection than would be presented using a different set of data.

Second, unlike surveys, many content analysis samples are selected from a population of the most popular examples of a media—the best-selling songs for various musical genres (often data gathered by Billboard), the most popular movies in terms of box office success (usually using the theater receipts from Box Office Mojo), the top prime time crime dramas on television (data usually derived from the Nielson ratings). In this case, the sampling is—obviously—representative of popular choices, but not representative of the total of what shows or songs could be watched or heard.

The reader, however, needs to keep in mind that audiences can self-select in very different patterns that could be critical for some research purposes. Would, for example, people with a low body image be likely to select a different media pattern than people with a normal or high body image? Are people who are troubled (in a variety of ways) systematically likely to view violent pornography or specific types of horror shows differently than people who are not troubled in the specific way under study? If so, then conclusions about the content of popular media selections would not necessarily apply to those specific groups of people, because these groups might systematically seek out a less popular set of viewing or reading material, or they could select just some specific types of content within the universe studied. Again, the same basic premise that was discussed for reading survey research also applies to reading content analysis research. The reader needs to carefully consider what exact population the sample is drawn from and not extend the findings beyond this group and the content studied, but also—if the reader is interested in applying the findings to the viewing or listening habits of a specific population of humans, they need to consider whether the population they are interested in has the same reading or viewing habits as those people who watch, listen to, or read the artifacts studied in the content analysis.

To illustrate, let’s say that you were developing a set of recommendations for parents of high school children with attention deficit disorder on teen use of social media and smartphones. It turns out that teens with ADHD are not affected in the same way as non-ADHD or depressed teens, so in order to make recommendations about what kinds of use parents of children with these specific needs need to limit, the studies used to develop recommendations should be specifically studying smartphone use by teens with ADHD. The same reasoning holds for specific kinds of content. Horror content is, by definition, different from sitcom content. And using a more general sample such as “prime time shows’” depiction of on-screen violence might not give an accurate reflection of how much violent content would be seen by a group that is specifically looking for horror in the shows they watch. So, again, a careful reader looking for the answer to the question “How much violence is my audience of interest (for example, depressed teens) exposed to?” would need to carefully delineate just how much they can abstract information from a content analysis of “prime time shows,” given that the audience of depressed teens can systematically select for more horror (or less horror) in the shows they watch. That is, when looking through research papers to answer a specific question about a specific audience, the reader needs to keep in mind whether their population of interest is the same as the audience who is viewing the content studied in the content analysis study.