9 Judgment Rule 1 for Content Analysis

Judgment Rule: Content analysis is valid for studying text, not people.

The first question that a reader needs to ask is, “What question is the researcher asking?” The researcher’s question then becomes the standard for judging the rest of the methodological procedure. Fortunately, the question is usually a simple rewording of the article title into question form.

For example, the article “Examining Diversity: A Content Analysis of Cancer Depictions on Prime time Scripted Television” answers the question, “What is the racial and ethnic heritage of television characters with cancer on prime time scripted dramas?”

The reader then will judge whether content analysis is the correct method to use to determine the researcher’s question. Since the researcher is interested in the content within television shows, content analysis is an appropriate method to use. (Reminder note: Content analysis is designed to study artifacts that humans produce, including texts, pictures, and videos.)

In Example Box 9.1, the research team is examining the visuals and lyrics of rap music videos to find out the themes of the rap music (conflict-oriented versus community-oriented) and the features of the performers (Eurocentric versus Afrocentric). Therefore, reasoning backwards, the research question is: “What is the content of popular rap music videos visually and lyrically?” Since the research question is about the content of something that humans have produced, rather than the humans themselves, content analysis is appropriate.

Usually, the researcher is extremely clear about what he or she is studying and how they are studying it.  The tricky part is that sometimes the researcher makes a mistake, and they use the wrong method to answer their research question. If you accept the most fundamental argument about data-driven science, then you—as the reader—are bound to reject the study’s conclusions if the researchers use the wrong method to answer their research question.

One of the most famous examples of a researcher making conclusions that go far beyond what his method can support is Lasswell’s study of military propaganda, one of the classic studies of propaganda.^[1] Lasswell, who studied the content of war material that the military developed for World War I (see Figure 9.1), found that military posters demonized the enemy (describing the enemy as “menaces” and “mad brutes”), appealed to fear (“loose lips sink ships”), and in other ways portrayed “our” side as good and noble and “their” side as animal and evil. Lasswell concluded from his study of the poster’s content that propaganda built a base of support for the war, sapped the enemy’s will to fight, is essential for modern warfare, and acted as a “flame to burn out the cancer of dissent …”

Since Lasswell studied posters, not people, he couldn’t—scientifically—make any claims about the posters’ impact on viewers. That is, he could demonstrate what kinds of claims the posters made, but not how the posters changed viewers.

When the researchers use the wrong method to answer their research questions, then their findings are useless for that research question; when the method they use is irretrievably flawed, then the findings are useless.

One additional note: The reader’s “job” is to carefully distinguish between what researchers found (the methodologically sound findings) and what the researchers suggest the implications of those findings might be (in the conclusion or discussion section of the paper). Readers should accept methodologically sound findings as valid, but not automatically accept the researcher’s discussion of the findings. In the discussion section of papers, researchers commonly suggest causes or implications of the findings. Both causes and implications are extensions beyond the actual findings and need to be studied further.