11 Judgment Rule 3 for Content Analysis

Judgment Rule: Coding categories should precisely measure the variable the researcher is studying.

Coding, in research terms, is where the rubber meets the road. If the theoretical framework is the map (the framework you use to tell you where you are going), and the road is the reality you are moving along, then coding rules are the wheels—the portion of the car that makes contact with what is “out there” and propels the research project (the car body) forward.

In all content analysis studies, researchers ask questions about the text. To answer these questions, the researchers need to describe precisely how they are going to measure each variable or characteristic of interest. To answer the question—Do female characters in strategy video games have unrealistic bodies—overdeveloped breasts and overly narrow waists?—researchers must determine precisely what is an “overdeveloped breast” and an “unrealistic narrow waist,” and describe each in enough detail that trained coders “from varied backgrounds and orientations will generally agree in its application.”^[1] Coders follow researchers’ instructions on how to code features of interest in the sample texts. (The set of instructions are called coding rules. Coding is the act of applying the rule to analyze a bit of text, video, or audio material.)

For example, how would a researcher develop coding rules to study Figures 11.1A-C (See Example 11.1B)? First, the researcher would need to have a research question such as, “Are women in magazine pictures portrayed as professionals less often than men?” Second, the researcher would need to develop a set of rules for all the variables needed to answer the research question. In the example below, the two major characteristics that the coder would need to record are “gender” and “ professional occupation.” Coding rules that could capture these two characteristics are shown in Example 11.1A below.

The next step is finding material to code.  An illustrative sample of pictures is shown in Example 11.1B.

Example 11.1B

The following figures are sample material to be coded using the instructions in Example 11.1A.

Once the researcher has developed the coding instructions and found the sample material, coders enter their analysis of the pictures into a coding answer form. (See Example 11.1C for an example of a blank sheet and Example 11.1D for an example of a coded answer form using Figures 11.1A through 11.C from Example 11.B.)

For the pictures in Figures 11.1A-C, their coded data (see Example 11.1D) shows no men, and seven women. More women are shown at leisure (5 women out of 7 total), but more pictures show women working (2 pictures out of 3).

When all of the material is coded, then the researchers will tabulate the results, showing summaries that illustrate their findings (Table 11.3). Now the research can say with confidence that the women dominated the coverage (100%), although the majority of the characters were at play (71%) rather than at work (29%).

Obviously, three pictures are too small a sample to draw any firm conclusions. If, however, after going over hundreds of pictures, the researchers found that 80 percent of pictures showed men at work and 90 percent showed women at work, then—clearly—women in magazines are shown as workers more often than men, albeit by a narrow margin.

Coding for “gender” is fairly straightforward. Producers of mainstream media are generally simplistic in how they assign gender, and most audience members within the culture are fairly good at figuring out the codes that the producers use to signal gender.

Notice, however, that even with characteristics that are fairly easy to code, coders might have difficulty in some situations. The very old and the very young (Figure 11.2) are often not visibly gendered, unless the message producer uses conventional signals (dressing a baby in blue, tying a head scarf on a female) to signify gender. In addition, the producer could deliberately work to blur gender lines (see Figure 12.1 in the next chapter).

By definition, an adequate coding rule should be precise enough that two coders would code the same object the same way most of the time.

Coding instructions range from unelaborated instructions, such as “Record hair color,” to highly complex instructions. The degree of instructions that coders will need usually depends on how socially ambiguous the coded material is. There isn’t a lot of ambiguity about whether a movie scene has some food in it or not. There is a great deal of ambiguity about many socially constructed concepts such as “tasty” or “attractive.”

In the avatar study discussed earlier, the population is, again, role-playing games. The specific characteristic of interest is the player’s ability to select for race when developing their avatar. The research question in this example implies that the coding scheme must include a way for the coder to determine whether or not a gamer has the capacity to choose the racial characteristics of the character when setting up the game conditions.

The coding scheme that the researcher used in the avatar study had three visible markers of race: skin color, hair style and color, and facial features (offered in games that allowed for greater detail). They measured these three markers using the following measures.

Skin tone: The research modified an older scale (the von Luschan scale) used to measure differences in human skin tone. The von Luschan scale distinguishes between 36 different levels of skin tone, ranging from very light to very dark. (The scale was modified to separate tanned from not-tanned skin.) To measure tone, the researcher created the darkest skin color possible to create a character and, using the scale, measured the how much the darkest character possible deviated from white.

Hair color and style: The researcher looked for whether a game allowed players to create a character with an African American hair style, defined as “any hairstyle typically associated with coarse hair (e.g., Afro, dreadlocks, or any hairstyle capable of being worn by someone with coarse hair (no straight hair or loose curls).”

Facial features: The last coding category the researchers used, “African facial features,” was based on examining the “size and shape of the nose and lips.”

In other words, the researcher was far more specific about measuring race than simply instructing a coder to check for what race the gamer could construct.

Returning to the earlier question of: Do female characters in strategy video games have unrealistic bodies—overdeveloped breasts and overly narrow waists? What would be an acceptable coding scheme for measuring breast to waist ratios? Look at the difference between Angelia Jolie’s Lara Croft and the character in the original Tomb Raider video game. Fairly clearly, using my subjective judgment, Angelina Jolie has a pretty good body shape; the original avatar’s body, however, has considerably more breast and far less waist that Jolie. Whatever coding scheme is developed needs to capture that difference clearly.

The researcher could allow the coders to use their own subjective judgment, but we know from earlier research studies that grappled with this question that the “use your own intuition” coding instruction will not work in this case:  some coders would code both bodies as unrealistic, some coders would code both as realistic, and some would code Jolie as realistic and the Lara Croft avatar as unrealistic. In other words, the coders will not agree, and thus the coding rule is not precise enough that two coders would code the same object the same way most of the time. However, coders will agree more often using a V body measurement. To measure V, coders should draw an arrow starting at the waist and extend out along the chest wall. The greater the angle of the V, the greater the breast-to-waist ratio is. If the researcher set a rule that “any breast measurement greater than a 50-degree arc is unrealistic,” the coders have a precise way to distinguish Jolie (40 degrees) from the Lara Croft avatar (50 degrees).

The task for the reader is to judge whether the coding scheme that the researcher has developed is a reasonable way to determine the specific characteristic of study. (The reader has to be able to answer the question, “If a game allows players to create a character with dark skin, an Afro hairstyle, and African facial features, am I willing to agree that the game will allow gamers to construct an African-race avatar? That is, does dark skin, an Afro, and African facial features adequately indicate a particular race? If the V measurement is greater than 50 degrees, does this indicate “unrealistic body measurement”?)

If the answer is “yes,” then you—the reader—should accept the coding scheme. If no, then you should reject the coding scheme for this variable and all of the findings based on this coding for this variable.

In other words, judging the adequacy of the coding rule means grappling with how the coding scheme itself defines reality and whether you are willing to accept that reality. Does “unrealistic” mean unreachable by any human, or does it mean unrealistic for most bodies? For example, would you consider Dolly Parton’s measurements to be “unrealistic” given that she is a real female who really has a well-endowed figure (and her V measurement is well over 60)? Or would you say that the standard should be the range of measurements for 95 percent of adult women?