13 Summary of Judgment Rules for Content Analysis

Question

What Is the Research Question?

You need to rewrite the title, or judge from the abstract, or read the article to determine the research question. Once you have the question, you can determine if content analysis is the correct method to use.

If yes, then fine, you can go on to examine more fine-grained aspects of the methodology.

If no, then the details of the method do not matter, because the researcher used the incorrect method to answer their question.

Sample

The Content Analysis Can Only Be Generalized Back to the Research Population Sampled

1. You need to check each time the researchers narrowed the sample. How was the sample selected? How good was the sampling frame? What subgroups would be left out? Would this make a difference?
2. Did the researcher check for external validity? (Did the survey researcher check their sample against the general population?)
   1. If there are no major flaws with selecting the sample, then you can trust that the results for clearly coded variables can be generalized back to the research population.
   2. If the survey sample is not representative of the population, then you have a range of answers.
      1. The results can be generalized back to the population, with the exception of specific groups that were not sampled. (In other words, the findings probably reflect the population that was actually sampled, and you have a good idea of how these specific sample is different from the general population.)
      2. The results should only be extended to the specified sample. The findings are probably pretty good for the sample, but there are significant unknown differences between sample and the population.
      3. The sampling procedure was so flawed that the findings might actually be misleading. Do not trust!

Coding Categories

1. Accept the findings when the coding categories are clear, easy to code, and not leading or biased.
2. Findings (or conclusions) that are based on flawed codes should be thrown out.
3. You can also use the codes to judge a researcher’s competence. If there is a consistent pattern of poorly worded or biased codes, you should assume that the researcher was equally sloppy (or biased) elsewhere in the study.

Intercoder Reliability

1. Accept only the variables which have an intercoder reliability in the 75 to 80 percent range (at least). For codes that are extremely straightforward, you should be concerned if the agreement between coders falls below 90 percent.