17 Judgment Rule 3 for Experimental Analysis
Judgment Rule: Accept the findings only if and when the treatment could reliably be expected to produce the change for which the experimenter is testing, and when the control would reasonably be expected to not produce this effect.
Key Takeaways
The treatment is the very specific way that the researcher manipulates the experimental subjects (see Example 17.1). The research question sets up what potential “cause” is being examined. Usually the cause is a category—high-fat, high-calorie food (Question 1); online news (Question 2); highly sexualized female avatars (Question 3). But researchers have to test something specific. In other words, they cannot just test a generic high-fat, high-calorie food; they must use a lasagna, or a burger, or a cake. The experimental treatment is the specific food, advertisement, movie, picture, news story, or avatar used. It is part of the researcher’s job to write a good enough description of the treatment that you, the reader, can visualize exactly how the researchers manipulated their subjects, and it is part of the reader’s job to think about—and judge—how well the specific treatment represents the general category that the researcher is looking at. Doom or Call of Duty would be appropriate treatment games for experiments that were looking at the impact of violent shooter games. Tetris, a tile-matching puzzle game, would not.
The reader, then, needs to first look at the research question to figure out what the researcher is testing. In the first research question in Example 17.1, the researchers are looking at the impact of high-density, high-calorie food. Therefore, any commercial for a specific food that had the qualities of high-density, high-calorie food would qualify as a reasonable experimental treatment—a Domino’s thin crust, 14-inch pizza with extra cheese (250 calories a slice, or 2000 for an entire pizza);[1] a Hardees half-pound Texas BBQ Thickburger (1030 calories)[2]; or a Culver’s chocolate, two-scoop waffle cone (657 calories).[3]
Example 17.1
Research Questions an Experiment can Answer
Research question: Does watching television commercials for high-density, high-calorie food increase preferences for high-density, high-calorie food in preteens?
Experimental treatment: Television commercials that feature high-density, high-calorie food. (Any of the following commercials could be used as examples of the kind of food needed for the experimental manipulation: a Domino’s pizza, a Hardees half-pound Texas BBQ Thickburger, or a Culver’s double waffle cone.)
Research question: Does watching news online decrease comprehension of issue-based (thematic) news stories?
Experimental treatment: In this experiment, the researchers would need to test reading comprehension of the same story online (experimental group) and in print (control group). Since the research question specifically called for the more complex stories that deal with the causes and the implications of social issues (thematic stories), the researchers would also need to select some longer, in-depth news stories for both groups to read, such as: What are the root causes of the Gaza occupation? What are the implications of keeping live samples of the smallpox vaccine in research facilities?
Research question: Does using highly sexualized female avatars decrease playing competence in first-person shooting games?
Experimental treatment: Having some video players use a highly sexualized avatar (experimental group), and some use a non-sexualized avatar (control group).
Research question: Does using Twitter increase subject’s ability to think logically?
Experimental treatment: Sending tweets in Twitter format (experimental group) versus having unlimited length in text message (control group).
The experimental treatment (for the experimental group) also needs to be paired with a control. The control mimics the same characteristics of the treatment without the specific aspect of the treatment that the researcher is interested in studying. The classic example is the sugar pill (placebo). Researchers have found out that just the act of swallowing a pill can improve people’s satisfaction with their health—in part because people believe that medicine will work, even when what they are actually given is a lump of sugar.[4] The control “treatment” runs a group through a similar process as the experimental treatment, without the specific features that the experiment is testing for. Going back to the high-fat, high-density food example, a number of different commercials could be used as controls—commercials for cars, perfume, cranberry, apple, or travel. The researcher should be careful to match as many characteristics of the two sets of commercials as possible—length, production quality, known brands—but the experimental treatment must have the distinguishing research characteristic, and the control cannot have this characteristic.
- Calories are for a 14-inch pizza with extra cheese and thin crust. “Cal-O-Meter,” Domino’s Pizza, last accessed June 2, 2023, https://www.dominos.com/en/pages/content/nutritional/cal-o-meter. ↵
- Calories for a half-pound Thickburger. “New Texas BBQ Thickburger from Hardee’s and Carl’s Jr.,” Grub Grade, archived January 27, 2015, https://web.archive.org/web/20150127080436/http://www.grubgrade.com/2014/07/21/new-texas-bbq-thickburger-from-hardees-and-carls-jr/. ↵
- “Culver’s Chocolate Waffle Cone (2 Scoop),” Fatsecret, last updated July 10, 2022, https://www.fatsecret.com/calories-nutrition/culvers/chocolate-waffle-cone-%282-scoop%29. ↵
- The placebo effect is powerful. When we think that something will work, a drug is, in fact much more likely to be effective. In act some searchers have suggested that a “substantial percent of the effects from antidepressants may be placebo effect.” For a quick discussion of the placebo effect, see Christopher Lane, “Placebos Do Work,” Psychology Today, June 26, 2009, http://www.psychologytoday.com/blog/side-effects/200906/placebos-do-work-lets-consider-why. ↵
