15 Judgment Rule 1 for Experimental Analysis
Judgment Rule: Experiments should be used to determine causality.
Key Takeaways
Using Experiments to Determine Causality
Once again, the first question to ask when looking a research paper is: What question is the researcher trying to answer, and what method is appropriate to answer this question?
If the researcher is interested in finding out the characteristics of a population—for example, how many people own and use smart phones—then a survey is the best approach. If the researcher is interested in finding out if using smartphones destroys people’s ability to cross the street safely, then the researcher is asking a causal question: “Do smart phones decrease pedestrian safety?”
Scientists use the experimental method to determine if A causes B. The core of the experimental method is to control all possible explanations of why the experimental group changes. If the scientist can create a situation where just one thing is changed (A), and that one change also changes something else (produces B), then we can fairly safely say that A caused B.
Each question in Example Box 15.1 is asking a causality question, which you can also think of as a question about sequence. In an experiment, as opposed to a survey, you know what happened first and what happened second. Pretty obviously, if watching television commercials for high-density, high-calorie food causes preteens to prefer high-density, high-calorie food, then the commercials should come first, and the increase in wanting high-density, high-calorie food second.
Example 15.1
Research questions that an experiment can answer
Does watching television commercials for high-density, high-calorie food increase preferences for high-density, high-calorie food in preteens?
Does watching news online decrease comprehension of issue-based (thematic) news stories?
Does using highly sexualized female avatars decrease playing competence in first-person shooting games?
Does using Facebook increase motivated performance on exams?
Does using Twitter increase the subject’s ability to think logically?
Let’s go into the difference between survey and experiment in a bit more detail.
Let’s say that a researcher surveyed a group of teenagers and college students about their video-gaming habits. The researcher asked students what games they play, what avatars they used, what those avatars looked like, and what their average scores were (or the highest levels they reached) on each of their games. The researcher then compared the average scores of the group of players who used highly sexualized avatars against the group of players who did not use highly sexualized avatars and found out that those people who used highly sexualized avatars did not score as highly (or reach as high levels) as those without sexualized avatars.
What has this researcher found out from this survey? Well, as it turns out, the researcher now knows exactly what was said above—that people who used highly sexualized avatars did not score as highly (or reach as high levels) as those without sexualized avatars. In other words, the researcher knows that sexualized avatars and low scores co-vary. (The term “co-vary” essentially means that when one thing changes, another thing changes in a systematic way.) What the researcher doesn’t know is what caused the change. Did the people who used the highly sexualized avatars do less well because they saw themselves as less capable, or did more skilled, more dedicated players choose less sexualized avatars because they were more interested in the gaming (the shoots, the kills, achieving levels) than in the presentation of the avatar? What came first? What influenced what?
In a well-designed experiment, the researchers know that the treatment alone caused the change. That is, the scientists have designed their experiment in such a way that they can rule out all other potential causes. Let’s take the simple research question: Does listening to rap music in the car slow braking time? To find out the answer, researchers gathered a bunch of students who were practicing for their first driving license, gave them driving tests in a virtual driving simulator, and found out that the students who listened to rap music improved their braking time.
To check whether the experimental treatment (in this case, rap music) caused the change, researchers controlled the experiment. One straightforward way to control whether the treatment or something else causes a change is to give the experimental treatment to half of the research subjects (the experimental group), and to not give the treatment to the other half (the control group) (See Figure 15.1). If the two groups start out as similar and both go through the same processes during the experiment, then the differences between the experimental condition and the control condition will be due to the treatment, and only the treatment.
Even before an experiment is run, the researchers should be able to describe what results they could get and what each of those results would mean in terms of answering their original question: Does listening to rap music increase braking time?
Potential result 1: If both the experimental and control groups improved braking speed, then “something else” caused the decrease in braking time (not the rap music). (The students probably learned to be better drivers, but this is a guess because the researchers didn’t control for learning how to drive.)
Potential result 2: If neither group improved, then the music had no effect.
Potential result 3: If the control group learned how to brake more effectively than the experimental group, then the treatment harmed the driver’s ability to brake, even if the experimental group’s braking scores improved over the course of the experiment.
Potential result 4: If the control group braked less effectively than the experimental group, then the rap music improved the driver’s ability to brake, even if the control group learned how to brake more effectively during the experiment.
Summary: Controlled experiments are, by definition, set up so that researchers can manipulate “A” to see if “B” changed. How the researchers run the experiment is called subject design, and is the next basic judgment you will need to make. Is the experiment designed appropriately?