19 Summary Judgment Rules for Experiments
Research Design and Sampling Method
The primary purpose of an experiment is to test for causality, not to describe a population. To determine causality, you need to be able to compare the difference between two groups—one treated and one controlled– or the change in one person from time A to time B.
Research Design
If you are looking between groups: look for whether the control and the experiment group were identical before the experimental treatment started. If there are no systematic differences between the control group and the experimental group, then you can trust that differences detected are due to the treatment. For between-group designs, researchers equalize the experimental and control groups by randomly assigning subjects to the groups. Look for whether the subjects were randomly assigned to groups.
If differences between the subjects in the control and experimental groups could also produce the same impact that the treatment is trying to produce, then the experiment is fatally flawed. DO NOT TRUST!
OR
For within subject designs, all participants are exposed to every treatment (or condition). Each participant becomes his or her own control. The researchers, however, need to check whether the order of the testing has made a difference— that is, whether something (other than the treatment) has caused a change during the experiment.
The reader should check whether the researcher has counterbalanced the treatment order and randomly assigned subjects to the different treatment sequences. IF NOT, DO NOT TRUST.
Treatment
Accept the findings only if and when the treatment could reliably be expected to produce the change for which the experimenter is testing, and when the control would reasonably be expected to not produce this effect. IF NOT, DO NOT TRUST.
Dependent Variable
Accept the findings only if the dependent variable can reasonably be expected to detect impact. IF NOT, DO NOT TRUST.
If all considerations for trust are met, then trust the experimental results.
