Block I Illinois Library Illinois Open Publishing Network

1 Introduction

Science (including its practical older sibling, engineering) has been a driving force for civilizations for centuries—arguably since humans learned to control fire and absolutely since the Age of Enlightenment. Science and engineering have altered virtually all aspects of what we are; what we wear, how we eat, how we protect ourselves, how we get from one place to another, how long we live, and how much pain we live in. Most of these changes are beneficial. Smartphones are great, binging on Netflix (or whatever) is a fine way to spend a lazy night, and having all the knowledge of the internet is fantastic. But the benefits come with risks.

Chart shows steep increase in opioid death rates from 2012 to 2020 primarily from synthetic opioids.
Figure 1.1. Overdose Death Rates Involving Opioids, by Type, United States, 1999-2020. Center for Disease Control and Prevention, 2020. Public domain.

Percocet, a combination oxycodone/paracetamol, is an opioid/non-opioid pain reliever used for moderate to severe pain. [1] When you have severe pain, Percocet is a blessing. It is also dangerous. Percocet can cause fatal complications even under a doctor’s supervision. Taken for pleasure outside of the medical system, Percocet and other opioids kill (42,000 lives in 2016 alone; see Figure 1.1).

In much the same way, my iPhone[2] is a wonderful thing. I use my phone for notes, music, games, light, texting, directions, internet, camera, fitness timers, and—rarely—a phone. Smartphones are so integrated into daily lives that businesses frequently assume that their customers will have one. Essentially, they require that assumed phone to verify purchases (credit card checks), identity (internet security checks), emergency messages (schools and hospitals), or that the cable guy or the plumber is actually on the way.

Nevertheless, just because smartphones are great does not mean that they are always great. Texting while driving increases car accidents. Texting while in lecture decreases grades.  For example, what would you do if research suggests that having the phone out while talking to friends decrease the satisfaction of interacting with friends?

Shalini Misra and her colleagues looked at how mobile phones affect the quality of social interactions in public.[3] Because the researchers were interested in real-world conversations, they asked groups of two who were waiting to order drinks in coffee shops if they would be willing to participate in a research study. If they agreed, they were seated on two chairs with a table between them. Half of the dyads in the study were asked to talk about “their thoughts and feelings” about plastic Christmas trees; the other half were asked to talk about the most meaningful events of the past year. A trained observer sat relatively far away and watched whether or not the participants had phones out during their conversation.

After ten minutes, both participants were asked to rate their satisfaction with the conversation. The mere presence of a phone dragged down the quality of the interaction. When one person in the conversation pulled out a smartphone, the participants felt that they were less connected to their partner and that the conversation was less fulfilling compared to conversations with no phone. The appearance of the phone hurt more for close friends than casual acquaintances. Satisfaction went down whether the participants were discussing plastic trees or the most important event of their lives.

Does this study mean that you should toss your phone out? No, not really. Does the study suggest that you should put the phone away when you are talking with your friends? No. “Should” is a verb signifying a moral directive. If you do not care about how your friends feel, then why should you put your phone away? Do what you want.

If you do care about the quality of your interactions with close friends and you think that the study is sound, then—yes—Put. The. Phone. Away.

Again, you need to put the phone away if you think that the study is sound; that is, if you accept that the researcher appropriately followed the rules of gathering evidence. This book is based on the belief that knowing how to judge the validity of scientific papers and using science as important background for making decisions are really important and useful skills to have. When you understand the rules of science, you have developed the context to decide for yourself when, how, and why to control your media use so that your media does not control you.

What Is Science?

Science is a set of rules on how to find things out.[4] These rules vary with every different scientific procedure, but every method is defined by a specific set of rules. Simplistically put, any scientific method is valid to the degree that the researcher follows those rules and weak to the degree that it does not. Therefore, the first task of a skilled reader is to learn what those rules are.

The rules tell you such things as:

What you can and cannot study using a particular scientific method or instrument. In medicine, doctors do not use a thermometer to measure blood pressure and do not give blood transfusions to cure a cold. Doctors select the tool that is most useful for what they want to measure. The principle is the same for social science, and both researchers and readers should know what each social science method is useful for——what the method can study and what it cannot.

Bananas
Figure 1.2. Fruits. Photo by Rodrigo dos Reis, Unsplash, Unsplash License.

What population the research can, and cannot, make claims about. The purpose of social science is to make claims about some group of people or things (for example, the population of fruits in Figure 1.2). But researchers can only make claims about the population they actually studied. If a physical scientist found out that fruits help people decrease stress, then an intelligent—and stressed—reader might want to start eating more fruits. So far, it is all very clear.

However, if you started looking at exactly what the scientist studied and—in this example—he only looked at bananas, then all you can really take from the study is that eating bananas will reduce stress. The researcher extended his results too far when he said “fruits.” Since the researcher did not look at apples, he (and you) should not assume that apples reduce stress. They may; they may not. You do not know. Of course, the difference between bananas, apples, and even pears is reasonably straightforward and hard to confuse.

Black and white photo of schoolboys wearing a school uniform with white piping around their jackets. All students appear Caucasion and are roughly in their early teens. Most of the boys are in long shorts, a minority are wearing long pants.
Figure 1.3. Schoolboys. Photo by A. H. Poole Studio. Poole Photographic Collection, National Library of Ireland via Flickr. Reproduced courtesy of the National Library of Ireland.

The social science version of this mistake is a bit trickier, particularly with humans’ self-centered tendency to forget that other groups of people are not exactly like them.

For example, let us say that a researcher is looking at whether reading graphic novels changes teenage behaviors. In her conclusions, she recommends policies that high schools should adopt to prevent harmful effects from graphic novels. Following the researcher’s recommendations is reasonable, assuming that you trust that the researcher studied students who are similar to the students the policy is being developed for; however, what if the sample studied were the students pictured above? (See Figure 1.3.)[5] Can she reasonably say that these teens fairly represent all teens?  Are some teenagers left out—African-Americans, Asians, females, or white males who wear tight jeans to school? Would you want to be subject to a policy made for white boys who go to private school (particularly schools that encourage wearing shorts and a blazer with white piping)?[6] Policies developed on the reactions of privileged, white boys might not be suitable for other groups.

Why Care about These Rules?

Essentially, using science means agreeing to accept a set of findings to the same degree that you trust the methodology. This means several things, all of which are important. First, you can (and should) decide on whether you are going to accept the findings before you know what the findings are.

If the methodology is crap (no matter how much you like the findings), then the findings are crap. If the method is good, then you must accept the findings as true, even if you hate what they imply.

Example: The Dark Tetrad

Let us say that you really like trolling online—the snarkier the comment

Illustration naming the four components of the Dark Tetrad: Narcissism, Psychopathy, Machiavellianism and Sadism
Figure 1.4. The Dark Tetrad. Illustration by author, licensed under CC BY-SA 4.0

is, the better. At the same time, you think that you are a nice person, an asset to your community, and a faithful follower of your religious principles. You are committed to both trolling and your self-image.

Buckels, Trapnell, and Paulhus’ study of internet trolling challenges the idea that trolling is a harmless hobby.[7] In a series of studies, Buckels and his colleagues looked at whether internet trolls show the personality characteristics of the Dark Tetrad, a series of highly negative personality characteristics: Machiavellianism (the willingness to manipulate and deceive others), narcissism (egotism and self-obsession), psychopathy (the inability to show remorse or empathize with others) and sadism (feeling pleasure in the suffering of others) (see Figure 1.4).

The study found that trolls were far, far, far more likely to be card-carrying members of the Dark Tetrad club than any of the other tested groups.

Now you have a conflict—you think that you are a nice person, but you really like to troll (and, to be fair, you have to admit that you enjoy getting a rise out of people). Figure 1.5 shows that trolls are not nice (for an important qualification, see footnote[8]), so one of these two things has to be wrong. Either you are not a troll, or you are not as nice as you think you are. In this case, science has challenged you to look beyond your biases about yourself and consider some hard truths, even if you hate it.

Figure 1.5. Dark Tetrad scores as a function of favorite activity when posting comments online. Erin E. Buckels, Paul D. Trapnell, and Delroy L. Paulhus, “Trolls Just Want to Have Fun,” Personality and Individual Differences, The Dark Triad of Personality, 67 (September 2014): 97–102. Illustration courtesy of Elsevier Publications.
Figure 1.5. Dark Tetrad scores as a function of favorite activity when posting comments online. Reprinted from Erin E. Buckels, Paul D. Trapnell, and Delroy L. Paulhus, “Trolls Just Want to Have Fun,” Personality and Individual Differences, The Dark Triad of Personality, 67 (September 2014): 97–102, with permission of Elsevier.

As illustrated above, science is—or should be—a way for you to break through your own biases. We humans have considerable psychological defenses, and we use them to defend our own prejudices, including the idea that we are personally exceptional. Communication science has numerous examples of situations in which people think they are much less likely to be influenced by media than they actually are:

Table 1.1. Examples of Cognitive Biases in Interpreting Impacts from Media

Examples of Cognitive Biases in Interpreting Impacts from Media

Example of situations where people do not think that they are affected, but research show they more likely are. Theory explaining the distortion
Other people are affected media messages, but I am not. Third person effect
Media images do not affect me. Priming
Reading stories of a robbery is not going to affect how much I blame the victim for being robbed.

 

Effects of reading episodic news

Science Is Also a Way to Get Deeper into a Conversation

Science is, or can be, a way to move beyond starting and ending arguments by trading personal opinions: the “Yes, you are”/“No, I’m not” quarrel.

Let us take a concrete example. Is the picture below sexist?

Marilyn Monroe for a publicity stunt advertising The Seven Year Itch. Monroe is standing on an air vent with her skirt billowing up. There are three men, at least one of whom is a photographer, in the background.
Figure 1.6. Marilyn Monroe, The Seven Year Itch. Photo by Associated Press. No known copyright restrictions.

Well, opinions could legitimately differ. One person could say yes, and another viewer might say no. Without elaboration, the debate could very quickly devolve down to a “Yes, it is,” “No, it is not” squabble.

The advantage of science is that the researchers have to spell out their reasoning. Let us take some of the reasons why the picture above (see Figure 1.6) could be sexist. One, the woman (Marilyn Monroe) is presented as a highly sexualized being. Two, the picture is voyeuristic. The skirt is lifted to reveal the hidden (forbidden) in her attire. Three, the bystanders are male—looking at Marilyn with a male gaze, including—if you look closely in the background—a male photographer. If the researcher developed a coding scheme that said, “A picture will be classified as sexist if it has four of the following characteristics (see below),” then that coding scheme would provide the readers with a detailed view of what “sexist” meant to the researcher.

Sample coding characteristics for determining sexism portrayals:

  1. Is the woman in the picture presented primarily as a sexualized object?
  2. Is the woman an object for males to gaze at?
  3. Is the woman nude or partially clothed?
  4. Is her facial expression sexual (a pout, a sultry gaze, fully lips, tongue extended)?

In looking at the picture, the coder could reasonably say, “Monroe is a sexualized object” (criteria 1). She is an object for males to gaze at (criteria 2). She is shown fully clothed, and she is in control of her skirt, pushing the skirt down with her hand and elbow, but she is—due to the wind—partially clothed (yes on criteria 3). She has no sexualized facial expressions, no exaggerated sexual pout, no sultry gaze, and no flushing (no on criteria 4.).

Reasonable (that is, sane) people would most likely agree that Monroe is partially clothed and she does not have a sexualized facial expression, but she is the subject of an observer’s gaze and viewed in a voyeuristic way. Therefore, using the sample coding scheme, all people who see the picture would have to agree Monroe is not portrayed in a sexist way. However, reasonable people might still think that the image of Monroe is sexist even though they have coded the picture as not sexist using the researcher’s code.

The people who think that the image of Monroe is in fact sexist have to (by the rules of science) agree that according to the methodology used the picture is not highly sexualized. However, they can argue that the method did not actually completely capture what can reasonably be defined as a sexist image. In other words, you turn a “Yes, it is,” “No, it’s not” moment to a deeper reflection on sexism; that is, “If I still think that this picture is sexist even though the method used classifies the image as not sexist, what criteria am I actually using to define sexist?” You still need to say that according to the criteria the scientist used, the picture is not sexualized, but you could say that the method is wrong, or incomplete, and according to the criteria that you develop, the picture would be sexualized. (For example, you might argue that pictures that have two of the four criteria are sexist, you might argue that the aspect of voyeurism alone is enough to classify the picture as sexist, or you might develop a whole new definition of sexism.)

You then have grounds to argue that the methodology of the study is incorrect, but you still have to accept that using the criteria that the researchers used the picture is sexist. Essentially, you have moved the argument to another “My method is right,” and “Your method is wrong,” which is simply another version of the “Yes it is,” “No it’s not” problem, but you have also clarified the meaning of what constitutes sexism to be used in another study.

Another study using the new methodology would have the same rules that accepting the methods means accepting the findings. If the new method suggests that the image is sexist, well, then, the two studies collectively mean that there is disagreement on what definition of sexism is acceptable. If the new method still does not classify the image as sexist, then it is even more likely that the image shouldn’t be considered sexist, because two studies with different methodologies have come to the same conclusion. (Maybe what you consider sexist is wrong or maybe your definition still hasn’t captured the specific things about the picture that indicate it is sexist to you.)  In this way, what we mean when we use a concept increases in precision.

Science as a Way to Develop Policy: Science, Tobacco, and Climate Warming

Relying on science to understand the world around you (which is science’s job, after all) is not the only way to develop an understanding of or form a plan to change the world. You can rely on luck, rely on what has worked in the past (tradition or custom), or apply a principle. However, relying on evidence gathering and data, which is the essence of the scientific approach, allows policy makers to understand and construct policies that cope rationally with the more subtle kinds of problems that spread throughout the nation (wide scope), penetrate into our cells (nonvisible effects), or take decades to develop (slow emergence). This approach intrinsically values facts and data over opinions (see Table 1.2), even when those facts are unpleasant.

Table 1.2. Differentiating Fact and Opinion
Differentiating Fact and Opinion
  A FACT can be VERIFIED An OPINION is what one feels,  thinks or  believes
Key Words Dates, historical events, numbers, science, data Believe, feel, think, just know, am sure, some people say…

Climate warming is already here—rising sea levels, wildfires, heat waves, coral reef die-offs, hurricanes, floods, virus/disease spreads, and mass species extinction. None of us can personally demonstrate global warming from experience. An older person can say that “there were more reefs when I was young,” but to demonstrate that the death of reefs is related to cutting down trees in the Amazon or cow gas (burps and farts)[9] is beyond what a single individual can reasonably determine from personal experience.

Instead, numerous people use agreed-upon procedures (that are open to inspection) to determine individual bits of information that make the connections between greenhouse gasses and methane, temperature increases, and coral reef death. Together, scientists agree that if they trust the methods that their fellow scientists are using, then they trust the findings. When study after study looks into different aspects of climate warming, and year after year the findings imply climate warming is/will be a problem, and further, when predictions scientists make on the impacts of climate warming repeatedly come true, then scientists develop a great deal of trust in the combined weight of these findings.

Our culture — and specifically our government — tends to put an enormous degree of trust in scientific findings. When scientific evidence becomes overwhelming, that evidence can be a counterweight to narrow interests profiting from a product or practice. The gradual development of a national consensus on the dangers of tobacco smoke is the classic case of the triumph of science and the public interest over the narrow interests of an industry.

At the end of the 19th century, both cigarette smoking and lung cancer were rare; cigarettes were considered “immoral” and “dangerous,” and ready made cigarettes were a luxury good. The development of a cigarette rolling machine[10] and a “brilliant businessman who virtually invented mass marketing and mass distribution”[11] transformed cigarette smoking from an upper-class vice to a common habit.

In a fifty-year period between 1900 and 1950, cigarette smoking soared, and so did lung cancer. By the half mark of the century, the scientific evidence that smoking caused lung cancer was developing rapidly. The surgeon general’s report in 1964 based on a review of over 7,000 published articles stated conclusively that smoking increased the risk of developing lung cancer. Neither government authorities nor university scientists have wavered from this conclusion.

The tobacco industry hired scientists, mostly scientists outside of the medical field, on their behalf. They also systematically funded and promoted studies that showed that secondhand smoke did not cause cardiovascular disease and that low-tar cigarettes are low-harm cigarettes (Tong and Glantz 2007);[12] however, the Big Tobacco playbook of challenging science has now lost the war on public trust,[13] and significant policy has been developed to reduce public smoking.[14] Science, credible and independently funded, had the power to stand up against extremely well-funded special interests, which makes science one of the few forces in society that can support the less powerful over corporate interests. To put the issue in the language of the populist movement of the 1900s, the “little guy” over “the man.”

Disadvantages

Science is not the perfect weapon for social justice. Building scientific knowledge is slow and expensive, and inherently cautious. The benefits of science are most likely to go to the educated, who are more likely to find the information and to benefit from its content sooner. However, in those cases where science does demonstrate harm, science (combined with activists and the law) is a powerful weapon to support change.

Understanding the rules that science operates by.

The goal of this book is to review the basic rules of judging three of the most used methods in communication sciences—content analysis, survey research, and experimental methods. These are not the only social science methods, but they are widely used to study texts, people, and impacts.

First, all methods have rules that allow the informed reader to decide how much to trust the study’s findings. These rules involve appropriateness, bias, and verisimilitude. Appropriateness translates into, “Is the method selected appropriate for the kinds of questions that the researcher is asking?”  In the same way that you do not use a tape measure to check how hot the oven is, the kinds of questions that content analysis answers cannot be asked with survey or an experiment. Bias refers to the multiple ways that the researcher can construct a study that favors one answer over another. A content analysis study that coded all pictures of adult humans with beards as male and all others as females would be biased towards[15] women because all adult males who shaved would be coded as female. Verisimilitude (what scientists call validity) is the degree to which the specific group of things that are studied (the sample) are similar to the actual thing. All three are important to establish when the findings from a single study can be extended to a different population.

Second, all studies have limits. The findings are good for the study, but extending beyond the study is problematic. For example, a beautifully constructed sample of South Carolina residents will reflect the views of the population in South Carolina, but is not necessarily trustworthy for Illinois. If the topic of the study is, “What is proper outdoor clothing for winter?”, then the two populations are likely to give very different answers. A careful reader should say that they trust the answers for South Carolinians, but not for Northerners. That does not mean that the study method was wrong; it means that a careful reader would recognize that the people studied (South Carolinians) are systematically different in important ways from the people for which the policy was developed (Illinois residents).[16]

Third, one study is one study—no more and no less. Because individual studies have limits, policymakers need to pay attention to how robust the scientific knowledge is. This is one of the major areas in which social scientists need to be more careful than physical scientists. In physical science, researchers (and research readers) generally assume that if someone drops a ball from a specific height and measures how many seconds pass before that ball hits the earth, then anyone can drop the ball from that distance, any place on Earth, any day of the year, and get the same results. Social scientists cannot assume that showing the same television show to any group of people will produce the same results, because the audience might view the show through different cultural lenses. In fact, scientists cannot assume that showing the same show to the same people on the same day will produce the same results (because the second showing loses the element of surprise). However, if several studies among different populations at different times have the same findings, we say that the findings are robust. Policy makers tend to trust robust findings.

The following chapters will cover rules for three major methods—survey, content analysis, and experiment. The final chapter looks at how to weave the findings (from sound studies) together to answer researchers’ questions, whether their question is “What is the effect of Twitter on rational discourse?” or “Are Instagram fitness pictures self-sexualizing?” or some other question entirely.


  1. “Percocet—oxycodone hydrocholoride and acetaminophen tablet” (Archived Version), DailyMed, National Library of Medicine, National Institute of Health, last modified July 2010,
  2. This is not an endorsement for the iPhone. It’s just what I happen to have, and it works for me.
  3. Shalini Misra et al., “The iPhone Effect: The Quality of In-Person Social Interactions in the Presence of Mobile Devices,” Environment and Behavior 48, no. 2 (February 1, 2016): 275–98, https://doi.org/10.1177/0013916514539755.
  4. —and the findings that come from applying these rules.
  5. The photograph is not of the actual study population.
  6. In this culture, shorts and blazers with white piping signify upper-class and wealthy attire for elite private schools, a very narrow class of people in the United States.
  7. Erin E. Buckels, Paul D. Trapnell, and Delroy L. Paulhus, “Trolls Just Want to Have Fun,” Personality and Individual Differences, The Dark Triad of Personality, 67 (September 2014): 97–102, https://doi.org/10.1016/j.paid.2014.01.016.
  8. The researcher used a survey, a method that cannot prove causality. A survey can only suggest that trolling and Dark Tetrad personalities appear together (covariance).
  9. Not a joke. Cow farts contribute to climate warming. Cows crop and chew grass and pass the plant down the esophagus to the first compartment of the cow’s stomach, which can hold twenty-five or more gallons or more of material. The rumen acts as a fermentation vat that churns the food microbes within the rumen digest. The process produces thirty to fifty quarts of gas per hour, mostly carbon dioxide and methane, which the cow must release or die. James Linn et al., “The ruminant digestive system,” University of Minnesota Extension, 2021, https://extension.umn.edu/dairy-nutrition/ruminant-digestive-system.
  10. The development of the machine was a direct result of a $75,000 prize offered by a tobacco manufacturing company. Allan Brandt, The Cigarette Century: The Rise, Fall, and Deadly Persistence of the Product that Defined America (New York, NY: Basic Books, 2009).
  11. During World War I, the military was giving cigarettes to the troops to “lighten the inevitable hardships of war.” Brandt, The Cigarette Century, 104. In the decades just after the war, filmmakers glamorized smoking, while doctors and athletes were featured in ads praising the cigarette they smoked. Martha Gardner and Allan M. Brandt, “The Doctors’ Choice is America’s Choice: The Physician in US Cigarette Advertisements, 1930-1953,” American Journal of Public Health 96, no. 2 (February 2006): 222-232, https://doi.org/10.2105/AJPH.2005.066654.
  12. Elisa K. Tong and Stanton A. Glantz, “Tobacco Industry Efforts Undermining Evidence Linking Secondhand Smoke with Cardiovascular Disease,” Circulation 116, no. 16 (October 16, 2007): 1845-1854, https://doi.org/10.1161/CIRCULATIONAHA.107.715888
  13. As of July 2017, ninety-five percent of Americans felt that smoking was harmful. Only two percent felt that it was not. Eighty-nine percent felt that second-hand smoking was harmful.
  14. Cigarette smoking is largely banned in enclosed workspaces (in twenty-five states), including bars and restaurants. Slightly over eighty percent of all Americans live with a ban on smoking at their workplaces, and/or restaurants and/or bars. American Nonsmokers’ Rights Foundation.
  15. “Biased toward” means that the researcher will overestimate in one direction. In this case, the coder will count more people as female than are female (females counted = females + beardless males).
  16. For those who are not familiar with either South Carolina or Illinois weather, South Carolina is warm in January, and Illinois can get quite cold.

License

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Reading Social Science Methods Copyright © 2023 by Ann Reisner is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.

Share This Book