7 Quick Overview: Research Designs Learning Objectives After reading and studying this chapter, students should be able to do the following: • Identify and differentiate between the concepts of independent and dependent variables as applied to social science research. • Understand the four basic components of research designs, and be able to identify what key features are related to the types of conclusions that can be drawn from different research approaches. • Differentiate between within-groups and between-groups research designs, and comprehend when each is used (and when a mixed-design approach would be beneficial). • Explain the benefits of various nonexperimental research designs, including quasiexperiments, observational designs, archival research designs, and survey research. • Comprehend the complexity of designing research, and be able to identify the key ideas needed before effective research design can occur. Lan66821_07_c07_p111-134.indd 111 4/24/12 9:35 AM Section 7.1 Variables CHAPTER 7 S o far, this text has explored how a student might analyze and synthesize others’ research. However, in this course—and likely at some point in his or her career—a student will have to design and perhaps conduct a research experiment to answer a question of interest. In research, the type of design chosen influences the type of conclusion that can be drawn from the research. This chapter begins with a basic overview of independent and dependent variables before moving on to major types of research designs and a brief review of different types of research approaches. 7.1 Variables W hen designing an experiment, the first thing a researcher must do is identify the variables at play. A variable is an entity that can take on different values (Harmon & Morgan, 1999). The term variable is probably familiar to most students. In fact, if a student studied any math in grade school, junior high, and high school, then the term is familiar; students might remember solving for variable X or variable Y. The term variable is not restricted to math and can be used as an adjective as well as a noun. There might be a variable speed drill press out in the garage, or some believe that a meteorologist has variable success in predicting the weather. Thus, variable can describe things that are inconsistent or capable of change, or it can refer to something that can be manipulated. In the social sciences, we build on this general definition of variable with more specificity. First, we divide the variables we are most interested in into two broad categories: independent variables and dependent variables. (There are other types of variables—extraneous variables, nuisance variables, confounding variables—but these are outside the scope of this text.) Again, the key idea to remember for now is that a variable—whether independent or dependent—must be able to take on different scores, numbers, outcomes, or values to be considered a variable. Independent Variables At its most basic level, the independent variable is the variable a researcher manipulates, controls, arranges, or organizes. For example, in studying the behavioral effects of caffeine in college students, a researcher may desire to control or manipulate the consumption of caffeine during the experiment. Different students receive different amounts of caffeine, measured in milligrams (mg). By doling out varying caffeine dosages, the experimenter is controlling caffeine consumption, defined as the number of milligrams consumed by the student. The researcher could designate specific levels of caffeine consumption, such as 50 mg, 100 mg, 200 mg, and 400 mg. This type of independent variable is called a non-subject variable (or an active independent variable; Harmon & Morgan, 1999) because the actual value of the independent variable—in this case, the number of milligrams of caffeine received—is not determined by the person receiving the caffeine but determined by the researcher. Sometimes the value or level of the independent variable is determined by the individual participant. For example, the experimenter does not manipulate a person’s level of extroversion; however, it may be measured and used to assign that person to a specific group: high, medium, or low extroversion. This type of variable is known as a subject variable Lan66821_07_c07_p111-134.indd 112 4/24/12 9:35 AM Section 7.1 Variables CHAPTER 7 (sometimes called an attribute independent variable). The researcher can only arrange or organize subject variables; they cannot be controlled or manipulated. Variables such as gender, personality traits, natural hair color, and race are subject variables: a characteristic each person possesses that can only be organized into different groups in a study and not controlled or manipulated. How can one tell the difference between a subject and non-subject variable? Use the coin-flip test. Imagine that students are arriving at the research study, each coming through the door one at a time. As each participant enters the room, the researcher assigns each to a level of the independent variable. If one can make this assignment based on a coin flip, then the variable is a nonsubject variable. For example, the participant is standing in the doorway, and the researcher says, “Heads 50 mg caffeine, tails 200 mg caffeine.” If that can be done, then the variable is a non-subject variable. However, one could not stand at the The researcher manipulates the door and say, “Heads you are male, tails you are independent variable, and it is classified as female.” Gender is a subject variable—a charac- a non-subject variable or subject variable. teristic that is inherent to each person. Students In an experiment testing the effect of can only be arranged or organized into groups caffeine on college students, caffeine is the of males and females, not randomly assigned by non-subject variable because the research a coin flip into the male group or female group. team can control it. Subject variables comprise those traits, qualities, and characteristics that we carry around with us—they are not “randomly assignable.” However, a researcher can organize or arrange a group of males and a group of females, and strive to determine if there is a difference between the two groups. Dependent Variables Just as there are different types of independent variables, there are different types of dependent variables. Remember that the dependent variable is the variable that is measured— hopefully the direct result of the manipulations of the independent variable. Dependent variables can be either qualitative or quantitative. A qualitative variable is one in which the responses differ in kind or type. That is, there is a difference in quality (what form) rather than quantity (how many), and the outcomes of these qualitative variables are usually described in words. On a survey, if a participant is asked to tell about an experience today at the mall and to write a few sentences about it, this would be qualitative data. Quantitative variables differ in amount; there is more or less of some known entity. Quantitative variables are usually described by numbers, and some social scientists tend to strive to develop measures of behaviors (dependent variables) that yield a number. On a survey, if a participant is asked to answer multiple questions about an experience at the mall where 0 =terrible experience and 10 = best experience ever, this approach would yield quantitative data. Lan66821_07_c07_p111-134.indd 113 4/24/12 9:35 AM CHAPTER 7 Section 7.2 The Basic Components of Research Designs Dependent variables can also be described in terms of the measurement process. See Table 7.1 for types and examples of dependent variables. An appropriate dependent variable is the result of careful, systematic observation, which is translated into a clear measure of behavior. Table 7.1: Types of dependent variables Dependent Variable Type Examples Frequency (how often a behavior occurs) Number of cigarettes smoked in a day; number of text messages sent in an hour; number of times studied before a test; number of times one hit the brakes upon approaching an intersection Latency (the amount of time until a behavior occurs) How long it took to learn the lyrics to a new song; after the semester started, how many days (weeks) it was until the textbook was opened; once a red light is observed, the amount of time it takes until braking commences Duration (the amount of time a behavior lasts) The amount of time spent playing XBOX 360; the amount of time studied (in minutes); the amount of time one’s foot was on the brake Amplitude (the intensity of a behavior) The amount of noise (in decibels) generated by a class of third graders; the degree of test anxiety (high, medium, low) exhibited by high school students taking the SAT; the intensity of braking (tapping the brakes vs. slamming on the brakes) Choice Selection (a decision from a number of alternatives) One’s answers to a multiple-choice test; responses on a personality inventory to determine introversion or extraversion; at a repair shop, which type of new brakes selected to be installed on one’s car When all goes well in a study, the measurements from the dependent variable are a function of the independent variable; in other words, the manipulations of the independent variable lead to changes in the values of the dependent variable. These terms for variables, independent and dependent, were popularized by Woodworth (1938) and later by Woodworth and Schlosberg (1954). The terms were used as a means of emphasizing the cause and effect relationship between what the researcher does (independent) and the subsequent outcome (dependent). However, not all studies yield cause-and-effect conclusions: Different types of studies can yield different types of conclusions for researchers. But before delving into the types of conclusions drawn from research, a basic understanding of the research design process is in order. The method by which a study is designed has a direct influence on the strength and type of conclusions that can be drawn when a research project is complete. 7.2 The Basic Components of Research Designs W • • Lan66821_07_c07_p111-134.indd 114 hen thinking about research designs, there are fundamental components or building blocks that need to be considered: whether the dependent variable is measured before and after the independent variable manipulation, or just after; whether or not there is one independent variable, or more than one; 4/24/12 9:35 AM Section 7.2 The Basic Components of Research Designs • • CHAPTER 7 whether the participants are exposed to just one level of an independent variable, more than one, or across two independent variables there is a mixture of both approaches; and finally, the method by which participants are assigned into the different independent variable groupings. Each of these is discussed here in turn. Pretest-Posttest or Posttest Only? The first of these is to consider whether the dependent variable is measured before and after (pretest-posttest) the introduction of the independent variable, or just after (posttest only). Researchers often like to use Xs and Os to describe research designs. The X in a research design stands for some sort of intervention or independent variable manipulation—X marks the situation where something is happening. The O in a research design stands for an observation or a measurement—the dependent variable. As an example, consider a course’s cumulative final exam as a posttest-only scenario. The X would be the course (Xcourse) and the O would be the cumulative final exam (Ofinal exam). These events can be placed in a linear sequence, as follows, and this is an example of a posttest only design reading from left to right. Xcourse Ofinal exam Notice that there was no pretest at the beginning of the course—if there had been, that would be a pretest-posttest design, and it would look like O X O. In fact, it could be interesting to give the course final exam on the first day of class and again on the last day of class. A pretest-posttest design looks like this: Ofinal exam Xcourse Ofinal exam Not to get too far ahead, but we could add a control group to this design—give the pretest and posttest to a different group of students not enrolled in the course. This control group is the comparison group—the group that does not receive the independent variable manipulation or treatment, which would be taking the course. The control group provides a meaningful baseline to which the effects of the treatment, intervention, or the independent variable can be evaluated. That design would look like this: Ofinal exam Xcourse Ofinal exam Ofinal exam Ofinal exam For more on these designs, see Meltzoff (1998). Factorial Design, or Not? A second basic component of knowing about research designs is knowing the number of independent variables being manipulated, controlled, or arranged. One independent variable is simply referred to as one independent variable, but more than one independent Lan66821_07_c07_p111-134.indd 115 4/24/12 9:35 AM Section 7.2 The Basic Components of Research Designs CHAPTER 7 variable is called a factorial design. There are some distinct advantages of factorial designs, namely, the ability to understand interactions between multiple independent variables. For example, a researcher may be interested in a person’s susceptibility to colds and flu. Working with just one variable, like parental status, might lead to a conclusion that parents are more susceptible to the flu than non-parents. One independent variable (parental status) can yield information about the dependent variable (flu susceptibility). But making this a factorial design by adding a second variable might lead to some interesting expectations and patterns of outcomes. For instance, now consider gender and parental status as independent variables. Might there be a combination of these two variables that yields an interesting finding? Given our complex world, factorial designs allow for multiple variables to be included in research studies. Between-, Within-, or Mixed-Groups Design? Another major component of the basic building blocks of experimental design is whether the research design is a between-groups design, a within-groups design, or the design includes a mixture of both—in that case, a mixed design. Between-Groups Design Briefly, the between-groups design is intended to measure differences between separate groups of participants in a study. For example, the researcher might expect males and females to behave differently; Republicans, Democrats, and Independents to vote differently; sociology and nursing majors to have different expectations and career paths; and so on. If a college offers a course to help students prepare for the Graduate Record Exam (or GRE, a test often required for admittance to many different types of graduate programs), and the college was interested in whether freshmen, sophomores, juniors, or seniors would benefit most from the GRE course, this would be a between-groups design. Four different, separate groups of individuals (freshThe between-groups design measures differences between men, sophomores, juniors, and participants in a study. For example, researchers will assess seniors) were utilized to see if behavior patterns in males and females. the GRE course was successful in helping students improve their GRE scores. In this case, the focus is on the difference between groups. Of course, there could be more than one between-groups independent variable. Researchers could add gender as a variable— would men or women benefit most from the GRE course? Lan66821_07_c07_p111-134.indd 116 4/24/12 9:35 AM Section 7.2 The Basic Components of Research Designs CHAPTER 7 A between-groups independent variable must result in two or more groups, and participants can be “claimed” in only one level of the variable for it to be between groups. Using the preceding examples, between groups is appropriate if a participant can be placed only in the male OR female group, the Republican, Democrat, OR Independent Party group, or in the sociology major OR nursing major group. If a student is double majoring in sociology and nursing, then a between-groups design would not be the appropriate design; the student’s membership spans more than one group. Within-Groups Design While the goal of a between-groups design is typically to examine how groups of people may differ from one another, the goal of a within-groups design is typically to examine how a person may change over time. The earlier pretest-posttest design (without the control group) is a good example. If a student were to take a “cumulative” final exam at the beginning of the course, take the course, then take the cumulative final exam again, this would be a within-groups design—specifically, a repeated-measures design. The goal of that research design is to see if the student changed over time; that is, if taking the course led to increased scores. Of course, there are all kinds of reasons why the scores could have changed over time, and such design issues are addressed later in the next chapter. Mixed Design Often the research question dictates the type of design used. For example, if a researcher wanted to test whether right-handed individuals have more legible handwriting than lefthanded individuals, this research question dictates a between-groups design (MacKenzie, 2008) because it will take two different groups of people to make the comparison. But let’s say the researcher wanted to know if practicing a new skill improves an individual’s skill level. In this case, the researcher is looking for a change in the participant over time. For example, the researcher might want to find out whether using a typing program causes an individual to become more proficient at keyboarding skills. To detect skill development over time, within-groups designs are used. But if a researcher wanted to look at skill development over time (within groups) depending on three different types of typing training programs (between groups), one could include both between-groups and withingroups design features into the research. This is called a mixed design. To better understand mixed design, let’s consider a type of mixed design called split-plot design. The easiest example of a split-plot design is one with two independent variables. One of the variables, variable “A,” has at least two levels, and each level has a group of randomly assigned participants. This is the between-groups aspect of the split-plot design, as there are two levels, or groups. The other variable, variable “B,” contains the same participants at every level of A. This is the within-groups aspect of the design. During 2008–2009, I worked with research assistants to design a booklet that helps students become more test-wise—that is, knowing the tips and tricks for test taking that will help students score better, even when students don’t know the answer. The research team developed a test-wiseness booklet as well as a “control” booklet that was about college in general but did not contain test-wiseness tips. Essentially, the experimental group, or treatment group, received the test-wiseness booklet, while the control group received the other. This is our “A” variable. Lan66821_07_c07_p111-134.indd 117 4/24/12 9:35 AM CHAPTER 7 Section 7.2 The Basic Components of Research Designs The dependent variable measurement involved the difficulty level of answers to general trivia questions, but within a set of 60 trivia questions, a third were easy, a third were medium, and a third were hard. The “B” variable here was item difficulty, and every participant was exposed to all three levels of difficulty (easy, medium, hard) no matter what condition of “A” they were in. See Figure 7.1. The experiment would then not only demonstrate whether test-wise tips helped a student’s overall test score but also whether it affected a student when the questions were more or less difficult. The split-plot factor is where a variable is divided in multiple subplots; in our example, that is the B variable, or the difficulty levels of the trivia items. Figure 7.1: Test-wiseness split-plot design A1 A2 Test-wiseness booklet Control booklet Easy Medium Hard Easy Medium Hard B1 B2 B3 B1 B2 B3 Between groups independent variable— random assignment Within groups independent variable— fixed assignment Randomization, Matching, or Blocking? A final key component of experimental designs concerns how participants are assigned to certain conditions or variations of the experiment. To use randomization (or a randomized approach) means that study participants are placed into different groups within a study without bias. In a drug study, for example, if the researcher needed an experimental group and a control group, the researcher could flip a coin for each participant and assign to groups based on the outcome of the coin flip. Randomization is typically considered the strongest or most powerful approach because it can allow for causal (cause-and-effect) conclusions. Take the case of the pretest-posttest design that includes a control group (here it is again as a reminder): Ofinal exam Xcourse Ofinal exam (Experimental Group) Ofinal exam (Control Group) Ofinal exam The issue of random assignment comes into play when researchers have to determine which group of students comprises the experimental group and which group of students comprises the control group. That process of how students are assigned to groups is very important because if true random assignment can be used, that helps to strengthen the conclusions drawn from our data. However, the preceding example is also a good example of a case where random assignment is not typically used. Researchers do not typically have the power to randomly assign students to course sections—students usually pick their own courses. Thus, Lan66821_07_c07_p111-134.indd 118 4/24/12 9:35 AM Section 7.3 Quantitative and Qualitative Approaches CHAPTER 7 in cases where random assignment may not be possible (Gribbons & Herman, 1997), researchers turn to other methods, such as matching in between-groups designs and blocking in within-groups designs. Matching occurs when individuals are paired on a variable of interest, and then the pair is randomly assigned. For instance, if age were a key variable, one could pair individuals with the same or approximate ages, and then randomly assign. Blocking is when one finds groups of individuals who are similar on a key variable, and then the group members can be randomly assigned to study conditions. There are also situations where random assignment may not be an ethical choice, such as in testing the effectiveness of a child welfare program and including a control group where benefits are withheld (Bawden & Sonenstein, n.d.). Even if complete randomization is not possible, researchers always want to know how participants were assigned to groups or conditions. This is important to know because if bias were present, individuals might behave differently due to preexisting conditions rather than because of the independent variable manipulation. 7.3 Quantitative and Qualitative Approaches O nce the basic processes of independent and dependent variables are mastered, researchers build on these concepts to conduct research. When thinking about the measurement of the dependent variable, the measurement process yields either quantitative or qualitative results. Depending on the research situations and questions of interest, social scientists rely on quantitative (e.g., numerical) data and qualitative (e.g., verbal and nonverbal) data. As discussed in prior chapters, social science researchers’ interest in measurement is universal, but researchers vary on the kind or type of data collected. It may seem natural to describe qualitative research methods in contrast to quantitative research methods, but many social scientists make optimal use of both methodological approaches to answer complex questions about social forces as well as individual behavior. Qualitative researchers do not rely on statistical approaches to draw conclusions, but they use complex descriptions of behavior or events; these descriptions often center on feelings about an experience (Beins, 2009). A qualitative research approach emphasizes that the researcher is part of the observation and not a separate and objective observer as is strived for in quantitative research. Qualitative researchers select a topic When conducting research, the results are described as quantitative and qualitative data. Here, this researcher at of interest and then attempt to the Langley Research Center evaluates information from an explore, elaborate, systematize, experiment. and illuminate the collective Lan66821_07_c07_p111-134.indd 119 4/24/12 9:35 AM Section 7.3 Quantitative and Qualitative Approaches CHAPTER 7 understanding of social scientists about the topic of study (Banister, Burman, Parker, Taylor, & Tindall, 1994). Said another way, the qualitative researcher studies individuals in their natural environment and attempts to gain understanding through descriptive means (Beins, 2009). Sometimes researchers discuss quantitative and qualitative approaches as an “either-or” proposition with clear preferences for one approach or another. Often this preference (or perhaps bias) is related to the method in which the researcher received his or her academic training. It is more constructive to think of these approaches as complementary rather than competitive. For instance, a quantitative approach might focus on measurement, explanation, and prediction, whereas a qualitative approach might focus on meaning, understanding, and interpretation (Larkin, n.d.). But both techniques can be used for insight concerning behaviors, attitudes, and opinions. For more on the so-called qualitative/quantitative debate, see Palmquist (2005). Remember that an appropriate dependent variable is the result of careful, systematic observation, which is translated into a clear measure of attitudes, perceptions, or behaviors—no matter if the measurement approach is qualitative or quantitative. Writing in Action: Practicing Variable Identification For each of the following scenarios, practice identifying the independent variable and the dependent variable: 1. A researcher was interested in the effects of reward on intrinsic motivation. Some children were told that they would be given a special award for drawing with magic markers (an activity they already enjoyed). Other children were simply asked to draw with the magic markers. One week later, the children were unobtrusively observed for how much time they spent drawing with the markers. The children who expected and received a reward for drawing with the markers were less likely to draw with them later. Independent Variable: Dependent Variable: 2. In an investigation of the fundamental attribution error, participants were given a speech to read that either favored or opposed Fidel Castro, the communist leader of Cuba. Participants were told that the speech was written by a student who had been assigned to the position taken in the paper (that is, the student writing the speech had no choice on which position to take). Nevertheless, participants believed that the student who wrote the pro-Castro speech had positive attitudes toward Castro, whereas participants who read the anti-Castro paper believed the writer had negative attitudes toward Castro. Independent Variable: Dependent Variable: 3. Within a classroom setting, participants were asked to listen to a guest instructor. All participants were given a description of the instructor. Some participants read a description containing the phrase “People who know him consider him to be a rather cold person . . .” while other people read a description where the word warm was substituted for the word cold (otherwise, the descriptions were identical). After the lecture, participants were asked to rate the (continued) Lan66821_07_c07_p111-134.indd 120 4/24/12 9:35 AM Section 7.3 Quantitative and Qualitative Approaches CHAPTER 7 Writing in Action: Practicing Variable Identification (continued) instructor. Participants who were told the instructor was warm gave him more favorable ratings compared to subjects who were told that the instructor was cold. Independent Variable: Dependent Variable: 4. Participants watched a videotape of a woman taking an SAT-like test. In all cases, she correctly answered 15 out of 30 questions. But participants who observed a pattern of initial success followed by failure perceived the woman as more intelligent than did those who observed the opposite pattern of failure followed by success. Independent Variable: Dependent Variable: 5. Participants read about a woman who used a particular title and then rated her on a number of traits. When the woman used the title Ms. rather than Miss or Mrs., she was assumed to be more assertive, achievement oriented, and dynamic, but also cold, unpopular, and unlikely to have a happy marriage. Independent Variable: Dependent Variable: 6. A stranger randomly approached people on the street and asked them to use his camera to take a picture of him for a school project. For half of the participants, the camera didn’t work—the stranger looked concerned, said that the camera was rather delicate, asked the participants if they touched any of the dials, and announced that it would have to be fixed. For the other half of the participants, the camera worked fine. Farther down the street, a woman dropped a file folder of papers. Forty percent of the participants who had no broken-camera experience helped the woman pick up her papers, whereas 80% of the participants who were led to believe that they broke the man’s camera helped. Independent Variable: Dependent Variable: Source: Anderson (n.d.) Answer Key: 1. Instructions given with markers (special award vs. none); time spent drawing 2. Opinion about Castro in the speech (favor or oppose); attitudes toward Castro 3. Phrase in description (warm vs. cold); instructor ratings 4. Response pattern (success-failure vs. failure-success); perceptions about intelligence 5. Title (Ms. or Miss, or Mrs.); ratings on traits 6. Camera status (work vs. didn’t work); likelihood of helping down the street Let’s say research was being done on the optimum learning conditions for students. For this example, learning is defined as a score on a test. In this research design, one betweengroups independent variable is where tests are administered: online or in a traditional classroom setting. A second independent variable is how the course is delivered: online or through a classroom setting. Both of these variables are between-groups variables because each results in two groups, and four different groups are needed to complete this research as described. Table 7.2 illustrates how this design would look: Lan66821_07_c07_p111-134.indd 121 4/24/12 9:35 AM CHAPTER 7 Section 7.3 Quantitative and Qualitative Approaches Table 7.2: Example of between-groups design Course Delivery Online Test Administration Classroom Online Classroom In this example, a researcher will look to see if there is a significant difference between scores for students tested online and those tested in a classroom. Essentially, the researcher looks to see if the scores in the Table 7.2’s rows are different from one another. The second effect a researcher will look for is whether teaching the course online versus in the classroom led to a difference in test scores. In this case, the researcher examines if the scores in Table 7.2’s columns are different from one another. However, what is more fascinating about this design is whether there is an interaction effect that is statistically significant—so unlikely to be a result of random chance that it may be due to the influence of the independent variable. In other words, is there a combination of rows and columns (a particular cell) that stands out and leads to superior student performance on tests? One can imagine how this information would be valuable. If the best combination of student learning occurs when online instruction is followed by online testing, that would be vital information for educators to have. However, it would also be important to know if there is no better combination than any other combination because then educators would know that students could benefit equally from different types of learning experiences. Researchers could experience a situation where just the test administration effect was significant, just the course delivery effect was significant, or different combinations, including a significant interaction. Figures 7.2–7.4 depict these different types of outcomes. Figure 7.2: Test administration main effect Classroom testing Scores on test Online testing Online delivery Classroom delivery In this example, regardless of how the course is delivered, classroom testing is superior, leading to higher test scores. In this case, there is not a significant interaction; a significant interaction would be evidenced by a different pattern of bars. Lan66821_07_c07_p111-134.indd 122 4/24/12 9:35 AM CHAPTER 7 Section 7.3 Quantitative and Qualitative Approaches Figure 7.3: Course delivery main effect Classroom testing Scores on test Online testing Online delivery Classroom delivery This is an example of data where there is a main effect of course delivery but no interaction. Scores on the test are higher no matter what type of testing is given, so long as the course is being delivered in a classroom. Although there are numerous examples of what an interaction might look like, this is an example of what the data would look like with an interaction taking place. Figure 7.4: Combination of test administration and course delivery main effects Classroom testing Scores on test Online testing Online delivery Classroom delivery In Figure 7.4, here is the place where an interaction is the most meaningfully interpreted. There is one combination of test administration and course delivery that leads to the best combination of tests scores—as the graph shows, that best combination is when the course is taught in the classroom but the test is administered online. This information would be highly valuable to educators and students alike. Lan66821_07_c07_p111-134.indd 123 4/24/12 9:35 AM Section 7.3 Quantitative and Qualitative Approaches CHAPTER 7 Pivotal Moments in Research: Cognitive Dissonance In 1957, Leon Festinger published an influential theory in social psychology called cognitive dissonance theory. As Festinger and Carlsmith (1959) originally characterized the theory, when a person privately holds an opinion but is pressured publicly to argue against it, a form of discomfort or dissonance will occur. As this theory was further studied and refined, cognitive dissonance was also thought of as a situation where a person’s attitudes and behaviors are in conflict. The amount of dissonance would help predict the degree of motivation to resolve the dissonance by changing either the attitudes or the behaviors (Aronson, 1992). What is so interesting about cognitive dissonance theory is that it makes specific predictions about changes in attitudes and behaviors, and sometimes counterintuitive results occur in determining what changes attitudes (Festinger & Carlsmith, 1959). Cognitive dissonance theory provided a wealth of opportunities for future research. In the Festinger and Carlsmith (1959) study, the actual participants were students who were asked to lie to other students who were about to perform a series of truly boring tasks. The participants were assigned to one of three conditions, which means a between-groups design—one of the basic building blocks. In the control/baseline condition, the participant was not asked to lie about the upcoming tasks. In the “$1 condition,” the participant was paid $1 to lie and tell the waiting participant that her upcoming tasks were interesting, enjoyable, and fun. In the “$20 condition,” the participant told the same lie but was paid $20 instead. In the $1 and $20 conditions, dissonance was present—the participants knew that the tasks were dull and boring, but lied about it. The participants were asked a number of questions about the study, and their responses were the dependent variables that Festinger and Carlsmith were most interested in. A key dependent variable question for Festinger and Carlsmith (1959) was “how enjoyable tasks were.” Participants (control, $1, $20) answered this on a −5 to +5 scale, with a +5 score indicating that they thought the tasks were very enjoyable. On average, both the control group and the $20 group still perceived the tasks as being quite boring, scoring an average −0.45 and −0.05, respectively. The $1 group, however, averaged a score of +1.35 in their responses. According to Festinger and Carlsmith, the participants in the $1 condition felt the most dissonance because they were lying for such a small amount of money, hence they changed their own perception of the experiment to match the lie they were telling. Participants who told the lie for a much larger amount were able to remember it was a lie and take the $20 without changing their own attitudes. “The greater the reward offered (beyond what is necessary to elicit the behavior) the smaller was the effect,” Festinger and Carlsmith wrote (1959, p. 208). To put it another way (Aronson, 1992), “People believe lies they tell only if they are under-rewarded for telling them” (p. 304). As Aronson (1992) and others have suggested, the theory of cognitive dissonance may be one of the most importance contributions of social psychology (see also Jones, 1976) and has inspired thousands of studies. Cognitive dissonance theory provided welcome relief for those who thought behavior was either not linked to cognition or was simply shaped by positive or negative reinforcement. It also offered alternative methods for changing behaviors. Prior to this research, it was generally considered that in order to change behavior, a person’s attitude had to change first—that is, our attitudes drive our behaviors (Aronson, 1992). Cognitive dissonance theory predicts that when attitudes and behaviors are in enough dissonance, behaviors may indeed change to match attitudes, but attitudes can also change to match behaviors. Aronson (1992) pointed to the desegregation of schools in the South in the 1950s. Some had suggested that the attitude of racism and prejudice needed to change first before the behavior of segregation could change. Cognitive dissonance theory allowed for the prediction that first changing the behavior can set in motion a change (continued) Lan66821_07_c07_p111-134.indd 124 4/24/12 9:35 AM Section 7.4 Quasi-Experimental and Observational Designs CHAPTER 7 Pivotal Moments in Research (continued) in attitudes—which in fact did occur (Aronson, 1992). Cognitive dissonance theory is still powerful today and has been used to analyze citizen responses to the events of September 11, 2001 (Masters, 2005). When attitudes and behaviors conflict (or simultaneously held cognitions conflict), we are motivated to resolve the dissonance. Questions for Critical Thinking and Reflection • Can you think of situations in your own life where an attitude you publicly held was not in sync with your private behavior? According to Festinger and Carlsmith (1959), one of those two conditions must be resolved in order for the dissonance to fade. In your personal situation, which won out—did you change your attitude or change your behavior? • Think about how cognitive dissonance might be purposely used to help attitude or behavior change. Would it be useful to point out to individuals how their attitudes and behaviors are not in sync? Thinking about your knowledge of psychology from this and other courses, what principles and theories would be useful to apply to achieve an intended attitude change? An intended behavior change? • Think about how the idea of cognitive dissonance applies to major problems that society faces. There is a heightened awareness about global warming and environmental concerns, but look around your local parking lot and check out the types of cars being driven? Is public transportation well utilized where you live? • We know about the negative effects of poverty and homelessness, but think about the efforts in your community (e.g., fund-raising, shelters). We hold certain attitudes, and we possess knowledge, but what facilitates behavior change? Why do so many people see the problems and fail to act? How might cognitive dissonance theory explain (a) a level of relative inaction, and (b) how dissonance might be leveraged for society-level changes? 7.4 Quasi-Experimental and Observational Designs T o be a bit more formal, a quasi-experiment is “a design that manipulates the presumed case and measures the presumed outcome but does not randomly assign participants to conditions” (Shadish & Cook, 2009, p. 619). For an overall comparison of the different types of nonexperimental designs, see Table 7.3. Although many tools are available for social scientists in the design of research studies, only a handful of those approaches are highlighted and reviewed here. These approaches may be particularly helpful to consider when designing a research project to answer hypotheses of interest. Nonequivalent Control Groups Nonequivalent control group designs are quite common throughout the social sciences, so the following is just a sampling of practical applications of these designs. For example, to measure the impact of a computer-based training program for nurses, Hart et al. (2008) administered a pretest questionnaire, delivered information about evidence-based Lan66821_07_c07_p111-134.indd 125 4/24/12 9:35 AM CHAPTER 7 Section 7.4 Quasi-Experimental and Observational Designs practice, and then administered a posttest questionnaire. This is the classic pretest-posttest design, and here is what it would look like graphically: Oevidence-based practice pretest Xcomputer-based education program Oevidence-based practice posttest This type of design lets the researchers know if the participants changed over time. However, it is hard to gauge the effectiveness of the intervention (X) without a control group. Sometimes the constraints of the situation make random assignment impractical. For example, a medical school decided to implement a new form of ethics training for its students based on small groups teaching (Goldie, Schwartz, McConnachie, & Morrison, 2001) but wanted to compare this new approach with the previous lecture-style large groups instruction. Rather than randomly assign students to different instructional conditions, new incoming medical students received the new curriculum, and students from the previous year were utilized as the control condition. The experimental design would look like this: Osurvey score Xnew curriculum Osurvey score (Experimental Group) Osurvey score Xold curriculum Osurvey score (Control Group) Luckily, under the old curriculum, an ethics and health care survey had been administered both pretest and posttest. These same instruments were utilized with the new small group ethics discussion sections. Goldie et al. (2001) found that the new curriculum led to greater consensus in considering ethical situations and concluded that small-group ethics teaching, in an integrated medical curriculum, had a positive impact on first-year students’ potential ethical behavior. It was more effective than a lecture and a large-group seminar-based course in developing students’ normative identification with the profession of medicine. (p. 295) Even though a true experiment was not conducted here, the benefit of the outcomes of the quasi-experimental design are obvious—researchers can learn much from these types of designs, even if unable to draw a cause-and-effect conclusion. Time Series Design In its simplest form, quasi-experimental research using a time series design “is simply a set of repeated observations of a variable on some entity or unit, where the number of repetitions is relatively large” (Mark, Reichardt, & Sanna, 2000, p. 353). For example (Garson, 2008a), the monthly calculation of the national unemployment index by the Bureau of Labor Statistics would be considered a simple time series design. In essence, one can think of this as an extended sequence of dependent variable measurements (O). A simple time series would look like this O  O  O  O  O  O  O  O  O  O  O  O Lan66821_07_c07_p111-134.indd 126 4/24/12 9:35 AM Section 7.4 Quasi-Experimental and Observational Designs CHAPTER 7 The preceding 12 observations could be the monthly reporting of the unemployment index, for example. As one can imagine, the time series design allows for the assessment of change over time—that is, looking for trends—but it can do much more than that (Mark et al., 2000). A time series design can also be used for forecasting. For example, if an economist is tracking unemployment rates, the economist may use this data to try to predict what will happen six months from now, based on the data accumulated leading up to this point in time. Time series designs can become more complex as independent variables (X) are introduced, such as a particular treatment or intervention. These types of designs are sometimes called interrupted time series designs (Cook & Campbell, 1979; Mark et al., 2000) because of the interruption (X) over the series of observations (Os). This type of design might look like this: O  O  O  O  O  O  X  O  O  O  O  O  O Note the independent variable manipulation in the middle of the sequence of observations. This interrupted time series design is often used to measure the impact of legislation and public policy, such as the implementation of a mandatory seat belt law or a ban on cigarette smoking on a college campus. So one can see the benefit of the interrupted time series design: to assess the impact of an intervention. But the drawback of the quasi-experimental design is that researchers cannot be overly confident about causality—a decrease in observed smokers could mean many things—some stopped smoking, some hid their smoking better, some switched to chewing tobacco, and so on. Observational Designs Within the scope of this chapter, a thorough and comprehensive review of observational research is just not possible. An observational design is one in which potential independent variables are not directly manipulated or controlled; dependent variable measures are collected, and depending on the research scenario, potential independent variables may or may not be inferred. This chapter focuses on key approaches—case studies, naturalistic observation, and archival research—with a brief overview of the terminology (Brown, n.d.; Garson, 2008b; Pope & Mays, 2006). Much of the research in the social sciences involves observational designs because of the inability to control or manipulate (or unethical) situations researchers could be placed in by attempting randomized control experiments. For instance, researchers do not purposely expose school children to possibly negative events, but observational designs allow researchers to study natural events as they occur. Lan66821_07_c07_p111-134.indd 127 4/24/12 9:35 AM Section 7.4 Quasi-Experimental and Observational Designs CHAPTER 7 Table 7.3: Summary of different observational designs Type Brief Description Field experiments A field experiment involves a research study where the actual data collection occurs in natural settings, that is, in the field. Case study An extensive observation of an individual or a single group is the hallmark of the case study approach. Case studies tend to look at a limited set of behaviors rather than the totality of the person or group. Naturalistic observation Using naturalistic observation, the researcher is involved in the direct observation of behavior as it occurs in its natural setting. In principle, the researcher does not interact within the environment being observed, but only observes. Participant observation In participant observation, the researcher inserts himself/herself into the environment being studied, which can be especially useful when studying group processes. Researchers using this technique must be careful to remain objective and avoid observer effects (those who know they are being observed may change their own behavior due to the observation). Action research Action research is a subset of participant observation in which the researcher in the natural environment works to change some aspect of behavior or the organization. These actions are designed to improve conditions for the participants or the organization. Rather than test a hypothesis, action research attempts to overtly change behavior. Archival research Archival researchers study the already existing records that were originally recorded in natural settings. Surveys Surveys are a versatile methodological approach because they can be administered to individuals in natural settings as part of a fieldwork approach. Program evaluation Program evaluation involves the evaluation of systematic programs in applied settings. That is, program effectiveness is determined by how patients or clients are served in the field. Ethnography Ethnography involves the direct observation of people during daily life. Sometimes used interchangeably with case study, ethnography refers to both a research process and the type of report written as a product of that research. Case Studies The case study approach focuses on a particular case of interest, and this case may be a person, a group, or perhaps an organization. Case studies can utilize qualitative and quantitative methods. In fact, a research strategy called triangulation encourages researchers to study the variable of interest from multiple perspectives and not over-rely on any one research approach. Researchers using a case study approach can be forward-looking (prospective) or look back in time (retrospective), they can approach theories inductively or deductively, and they can strive to describe, evaluate, or explain behavior (Garson, 2008b; Walshe, Caress, Chew-Graham, & Todd, 2004). Case studies can be very influential in helping to understand the historical background of the topic under study, explore unexpected outcomes, delve into the complexity of interrelationships among people and entities, explore gaps between what is intended and what happens, and in general, obtain a comprehensive look at the big picture. Although there are limitations to the case study Lan66821_07_c07_p111-134.indd 128 4/24/12 9:35 AM Section 7.4 Quasi-Experimental and Observational Designs CHAPTER 7 approach, such as the inability to broadly generalize (more on limitations at the end of this section), case studies can be particularly useful in generating hypotheses and theories in newer fields (Garson, 2008b). In other words, when little is known about a topic, the case study can be extremely useful in providing context about a new idea and about how variables may affect the measure of interest (the dependent variable). Naturalistic Observation The term naturalistic observation typically implies an observational situation where the researcher does not interact in the environment, but merely observes it. However, observation-based research is much more complicated. Naturalistic observation tends to fall into the category of qualitative research, and the goal of qualitative research is to understand human behavior holistically (rather than analytically) and consider the social and cultural context in which we behave (Angrosino, 2007). The development of systematic observational protocols is a key component in naturalistic observation research, and these studies tend to fall into one of three broad categories: (a) nonreactive (unobtrusive) research, where the researcher does not participate in the events under observation; (b) reactive research, in which the researcher is immersed and clearly present in the environment but strives for the role of outside observer; and (c) participant research, where the researcher embeds himself or herself into the environment and participates as an active member of the group being studied (Angrosino, 2007). Naturalistic observation enables researchers to study subjects in their natural habitat without any manipulation from the researcher. For example, researchers could collect data using naturalistic observation for a study of prison inmates. • • Lan66821_07_c07_p111-134.indd 129 So what would a naturalistic observation look like, or, in other words, what basic steps are followed? Angrosino (2007) described a typical sequence of events followed in naturalistic observation research: In the descriptive phase, the researcher is interested in reporting initial observations related to the general research questions under study, as well as providing descriptions of the environment being studied, including the people and the place. At this point, observations should be as value-free as possible, without interpretation—the goal would be statements of fact based on direct observation. Once a broad base is established, the focusing phase begins as researchers strive to sort out relevant observations from irrelevant observations, especially in how these observations relate to the hypotheses and key questions under study. These observations would be more focused on well-defined activities (e.g., traditions, rituals, events) rather than one-time random occurrences. The goal here is to identify patterns, especially how the observed patterns relate to the research question of interest. 4/24/12 9:35 AM Section 7.4 Quasi-Experimental and Observational Designs • • CHAPTER 7 The selective phase might be analogous to a “highway merge” in a large city, where six lanes of traffic are funneled into two lanes. There are still observations to be recorded, but now the key behaviors have been focused on and are under careful scrutiny. Although the entire field of the environment is still under observation, selected observations are used to help provide possible explanations for behaviors. Finally, by the time the saturation point is reached, no new findings are being discovered. The major patterns of behavior are well established and rarely change. Further observation after reaching the saturation point has a relatively low probability of discovering anything new. At this point, the data analysis and interpretation phase is about to begin. Success during this phase of the observational research will very much depend on the coding scheme used and the researcher’s attention to detail in following the protocols established prior to beginning the naturalistic observation. Naturalistic observations can be great sources of new ideas. Take, for example, the study by Chiang (2008) where children with autism were studied using naturalistic observation. Autistic children often can be taught communication skills, but these children often lack the ability to spontaneously utilize verbal and nonverbal skills. The goal of Chiang’s research was to document and categorize the levels of communicative spontaneity in autistic children. Thirty-two diagnosed autistic children, ranging in age from 3 to 16, were videotaped in their natural settings, which included special schools for autistic children, special education classrooms, and general education classrooms. One of the key findings from this research was that autistic students exhibited higher levels of communicative spontaneity in nonsymbolic forms (e.g., keeping an item, pushing) than symbolic forms (i.e., writing, speech). These types of results, while interesting on their own merit, can provide fertile grounds for additional researchers to formulate ideas about how to better understand autistic children and their communication patterns. As with every methodological approach to studying human behavior, case studies and naturalistic observations have drawbacks as well. Because the participants studied may not be representative of the population, these types of studies lack external validity— the ability to be generalized beyond the participants studied. Case studies and naturalistic observations are typically difficult to replicate unless extreme care has been taken to explicitly record the procedures used. Since these approaches do not follow a true experimental protocol (including random assignment), causal inferences are not possible. Finally, these types of studies rely on highly skilled researchers because the potential for influencing the outcome of the study is great, whether it would be experimenter bias in a case study or experimenter reactivity (Brown, n.d.). Archival Research Archival research is a broad term that can be used to describe a wide range of studies. Essentially, archival research involves analysis of data from already existing records— records that were made in natural settings. For instance, by reviewing records from professional baseball and basketball championship games, Baumeister (1995) found that home teams are more likely to “choke” (i.e., perform badly), perhaps due to the burden of high expectation made by playing in front of hometown fans. Riniolo, Koledin, Drakulic, and Payne (2003) used archival records from 1912 United States Senate hearings (and from Lan66821_07_c07_p111-134.indd 130 4/24/12 9:35 AM Questions for Critical Thinking CHAPTER 7 the British Board of Trade) to compare eyewitness accounts of the Titanic sinking to the forensic data we now have, particularly examining the claim that the Titanic was breaking apart as it sank. Riniolo et al. (2003), after carefully screening testimony that indicated clear observations, found that 15 of 20 eyewitnesses accurately reported this tragic event. Related to more recent events, Martz, Bodner, and Livneh (2009), in using archival data available from the National Vietnam Veterans Readjustment Study (a national random sample of over 3,000 veterans drawn from 8.2 million veterans who served in Vietnam), found that for veterans with disabilities, teaching these veterans problem-solving skills was beneficial only for veterans with mild to moderate disabilities (the intervention was ineffective for those veterans with severe disabilities). These three studies illustrate the versatility of archival research. None of these researchers actively collected data; these researchers examined baseball box scores, congressional testimony, and preexisting survey data, respectively. Archival research can be invaluable in providing real-world examples of phenomena studied in controlled, laboratory settings. As Baumeister (1995) put it, “Perhaps the best compromise is that these [archival studies] should be regarded as extending, illustrating, and confirming laboratory studies rather than as primary, direct tests of theory” (p. 646). Chapter Summary E mpirical research is at the heart of the social sciences, and it is important for students to have the appropriate appreciation for the contribution of research as well as a comprehension of the variety of research approaches available. The research process begins with the identification of independent and dependent variables, and measurement processes are utilized henceforth. Although there are four basic building blocks (components) to every research design, this chapter presents a bit more detail about the major approaches to social science research, including between-, within-, and mixed-experimental designs, quasi-experimental designs, observational designs, different types of archival research, and survey research approaches. Each of these approaches has its own advantages and disadvantages. No research design is perfect, and no research effort is perfectly executed. However, by maximizing our adherence to research methods best practices in the social sciences, we can minimize errors and minimize inaccurate conclusions drawn from evidence. Knowledge obtained from the social sciences moves at a slow pace, following the sample principles of how knowledge advances in the sciences. Knowledge of these different research designs should be helpful as budding social scientists endeavor to capture the big picture, 30,000-foot view of the discipline. Questions for Critical Thinking • • Lan66821_07_c07_p111-134.indd 131 Part of your task in this course is to design a research study. How will the topics presented in this chapter be applicable to your study? Try to identify the independent and dependent variables in your proposed research plan. Over the course of studying for your sociology degree, how has your perception of research changed, if at all? Do you now see the value of new research findings, as well as the complexity of obtaining new (and reliable) research findings? 4/24/12 9:35 AM Concept Check • CHAPTER 7 As you self-reflect, when was the last time you changed your mind based on a research report? What does it take for you to change your attitudes toward a topic or your behavior in a particular situation? Do you think your flexibility in changed thoughts and behaviors is typical of others you know? How or how not? Concept Check 1. Which of the following is NOT a function of the independent variable? a. arranged by experimenter b. controlled by experimenter c. measured by experimenter d. manipulated by experimenter 2. Qualitative variables are likely to be expressed as ______, whereas quantitative variables are likely to be expressed as ______. a. integers; words b. words; numbers c. quantities; qualities d. descriptions; expressions 3. Which of the following is the key trait in a factorial design? a. There are two or more independent variables. b. The design is decidedly a pretest-posttest design. c. Randomization is achieved by blocking rather than matching. d. There are a minimum of three dependent variables. 4. The signature approach of a case study is that in a case study the focus is typically on a. the case that involves the most participants. b. drawing broad conclusions that apply to all cases. c. the accumulation of forensic case evidence. d. one particular person or case of interest. 5. Which of the following is a key characteristic of archival research? a. Its findings are easier to publish than experimental research. b. Nonequivalent control groups designs can be optimized. c. The data are obtained from already existing resources. d. Cognitive dissonance is typically present in the data. Answers: 1) c, 2) b, 3) a, 4) d, 5) c Lan66821_07_c07_p111-134.indd 132 4/24/12 9:35 AM Key Terms CHAPTER 7 Web Links This website describes pretest-posttest design and explains the different scenarios in which this experimental design is appropriate to use. Additionally, the challenges associated with this experimental design are also outlined: http://www.experiment-resources .com/pretest-posttest-designs.html This website describes the features of experimental designs and offers a variety of definitions for the key terms used in this chapter: http://www.sportsci.org/jour/0001/ wghdesign.html This website describes a sample chapter regarding experimental research and variable manipulation with a strong emphasis on test design and the different types of designs a researcher could choose to use: http://www.southalabama.edu/coe/bset/johnson/ lectures/lec9.htm This website details 2x2 factorial design by giving examples and visual representation of different design development strategies: http://web.mst.edu/~psyworld/mixed_designs .htm This website includes a nice summary of cognitive dissonance research and why this effect is an important phenomenon to understand: http://changingminds.org/ explanations/theories/cognitive_dissonance.htm Key Terms between-groups design A method of study design intended to measure differences between separate groups of participants in a study (e.g., freshmen, sophomores, juniors, and seniors). blocking A process of data analysis that turns a potentially extraneous variable into an independent variable, which permits the examination of whether or not the variable interacts with the intended independent variable. cognitive dissonance theory A theory developed by Festinger and Carlsmith; occurs when a person privately holds an opinion but is pressured publicly to argue against the privately held opinion, and a form of discomfort or dissonance occurs because of the conflict. Lan66821_07_c07_p111-134.indd 133 coin-flip test If participants can be assigned to different groups on the basis of a coin flip, then the independent variable being used is a non-subject variable. control group The comparison or baseline group of participants to which the experimental group is compared. dependent variable The variable that is measured. factorial design The statistical experiment design in which more than one independent variable is being manipulated, controlled, or arranged. This enables the experimenter to understand interactions between multiple independent variables. independent variable The variable that is manipulated, controlled, or arranged/ organized by the researcher. 4/24/12 9:35 AM Key Terms matching The pairing of participants based on similar measures on a targeted variable. mixed design When an experimenter includes both between-groups and withingroups design features into the research. non-subject variable When the value of the independent variable is not determined by the participant but determined by the researcher. observational design A research design where there are typically no independent variables controlled by the experimenter (non-subject variables), only dependent variables and preexisting subject variables/ independent variables. posttest only design When the independent variable is measured only after the experimental intervention has been administered. pretest-posttest design When the independent variable is measured both before and after the experimental intervention has been administered. CHAPTER 7 quantitative variable A variable to be measured that varies by degree or magnitude, often as a numerical response. random assignment When participants are randomly assigned to a group or condition in an attempt to control for any significant differences among groups. randomization When individuals are assigned to a study group by chance and not in a predictable manner. split-plot design When a variable is divided in multiple subplots. subject variable A characteristic, such as GPA, that an experimenter cannot randomly assign because the participant already has that characteristic previous to participation in the study. variable A unit of measure that is free to vary. within-groups design An experiment design that aims to measure the change within a participant over time. qualitative variable A variable to be measured that varies by kind or type, often as a verbal response. Lan66821_07_c07_p111-134.indd 134 4/24/12 9:35 AM
Purchase answer to see full attachment