CHAPTER 2 TIME PRESSURE AND TIMELINES Whatever the length of time you have to complete a research project, start the process by creating a timeline of due dates for components of the project: coming up with ideas, consulting the literature, settling on an approach (research design), identifying and obtaining measures, securing IRB approval, recruiting participants, collecting the data, entering and/or analyzing the data, presenting the results, and explaining the results. As explained earlier in the chapter, the written report (manuscript) includes the Introduction, Method (participants, measures, procedure), Results, and Discussion. Given time pressure, it is a wise idea to write sections along the way, rather than wait until after you analyze your data. Figure 2.13 Steps for Managing the Research Process Source: Adapted from Devlin, A. S. (2006). Research methods: Planning, conducting, and presenting research. Belmont, CA.: Wadsworth/Thomson. Divide the semester (12–13 weeks) into weekly segments; then work backward from the final deadline. If your instructor reads drafts, incorporate that into your timeline. Figure 2.13 provides one model to manage the steps in the research process. You can control your own behavior, but receiving approval from the IRB or feedback on your draft may not happen according to schedule. Manage the time for the activities you can control (generating ideas, consulting the literature, writing sections of the manuscript). One other challenge is worth noting. One of the biggest obstacles is obtaining complete measures (with all items and scoring instructions), which is not as straightforward as you might think. The pitfalls of this task will be covered in Chapter 5. Chapter 3 RESEARCH QUALITY AFFECTS RESEARCH ANSWERS Good research involves asking worthwhile questions that combine tradition and innovation, as illustrated in Chapter 2. Good research also involves selecting an approach that is appropriate for the question(s) you want to answer. Some research questions can be answered with a high degree of certainty; others cannot. In some instances, researchers are interested in what might be called exploratory research; they want to gain a preliminary understanding of a topic and may not formulate specific hypotheses. Often, researchers use interviews, focus groups, and case studies in such research. These approaches are usually described as qualitative and will be discussed more fully in Chapter 8. Experimental design: Research approach with manipulated variables and random assignment. Quasi-experimental design: Research approach that resembles experimental research but is based on the use of preexisting groups (quasi-independent variables [IVs]). Internal validity: Extent to which a research design allows you to test the hypothesis adequately. Type I error: Incorrectly rejecting the null hypothesis when it is true. Whatever your goal and approach, the manner in which you conduct the research affects its quality. As a number of authors have pointed out, simply doing a large and well-funded study does not guarantee that the research questions will be appropriately framed (see R. Kaplan, 1996). Even in exploratory qualitative research where you are trying to get a sense of the issues that matter to people (e.g., reactions to building high-rise, high-density housing near a neighborhood park), many aspects of your approach, such as the facility where the questions are asked and who is included in the interviews, can affect how people respond. This chapter will emphasize quantitative approaches, but many of the same issues dealing with the internal validity of your research (whether the design enables you to answer the research questions) apply to qualitative research as well. WHAT RESEARCH CAN TELL YOU: THE CONTINUUM OF CERTAINTY The degree of control you exercise in conducting your research is usually related to the level of certainty you can have in your results—greater control is related to greater certainty. On one end of the research continuum, we do not manipulate any variable; if we want to quantify people’s answers (i.e., quantitative approaches), we typically ask people to answer questions about the topic of interest on a rating scale. For example, we might ask whether there are relationships between the number of times students work out each week, their caloric intake, and their body esteem. Using measures such as rating scales with numerical values, we would be interested in the correlations or associations between variables. Nothing is manipulated. Even in this type of study, there are additional questions we can ask that will increase our confidence in the relationships we see (perhaps we should ask about height and weight, for example, which might affect the relationships). Type II error: Failing to reject the null hypothesis when it is false. At the other end of the continuum, we manipulate specific variables, keep everything else the same, and try to infer causality from the study (e.g., randomly assigning students to different workout schedules before measuring their caloric intake and body esteem). You can see that even in this second case, there are still a lot of other “unmeasured” variables (e.g., students’ workout schedules prior to the study or whether they are varsity athletes) we would need to assess to be confident that differences in workout schedules caused differences in caloric intake and body esteem. CORRELATION VERSUS CAUSATION You may have heard the phrase “Correlation is not causation.” These two concepts lie at different ends of the spectrum of certainty about relationships. That does not mean that one kind of relationship is always preferable to the other; each is suited to different research questions and situations. Ultimately, most researchers seek to understand causes of behavior, it is true, but in some kinds of situations, research that would result in making statements about causality is not possible. When the research approach is correlational, the focus is on the relationships between variables. We change nothing about the situation of interest and simply assess whether relationships exist. We have no evidence that a change in one variable caused a change in the other. Because the variables have not been manipulated, there is no opportunity to assess causality; there is no evidence of influence (that is, one variable cannot be said to affect another). Rather, the concepts of interest are associated or related to one another. When the research approach is causal, there is evidence of influence. In this situation, there is an explicit manipulation of (i.e., change to) one or more variables. This change allows us to assess causality. Correlational research: Approach to research where no variables are manipulated. Causal research: When the research design enables you to test cause-and-effect relationships. To illustrate the difference, we might first investigate the relationship between students’ GPAs and the distance of the college they attend from the students’ hometowns. We cannot randomly assign people to a given GPA, nor can we randomly assign them to living in a specific hometown. Students “come that way.” We might hypothesize that these variables (GPA and distance of the hometown from the college) co-vary, such that changes in one are associated with changes in the other—for example, that students who have higher GPAs live farther away from their hometown (and those with lower GPAs live closer). In this case, we are predicting a positive relationship (higher GPAs correlate with longer distances), but we cannot infer causality. Why? Because there are many other explanations other than distance for that GPA. Try This Now 3.1 List another variable that might be used to help explain this significant relationship between students’ GPAs and the distance of the college from their hometowns. Perhaps you said, “attended boarding school.” Perhaps going to boarding school prepares you for attending college a far distance from home, and it is this boarding school preparation, not the distance from home itself, that better explains this relationship with GPA. The essence of research in which causality can be inferred is control—control over every aspect of the research endeavor that can be controlled. When control is not possible, we have other ways we try to spread out the variability or differences in humans that can interfere with the factors we are studying. For example, when we do a study where people are exposed to different stimuli (e.g., pictures of natural and built environments) to measure environmental preference (i.e., how much they prefer particular environments), we randomly assign the participants to the different pictures (i.e., conditions) to spread out or distribute the variability that exists in the population (our participants). We do this random distribution to try to make sure, for example, that all the environmental studies majors don’t end up in the condition with pictures of nature! If they did, a higher preference for pictures of the natural environment might be explained by the students’ major, not by qualities depicted in the pictures themselves. When we make a statement about causality, we have to persuade our audience that there are no other likely explanations; we have to rule out what are known as alternative variables represented by unmeasured or third variables, which was discussed at the end of Chapter 2. In correlational studies, such variables may lead us to infer incorrectly relationships between our variables of interest when, in fact, it is the third variable at work. Excerpts from a wonderful video from Frans de Waal’s TED Talk, “Moral Behavior in Animals,” show what happens when two capuchin monkeys were rewarded unequally for the same “work” and the less-well-rewarded monkey tests out an alternative explanation (https://www.youtube.com/watch?v=meiU6TxysCg). One monkey is rewarded with a grape (the preferred food); the other monkey is rewarded with a piece of cucumber for the same task (handing a small stone to the experimenter). When this happens, the monkey that received the cucumber is quite unhappy (throwing the cucumber back at the researcher) and next tests a stone against the side of the cage to make sure that the stone hasn’t somehow produced the inequitable result. What this monkey is doing is testing for a third variable, as if to say, “maybe it’s the stone that’s the problem.” Even capuchin monkeys are capable of thinking about alternative explanations! Combing two previous examples, how might you infer causality in a study involving distance and working out? In the earlier example, we could not randomly assign people to their hometowns, but we might be able to assign first-year students randomly to residence halls at different distances from the campus fitness center. Then our research question might be whether living closer to the fitness center has an effect on the number of times a week students go to the center to work out. There are many other variables (representing alternative explanations; see Chapter 2) that we might need to rule out (whether the student is a varsity or club athlete, any health restrictions, athlete status in high school, and so on), but we have manipulated and controlled a variable (distance from the fitness center) and randomly assigned people to the conditions (different residence halls located at different distances from the fitness center). If there is a result showing that people who live closer to the fitness center work out more times per week than do those who live farther from the fitness center, then we might infer causality related to the variable of distance. WHY CONDUCT CORRELATIONAL RESEARCH? You might be asking yourself why people do correlational research if the goal of research is to explain behavior (hence, to determine causality). Correlational research has important purposes. First, it is sometimes used as exploratory research to see whether relationships exist before investing more resources in experimental research. Second, there are many instances in which it is not possible to manipulate variables (such as people’s hometowns). Third, it may be unethical to manipulate variables. For example, we could not tell people that a car accident had occurred to a member of their family or that they had failed a final exam to determine their emotional reaction. As Chapter 4 will explain, research on human participants is monitored by review boards to make sure ethical principles are followed. There are innumerable situations where it is not feasible to manipulate variables. For example, we can’t change the condition of the sidewalks in people’s neighborhoods to examine the impact of sidewalk upkeep on an activity such as walking. What we would do instead is find neighborhoods (and the people in them) that are alike in as many ways as possible (e.g., health), except for the condition of the sidewalks, and then look at the differences in activity levels across neighborhoods. Again, we would also need to measure background variables such as age and car ownership that could be related to activity levels. A fourth reason to use correlational research is to study relationships that are naturally occurring. You might assess the relationship between student test scores and involvement in extracurricular activities, without wanting to manipulate either one of those variables. Relatedly, assessing naturally occurring relationships may complement experimental studies conducted in the lab that have been described as artificial. THE LANGUAGE OF CORRELATION AND CAUSATION The words you use to describe your research have certain implications. When you use such words as impact, influence, determine, control, regulate, shape, alter, or modify, you suggest causality. When you use such words as association, relationship, link, or correspondence, you suggest correlation. In your writing, it is important to use the language that matches the kind of research you have done (see also Chapter 13 on writing up research). CORRELATIONAL RESEARCH APPROACHES: CORRELATIONAL AND QUASI-EXPERIMENTAL Correlational research approaches can be divided into two general categories: correlational and quasi-experimental. What the two approaches have in common is that only correlational relationships are involved; that is, no causality can be inferred. What is typically called a correlational approach involves questions about a randomly selected sample as a whole in which two or more variables are measured (Category 1 from Figure 2.1); what is called a quasi-experimental approach typically involves questions about differences between preexisting groups (Categories 2 and 4 from Figure 2.1). Although there are groups in quasi-experimental designs, they have not been randomly assigned. Quasi means “resembling.” Even though a quasi-experimental design mirrors a true experimental approach in some aspects (e.g., asking questions about group differences), it does not include the critical aspect of random assignment to condition and the groupings preexist (e.g., gender) or can be formed from preexisting situations (e.g., dividing students into those who own cars vs. those who do not based on responses to a questionnaire). Examples of such preexisting groups in quasi-experimental research are gender, class year, marital status, athlete status, coffee drinker or not, or almost anything where the characteristic in question preexisted or was naturally formed. True experimental approach: Research approach in which one or more variables are manipulated and participants are randomly assigned to condition. Random assignment: When participants are randomly assigned to the conditions of the study. HALLMARKS OF TRUE EXPERIMENTAL APPROACHES In experimental approaches, participants are randomly assigned to conditions in which one or more variables of interest have been manipulated. There is an attempt to control extraneous variables and to measure as many potential third variables as necessary. The outcomes you measure test the effect of these manipulations. As an example, in Devlin et al. (2013), participants (students and adults from the community) viewed one of four photographs of the office of a psychotherapist that varied in the kind of art displayed (Western vs. Multicultural) and the number of art objects on view (1 vs. 6). Participants were randomly assigned to one of the four conditions generated by crossing the two art traditions (Western vs. Multicultural) with the two different numbers of art objects (1 vs. 6) (see Figure 3.1). Between-subjects design: Research in which the conditions of an experiment are distributed across participants such that each participant is in only one condition. Within-subjects design: Type of experimental design in which participants are exposed to all of the conditions. Participants answered a series of questions about the characteristics of the therapist whose office they viewed; the office was created for the purposes of the research. The research question was whether the display of art that differed in (a) cultural tradition and (b) number of art objects would impact participants’ judgments of therapists, in particular, their openness to multiculturalism. This experimental approach is called a between-subjects design because the conditions are distributed between (across) participants. In the between subjects approach, each person participates in only one condition. This approach is often used because the researcher is concerned that participating in more than one condition would produce different results than participating in a single condition and that the impact of a given condition could not be isolated. Moreover, the between subjects approach reduces the likelihood that participants will guess the hypothesis of the research. In contrast, in the experimental approach called a within-subjects design, all participants would have seen all four photographs; that is, they would have been exposed to all of the conditions. Researchers often select a within-subjects design when (a) effects of participating in one condition on another are unlikely or (b) there are such effects that carry over and researchers want to study them. Between- and within-subjects approaches are covered in more depth in Chapters 9 and 10, respectively. DIFFERENTIATION OF INDEPENDENT AND DEPENDENT VARIABLES A course in research methods exposes you to specific terms that communicate important information. In this chapter, we have already seen important terms, like correlational and quasi-experimental designs. Two critical terms to understand are independent and dependent variable. Often these are referred to as the IV and DV, respectively. An independent variable is manipulated or varied (like our example of art in the previous section). You could think of this as the variable that is independent or “free to differ.” A dependent variable is the outcome of (depends on or is constrained by) exposure to the independent variable. Some researchers look at the independent variable as preceding an effect and, hence, as a cause; the dependent variable reflects the impact of the independent variable and is the outcome or effect. Independent variable (IV): Variable that is manipulated in an experiment. Dependent variable (DV): Variable that reflects the impact of the manipulated or independent variable in research. Quasi-IV: Independent variable (IV) that is naturally occurring (e.g., race and gender) and as a consequence is not assigned at random. We also need to identify what is called a quasi-IV. You remember that we talked about the difference between quasi-experiments and true experiments (where variables were manipulated and subjects were randomly assigned to condition). Here we will differentiate the parallel terms quasi-IV and “true” IV (normally just referred to as the IV). A quasi-IV is a grouping variable that has not been manipulated (like race or class year). A true IV has been manipulated (like our art example). As we will see later in this chapter, the statistical analyses for research involving quasi-IVs and IVs are identical; what differs is the language we use to describe the results. When we use quasi-IVs, we use the language of correlation. Thus, if we have sailing team members and nonsailing team members take a cognitive task known as the Mental Rotations Test (MRT; see Vandenberg & Kuse, 1978) and sailing team members score significantly higher than do nonsailing team members on this test, can we state that being a sailing team member caused this higher performance? No. What we can say is that there is a relationship between being a sailing team member and scoring higher on the MRT in comparison to the performance of nonsailing team members. REFRAMING A RESEARCH IDEA First come research questions; then come research designs; last come statistical analyses. The manner in which your research question is stated guides the research design. Let’s start with a correlational example and transform it into a quasi-experimental and finally an experimental design. This transformation will illustrate that there is usually a way to approximate an experiment based in the real world, even when the specific real-world variables cannot be manipulated. As discussed earlier, if you ask a question about the sample as a whole without any manipulation that forms groups, you will have a correlational research design. For example, if you ask whether the number of magazine subscriptions in a home is related to reading scores in fourth graders (see Figure 3.2), you will have a correlational design; the statistics will be Pearson’s r. If you ask a question about group differences, and the groups preexist (like subscribing to print magazines or not), you will have a quasi-experimental design. For example, if you ask whether there are higher fourth-grade reading scores in the homes of people who subscribe to print magazines than in the homes of people who do not, that is a quasi-experimental design (the preexisting groups are composed of people with and without print magazine subscriptions; see Figure 3.3). In that particular situation, if your quasi-independent variable is print magazine subscriptions, which has two levels (yes or no) and your dependent variable is the fourth-grade reading scores of the children from those homes (one DV), your statistical analysis will be an independent samples t test. There is still no causality. If you ask a research question about group differences, and the groups are created through manipulation, you will have an experimental research design. For example, if you create written scenarios (written text describing situations) in which participants are randomly assigned to read a scenario about a home with no (zero) magazine subscriptions/month versus a home with 10 magazine subscriptions/month, and you ask participants their estimate of the reading scores of the fourth graders in the home (chosen from a scale of the possible reading scores), that is an experimental design (see Figure 3.4). In this situation, there is one IV (magazine subscriptions) with two levels (zero vs. 10 subscriptions/month) and one DV (the reading score estimate). Again, your statistical analysis will be an independent samples t test. The statistical test in the examples of the quasi-experimental and true experimental designs are the same, but the language you use to describe the results will differ. In the case of the quasi-experimental research, you will use the language of correlation—for example, probably that subscribing to print magazines is associated with having higher fourth-grade reading scores than is the case in a home where there are no print magazine subscriptions. With this experimental design, you have the language of causality, for example, that reading about a household with 10 magazine subscriptions/month led to judgments of higher reading scores in fourth graders than did reading about a household with no (zero) magazine subscriptions/month. TYPE I VERSUS TYPE II ERROR Research in the social and behavior sciences typically uses inferential statistics, which means that we use samples to make informed guesses about the characteristics of the population from which the sample is drawn. In other words, we don’t know for sure about an answer to a particular outcome because we haven’t asked or assessed every person in the population of interest. We have asked what we hope is a representative sample. But we could be wrong because we are using inferences about our statistical hypotheses. Type I and Type II errors describe the ways in which we could be wrong. In a Type I error, we claim that we have a significant statistical result when that is not the case. Formally, we reject the null hypothesis (of no statistical difference or relationship between groups) when we should not have done so. In a Type II error, we have missed a finding that is there. Formally, we fail to reject the null hypothesis when there is a statistical difference, that is, when we should have done so. Figure 3.5 represents the four possible outcomes. We can be correct in two ways: We are correct when we reject the null hypothesis when there is a finding; we are correct when we do not reject the null hypothesis because there is no finding. We can also be incorrect in two ways (our Type I and Type II errors). We are incorrect when we reject the null hypothesis (say there are statistical differences or relationships; a false alarm) when there are none (Type I error). We are also incorrect when we do not reject the null hypothesis and we should have, that is, such statistical differences or relationships are there (Type II error; a miss). Bonferroni adjustment: Adjustment for Type I error by dividing the alpha level (.05) by the number of statistical tests performed to create a new more stringent alpha level. Two-tailed significance test: When the critical region for significance is spread across both tails of the distribution. One-tailed significance test: When the critical region for significance is limited to one tail of the distribution (more lenient than two-tailed tests). Both Type I and Type II errors should be avoided, but Joseph Simmons et al. (2011) view Type I as more problematic. Their argument is once these false positives or incorrect statements that a finding exists appear in the literature, they tend to stay there. Such findings can encourage other researchers to investigate the phenomenon further, which may be a waste of resources. Sources of Type I Error and Remedies Remember that in inferential statistics, we are estimating likelihoods or probabilities that the data represent the true situation, but we set that likelihood at a given level, called the alpha level. By convention, the alpha level is set at .05. What that means is that there are only 5 opportunities in 100 (5 / 100) that we are mistaken in saying that our results are significant when the null hypothesis is true (see also Chapter 2 on this topic). Standard approaches to reduce the likelihood of a Type I error are as follows: adjusting the alpha level; using a two-tailed versus a one-tailed test; and using a Bonferroni adjustment for multiple analyses (dividing the alpha level by the number of statistical tests you are performing). In theory, you could set your alpha level at a more stringent level (e.g., .01) to avoid a Type I error, but most researchers do not, fearing that a Type II error will occur. A second approach is using a two-tailed rather than a one-tailed significance test. Please note that the decision to use a one- versus a two-tailed test is made prior to conducting analyses (and is typically indicated in the hypotheses). The difference between a two-tailed significance test and a one-tailed significance test deals with how your alpha is distributed. In a two-tailed test, the alpha level (.05) is divided in two (.025), meaning that each tail of the test statistic contains .025 of your alpha. Thus, the two-tailed test is more stringent than a one-tailed test because the critical region for significance is spread across two tails, not just one. A one-tailed test is not adopted in practice unless your hypothesis is stated as uni-directional rather than as bi-directional. Again, that decision has to be made prior to conducting analyses. Another way in which a Type I error occurs is the use of multiple statistical tests with the same data. This situation may happen in research because there are not enough participants with specific demographic characteristics to run a single analysis. Here’s an example. Suppose you want to examine issues involving number of majors (e.g., for students who identified themselves as having one major, two majors, or three majors), class year (first vs. fourth year), and athlete status (varsity athlete [A] or not [NA]) and the dependent variables of interest were GPA and career indecision (see Figure 3.6). What this table shows is that we have 12 cells (look at the bottom row) to fill with participants with the appropriate characteristics. A minimum number of participants per cell might be 15 individuals. But we don’t need just any 15 individuals; each cell must be filled with the 15 who have the required characteristics for that cell. For Cell 1, we need 15 students who identify as having one major, are in their first year, and are varsity athletes. Cell 2 is 15 students who identify as having one major, are in their first year, and are not varsity athletes. You can see the difficulty in filling all of the cells with students who have the sought-after characteristics. A single analysis might not be possible. We might have to ignore the athlete status in one analysis and look at class year and number of majors, which is six cells (against the DVs, GPA, and career indecision). In another analysis, we might look at number of majors (three) and athlete status (two) against the DVs (six cells again). In the full analysis, testing all variables at once, you would have 2 (class year) × 2 (athlete status) × 3 (number of majors) = 12 cells. Thus, if we did only two of those at a time (and given the selected examples) we would have fewer cells than in the full analysis. In a third analysis, we would look at athlete status (two) and class year (two), which has four cells. If we did all of these analyses, we would have run three analyses instead of one. The likelihood that a Type I error would occur has increased because of these multiple tests. For that reason, many researchers recommend using a Bonferroni adjustment, which resets the alpha level (more stringently). To use a Bonferroni adjustment, you divide the conventional alpha level (.05) by the number of tests you have run (here 3) to produce a new alpha level—here .017. Now, to consider a finding significant, the result would have to meet the new (more stringent) alpha level. (For those who want to read in more detail about this issue, articles by Banerjee et al. [2009] and by Bender and Lange [2001] may be helpful.) TYPE II ERRORS: SAMPLE SIZE, POWER, AND EFFECT SIZE In a Type II error, we fail to reject the null hypothesis when we should have done so. Often the problem is having too few participants; therefore, having an adequate sample size is the primary way to address this problem. In general, larger sample sizes produce more power (see next section). Power is the ability to evaluate your hypothesis adequately. Formally, power is the probability of rejecting Ho (the null hypothesis), assuming Ho is false. Power: Probability of rejecting Ho (the null hypothesis), assuming Ho is false. When a study has sufficient power, you can adequately test whether the null hypothesis should be rejected. Without sufficient power, it may not be worthwhile to conduct a study. If findings are nonsignificant, you won’t be able to tell whether (a) you missed an effect or (b) no effect exists. There are several reasons why you might not be able to reject the null hypothesis, assuming Ho is false. Your experimental design may be flawed or suffer from other threats to internal validity. Internal validity refers to whether the research design enables you to measure your variables of interest rigorously. All aspects of your research may pose threats to internal validity, such as equipment malfunction, participants who talk during the experiment, or measures with low internal consistency (see Chapters 2 and 5). Low power is another threat to internal validity. Four factors are generally recognized as impacting the power of the study. In discussing these, David Howell (2013, p. 232) listed (1) the designated alpha level, (2) the true alternative hypothesis (essentially how large the difference between Ho and H1 is), (3) the sample size, and (4) the specific statistical test to be used. In his view, sample size rises to the top as the easiest way to control the power of your study. Alternative hypothesis: Hypothesis you have stated will be true if the null hypothesis is rejected. Power is associated with effect size (as defined in Chapter 2; Cohen, 1988), which is discussed next. Effect size describes what its label suggests: whether an intervention of interest has an impact or effect. Consider two means of interest (when Ho is true and when Ho is false) and the sampling distribution of the populations from which they were drawn. What is their overlap? If they are far apart and there is little overlap, you have a large effect size; if they are close together and there is a lot of overlap, you have a small effect size. Effect size is indicated by (d) and represents the difference between means in standard deviation units. Statistical programs generally have an option for providing estimates of both power and effect size, and authors are often asked to include an estimate of effect size in their manuscripts (Howell, 2013). In the literature, you will see descriptions of effect sizes as small (.20), medium (.50), and large (.80 and above). Jacob Cohen (1988) is usually the source cited. These three sizes represent different degrees of overlap: 85% (small), 67% (medium), and 53% (large). You can see that these percentages of overlap relate to the idea that there is a lot of overlap when the effect size is small and much less when the effect size is large. Without doing a power calculation, you can still get some sense of the sample size needed in your topic area (with implications for power) by reading the literature. More participants are needed if the effect size reported in your topic area is small. If no effect size is reported for your area of study, you could make a guess about whether you think it is likely to be small, medium, or large. Without information to the contrary, a conservative estimate (i.e., a small effect size) is probably prudent. Cohen (1988) has published tables that indicate how many participants you will need to detect a difference (i.e., to reject H o, assuming it is false) for a specific effect size. Power is discussed further in Chapter 9, including the use of an online power calculator. INTERNAL VALIDITY The validity of research refers to the degree to which the research evaluates what we claim it does. When we talk about the internal validity of research, we are talking about the degree to which the research was conducted in a manner that allows us to rule out alternative explanations; in other words, we are talking about the quality of the research process. Researchers are always vigilant to the occurrence of what are called threats to internal validity (additional types of validity related to measures are discussed in Chapter 5). A wellknown list of these threats was produced by the researchers Donald Campbell and Julian Stanley in 1963. Threats to internal validity: Factors that undermine the ability of your research to ascertain the influence of an independent variable (IV) on a dependent variable (DV). Table 3.1 presents these threats to internal validity discussed by Campbell and Stanley (1963). Let’s expand on each one of these threats and identify some others. In truth, some of these threats are beyond your ability to control, but at least you will be aware of them. History History is one of those threats you can’t control. Events in the world that occur during the course of research (e.g., terrorism or illness) may impact respondents’ answers to your surveys or problem-solving skills, as examples. If your sample is large enough, events specific to individuals are likely to be randomly distributed across conditions. In the case of terrorism or some other event that affects the population, you may not be able to tell whether the event interacted with and differentially affected the responses of one group relative to another, especially if the sample size is small. History: One of Campbell and Stanley’s (1963) threats to internal validity in which something happens between experimental treatments to influence the results. Maturation In the case of maturation, you have changes to participants that affect their performance. One aspect of maturation you can control is fatigue. If a long battery of measures is administered to participants (for example, of 60 minutes), they may lose interest, and their performance on the later measures would be different than if there were fewer measures. As a researcher, you can pilot test your questionnaire batteries and ask for feedback about fatigue. Solutions might be to administer the questionnaires in two sittings or to reexamine whether all of those surveys are, in fact, essential. Maturation: One of Campbell and Stanley’s (1963) threats to internal validity in which capacities of the participants may change as a result of fatigue, illness, age, or hunger that affect the intervention. Testing In testing, exposure to one instrument during a pretest or to an assessment that comes between a pretest and a posttest can change people’s responses to the posttest. To determine whether this is the case, some researchers use what is known as the Solomon four-group design (see Chapter 10), which evaluates what would happen with and without the pretest. This approach is expensive in terms of time and resources because there are four groups to run. Another option is to consider whether the pretest is needed. Testing: One of Campbell and Stanley’s (1963) threats to internal validity involving multiple testing situations in which the first test affects how participants respond to subsequent tests. Solomon four-group design: Pretest, posttest research design involving four conditions; takes into account the possible effect of sensitization in responding to the pretest measures. Instrumentation Instrumentation is a threat that is more easily managed, in principle. With regard to instrumentation, for example, the conditions of projecting images for participants to view, being prepared is the best course of action. Knowing how to use all of the equipment is important (e.g., what to do when you get a “no signal” message for LCD projection or what to have participants do when a survey link doesn’t load). Routinely checking calibration of equipment is advisable. Another possible threat to internal validity in the category of instrumentation involves your measures. Make sure you include all of your items, and make sure that your participants have looked at all the pages of the questionnaire, if you are administering a paper version. When administering questionnaires online, it is possible to prompt participants to check that they have answered all of the items they intended to answer; such prompts help cut down on missed items. Another kind of “instrument” is the researcher. If the researcher is giving task instructions, it is important to follow a script to make sure every participant receives the same information. Some researchers record instructions and other material delivered in spoken form to ensure that participants hear the same speaking voice, with the same pace. Instrumentation: One of Campbell and Stanley’s (1963) threats to internal validity in which changes in equipment and/or observers affect judgments/measurements that are made. Operational definitions: Describes a variable in terms of the processes used to measure or quantify it. Statistical regression: One of Campbell and Stanley’s (1963) threats to internal validity when participants are selected on the basis of extreme scores (e.g., high or low intelligence) and their scores move toward the mean on subsequent testing. Differential selection (biased selection of subjects): One of Campbell and Stanley’s (1963) threats to internal validity in which participants assigned to groups are not equivalent on some important characteristic prior to the intervention. Another kind of issue involving the researcher is the subjective evaluation of participants’ responses. Consider the situation where participants are giving responses to open-ended questions (i.e., questions where participants are free to answer as they wish and do not have preset categories from which to select) and the researcher is categorizing those responses. It is essential that the criteria for each category remain consistent across coders. One way this is accomplished is by creating clear operational definitions for each category. Operationally defining a variable is describing it in terms of the processes used to measure or quantify it. Imagine if researchers were categorizing qualities of the hospital environment in terms of Roger Ulrich’s (1991) theory of supportive design: positive distraction (PD), social support (SS), and perceived control (PC). If patients mentioned that having access to the Internet improved their experience, we would need an operational definition of each category to place the Internet in one of them. Is the Internet an aspect of positive distraction (something that redirects your attention away from worries and concerns), or is it an aspect of social support (a way to connect with others or encourage interaction)? Arriving at an operational definition can be challenging. OVERVIEW When thinking about research, one question that arises fairly early is whether there are rules or regulations that govern what you can and cannot study and how you can study it. The answer is “yes.” Regulations, starting at the federal level, have been developed to protect people who participate in research (and animals used in research), the researchers themselves, and the institutions they represent. There are also specific definitions of what constitutes research and what a human subject is. Research is essentially a systematic investigation designed to contribute to generalizable knowledge. A human subject is a living individual from whom the researcher gathers information or biospecimens through interaction or intervention or about whom the researcher has access to identifiable private information or identifiable biospecimens. A biospecimen from humans is material such as blood, tissue, urine, cells, or protein. These definitions of research and human subjects come from the federal regulations known as the revised Common Rule (45 CFR Subpart A 46.102[l] and [e][1], respectively), which is discussed in more detail later in the chapter. Not every study rises to the level of research (e.g., a study on food preferences in a single residence hall might not be designed to contribute to generalizable knowledge), but the ethical treatment of people who participate in studies is an important aspect, whether or not the study is technically “research.” After a lengthy review process that generated over 2,000 public comments in response to the proposed rule changes, the Common Rule has been substantially revised. The essential components of Subpart A of the revised Common Rule (also called the 2018 Requirements and the 2018 Rule), which went into effect on January 21, 2019, are presented in this chapter. Additional subparts (B–D) dealing with subjects in the revised Common Rule include Subpart B: Additional Protections for Pregnant Women, Human Fetuses and Neonates Involved in Research; Subpart C: Additional Protections Pertaining to Biomedical and Behavioral Research Involving Prisoners as Subjects; and Subpart D: Additional Protections for Children Involved as Subjects in Research. While this chapter focuses on the material in Subpart A, additional information regarding the protections for children involved in research will also be highlighted. Research: With respect to the federal definition (45 CFR 46), research involves a systematic collection of data with the goal of generalizable knowledge. Human subject: A human subject is a living individual from whom the researcher gathers information or biospecimens through interaction or intervention or about whom the researcher has access to identifiable private information or identifiable biospecimens. Revised Common Rule: Set of federal regulations that govern research with human subjects (also called the 2018 Requirements and the 2018 Rule). Deception: In research, when participants are not fully informed of the purposes and/or procedures; receives close attention in institutional review board (IRB) review. Many students have heard of Stanley Milgram’s (1963) research on obedience to authority, either in class or through the 2015 movie about the research (i.e., The Experimenter). In the research, participants are deceived, believing they are giving potentially lethal shocks to unseen “learners,” who are performing a word association task and are shocked for their incorrect answers. In fact, these learners are confederates, or collaborators of the experimenter, and no such shocks are being administered. Conducted after World War II, the Nazi war crimes were a clear motivation for Milgram’s studies; he mentioned them in the introduction to his paper about this research, published in 1963. Milgram questioned the extent to which ordinary people would potentially inflict harm on others when instructed to do so by someone in authority, in this case, an “experimenter.” Milgram’s (1963) research raised a host of questions, many of them about deception and the extent to which we can ethically mislead people in research. The point of the research was determining the extent to which people follow authority, but concerns were raised about the immediate and long-term effects on the participants, who believed they were inflicting potentially harmful levels of shock. Institutions are sensitive to such issues and have established boards that evaluate the ethical issues raised by research. In considering ethical aspects of research, this chapter will explain why these boards exist and how they work. This chapter will explain why we have ethical review boards (typically called institutional review boards or IRBs) and how to prepare a research proposal to undergo IRB review. Institutions that conduct research with humans and/or animals and receive federal funding to conduct or support that research are required to have IRBs and must comply with the revised Common Rule to determine that ethical guidelines are being followed. There are separate boards for humans and animals (i.e., infrahuman species). This chapter will concentrate on research with humans. Institutions where there is research on animals have a separate review committee for that research, often called the Institutional Animal Care and Use Committee (IACUC). Whether or not federal funding is involved, most institutions have IRBs for evaluating research with human subjects and/or with animals. Evaluating the ethical parameters of research protects all parties involved. In addition, if research is submitted for publication, most journals require certification that the research has undergone IRB review or its equivalent. Institutional Animal Care and Use Committee (IACUC): institutional committee that reviews research with animals and their care. The parameters of the research (most importantly the kinds of participants and the degree of risk involved in the research) determine the level of review required. The level of review makes a practical difference for the researcher. Meetings for the highest level of review require all IRB committee members, and those meetings are held infrequently. By learning about the different levels of review, you will be able to gauge how long the review of your proposal is likely to take. Materials included in appendices at the end of this book explain the information needed to submit a proposal for IRB review (Appendix B); Appendices C (Informed Consent) and D (Debriefing) provide you with sample documents that are required parts of research. Informed consent explains the nature of the research and states participants’ rights. By doing so, participants can make fully informed decisions about whether to participate. A debriefing or explanation of research document provided at the end of the study explains the specific hypotheses and aims of the study in more detail. If the study involves deception, the debriefing explains why that was necessary. Often, institutions provide templates for informed consent and debriefing documents on an institutional website along with other information about the IRB (e.g., committee meeting dates, timelines for review, and application materials). Informed consent: Document given to potential research participants that outlines the nature of the research; participants must agree and sign or otherwise provide evidence of consent in order to take part in research. Debriefing: Document given to participants at the conclusion of a research project that explains the hypotheses and rationale for the study. WHAT IS THE IRB, AND WHY DOES IT EXIST? Human subjects IRBs are charged with protecting the welfare of individuals who participate in research. An IRB has jurisdiction over the research proposed by any member of that institution. Regulations to protect human subjects in research emerged in the mid-20th century (Frankl, cited in J. G. L. Williams & Ouren, 1976). Since 1950, several pieces of legislation have targeted protecting the welfare of human subjects (e.g., the National Research Act in 1974) and established commissions to monitor research. One important commission is the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (Office for Protection from Research Risks [OPRR], 1993). Among other duties, this commission was charged with determining the risk–benefit criteria to evaluate research with human subjects (National Research Act, 1974, Section 202. B.2). The commission issued a report in 1979, known as The Belmont Report (named after the location where the meetings were held; https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report). The Belmont Report: Federal report issued in 1979 that outlines the principles that guide ethical treatment of research subjects; three major principles are respect for persons, beneficence, and justice. Title 45 Part 46 of the Code of Federal Regulations: Federal code that governs the protection of human subjects. Respect for persons: One of the three principles of The Belmont Report that emphasizes people are autonomous agents; specifies the use of informed consent. Beneficence: One of the three principles of The Belmont Report that emphasizes doing no harm (maximizing benefits while minimizing risks). Justice: One of the three principles of The Belmont Report that emphasizes the fair distribution of risks and rewards of participation. The report provided guidelines to help resolve ethical problems in evaluating research with human subjects. The regulations took on more significance when they were codified as Title 45 Part 46 of the Code of Federal Regulations by 1981 (Landrum, 1999; Pattullo, 1984). That code will be identified here as 45 CFR 46. The Belmont Report Published in the Federal Register in 1979, The Belmont report is “usually described as an ethical statement about research with human subjects” (Maloney, 1984, p. 21). Going beyond the ethical points in the Nuremberg Code, The Belmont Report offers direction in how to protect human subjects, including the following (Maloney, 1984): • • • • Assessing risk–benefit criteria Creating guidelines for selecting subjects Monitoring and evaluating the work of IRBs Creating informed consent guidelines for vulnerable populations (children, prisoners, those institutionalized with cognitive impairments) The Belmont Report is useful to read in its entirety; at the very least, researchers might read Part C: “Applications.” Three ethical principles underlie The Belmont Report: respect for persons, beneficence, and justice. These three core values are reflected in the way we do research (see Figure 4.1). Respect for persons is the leading principle, which addresses two core ethical guidelines. First, people are to be treated as autonomous agents. Autonomous agents are capable of acting independently—that is, making their own decisions and given the opportunity to do so. Second, when people cannot act as autonomous agents, for example, because of diminished capacity, they are provided with additional protections. How is this principle of respect for persons manifested in research? Through the process of informed consent; the elements of informed consent are covered fully later in the chapter. Informed consent is a document outlining what will happen in the research project and what protections are offered for participants. Autonomous agents: Part of respect for persons, one of three guiding principles in The Belmont Report, stressing that people need to participate voluntarily in research. The second guiding principle listed is beneficence. This word points to the importance of doing good, expressing kindness, and safeguarding well-being. Maloney (1984, p. 34) listed as extensions of this category “1) do not harm and 2) maximize possible benefits and minimize possible harms.” In this principle, there are links to what is known as the Nuremberg Code (discussed in the next section in the chapter). Being able to assess the risks and benefits provided by the research is an aspect of beneficence. The third principle is justice. Stated in The Belmont Report, the essence of this principle is “Who ought to receive the benefits of research and bear its burdens?” Another phrase used in the report framing this principle is “Fairness in distribution.” Risks and benefits should be equally distributed across participants. Thinking about the selection of participants and who takes part in the research is a central task related to the principle of justice. Institutions receiving federal funds for biomedical or behavioral research with human subjects must have an IRB according to federal law (National Research Act, 1974). The funds need not be for the specific research under review. In practice, many colleges and universities, whether or not they receive such funds, have established an IRB. As you might have guessed, there are a number of reasons for having an IRB: • • • Protecting the welfare of research participants Improving the quality of research proposals Addressing liability issues HISTORY OF ETHICAL OVERSIGHT Nazi biomedical experimentation on concentration camp prisoners during World War II and the judicial hearings of those events by the Nuremberg Military Tribunal led to the creation of the Nuremberg Code. These events are often listed as the reason the United States established its own code to protect human subjects. For example, The Belmont Report specifically mentions the Nuremberg War Crimes Trial and subsequent establishment of the Nuremberg Code. The substance of the 10 points in the Nuremberg Code is as follows (http://wayback.archiveit.org/4657/20150930181802/http://www.hhs.gov/ohrp/archive/nurcode.html): Nuremberg Code: Ten guidelines for the ethical treatment of human subjects in research that were codified in 1949; emerged out of the Nazi War Crimes Tribunal in Nuremberg. • • • • • • • • • • Voluntary consent is critical The results should be beneficial to society and not procurable in any other way A knowledge base from animal studies or the literature should suggest that the likely results justify the experiment No unnecessary physical or mental suffering or injury is involved No research should occur if there is a priori knowledge that death or disabling injury may occur The degree of risk should never outweigh the importance of what is to be learned Every precaution should be taken to prevent even the remote possibility of injury, disability, or death Only scientifically qualified individuals should conduct experiments Subjects should have the right to stop participating if they have reached a physical or mental state where they judge it impossible to continue Scientists must be prepared to stop the research if in their judgment continuing presents the risk of injury, disability, or death to the subject Tuskegee syphilis study: Study of African American men with syphilis that demonstrates violations of ethical principles because the men were left untreated even when a treatment was available; sponsored by the U.S. Public Health Service from 1932 to 1972. These stipulations may strike you as emphasizing the risks associated with medical research, but research in the social and behavioral sciences has also raised concerns about the safety and protection of human subjects. For example, The Belmont Report highlights the Tuskegee syphilis study: In this country, in the 1940s, the Tuskegee syphilis study used disadvantaged, rural Black men to study the untreated course of a disease that is by no means confined to that population. These subjects were deprived of demonstrably effective treatment in order not to interrupt the project, long after such treatment became generally available. (The Belmont Report, B.1.3; http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.html#xethical) This study lasted from 1932 to 1972, although by 1947, penicillin was known to be an effective treatment for syphilis. This research was sponsored by the U.S. government (the U.S. Public Health Service), but no formal amends for this unethical undertaking appeared until 1997 when President Bill Clinton offered an apology, calling this episode part of the country’s “shameful past” because the men in the Tuskegee study participated without knowledge and consent. These African American men thought they would receive free medical care when, in fact, the real purpose of their participation was to chart the course of syphilis. Thus, not only did this study violate fundamental principles about informing participants about the nature of the research, but in addition, the undertaking was racist in that the burden of participation fell to one racial group. Numerous resources exist about the Tuskegee study, including the Centers for Disease Control and Prevention (CDC) website (http://www.cdc.gov/tuskegee/index.html), which has links covering the timeline of the study, the presidential apology, research information, and information about syphilis as a disease. This CDC website also offers links to external websites, such as the National Archives (which has records from the Tuskegee study) and Tuskegee University’s own National Center for Bioethics in Research and Health Care (http://www.tuskegee.edu/about_us/centers_of_excellence/bioethics_center/about_the_usp hs_syphilis_study.aspx). Other recommended resources include the book Bad Blood (Jones, 1981), which traces this research sponsored by the U.S. Public Health Service, and the television movie Miss Evers’ Boys (Kavanagh & Sargent, 1997), which depicts the (fictitious) experiences of a nurse and four of the participating tenant farmers. Milgram’s Obedience to Authority Study Another well-known case in the United States that involves ethical issues (and the one mentioned near the beginning of this chapter) is the work of Stanley Milgram (1963) on obedience to authority. Milgram’s later research (1974) highlighted the importance of questioning what is ethical research and when deception in research is appropriate (Fisher & Fyrberg, 1994). As many of you may recall from hearing about this study in introductory social science courses, the experiment at Yale University consisted of people in two roles: teachers or learners. In fact, the learners were all confederates in the study, which was stated to be on learning and memory. Only the teachers (adults recruited from the surrounding New Haven, Connecticut, community) were participants. The basic question in the study was the extent to which people (the teachers) were willing to administer “shocks” to the “learners” when learners answered a question incorrectly. Even after hearing a cry from the learner, more than 80% of the teachers continued administering shocks when a question was incorrectly answered. Unbeknownst to the teachers, there were no actual shocks, and the learners’ cries and protestations to stop were staged. It is unlikely that the identical circumstances of Milgram’s (1974) research would receive IRB approval today (Gillespie, 1999), but a modification of Milgram’s research, including setting the top “shock” level at 150 volts (in Milgram’s original research, it was 450 volts) was approved (Burger, 2009). Readers interested in the safeguards Burger took in this IRB-approved replication are directed to his 2009 article. Zimbardo’s Prison Simulation Related to the obedience to authority research and receiving renewed public interest is the work of Philip Zimbardo (Haney et al., 1973) on what is known as the Stanford prison experiment, which marked its 40th anniversary in 2013. Zimbardo cited Milgram’s (1963) work on obedience to authority as an influence on the Stanford prison study (Sparkman, 2015). In 2015, a film called The Stanford Prison Experiment premiered at the Sundance Film Festival (http://www.prisonexp.org). A focus of Zimbardo’s research was the extent to which the personality versus the context of the setting and its social roles influenced behavior. The experimenters (Haney et al., 1973) approached this prison research through a simulation, creating a prison in the basement of one of the Stanford University buildings (home of its psychology department), recruiting volunteers for the role of prisoner or guard, screening them for pathology, providing them with role-consistent attire (i.e., uniforms for the guards and prison clothing for the prisoners, but without underclothing), and then watching their behavior unfold. Because some prisoners were judged to have become acutely emotionally agitated, they were released early from the two-week-long simulation. The entire project was halted on Day 6 after Christina Maslach, a newly minted Stanford PhD, raised concerns about the emotional suffering exhibited by the “prisoners.” Only then did Zimbardo, who was fulfilling two roles, one as researcher and the other as superintendent of the “prison,” stop the simulation. Social roles in this physical setting appeared to have had a substantial impact on participants’ behavior. Stanford prison experiment: Research conducted by Zimbardo and colleagues showing the effect of obedience to authority in a simulated prison environment (Haney et al., 1973). Kennedy Krieger Institute Lead Paint Study The Milgram (1963) and Zimbardo (Haney et al., 1973) research took place decades ago, but a more recent example shows the need for ongoing vigilance to protect human participants. The research in question involved children in what is often called the Kennedy Krieger Institute Lead Paint Study, conducted in Baltimore, Maryland, under the auspices of the Johns Hopkins University (D. R. Buchanan & Miller, 2006). Funded by the Environmental Protection Agency, the focus of the Lead-based Paint Abatement and Repair and Maintenance Study was the effectiveness of different steps to rid housing of the toxin lead. Families with children in this study were recruited to live in housing with various levels of lead abatement; thus, some families were enticed to live in housing where lead was present. A justification for exposing children to this potential risk was that these were children who likely already lived in or eventually would live in housing where the children would be exposed to lead. This situation ostensibly existed because of the high percentage of housing stock in Baltimore that contained lead-based paint. Kennedy Krieger Institute Lead Paint Study: Research in Baltimore, Maryland, conducted by the Johns Hopkins University and funded by the Environmental Protection Agency (EPA) that exposed some children to lead paint dust. There are many aspects of this study, and rulings were rendered in the trial of Grimes v. Kennedy Krieger Institute (2001). Children in the study were all low income, some lived in housing where only partial lead abatement was undertaken (hence, exposing the children to risk), and incentives were given to encourage families to participate. This research has been referred to as nontherapeutic in that it offered “no prospect of direct benefit” to the children (Glantz, 2002). Two parents who argued that information about the levels of lead paint dust in their homes was not provided in a timely manner brought lawsuits. The Court of Appeals reversed the original lower court ruling against the plaintiffs (the parents) and stated, “in Maryland, a parent, appropriate relative, or other appropriate surrogate, cannot consent to the participation of a child or other person under legal disability in nontherapeutic research or studies in which there is any risk of injury or damage to the health of the subject” (cited in Glantz, 2002, p. 1071). This ruling should remind you of the safeguards stipulated in the Nuremberg Code. In fact, in his article about this case, Leonard Glantz (2002) pointed out that the ruling marked the first time in the U.S. court system that the Nuremberg Code had “so explicitly” been adopted “as a source of legally enforceable ethical standards” (p. 1071). Although this research took place well over a decade ago, the Kennedy Krieger Institute is still facing “lead-paint” lawsuits related to the research (Wheeler & Cohn, 2014). The point demonstrated by just these few examples is that research needs to be reviewed and monitored to ensure that it meets ethical standards. THE APA CODE OF ETHICS Before moving on to concrete aspects of ethical research and IRBs, one more ethical code will be described: the Ethical Principles of Psychologists and Code of Conduct from the American Psychological Association (2002, with amendments in 2010 and 2016), also known as the American Psychological Association (APA) Code of Ethics. The APA has published an ethics code since 1953. In the most recent document (effective January 1, 2017), there is a preamble, general principles, and 10 standards. Here, the focus is on the standards that pertain to ethical issues in research. As general principles, the code overlaps with The Belmont Report in five principles: Beneficence and Nonmaleficence (do no harm or inflict the least harm); Fidelity and Responsibility (operate within your areas of competence); Integrity; Justice; and Respect for People’s Rights and Dignity. Of the standards, Standard 8 (Research and Publication) is the most applicable to issues of ethics in research. Aspects covered in the subsections include informed consent, inducements, deception, debriefing, reporting, and publication credit. The list of items that should be included on an informed consent document is important to read. In conjunction with stipulations in the revised Common Rule, you could use this list to develop an informed consent document: American Psychological Association (APA) Code of Ethics: Ethical code of conduct guiding the behavior of psychologists; comprises 10 standards. • • • • • • • • • The purpose of the research, expected duration, and procedures The right to decline to participate and to withdraw from the research once participation has begun The foreseeable consequences of declining or withdrawing Reasonably foreseeable factors that may be expected to influence their willingness to participate, such as potential risks, discomfort, or adverse effects Any prospective research benefits Limits of confidentiality Incentives for participation Whom to contact for questions about the research and research participants’ rights Opportunity for the prospective participants to ask questions and receive answers (https://www.apa.org/ethics/code/ethics-code-2017.pdf; 8.02) OVERVIEW As a central theme of this book, it is important to remember that the research question guides the research design. This chapter emphasizes the kinds of research questions that are appropriate for a correlational approach. In describing approaches to research design, Chapter 3 provided an overview of correlational research; this chapter will provide more depth on the topic and will include more information about statistical approaches. A secondary emphasis is specialized nonexperimental approaches; in nonexperimental research, there is no manipulation of variables or random assignment. Included in this chapter are such approaches as time-series analysis, longitudinal design, cohort-sequential design, and cross-sectional design. Longitudinal, cross-sectional, and cohort-sequential designs are sometimes referred to as developmental designs. While these designs do not typically involve the manipulation of variables, there are instances, for example in longitudinal research, where an intervention may be added (and the chapter will include such an example). By the end of this chapter, you will have a good sense of when and how these approaches are used in the social and behavioral sciences. CORRELATIONAL RESEARCH: GENERAL CHARACTERISTICS The hallmark of correlational research is the study of relationships among variables. In correlational studies, researchers are interested in the extent to which changes in one variable are associated with changes in another. These variables can represent categories, such as class year or gender, or they can represent quantities, such as scores on a scale. In the case of categories, we could, for example, study the relationship between being a first- or fourth-year student and living on or off campus, predicting that first-year students will live on campus whereas fourth-year students will live off campus. We can measure the strength of these relationships, that is, the proportion of times these predictions were correct in our sample. In the case of these categories, there is no continuum of membership; one either is or is not a first- or fourth-year student; one either lives on or off campus. We could also approach questions with “quantities,” that is, using scores on a scale. For example, we could measure the relationship between students’ self-reported household income and the amount of student debt they will have at graduation. In the case of quantitative variables, we are now able to say something “more” about the relationship because we can say whether the variables relate to each other positively (e.g., that as household income goes up, student debt goes up) or, as is more likely the case, negatively or inversely (as household income goes up, student debt goes down). Thus, with quantitative variables it is possible to say something about value and direction in correlational research (Singleton et al., 1988). Nonexperimental research: Research in which there is no manipulation of variables or random assignment. Time-series analysis: Analysis of data points collected repeatedly over time. Longitudinal design: Research design in which the same participants are followed over a long period of time (e.g., Terman’s Termites). People sometimes equate correlational research with the statistics of correlations (e.g., Pearson’s r), but correlational research is an umbrella term that refers to a range of approaches in which researchers examine whether two or more variables covary and no variables are manipulated; in addition, a range of statistics, not simply Pearson’s r, are used to assess the associations that exist. WHAT CORRELATIONAL DATA CAN TELL US Correlational approaches have distinct strengths, and there are numerous situations in which correlational research is the preferred if not the only approach. While one often reads that correlational research is a good approach at the beginning (italics added) of an investigation into a topic area to see if relationships do, in fact, exist (and the strength of their association), that statement undervalues the contribution that correlational research provides. First, correlational research is suited to studying naturally occurring behavior, often on a large scale, which occurs without any intervention or manipulation of variables. For example, correlational research can help examine such topics as the relationship between crime and temperature (e.g., Mares & Moffett, 2019), often using large data sets (the FBI’s Uniform Crime Report and historical weather data from the Global Historical Climatology Network). In this study by Dennis Mares and Kenneth Moffett, “data from 1,087 weather stations were combined with data from 18,297 law enforcement agencies over a period of 648 months” (p. 507). Second, correlational research can handle the simultaneous measurement of a vast number of variables. We see this in large surveys such as the European Social Survey (ESS), a crossnational survey of more than 30 nations, which takes close to 1 hour to administer in “British English” according to the ESS website (https://www.europeansocialsurvey.org/about/faq.html). The ESS and similar surveys are valuable in tracking changes in attitudes and values across Europe (and elsewhere) on such topics as social trust, politics, religion, perceived discrimination, subjective well-being, immigration, media use, Internet use, sociodemographic characteristics, and a variety of other issues. The data are generally free for noncommercial use and provide an incredible resource to understand people’s lives. Cohort-sequential design: Research design that combines both cross-sectional and longitudinal components; cohorts are selected at different points in time and followed longitudinally; used to address drawbacks to cross-sectional design. Cross-sectional design: Correlational approach, typically when different populations are measured at the same point in time (e.g., first year students and fourth year students). Third, there are innumerable situations where you cannot manipulate variables of interest, either ethically or practically, and instead decide to assess attitudes, beliefs, and/or behaviors related to the topic. Researchers often use correlational approaches to study variables that are difficult to manipulate. For example, you might be interested in the relationship between gambling (and perhaps more specifically problem gambling) and college student age (18–22), wondering whether older college students are likely to exhibit more problem gambling behaviors because they have more independence from parental supervision than do younger college students. You can neither manipulate age nor current problem gambling. The legal age for gambling varies by state (and type). In Connecticut, you have to be 18 to buy a lottery ticket or wage a pari-mutuel bet (e.g., for horseracing), but you have to be 21 to gamble in casinos (this requirement is typically linked to drinking age). Some standardized instruments are available to measure gambling behavior. The South Oaks Gambling Screen (SOGS) is one such well-known scale (Lesieur & Blume, 1987). Because college students in southeastern Connecticut have two major casinos (Foxwoods and Mohegan Sun) less than 10 miles apart, a correlational study of realworld gambling behavior and students’ age is a workable idea for a research methods project at colleges in southeastern Connecticut. True, you could study gambling behavior in the laboratory, but it might have limited ecological validity (refer to Chapter 3). If you want to try to understand whether students who live close to casinos have a gambling problem as measured by their score on the SOGS (which indicates levels of gambling severity) and further whether there is a correlation between student age and score on the SOGS, then a correlational route makes sense. CONSIDERATIONS OF INTERNAL AND EXTERNAL VALIDITY IN CORRELATIONAL RESEARCH As discussed in Chapter 3, while there is no manipulation of variables or intervention as part of correlational research, internal and external validity are still concerns. Internal validity is the ability of the research design to test the hypotheses adequately. External validity is the ability to apply the research results more broadly. As an example of concerns about internal validity, consider the European Social Survey (ESS) (mentioned earlier in this chapter and also in Chapter 3 when survey research was described). Because the survey questions are answered by people who are individually interviewed, the training of the interviewers and the execution of the interview is obviously important in the integrity of the responses (i.e., the data collected). On the ESS website, a working paper is mentioned that documents various ways possible data falsification might be addressed. The ESS website also states how the Core Scientific Team is working to prevent data falsification, including “extensive pretesting of survey questions, guidelines for briefing interviewers, for selecting and recruiting respondents, and for monitoring fieldwork, analyses of the process of obtaining interviews and of the answering process during interviews” (https://www.europeansocialsurvey.org/about/singlenew.html?a=/about/news/essnews0065. html). There are a variety of strategies to combat data falsification in such interviews (which is typically defined as intentional departure from the prescribed interviewer procedure). The strategies for the ESS include back checks for 10% of the respondents in each country (in a back check, the respondent is re-interviewed, either in person or by phone, often with a subset of the original questions). Computer-assisted personal interviewing is also utilized, providing a time stamp of the interaction, which would allow researchers to analyze the speed of interviews and further investigate those that were “fast,” perhaps reflecting poor interviewing techniques (such as skipping the introduction). Future approaches mentioned on the website might include GPS tracking of interviewers or audio-recording, although the latter might be off-putting to some respondents. You can see that there are many aspects of correlational research where validity might be threatened. Care in the development and implementation of surveys and interviews, among other steps, can help to address these problems that undermine the quality of the research. DRAWBACKS TO CORRELATIONAL APPROACHES Correlational approaches have advantages, for example, the ability to study naturally occurring behavior, but there are some distinct disadvantages. In addition to the lack of ability to infer causality from using a correlational approach, there are two related issues: third variables and directionality. Third variables were introduced in Chapter 2 but are reviewed here because of their importance. Directionality: In correlational design, inability to determine the direction of cause and effect. The Third Variable Problem A third variable is an example of a confounding variable, that is, a variable that influences the associations between measured variables of interest leading you to infer incorrectly a relationship between your variables of interest. Continuing the gambling example, imagine there is a significant relationship between students’ age and gambling (such that older students participate in more gambling than do younger students). We have a confounding variable in this situation because students’ maturational age is associated with their legal age for gambling (21); in other words, there may simply be more gambling among older students because there legally can be, not because being older is associated with particular maturational processes that are related to gambling. In this example, we were unable to measure legal gambling behavior in younger students—that is, our unmeasured third variable. To evaluate that issue, we would need to conduct the study in a location in which gambling could legally occur at a younger age (for example, Canada). In other words, we would need to take into account this problematic variable of legal gambling age. Confounding variable: Extraneous variable that is associated with both the independent and the dependent variable and undermines the researcher’s ability to pinpoint causality. The Problem of Directionality In experimental research, there is a claim of cause and effect, and typically the independent variable, which is manipulated, is the presumed cause and the dependent variable, the outcome, is the effect. In correlational research, there is no manipulation; hence, there is no effect. In a relationship that appears, we have no way of knowing which direction it flows, that is, which variable is more likely to be a cause or which variable is more likely to be an effect. The following example from Ann Devlin (2006, pp. 50–51) clearly shows this difficulty. My colleague used this example in teaching the relationship of cause and effect to correlation. A graduate student of this professor found a relationship between conflict in a marriage (from questionnaire answers) and aggression of the couple’s child documented through playground observations. Three possible causal paths exist for this relationship (see Figure 7.1; from Devlin, 2006, p. 51): • • • “A (marital conflict) causes stress in the child (or is imitated by the child), B. Or, a temperamentally difficult child (B) causes marital conflict (A). Or, poverty or another stressor (C) causes both A and B.” Multiple causal explanations are available for a correlation, and the approach cannot tell you which specific cause and effect. When you choose a correlational design, make sure your research question is appropriate. You must manipulate an independent variable to determine cause and effect. A Word About Causality Some researchers argue that causal inferences should not be dismissed out of hand in nonexperimental research. Advances in statistics, such as those in structural equation modeling (SEM) make such arguments possible. These statistical techniques are well beyond the scope of this book, but it is worth emphasizing the concept of modeling in the SEM label. The techniques allow mathematical modeling of the various ways in which variables can transmit changes to other variables (the idea of transmitting changes, rather than talking about effects, was outlined in Judea Pearl’s [2010] useful article). Importantly, Pearl (2010) reminds us that the logic and soundness surrounding the causal relationships you are testing is more important than the statistics: “Every claim invoking causal concepts must rely on some premises that invoke such concepts; it cannot be inferred from, or even defined in terms of statistical associations alone” (p. 2). Thus, we return to R. Kaplan’s (1996) reminder that conceptualization is at the core of research worth doing (see Chapter 2). For further reading, David Kaplan (2009) has a useful chapter outlining the importance of causal inference in nonexperimental educational policy research. The author talks about this issue in the context of an important social initiative (the No Child Left Behind Act) and further presents a case about the fallacy of the argument that randomized experimental designs necessarily represent a “gold standard” in terms of establishing causality. Particularly in an area of research such as education that is exceedingly complex, the author argues there is an important role for nonexperimental work and outlines the contributions from the econometric approach (path analysis and structural equation modeling) that permit the argument of causal inference. Helpfully, the chapter also includes a brief overview of the early historical foundation of causation starting with David Hume before moving on to the experimental design tradition including John Stuart Mill, and then Donald Campbell and Julian Stanley, and others (also see Chapter 3). CORRELATIONAL DESIGN: QUASI-EXPERIMENTAL DESIGN (I.E., QUESTIONS ABOUT GROUPS) Correlational research can take the form of posing questions about the sample as a whole; it can also take the form of posing questions about differences between preexisting groups (e.g., gender). If you ask questions about group differences in terms of some relationships of interest but use preexisting groups based on demographic characteristics, such as gender, or affiliation, such as being a varsity athlete, you are still using a correlational design. This approach is called quasi-experimental, which was introduced in Chapter 3. The statistics to analyze your data will be identical to those used if you formed the groups of interest through manipulation, but the fact remains that you have no intervention or manipulation; hence, you have no opportunity to assess causality. Even in a design that has an experimental component (that is, where you manipulated one or more variables), you may include one or more quasi-experimental variables such as gender. Then, when you analyze your results, you can make causal statements related to the manipulated variable but you must use the language of association when you talk about the correlational variable. As an example, imagine you did the study mentioned in Chapter 2 “Perceived Femininity of Women Weightlifters” in which participants are randomly assigned to view a photograph of a woman lifting a 5-lb. (80-oz.) weight or the same woman lifting a 25-lb. (400-oz.) weight. Weight is your experimental variable with two levels, 5 lb. or 25 lb. Participants are asked to judge the sex role characteristics (specifically the femininity and masculinity of the woman lifting the weight, obtained through responses to the Bem Sex Role Inventory [BSRI]; see Chapter 5); those are dependent variables. At this point, this study qualifies as a true experiment (manipulated variable; random assignment to condition). But imagine that your participants came from a participant pool that included both varsity athletes and students who were not on sports teams and you wanted to take that variable into consideration. If you did so, you would have one true IV (weight lifted) and one quasi-IV (athlete status; see Figure 7.2). Now, we have a study with one true IV and one quasi-IV (the correlational aspect). Imagine your results showed that judgments of masculinity were significantly higher for the woman lifting the 25-lb. weight than the 5-lb. weight. Also imagine that nonathletes judged the woman to be significantly more masculine, over conditions, than did the athletes. In your Discussion section regarding the effect for weight lifted, you could talk about the picture having affected or influenced people’s judgments of masculinity; nevertheless, with regard to the difference between the perceptions of masculinity exhibited by athletes and nonathletes, you would be limited to talking about athlete status as being associated with judgments of lower masculinity. Why? In this research, you did not randomly assign participants to their status of being athletes or not. Participants arrived with those characteristics. STATISTICS USED IN CORRELATIONAL DESIGNS Correlation analysis: Statistical approach that assesses the relationship between two variables, typically interval scale. Regression analysis: Estimates the ability of a variable (called a predictor) to predict an outcome (called the criterion). The types of statistics used in correlational designs are linked to whether the research addresses questions about the (a) sample as a whole or (b) groups. Sample as a Whole In the case of the sample as a whole, there are two typical approaches: correlation analysis and regression analysis. Ranked data can also be used to address questions about the sample. Ranked data will be discussed after correlation and regression. In the case of correlation, you are asking whether two measures given to the participants are statistically correlated to each other. To answer this question, you would typically use Pearson’s r. As an example, let’s assume we are interested in the relationship between participants’ self-esteem and their attitudes toward disordered eating. We might have given all participants Morris Rosenberg’s Self-Esteem Scale (Rosenberg, 1965), a widely used 10-item scale to assess self-esteem and the Eating Attitudes Test (Garner & Garfinkel, 1979), a widely used measure to assess disordered eating; the original version of the scale has 40 items. Using statistical software such as SPSS Statistics, we would correlate the scores on the two measures for the entire sample to produce a Pearson’s r value, which would indicate the degree of association between the two measures. Pearson’s r: Statistical measure of correlation between two variables. If we wanted to see whether scores on the self-esteem scale predicted those on the disordered eating scale, we would use regression analysis, in this case, simple regression because we have only one predictor, self-esteem. The analysis produces a regression coefficient, b. We could also have asked the question the other way around: whether scores on the disordered eating measure predict those on the self-esteem measure; a reasonable theoretical case could be made for either predictor. When we move to multiple regression, we add more predictors (you need at least two for multiple regression) and ask whether each of these variables predicts the outcome (called the criterion in both simple and multiple regression) of interest. In addition to the measure of disordered eating, we might have added grade-point average (GPA) and have asked to what extent GPA and self-esteem predict disordered eating, as well as whether one of those predictors is stronger than the other. In multiple regression, researchers often use the standardized regression weight, beta (β), because it employs a common unit across measures that have different units (that is, it standardizes them), whereas B, the unstandardized regression coefficient, is given in the units associated with that particular measure. Predictor: In regression analysis, the variable being used to predict the criterion (outcome). Criterion: Outcome measure of interest (e.g., in regression analysis). Standardized regression weight: Beta (β) employs a common unit across measures with different units (i.e., it standardizes them). Unstandardized regression coefficient: Coefficient (B) that employs the units associated with the particular measures. OVERVIEW This chapter will focus on qualitative research. Qualitative research emphasizes that the whole of human experience cannot be adequately represented through quantification, that is, through the measurement of variables. Qualitative approaches such as interviews, focus groups, and case studies provide opportunities for in-depth investigation with an emphasis on the whole of the experience as a means to understanding behavior. QUALITATIVE RESEARCH As its label suggests, qualitative research is characterized by a de-emphasis on quantitative analysis (i.e., the reduction of data to numbers), although there may be instances where quantitative information is added. While qualitative research is sometimes referred to as nonexperimental research, that label is a categorization by exclusion rather than inclusion. This chapter seeks to help students understand the unique contributions that can be made through qualitative research. Qualitative research takes a variety of forms, but there is an emphasis on understanding behavior and subjective experience as it occurs, without having intervened in the research. In some cases, there are no live participants (e.g., archival research and physical traces); in other cases, the participants are present but the researcher has varying degrees of involvement (e.g., participant and nonparticipant observation); in still other cases, there may be a high degree of interaction with participants (e.g., interviews, focus groups, and case studies; see Figure 8.1). QUALITATIVE RESEARCH AND THE CONCEPT OF REFLEXIVITY Qualitative research: In-depth investigation of topics using techniques such as focus groups, interviews, and case studies; emphasis on the whole of the experience. Nonexperimental research: Research in which there is no manipulation of variables or random assignment. Archival research: Research based on existing records such as newspapers, medical records, yearbooks, photographs, audio- or video recordings, or any unpublished or published materials that can be evaluated. Physical traces: Leftover physical elements (e.g., trash) in the environment that can be used as sources of data for research. Participant observation: Type of behavioral observation in which the observer is a member of the group being observed. As discussed in Chapter 3, the role of the researcher has to be considered in research, no matter the approach. In qualitative research, an important characteristic of researchers is the concept of reflexivity, in which researchers reflect on their experience as part of the research process. For researchers, this concept of reflexivity means reflecting on their role and reactions to the research process. Researchers need to consider their potential for bias given that they may have different status and power than the people in the communities they seek to understand (Nagata et al., 2012, p. 261). David Rennie (1999) saw reflexivity as one of the main differentiators of qualitative research from the tradition of positivism, where the subjectivity of the experimenter (in theory) is eliminated from the research process. As a philosophical approach, positivism asserts that rational claims about behaviors can be scientifically verified. ACCEPTANCE OF QUALITATIVE METHODOLOGY IN THE SOCIAL AND BEHAVIORAL SCIENCES It is important to consider the standing of qualitative research. Although disciplines such as anthropology place heavy emphasis on qualitative approaches (e.g., ethnographic research), some branches of social science such as psychology have a complex relationship with qualitative research. In the social sciences generally, there is increasing acceptance of qualitative methodology, reflected in the emergence of journals that exclusively publish qualitative research (e.g., Qualitative Research, begun in 2000) or publish work that combines quantitative and qualitative methods, called mixed methods (e.g., Journal of Mixed Methods Research, begun in 2007). Nonparticipant observation: Observing participants without taking part in the activities. Interviews: Qualitative research approach in which questions are asked of an individual; types are structured, unstructured, and semistructured. Focus groups: Form of qualitative research in which people are asked their views on a target issue. Case studies: Research approach for in-depth exploration of an event, program, process, or of one or more individuals. Reflexivity: Aspect of qualitative research in which researchers reflect on their experience as part of the research process. Positivism: Philosophical approach to science that stresses information gained through the senses (direct experience). At the same time, there has been resistance to embracing qualitative methods in psychology in full. In an article titled “The Qualitative Revolution and Psychology: Science, Politics, and Ethics,” Frederick Wertz (2011) explained the history and sources of resistance to accommodating qualitative methodology in psychology. The use of the word “politics” in the subtitle of the article highlights one of those sources. Wertz and others have pointed out that although psychology has a history of qualitative research (think of Sigmund Freud’s case histories or the psycholinguistic study of Genie by Susan Curtiss [1977]), qualitative methods “have been consistently devalued and marginalized in psychology” (Wertz, 2011, p. 77). The greater influence, Wertz (2011) explained, is psychology’s alignment with the natural sciences and the hypothetico-deductive method. The hypothetico-deductive method is commonly called the scientific method, in which testable hypotheses are formulated in a way that could be falsifiable. With this alignment with the natural sciences and the scientific method comes funding and power. “The qualitative revolution, inasmuch as it risks transforming psychology’s disciplinary identity, threatens the discipline’s cultural power and economic wellbeing” (Wertz, 2011, p. 86). In psychology, to the extent that qualitative methods have become more acceptable and the critiques of positivism more pronounced, this change has been linked to recognition that “the great experiment of applying natural science methods to complex social problems was a failure” (p. 88). Qualitative methods are integrated into a variety of disciplines, including cultural anthropology, education, social work, nursing, history, political science, and theology, among others. As pointed out in an article dealing with infertility care (Hammarberg et al., 2016) and echoing a theme in this book, it is the research question (i.e., when to use qualitative methods) that determines their appropriateness. The areas in psychology that have been most receptive to qualitative methods are “professional practice, women’s issues, and multicultural concerns” according to Wertz (2011, p. 101), which may reflect the traction of qualitative approaches among people not previously well integrated into the discipline, he argued. Other data reinforce this pattern that qualitative and mixed methods approaches have been more widely adopted in applied domains such as nursing and education than in pure domains (psychology, sociology) and are less likely to appear in what are described as “elite/prestigious” journals (Alise & Teddlie, 2010, p. 121). Mixed methods: Approach to research that typically combines quantitative and qualitative approaches. Hypothetico-deductive method: What people refer to as the scientific method in which hypotheses are formulated that can be tested in a potentially falsifiable manner. Qualitative data are rich; they often provide insights into human behavior that are unavailable when people respond to standardized scales and are limited to close-ended questions (refer to Chapter 5). The increasing use of mixed methods, combining quantitative and qualitative approaches, may create a fuller understanding of the human condition. QUALITATIVE APPROACHES TO RESEARCH There are a variety of approaches to qualitative research. The chapter will start with those with little involvement of the researcher and move to those where the researcher is deeply involved in the data collection. Archival Research and Document Analysis In the examples of cor...
Purchase answer to see full attachment