4 Strategies for Reviewing the Literature Learning Objectives After reading and studying this chapter, students should be able to do the following: • Appreciate both the depth and breadth of social science research and how these dimensions influence research and literature reviews. • Use and apply several informational search strategies, including how to find key journal articles, locate and judge information from the Internet, and leverage the expertise of local experts and previously published works. • Filter and sift through volumes of information using the tools available to improve one’s precision in completing efficient database searches, making reasoned judgments about material to include and exclude in research papers. • Utilize some of the advanced database search tools available to optimize the identification of research reports relevant to the topics being reviewed. Lan66821_04_c04_p051-070.indd 51 4/24/12 9:34 AM Section 4.1 Locating Resources CHAPTER 4 O nce a topic is identified and honed into a workable thesis, research begins. The ability to analyze information from a variety of complex sources, synthesize or summarize that information, and observe common themes is an incredibly valuable skill in college. It is also necessary to employers and for advancement within a profession. As has been mentioned in prior chapters, in the social sciences, researchers do what is called a “literature review” or “literature search” to determine the ideas that have already been made public or vetted by scholars in the field. These ideas could be personal opinions or anecdotal reports, but researchers prefer scholarly, peer-reviewed literature and learned opinions from experts in the field. The process of reviewing previous research allows a writer to set the theoretical foundation for his or her topic. When students review the literature for their assignment, ideally, each aspect of the literature review is related to specific components of the research question or thesis statement. The author presents past research outcomes and guides the reader through past results in order to provide a context or orientation for the research that is still needed. Depending on the assignment, the literature review may end with recommendations for future research, or if the literature review is a part of an empirical project, this section sets the stage for the research to be conducted and describes how those outcomes will help inform what is known about a topic. When crafting a literature review, a scientific writing approach relies on the presentation of facts, theories, and hypotheses in an impartial and objective manner. Scientific writing generally strives to avoid the personal opinions of the writer and attempts to test the validity of ideas. A major component of this approach is the ability to incorporate evidence—specifically, research-based evidence—into the writing product. Because of the unique nature of studying the social sciences, a social scientist might take a broad approach, which can span many disciplines, or a narrower approach, which can lead to very specialized work more applicable to one discipline than others. Whatever approach the writer chooses ultimately affects the nature of the literature search and the type of capstone paper or project. 4.1 Locating Resources A student must then find the appropriate literature to research for his or her topic. Local public and university libraries—as well as a careful, discerning Internet search (more on this in the next section)—are all prospective options, but Ashford University also offers a number of convenient resources for the online student, including an e-book database called ebrary. In literature reviews, journal articles are commonly utilized and summarized, but other types of resources can provide valuable academic contributions as well: books, monographs, recorded interviews, book chapters, web sites, and so on. Later in this chapter the criteria by which to evaluate the credibility of information (retrieved online or otherwise) are presented. The Online Library and Other Online Options Many institutions, including Ashford University, pay for institutional access to online databases that help search what is available and then help locate the materials for direct Lan66821_04_c04_p051-070.indd 52 4/24/12 9:34 AM CHAPTER 4 Section 4.1 Locating Resources access. These databases are housed in Ashford’s online library (login via the Student Portal) and include electronic books, journals, and other types of articles. Social science students will likely use EBSCO, JSTOR, ProQuest, Project MUSE, and ebrary most often—see the library’s Database List by Subject—but should not feel limited to those databases. It should also be noted that a local public library and university library might have access to databases that Ashford University does not subscribe to. However, the Ashford online library does offer descriptions of each database as well as tip sheets and guides for research and advanced search techniques, which may be helpful to the amateur researcher. Knowing the important keywords and exhausting relevant databases will be essential first steps prior to the actual research, source evaluation, and extraction and analysis of information. For additional advanced techniques on searching databases, see Tips & Tools: Making the Most of Databases—Advanced Features. Tips & Tools: Making the Most of Databases—Advanced Features Databases like ProQuest and others are search engines indexed on a number of criteria. This indexing is used to facilitate quicker retrieval when a user enters the search terms he or she is interested in. However, a literal search engine may miss relevant documents because the wrong term was used or even if the term is misspelled. Unless a user asks ProQuest to return hits that are variants of spellings, the database will find only literal, exact matches—hence the name. To provide more flexibility, two advanced features to consider using are the commands for truncation and wildcards, shown in Table 4.1. Table 4.1: Truncation and wildcard characters Character Explanation Example * The symbol * is used as a right-handed truncation character only; it will find all forms of a word. Searching for econom* will find “economy,” “economics,” “economical,” etc. ? The symbol ? is used to replace any single Searching for “wom?n” will find character, either inside the word or at the “woman” and “women.” Searching for end of the word. ? cannot be used to begin “t?re” will find “tire,” “tyre,” “tore,” etc. a word. The * symbol (asterisk) is used as a type of wildcard that will search for words that start with the same root letters but may end differently. For example, a database search on the word depression might not turn up related words like depressing, depressed, or depressive unless these latter words were also mentioned in the journal article. But a search on depress* would return all possible options for words starting with d-e-p-r-e-s-s. Using the ? (question mark) feature can be helpful as well, as in the boxed example from ProQuest using the term woman and women in one search. Boolean operators are mathematical terms and descriptors that can be used in database searches to widen or narrow the scope of the search. The most common Boolean operators used for database searches are and, or, and not—but there are other advanced features as well. For instance, the terms woman and women can be searched for simultaneously using woman or women, in addition to the wom?n feature. This means that the database should retrieve any journal article that contains either term, as opposed to woman and women, which indicates that the journal article (continued) Lan66821_04_c04_p051-070.indd 53 4/24/12 9:34 AM Section 4.1 Locating Resources CHAPTER 4 Tips & Tools: Making the Most of Databases—Advanced Features (continued) must contain both terms (not either term). The not function can be useful as well, such as a search including not dissertation, which means dissertations should not be included in the list of database matches (hits). Such searches can also be combined on the same line, such as (woman or women) not dissertation. To find database matches with terms within the proximity of one another, see the advanced options in Table 4.2. Table 4.2: Operators used in database searches Operators Boolean, proximity, and adjacency operators are used to broaden and narrow your search. AND AND NOT Find all the words. When searching for keywords in “Citation and Document Text,” AND finds documents in which the words occur in the same paragraph (within approx. 1,000 characters) or the words appear in any citation field. Use W/DOC in place of AND when searching for keywords within “Citation and Document Text” or “Document Text” to retrieve more comprehensive results. Example: Internet AND education Find documents that have the first word but not the second word. Example: Internet AND NOT html OR Find any of the words. Example: Internet OR intranet W/# Find documents where these words are within some number of words apart (either before or after). Use when searching for keywords within “Citation and Document Text” or “Document Text.” Example: computer W/3 careers W/PARA Finds documents where these words are within the same paragraph (within approx. 1,000 characters). Use when searching for keywords within “Document Text.” Example: Internet W/PARA education W/DOC Find documents where all the words appear within the document text. Use W/DOC in place of AND when searching for keywords within “Citation and Document Text” or “Document Text” to retrieve more comprehensive results. Example: Internet W/DOC education NOT W/# Find documents where these words appear but are not within some number of words apart (either before or after). Use when searching for keywords within “Citation and Document Text” or “Document Text.” Example: computer NOT W/2 careers PRE/# Find documents where the first word appears some number of words before the second word. Use when searching for keywords within “Citation and Document Text” or “Document Text.” Example: world pre/3 web The possibilities may seem nearly endless with all of the various options and features in each database. One more useful tool that might prove helpful is ProQuest’s built-in thesaurus feature. This feature helps users identify other search terms that may be synonymous with the initial search term or related to the term in some way. For example, a very broad search term, such as economic development, would likely return hundreds of thousands of hits. But perhaps economic development isn’t quite the right term; the thesaurus feature will offer conceptually related ideas (and this feature is quite different from a literal search engine): broader terms, narrower terms, and then related terms. So if the true topic of interest is really economic development in third-world countries, clicking on the suggested Developing countries—LDCs will make a search more precise. See Figure 4.1. (continued) Lan66821_04_c04_p051-070.indd 54 4/24/12 9:34 AM Section 4.1 Locating Resources CHAPTER 4 Tips & Tools: Making the Most of Databases—Advanced Features (continued) Figure 4.1: ProQuest’s Thesaurus feature Access ProQuest’s Thesaurus feature by clicking on “Thesaurus” on the Advanced Search page or at the bottom of a search results page. Note that “Suggested Topics” also provides additional search terms and related topics. As with other search engines, ProQuest includes drop-down menus of Boolean operators to help widen or narrow the scope of the search. The power and complexity of the available databases can be overwhelming to a first-time researcher. The key is to practice with these new tools, especially during a time when there is not pressure to hand in a paper the next day. This can be an investment that yields returns later in the course and also helps a budding researcher attain search skills that are useful well beyond the completion of the course. In addition to the library resources available to students, another fruitful route for discovering research of interest involves using Google Scholar, which searches scholarly literature and abstracts available on the web, and even Google itself. Considering the multidisciplinary nature of the social sciences, a library database may at times be too narrow and too focused, so resources that are more broad-based (such as Google Scholar) might identify research efforts not captured in a formal database. There are also numerous governmental Lan66821_04_c04_p051-070.indd 55 4/24/12 9:34 AM Section 4.2 Breadth of Field: Broad Versus Narrow Approach CHAPTER 4 depositories of resources (e.g., the Educational Resource Information Clearinghouse, or ERIC) that may be helpful to a researcher. A reference librarian, who has been educated and trained to help others locate reference materials, is a great resource to consult to see if government databases may be of value. Knowledgeable People Faculty can also act as a resource—or at least a starting point for a student. Depending on the course, the instructor may have more expertise about a given topic, or at least be able to point students in a direction of additional sources. A reference librarian also has special talents that can be particularly useful as students use the library to obtain the research necessary for scientific writing assignments. Ashford University’s team of librarians is familiar with the university’s available databases and can assist students with locating resources, generating search terms, and narrowing paper topics. Ashford librarians are available via e-mail at library,ashford.edu, chat, or phone. Treeing Treeing is a technique by which a student or researcher examines the references section of an article to determine if some of the sources cited might also be of value as part of his or her cur- A reference librarian can be a useful source rent research project. This technique is particu- for the student. Oftentimes, an instructor larly effective when the student identifies a spot- may offer expertise on a given topic, but on research article directly addressing the issue a reference librarian can also point the of interest; odds are, some of the references in student in the right direction. that article may be references the student wishes to consult for his or her own research paper. Even if a particular research article that is retrieved is not ultimately cited in the research paper, some of the references from that article may be valuable for treeing. Thus, be sure to save PDFs and paper copies because those resources may be valuable in a different context. 4.2 Breadth of Field: Broad Versus Narrow Approach O ne of the early decisions to consider in writing a social science research paper is whether to take a broad approach or a narrow approach. As mentioned in Chapter 3, the social science field encompasses a wide range of disciplines. Naturally, a broad approach covers a breadth of subjects; a social scientist often attempts to integrate multiple perspectives and disciplinary specialties in examining phenomena and events of interest. Lan66821_04_c04_p051-070.indd 56 4/24/12 9:34 AM Section 4.2 Breadth of Field: Broad Versus Narrow Approach CHAPTER 4 This sort of broad viewpoint can attempt to span the social, behavioral, historical, and cultural impact of events that influence our daily lives. For example, consider the topic “relationships.” A researcher might explore the context of how heterosexual couples are often expected to start a family (social), or the differential struggles of monogamy between men and women (behavioral), or how expectations have changed over time about the longevity of marriage and the frequency of divorce (historical), or the difficulties faced with a marriage that spans two different yet strongly religious families (cultural). To take a broad approach, the writer might address all of these aspects on one literature review. A narrow approach, on the other hand, explores a subject in depth. In the earlier example about relationships, a researcher taking a narrow approach would perhaps explore only the social perspective or only the cultural perspective—but explore it more deeply. This type of approach is common in any one of the social science disciplines. Often, an academic’s training or background might be in only one of the many social science fields, but over time he or she might choose to adopt a broader approach. Perhaps the researcher wants to understand behavior from a “big picture” or trace the development of a theoretical orientation, or perhaps the researcher is seeking out new ways to see an old research problem. The type of approach would undoubtedly influence the types of journal articles a researcher might review in the preparation of a paper. There are journals such as The Social Science Journal (mentioned in the prior chapter) which publish a broad spectrum of articles and reflect the integrative work that social scientists often do. (See Chapter 3 for a list of studies that appeared in one issue.) This journal is a publication of the National Social Science Association (www.nssa.us), which is the largest interdisciplinary organization of social scientists in the United States (NSSA, 2010); there are regional social science associations as well. In contrast, each of the social science disciplines may have hundreds of specialty journals targeted toward its respective key issues and methodological approaches. In the sociological field alone, consider these specialized journal titles: The social science disciplines have hundreds of specialty journals to choose from when researching articles. • • • • • • • • Lan66821_04_c04_p051-070.indd 57 American Journal of Economics and Sociology Comparative Studies in Society and History Deviant Behavior International Review of Social History Journal of Artificial Societies and Social Simulation Journal of Mundane Behavior Mobilization: The International Quarterly Review of Social Movement Signs: Journal of Women in Culture and Society 4/24/12 9:34 AM Section 4.3 Evaluating Sources CHAPTER 4 The various conferences social scientists can attend also reflects this idea of breadth versus depth in research. For example, in the discipline of psychology, one could attend a generalist conference, which probably has something for everybody, such as the American Psychological Association annual convention, or a regional convention like the Midwestern Psychological Association annual meeting. Or one could attend a specialty conference on a topic like alcohol use (e.g., Research Society on Alcoholism) or child development (e.g., Society for Research on Child Development). Interestingly, sometimes specific specialty topics and societies bring together a cross-disciplinary range of individuals: In thinking about alcoholism, for example, solutions to that issue would involve behavioral, social, cultural, and historical influences, among others. So what appears to be a narrow topic can attract a broad spectrum of social scientists. Perhaps this broad and narrow approach is one method in approaching research papers that a student writes for the social sciences. Ultimately, depending on the topic and nature of the assignment, the best approach might be a hybrid approach where the paper simultaneously looks at the big picture as well as the important details. Being able to see the big picture as well as attend to the important details is a good skill to possess. 4.3 Evaluating Sources H owever narrow or specific a topic might be, the student will still face the task of sifting through the numerous potential sources or “hits” on a database search. It would be tedious, tiring, and simply impractical to read hundreds of sources for a short research paper—although a longer-term project with a wider scope may demand such a task. In evaluating resources, students should consider the source’s relevance to the topic at hand as well as its credibility, or trustworthiness. This is particularly true of Internet sources. Search engines such as Google Scholar and JSTOR can be useful in uncovering sources unavailable on library databases, such as government records and sources at college libraries. However, because it is more difficult to determine an Internet source’s veracity, take extra care in evaluating these sources. (This is why some instructors discourage students from using Internet searches altogether.) The following sections recommend specific criteria for making the decision about the credibility of the source. In truth, sometimes it takes quite a bit of detective work to determine if the information from a particular source is believable or not. Relevance Imagine that a student is working from home, using library resources and search engines to research a topic of interest. A search result emerges with many good leads on the topic. How does one decide which materials to retain and which to ignore? To be able to do this efficiently is a skill, and like any skill, it will take practice before anyone can become proficient at it. But the research paper is due in three weeks—how does a student make those decisions now? Honestly, it is a bit of trial and error. For those whose research skills are in the developing stage, it would be better to err on the side of retaining too much information (for possible inclusion in a research paper) than to not have enough information available. It is easier Lan66821_04_c04_p051-070.indd 58 4/24/12 9:34 AM Section 4.3 Evaluating Sources CHAPTER 4 to discard a source than to go back and generate more searches and reviews. A strong outline and a clear thesis statement can be the litmus test for how relevant a lead or source might be. That is, a student can compare the “gist” of a located resource to the thesis statement and ask whether the resource under consideration could contribute to supporting or refuting one’s thesis statement. It may seem odd to include sources that might refute a thesis statement, but to make a learned argument in a research paper, one effective strategy is to honestly present both sides of an issue—then provide the reasoning and the research that shows why the paper’s thesis statement should be supported. Remember that throughout the research process, one must strive to be an open-minded, critical thinker, looking for evidence on all sides of an issue—including sides that may not conform to presently held beliefs. Do not ignore contradictory research just because it is contradictory; be sure to acknowledge alternative points of view. Critically thinking about social science issues almost always means there are multiple points of view to consider. Primary Versus Secondary Sources As much as possible, the paper writer will want to retrieve primary sources. What are primary sources? A primary source is an original document. For example, if there is a journal article about original research and the researcher wrote the article—that is a primary source. A secondary source would be if the article were about the original research, but was not written by the original researcher; perhaps it appeared in another journal article or a textbook. Essentially, authors cannot control how others interpret their work—they can only control what they write. Social scientists prefer primary sources whenever possible. Primary sources provide the opportunity to understand the writer’s perspective in his or her own words, as opposed to someone else’s perspective. Authorship For any information to be considered credible (including a journal article), it is important that an author be listed (Ormondroyd, Engle, & Cosgrave, 2011). Why? When an author (or authors) wants to make an evidence-based claim in the sciences, the credibility of that claim is based in part on the author’s identity. (This is related to expertise, discussed in the next section.) In general, scholars should be willing to attach their name to their own work. There are rare occasions when a social scientist might publish a work as “Anonymous,” but a student should question why. It could be that a topic is so controversial that an author fears some sort of academic retribution, such as the denial of tenure or a promotion. In some limited cases, it might make sense to publish anonymously, but those extenuating circumstances would be rare. So when authors are unwilling to attach their names to an idea or claim, one might start to think, “Is this idea so flawed that no one was willing to put their name to it?” This circumstance happens frequently on web pages. A student might locate some great web-based information in the process of doing research for a term paper or literature review, but there is no author listed on the page (or document) of interest. It may take some extended detective work to ascertain the author, but that process is important. If no Lan66821_04_c04_p051-070.indd 59 4/24/12 9:34 AM Section 4.3 Evaluating Sources CHAPTER 4 one is willing to attach their name to the work (either through omission or obfuscation), this raises a red flag. The question of authorship is why many educators become concerned when students cite Wikipedia entries in their term paper assignments. Wikipedia contributors often do not identify themselves. Couple this anonymity with the unrestricted ability to change an entry (at least temporarily), and doubts about credibility emerge. This is not to say that Wikipedia has no value—it can be a good source for generating ideas or seeing what others believe about a topic. In essence, the importance of Wikipedia’s veracity hinges on how that information will be used. If a student were to look for the capital of Saudi Arabia and Wikipedia provided the wrong answer, that incorrect information might hurt a student on a test or lead to embarrassment when repeated. However, getting the wrong information in this instance would not likely be life threatening. But if one were to look up information about colon cancer on Wikipedia and make health-based decisions on potentially wrong information, then the credibility of the information truly matters. Also, when authorship is tied to information, accountability is possible. To be fair, an oncologist could also provide incorrect information about colon cancer, but that oncologist can be held accountable: Physicians can be sued for malpractice for providing incorrect advice, but has anyone on Wikipedia ever been sued for providing incorrect information? The authoring of ideas and claiming that authorship are important aspects of credibility and accountability. Expertise In addition to knowing the identity of an author, it is also important to be able to gauge the relative expertise of the person or organization making the claim. That is, does the person making the claim have the expertise—credentials, and knowhow—to make an appropriate and accurate claim? Every source has a point of view, whether that be a sociologist or psychologist at a university, a nonprofit agency working to reduce homelessness, or a government agency charged with reducing air pollution. No source is completely objective, so it is important to understand an author’s underlying motivations. One could examine a person’s track record in publishing, conference presentations, training in a particular area, and so on. Harris (2005, p. 38) recommended the expertise test: “Is there evidence that the source knows the subject?” But, to be clear, here expertise is used in the context of a connection to a scholarly discipline or community. Serial killers can develop “expertise,” but in this context expertise means as applied to scholarly and academic pursuits. Two gentle warnings to consider: First, just because a source has more education (master’s degree, Lan66821_04_c04_p051-070.indd 60 Harvard professor and African-American scholar Henry Louis Gates Jr. has been at Harvard since 1991 and undoubtedly has gained considerable credibility in his area of expertise. 4/24/12 9:34 AM Section 4.3 Evaluating Sources CHAPTER 4 doctorate) does not necessarily make that person an expert. In fact, sometimes the opposite trend emerges: As is sometimes said, “The more you know, the more you know you don’t know.” As one learns more about a topic, one often comes to appreciate the complexity of the topic and realizes that simplistic generalizations are rarely accurate in depicting reality. Sometimes when experts speak or write about a topic, they often “hedge their bets” about making definitive claims, perhaps couching a claim as their opinion or qualifying a statement by using phrases such as “generally speaking” or “odds are.” Second, just because an Internet source has “.edu”—a domain reserved for educational institutions—as part of the URL does not necessarily lend expertise to authors or the information they provide. Ideally, the information retrieved from an educational entity would be accurate and credible, but one cannot make that universal judgment. Ultimately, researchers need to do due diligence (fully and thoroughly explore the accuracy of information from multiple sources) and utilize all the tools provided in this chapter to determine what to believe and what to be skeptical about. Currency Currency refers to when the journal article (or other type of information) was published or made public. With a journal article, the year of publication should be clearly identifiable, as should be other information, such as the journal’s volume number and issue number. With web-based information, it is often much more difficult to determine its publication date, which is why “n.d.” is used in references at times, meaning “no date.” The date provides yet more evidence about the credibility and believability of information; undated information can still be useful, but then a researcher needs to rely on other criteria (authorship, expertise, etc.) to make the case for credibility. Depending on the type of writing assignment, currency may or may not be an issue (Ormondroyd et al., 2011). For example, if a student were writing about the idea of cognitive dissonance from social psychology, it would make perfect sense to review the work of Leon Festinger from the late 1940s and 1950s—currency would not be an issue with this type of writing assignment or journal article review. Currency is one factor in determining credibility and relevance. For example, the impact of Facebook has significantly changed since its inception in 2004, and articles from 2007 may no longer be relevant. Lan66821_04_c04_p051-070.indd 61 But if a student were asked for a review of how time spent on Facebook is affecting the frequency of letter-writing between friends, then the currency of sources would matter. Even in considering Facebook, its impact seems to have considerably changed from 2007 to 2012, so citing a 2007 journal article about Facebook may not be current at all and could lead research 4/24/12 9:34 AM CHAPTER 4 Section 4.3 Evaluating Sources conclusions astray. Depending on the assignment and the ultimate goal, sometimes currency may matter and sometimes it may not, but social science writers prefer to have a publication date available rather than no date. Accuracy To be fair, judging the accuracy of academic sources is a complex task requiring some expertise on the researcher’s part to determine if an author “got it correct.” In some ways, this process is a bit analogous to looking up a word in the dictionary that one knows absolutely nothing about. Since no prior knowledge is available, one may need to just trust the dictionary definition until more knowledge about the topic is acquired. So a student in the social sciences may need to rely on other indicators (at first) concerning the accuracy of a journal article until acquiring enough expertise for independent judgments. However, one of the key ideas underlying accuracy is the lack of bias, or prejudice, in the presentation of information (Harris, 2005). The more a student reads journal articles, the better he or she may become at identifying such bias; it may be slight at times and at other times flagrant. Useful information can be extracted from a biased source, but it is important to remember that the bias colors any ultimate interpretations. For different types of bias that can occur within the realm of research, see Table 4.3 (from Cochrane Collaboration, 2011): Table 4.3: Types of bias Type of Bias Description Publication bias (Positive results bias) The tendency on the parts of the investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings. Language bias Languages of publication depend on the direction and strength of the study results. Funding bias The biases in the design, outcome, and reporting of industry-sponsored research in order to show that a drug shows a favorable outcome. Outcome reporting bias A study in which multiple outcomes were measured reports only those that are significant and not those that were insignificant or unfavorable. Database bias Biased indexing of published studies in literature databases. The literature search will be biased when it is based on a database in which the results of indexed studies are systematically different from those of non-indexed studies. Study quality bias Studies of lower or higher quality are associated with positive or favorable results. Grey literature bias The results reported in journal articles are systematically different from those presented in reports, working papers, dissertations, or conference abstracts. Lan66821_04_c04_p051-070.indd 62 4/24/12 9:34 AM Section 4.3 Evaluating Sources CHAPTER 4 Another consideration is the distinction between assertion and evidence (Galvan, 2006). An assertion is an author’s opinion about the meaning or interpretation of data; it may be the author’s own data reported in an empirical study or an assertion about the data of others. Evidence is the quantitative or qualitative information gathered using appropriate research methods; the types of information one might see in the results section of an empirical journal article. Solid evidence should stand on its own; the assertions made based on that evidence should be carefully considered. For instance, based on the same evidence, a different social scientist might have very different assertions to make about the practical applications of the evidence. When evaluating the accuracy of an academic source, take care to differentiate between assertions and evidence. Corroboration If research findings or results from a journal article seem too good to be true, they probably are. Another old saying in science is that “if it hasn’t happened twice, it hasn’t happened.” Advances in science are based on the accumulation of corroborating, or supporting, evidence. When much evidence for a particular idea or proposition exists, we tentatively believe it until solid contradictory evidence is available. Scientists never prove a hypothesis or a theory but attempt to disprove competing hypotheses (an idea called falsificationism) until the only plausible idea left standing is the working hypothesis—think of this as an academic version of the game “king-of-the-hill.” So in a critique of a resource, if the new findings can help to confirm previous findings, then the journal article under review is helping to corroborate existing evidence. At times a newly published article may contradict existing research, and in that case, it is the obligation of the researcher to explain why these new results do not corroborate existing research. Maybe a new research approach was utilized or a survey scale was refined or some other methodological improvement was leveraged; in any case, it is incumbent upon the researcher to explain how the new evidence contributes to the current understanding on the topic (these explanations would typically appear in the discussion section of an empirical journal article). The more corroboration that exists for an idea, the more researchers can rely on that idea being correct. For Harris (2005, p. 39), the key idea is this: “Is the information supported by other sources?” For a slightly different conceptualization (and summary) of the preceding points that contribute to determining the value of information (especially online information), see the table here in which librarians (e.g., University of Iowa, 2011; University of the Fraser Valley, 2009) refer to what is known as the CRAAP test (Table 4.4). Lan66821_04_c04_p051-070.indd 63 4/24/12 9:34 AM Section 4.4 Journal Articles and the Peer Review System CHAPTER 4 Table 4.4: CRAAP test Currency: the timeliness of the source and the information • • • • • When was the information posted? When was it last revised? Are links functional and up-to-date? Is there evidence of newly added information or links? Is the information still considered accurate? Has more recent research challenged this information? Don’t exclude articles or information because of the publication date; instead think about the currency and relevance of the arguments presented. Relevance/ Coverage: the importance and scope of the information • Does the information relate to your topic or answer your question? • Is the topic covered with sufficient depth and breadth? Is the information comprehensive enough for your needs? Are the complexities of your topic adequately addressed? • Could you find the same or better information in another source? • Is the information relevant to current scholarly discussions on the topic? Do scholars refer to this source? Authority: the source of the information • Is the author/sponsor clearly identified? Is contact information easy to find? • What are the author’s credentials? Is the author knowledgeable in his/her field (based on employment, publications, sponsorship by reputable organizations)? • Has the author published works in traditional formats? (Look up the authors in Google Scholar.) • Is the author affiliated with an organization? Does this organization appear to support or sponsor the page? (Google the authors and/or sponsoring organizations.) • What does the sponsoring site (e.g., www.noaa.gov, www.uiowa.edu) and domain name (e.g., .com .edu .gov .org .net) reveal about the source of the information, if anything? Accuracy: the reliability, truthfulness, and correctness of the content • Where does the information come from? Can you verify any of the information in independent sources or from your own knowledge? • Are the original sources of information listed? • What evidence is presented to support claims made? • Has the information been reviewed or refereed? • Does the language or tone seem objective and unbiased? • Is the information free of spelling, grammar, and typographical errors? Purpose: the reason for the web site’s existence • Is the purpose of the page stated? Is the purpose to: inform? teach? entertain? enlighten? sell? persuade? Are possible biases clearly stated? • Is advertising content vs. informational content easily distinguishable? • Are editorials/opinion pieces clearly labeled? 4.4 Journal Articles and the Peer Review System J ournal articles are perhaps the most important communication mode for social science research. Journals have a more timely publication frequency, can reach large numbers of people, have a rigorous acceptance and publication process, and are a well-established means of information distribution. It is important that social science students become familiar with journal articles and their format, as journal articles will likely provide a significant amount of research fodder. For some tips, see Tips & Tools: How to Read a Journal Article. Lan66821_04_c04_p051-070.indd 64 4/24/12 9:34 AM Section 4.4 Journal Articles and the Peer Review System CHAPTER 4 Tips & Tools: How to Read a Journal Article Social psychologists Christian Jordan and Mark Zanna (1999) wrote a book chapter about how to read a journal article, and although their emphasis was on how to read journal articles in social psychology, their advice is applicable to all of the social sciences. For instance, When approaching a journal article for the first time, and often on subsequent occasions, most people try to digest it as they would any piece of prose. They start at the beginning and read word for word, until eventually, they arrive at the end, perhaps a little bewildered, but with a vague sense of relief. This is not an altogether terrible strategy; journal articles do have a logical structure that lends itself to this sort of reading. There are, however, more efficient approaches—approaches that enable you . . . to cut through peripheral details, avoid sophisticated statistics with which you may not be familiar, and focus on the central ideas in an article. Arming yourself with a little foreknowledge of what is contained in journal articles, as well as some practice advice on how to read them, should help you read journal articles more efficiently. (p. 461) Reading and extracting information from journal articles is an acquired skill, and like all skills, it takes time and practice to “get good” at it. In order to practice this skill development, Jordan and Zanna (1999) offered these tips for reading the major sections of research articles: INTRODUCTION (Try to answer these questions when reading an introduction section.) • • • • What was the problem studied, and why was it studied? How did this study go beyond previous studies of this same problem? How did the researchers arrive at their hypotheses? What questions do the researchers hope to answer by the study’s conclusion? METHOD (Try to answer these questions when reading a method section.) • How were the hypotheses translated into testable questions? • How were the variables manipulated and/or measured? • Did the measures of the variables accurately represent the actual variables of interest? RESULTS (Try to answer these questions when reading a results section.) • • • • Did the researchers provide evidence that the independent variables worked? What were the major outcomes of the study? Were the researchers’ hypotheses supported by the results of the study? Did the researchers explain the findings that did occur (regardless of hypotheses)? DISCUSSION (Try to answer these questions when reading a discussion section.) • • • • • What conclusions can be drawn from the study? What new insights are now available about this topic because of these research findings? Did this study help fill the previous knowledge gap about the topic? What are the practical implications of these results? Did the results contradict previous findings, and if so, why? Journal articles are a key means of communication for researchers in all of the social sciences. Advances can certainly be made in other communication venues, but journal article publication is key to the advancement of our knowledge about human behavior in context. The famous social psychologist Elliot Aronson (1995) once stated that when his colleagues read his journal articles, his colleagues will “look at it, be stimulated by it, be provoked by it, annoyed by it, and then go ahead (continued) Lan66821_04_c04_p051-070.indd 65 4/24/12 9:34 AM Section 4.4 Journal Articles and the Peer Review System CHAPTER 4 Tips & Tools: How to Read a Journal Article (continued) and do it better . . . that’s the exciting thing about science; it progresses by people taking off on one another’s work” (p. 5). In other words, in some cases the journal article publication can set the bar for other researchers; if a journal article stimulates the thoughts of others, then the writer has achieved a noble goal indeed. Jordan and Zanna (1999) alluded to the different ways that a newly published research article could help stimulate such advances: • • • • • Test better hypotheses to rule out alternative explanations. Look to see to what degree the results can be generalized beyond the setting of the study. Lead to a direct test of a pre-existing theory. Inspire a researcher to attempt to generalize the findings to a novel or innovative setting. Encourage a researcher to seek a practical application of a theory or hypothesis to see if the findings hold up outside of the laboratory. When done well, the journal article is the means by which our knowledge of human behavior advances, one journal article at a time. Questions for Critical Thinking and Reflection • Note the reading and re-reading of a journal article alluded to at the beginning of this case study. Students should be prepared to “dig in” and keep working while reading until understanding the basics of a journal article; that is, the key questions provided here can be answered. How often do you do this? • Do you often skip over the method and results sections of journal articles, just reading the abstract and perhaps the last paragraph of the discussion section if necessary? If you are just skimming the surface of the published work, do you realize the depth and benefits from the research that you may be missing? • Think about your career goals after graduation. Will you be entering the type of career where a journal article publication could be useful to your future, such as promotions and raises? Are there professional publications in your field that are widely read by those sharing your same job description? • How do you think advances will be made in your future workplace? Will it be an environment of trial and error, or do you think ideas (e.g., hypotheses) will be systematically tested? What might be the benefits of applying the research-based approach from your social science education to a future workplace? Journals in the social sciences operate under a peer review system where several professionals review article submissions before an acceptance decision is made. A social scientist who wants to professionally publish the results of their research starts by selecting a journal to send a manuscript to—not always an easy task—and sending multiple copies to the journal editor. (A manuscript may be submitted to only one journal at a time.) The editor sends copies of the manuscript out for review. Here is where the peer review process begins. The editor asks the social scientist’s peers in the field to review the manuscript and decide if it is suitable for publication. The peers are also called referees; a “refereed journal” means the journal follows this peer review process. This process differs from that of a magazine in that magazines pay people to write articles; authors of journal articles are not paid and sometimes even help defray the cost of journal publishing. Whereas magazine articles may be checked for accuracy, they do not undergo Lan66821_04_c04_p051-070.indd 66 4/24/12 9:34 AM CHAPTER 4 Chapter Summary the same scrutiny, examination, and review as journal articles. The majority of journal articles are well documented with supporting references noted as to when an idea has been adopted from another source. A magazine article is rarely as extensive in documenting the academic and scholarly work of the author. For some excellent guidelines for understanding the difference between a scholarly journal article and a popular magazine article, see Table 4.5 (University of Texas at Austin, 2007). Table 4.5: Differences between magazine and journal articles Characteristic Popular Magazine Article Scholarly Journal Article Length Shorter articles, broad overview of topics Longer articles providing in-depth analysis of topics Authorship Author often a staff writer or journalist; sometimes name and credentials not provided Author or authors usually an expert or specialist in the field, with name and credentials always provided Language and audience Written in a nontechnical language for anyone to understand Written in the jargon of the field for scholarly readers (professors, researchers, students) Format and structure Articles do not necessarily follow a specific format or structure Articles are usually very structured, with abstract, introduction, method, results, discussion, references Special features Illustrations with glossy or color Illustrations that support the textual prephotographs, usually for advertising sentation, statistical tables, graphics purposes Editors Articles not evaluated by experts but by editors on staff Articles reviewed and critically evaluated by experts in the field (called peer reviewers) Credits A bibliography is usually not provided, although names of reports or references may be mentioned in the text A reference section is always provided to document research sources thoroughly Chapter Summary R esearch in the social sciences encompasses various disciplines from which to draw, with the primary disciplines of interest being sociology, psychology, and history. Within each of these disciplines there are specialties and subspecialties, demonstrating the potential depth of any one particular discipline. The additional challenge for social sciences is to bridge these depths to achieve breadth and to work to understand events, peoples, and cultures from various perspectives, using a host of methodological approaches. The skills and abilities of an effective literature review are essential to identifying past work in the field that can provide insight for future directions and new destinations. Evaluation of online information, regardless of source, should involve the investigation of the currency, relevance, authority, accuracy, and purpose of the information available (i.e., the CRAAP test). The process of reviewing the literature can be enhanced when a thesis statement or research question is available and clearly defined, which can drive and inform Lan66821_04_c04_p051-070.indd 67 4/24/12 9:34 AM Concept Check CHAPTER 4 the various search strategies that are employed. By gaining the skills and confidence to navigate online databases, including the use of advanced features, students in the social sciences acquire valuable skills that will serve them well beyond graduation. Questions for Critical Thinking • • • In a self-evaluation of your own resource search strategies, what are your strengths and weaknesses? Are you an efficient user of online databases? What would be the approximate investment (in time) in your opinion that it would take to become proficient at this skill? Are there any drawbacks or limitations to exclusively searching the literature for scholarly, peer-reviewed journal articles, as is a feature of ProQuest Online? What types of information might you be using when restricting your search to an academic database search? Other than using Google, what other search tools might you use to expand your literature review horizons? During your studies as a social science major, would you consider yourself to be a generalist or a specialist? That is, do you find yourself frequently reaching across connections for the common thread or interconnections that may exist (generalist), or do you tend to live in the comfort zone of one particular discipline and venture out of your comfort zone only when necessary (specialist)? Why do you think you prefer one strategy over another? What types of assignments or course situations would it take to help you become more practiced in your non-preferred zone? Concept Check 1. The research method of looking at the references section of journal articles in an attempt to locate additional journal articles on the topic, is called a. truncation. b. treeing. c. Boolean operation. d. callbacks. 2. Which of the following best matches what social scientists mean by the “credibility” of the source? a. ingenuity b. pervasiveness c. trustworthiness d. creditworthiness 3. The originally published journal article is a/an ______ source. A source with information that someone else has written about the original journal article— such as the Pivotal Moments in Research features in this text—is considered a/an ______ source. a. internal; external b. central; peripheral c. abstract; original d. primary; secondary Lan66821_04_c04_p051-070.indd 68 4/24/12 9:34 AM Key Terms CHAPTER 4 4. A journal article published one year ago would be thought to have a high amount of a. currency. b. anonymity. c. search terms. d. citations. 5. Which of the following is true about the differences between popular magazine articles and published journal articles. For a popular magazine article, a. peer reviewers must approve prior to publication. b. the author is usually paid for writing the article. c. jargon is used predominately throughout the article. d. the author name and credentials are prominently displayed. Answers: 1) b, 2) c, 3) d, 4) a, 5) b Web Links This website outlines tips for evaluating web content and determining an author’s expertise: http://library.albany.edu/usered/eval/evalweb/ This website explains the criteria by which researchers evaluate print information: http://guides.library.jhu.edu/evaluatinginformation This website outlines ways to analyze the usefulness of a web page and presents questions to ask yourself when visiting web pages for the purposes of obtaining information: http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/Evaluate.html This website contains a detailed tutorial on how to use Boolean operators to maximize searches: http://www.internettutorials.net/boolean.asp This website provides videos, definitions, and tips and tricks for identifying scholarly articles versus popular periodicals: http://olinuris.library.cornell.edu/ref/research/ skill20.html Key Terms anonymity The absence of a connection between a specific participant and the data that he or she provides. bias Prejudice or favor for one outcome versus another, generally unintentionally caused by the outcomes a researcher hopes to find. Lan66821_04_c04_p051-070.indd 69 Boolean operators Mathematical terms, phrases and descriptors that can be used in database searches to widen or narrow the scope of the search. corroborate A process of verification by ensuring that previous research supports claims that a researcher is attempting to make in a study. 4/24/12 9:34 AM Key Terms CRAAP test A tool used to interpret the value of information using the characteristics of currency (timeliness), relevance (importance), authority (source), accuracy (correctness), and purpose (reason). credibility The trustworthiness of a source based on where the information was obtained and how recent and relevant it is. currency Refers to when a journal article or other source was published. expertise The experience, credentials, and know-how that a person or organization has in order to make their information credible. peer review A practice in the social sciences where several professionals review article submissions before an acceptance decision is made. Lan66821_04_c04_p051-070.indd 70 CHAPTER 4 primary source The original document about the research, written by the researcher. reference librarian An individual educated and trained with the ability to help others locate reference materials. search engines Databases that index material in order to facilitate a user’s fast retrieval. secondary source An article or source about original research that was not written by the individual who completed the research. treeing A technique by which a student or researcher examines the references section of an article to determine if some of the sources cited might also be of value as part of his or her current research project. 4/24/12 9:34 AM 5 The Critique of a Journal Article Learning Objectives After reading and studying this chapter, students should be able to do the following: • Understand why the ability to critique a journal article is a high-level skill developed only with repeated practice over time. • Use the holistic critique approach and the question-answering method as possible organizational schemes to coordinate a thorough journal article critique. • Appreciate the historical significance of previous critiques in the detection of fraud. • Understand the context in which journal article critiquing falls within the higher-level inquiry skills of analysis, evaluation, and creation. • Gauge if the type of data collected in a particular study appropriately matches with the conclusions being drawn from the study by its authors. • Comprehend the basic concepts necessary to be a savvy consumer of research, appreciating the role of critical thinking skills and their application to daily life. Lan66821_05_c05_p071-092.indd 71 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 I n the previous chapter, we focused on the search for relevant sources related to the topic of study. A student may be searching far and wide for information to include in a research project, but once a student has located that information and confirmed its veracity, what’s next? At some point, a student must carefully review the sources and determine if the evidence is worth including in the research project—that is, a critique, or critical evaluation, of the retrieved information is conducted. Although we focus on the critique of a journal article in this chapter—because journal articles tend to be key in making advances in the social sciences—many of these critical analysis techniques are applicable to books, websites, government reports, etc. The ability to analyze the information presented is a valuable skill that can be used far into the future. 5.1 Evaluating Journal Articles: The Big Picture A ccording to Bloom, the ability to critique is a so-called “higher order skill:” It requires more thought, memory, attention—cognitive processing—but also has more benefits (1956). Granted, the applications and benefits in the academic arena are immediately obvious: Students must be able to critically evaluate others’ work for their research papers and in their broader studies, and professors often advance their careers based on critiques (also called peer reviews) of their own scholarly work (Marshall, 2005). Beyond that, however, the ability to critique is a skill for all because it serves lifelong learning: Everyone should be able to think critically and analytically in everyday situations and about everyday information. Thus, critiquing is a skill that all should—and can—practice, particularly at the undergraduate level. Critiquing a journal article is different from simply reading a journal article, as described in Chapter 4. One can read a journal article just for the extraction of ideas or to better understand the context by which issues, variables, and behaviors are studied. A journal article critique is a deeper analysis than just information extraction; during a journal article critique, a reader would be prompted to think about possible improvements to the study or flaws to the study that must be considered in the context of limitations. Of course, understanding the anatomy of a journal article can help organize the critique of that article. See Table 5.1 here for a quick depiction of the typical anatomy of a researchbased journal article. (Recall that we covered this in depth in Chapter 2.) These are just the typical structures or anatomy of a journal article—student-written products will vary depending on the assignment guidelines. Lan66821_05_c05_p071-092.indd 72 4/24/12 10:06 AM CHAPTER 5 Section 5.1 Evaluating Journal Articles: The Big Picture Table 5.1: Quick anatomy of the journal article Title page (take credit) • • • • Author’s name Author’s affiliation Other information as your professor requests Running head information Abstract (quick summary) • Usually 120–150 words • Keywords may need to be included • Some assignments will not need an abstract Introduction (what you are studying) • Introduce the problem • Develop the background • State the purpose and rationale for the present study Method (what you did) • • • • Results (what happened) • Tables and/or figures if necessary • Presentation of statistical outcomes • No interpretation, just presentation Discussion (what it means) • • • • Reference page (give credit where credit is due) • Starts on its own page • Authors listed alphabetically by last name of first author • No first names used • This section shows your scholarly ability and how you did your homework Participants Apparatus or materials Procedure Should be in enough detail to replicate if desired Was there support for the research idea? Did the study help resolve the original problem? What conclusions can be drawn? Suggest improvements, avenues for further/new research Note that critique does not mean only criticize, for it can be relatively easy to criticize anything and tear it apart. The goal of a critique is not to go hunting for the flaws of the written work, but to analyze and evaluate that work on multiple dimensions to determine the relative impact of that work on our thoughts and beliefs about human behavior. By conducting a scholarly critique, a student should be able to both praise the work for its positive contributions as well as identify any potential flaws that might limit the positive contributions of the work (Collins, n.d.). Journal articles, and the connections we make between journal articles, comprise the brick-and-mortar of the social sciences. We build our cumulative understanding about human behavior one study at a time. If the individual components are questionable, then the overall outcome will be shaky at best, whether that be a building or a sociological theory. For many social scientists, much can be riding on the publication of one’s work. To see where that process can go astray, see Pivotal Moments in Research: Sir Cyril Burt and the Question of Fraud featured later in this chapter. Lan66821_05_c05_p071-092.indd 73 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Odds are that when asked to complete a journal article critique, an instructor will provide a specific format to follow. Often, the best way to approach the critique of a journal article is to break the task down into a series of questions—sometimes called leading questions (see examples later in this chapter). The typical journal article critique has three main sections: 1. the identification of the author and the formal citation of the source (as it would appear in a references section), 2. a summary of the main points of the journal article, and 3. the analysis and evaluation (the “critique”) of the article, which is typically the bulk of the document (Grand Prairie Regional College [GPRC], n.d.; University of Illinois at Springfield, 2009). Although the leading question method is effective and provides a nice organizational structure for a journal article critique, most of the points raised in a critique center around strategies for evaluating sources discussed in Chapter 4: authorship, expertise, currency, accuracy, and corroboration. Another method presented here is a more holistic (general) approach that can be used as a critique strategy. A Holistic Critique Approach An overall, global assessment of a journal article is often a good way to start a critique (Pyrczak, 2008). In a sense, that global assessment was performed when the journal article was selected for possible inclusion in the research paper. A specific evaluative scale might be used for each of the following prompts (such as very unsatisfactory to very satisfactory); or one might use a global, qualitative approach in critiquing the journal article. In either case, think about how the following questions might be answered regarding a specific journal article during a holistic critique (Pyrczak, 2008). (See Writing in Action: Sample of a Holistic Critique for how this might be done.) 1. 2. 3. 4. 5. 6. 7. 8. 9. Has the researcher selected an important problem? Were the researchers reflective? Is the report cohesive? Does the report extend the boundaries of the knowledge on a topic, especially for understanding relevant theories? Are any major methodological flaws unavoidable or forgivable? Is the research likely to inspire additional research? Is the research likely to help in decision making? All things considered, is the report worthy of publication in an academic journal? Would you be proud to have your name on the research article as a coauthor? A student could use most of these same criteria to think about the quality of a research paper in preparation as to a holistic assessment of the quality of the work. Lan66821_05_c05_p071-092.indd 74 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Writing in Action: Sample of a Holistic Critique Article Critiqued (available through a Google Scholar search): URL for Google Scholar: http://scholar.google.com/ URL for this specific article: http://www.sscnet.ucla.edu/soc/faculty/Moore/MooreChase.pdf APA Citation/Reference Moore, M. R., & Chase-Lansdale, P. L. (2001). Sexual intercourse and pregnancy among African American girls in high poverty neighborhoods: The role of family and perceived community environment. Journal of Marriage and Family, 63, 1,146–1,157. Have the researchers selected an important problem? Yes. These researchers addressed the onset of sexual activity by African American girls (ages 15 to 18) in a high-poverty environment. Specifically, the researchers set out to address (1) how do role models in the home (married parents or divorced parents) influence the onset of sexual activity, and (2) when there is more social support from family and neighborhood influences, does this delay the onset of sexual activity? Knowing the answers to these important questions would be valuable to social scientists and policy makers as to those factors that lead young adults to make thoughtful, planned decisions about sexual activity. Were the researchers reflective? The researchers were reflective, as evidenced by an extensive review of the literature, encompassing three different theoretical approaches to family influences on adolescent sexuality: a family structure approach, a parent-child relationship approach, and neighborhood/community context approach. Moore and Chase-Lansdale tested the hypothesis that “the risk of sexual debut and pregnancy is greater for disadvantaged youth in singlemother households compared with married households” (2001, p. 1,149), as well as testing four additional hypotheses. Is the report cohesive? In general, yes. The report follows the typical anatomy of a research report, with the major sections of introduction, method, results, and discussion. However, this work has a heavy statistical basis, thus some of the tables will take a bit of time for the reader to comprehend and digest (especially Tables 3–5 and the use of odds ratios with logistic regression data). Does the report extend the boundaries of the knowledge on a topic, especially for understanding relevant theories? These authors tackle a complex topic with the appropriate tools for capturing that complex picture. For example, there are situations where more than one variable combines with other variables that leads to an explanation of the self-report survey data. For example, it was hypothesized that the odds of an earlier (younger) sexual debut and pregnancy would occur in single-mother and cohabitating households, in part due to the available social structure. This pattern of results held true for teenagers in single-parent households, but those teenage girls in cohabitating households did not have significantly higher odds of pregnancy. Understanding the complexities of these behaviors and their influences does indeed help extend the knowledge available on this topic. (continued) Lan66821_05_c05_p071-092.indd 75 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Writing in Action: Sample of a Holistic Critique (continued) Are any major methodological flaws unavoidable or forgivable? There were not. The researchers were fair and measured with their conclusions, and they noted that studying impoverished neighborhoods in Chicago was a limited sample that does influence the generalizability of their results. However, with the substantial case study data available (N = 289), the statistical approach is appropriate as are the conclusions drawn. Is the research likely to inspire additional research? Yes. This is an important social issue, and understanding the variables and social forces at work here will continue to be a concern to society for some time. Pregnancy is not inherently a negative event; optimally, however, it should occur in a context where both mother and child have the greatest odds to thrive. Understanding the dynamics that lead some African American girls to an early sexual onset and pregnancy will be important for educators and policy makers; as culture changes, so will the relative impact of social structures on this very important and very personal aspect of decision making. Is the research likely to help in decision making? Yes. The findings reported here are most valuable to educators, as these results can help in the design of sexual education programs and identifying sexually at-risk youth. All things considered, is the report worthy of publication in an academic journal? Yes, absolutely. This is important research; the methodology presented is important for future researchers, and the results are important for educators, policy makers, and social scientists interested in these issues. Would you be proud to have your name on the research article as a coauthor? Absolutely, yes. Using the “Answer These Questions” Critique Method A student seeking advice about how to critique a journal article would find many sources available to provide long listings of key questions to ask while reading and reviewing a journal article. These listings (GPRC, n.d.; Ballantine & Roberts, 2009; Salkind, 2009; University of Illinois at Springfield, 2009) do share common characteristics, and Table 5.2 thematically presents the prompts a student might use if this “answer these questions” method is selected as the critique approach. Think of these as a more detailed approach than the holistic questions method presented previously. Especially as a student learns to critique a journal article, this list of detailed questions—the question-and-answer critique—can be particularly helpful in building confidence in what to look for throughout the critique process. Lan66821_05_c05_p071-092.indd 76 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Table 5.2: Critique questions thematically sorted by topic Review of the Existing Literature 1. Is there a theoretical perspective being followed here? If so, which perspective? 2. Was the author correct in his or her interpretation of others’ work? 3. Has the author cited the pertinent references necessary? 4. How closely was the literature cited in the journal article actually related to the previous literature? Was it a good match or a mismatch? 5. Is it an up-to-date literature review? 6. Are there any classic or seminal research works that should have been included in the literature review but were left out? Problem and Purpose of the Study 7. Briefly describe the topic of the journal article, which is usually described in its title. 8. Identify the important concepts the author presents. 9. What are the author’s underlying assumptions that may influence the study’s outcomes? 10. Can you understand the statement of purpose of the study? 11. Is there a rationale that explains why the study’s outcomes would make an important contribution to the published literature? Thesis Statement or Hypothesis 12. Are the key concepts clearly defined and explained? 13. What is the thesis of the work presented in the journal article? 14. Did the author’s approach to supporting or refuting the thesis make sense? 15. Can the hypothesis or hypotheses presented actually be tested? 16. Do the hypotheses state a clear, expected association between the variables under study? 17. Are the hypotheses grounded in theory or as part of a review of relevant literature? Methodology, Including Sample 18. What was the author’s general method for drawing conclusions about this topic? Was this an empirical study? 19. What was the author’s method collecting data? Qualitative, quantitative, or both? 20. Is there a clear rationale provided for how the key questions of interest can be adequately addressed by the research methodology utilized? 21. Are the research methods adequately described to the extent that another researcher could replicate this research study? 22. How large or representative was the sample that was used? 23. Are independent and dependent variables clearly defined, or at least clearly identifiable? 24. Is it clear how the study was conducted—that is, the participants selected, the materials used, and procedures followed throughout the study? 25. Was the sample selected such that you believe it to be representative of the greater population of interest? 26. How similar are the participants selected here to participants utilized for past studies? (continued) Lan66821_05_c05_p071-092.indd 77 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Table 5.2: Critique questions thematically sorted by topic (continued) Analyses and Results 27. Are the statistical calculations used appropriate and correct? 28. Did you discover any errors in the analyses selected? 29. Were the results reported fairly and accurately? 30. Are the results reported relatable to the hypotheses intended to be tested? Discussion, Conclusion, and Interpretations 31. Do the data analytic outcomes support the assertions made by the author? 32. Which results are the most thought provoking? 33. Are limitations to the study mentioned, and are they reasonable and plausible limitations? 34. Does the author make suggestions for future research? What are the unanswered questions? 35. Did the author draw appropriate generalizations? 36. Did the author present any counterevidence or alternative explanations for the data collected or conclusions drawn from this research? 37. Does the discussion provide closure regarding the initial hypotheses presented earlier in the journal article? 38. What would you do to improve this research? 39. What are the primary implications of these research outcomes? References 40. Are the references recent and relevant? 41. Are the references properly formatted and complete? 42. Does the list of references reflect the most important reference sources available in the field currently? 43. Is there enough information provided in each reference such that the reader can retrace the exploratory steps of the author and locate the original source information? General Comments 44. Depending on your instructor’s preference, provide the full citation of the journal article being critiqued. 45. Is the title of the article understandable? 46. Is the abstract correctly formatted and helpful? 47. Did you come away with at least one big-picture, take-home message? 48. Is the overall writing style clear and understandable? Is the writing style appropriate for the audience? 49. Does the author strive for neutrality and objectivity in word choice, or is the language selected emotionally charged and biased? 50. Do you agree or disagree with the author’s conclusions about the topic? Why? 51. What are the strengths and weaknesses of the research presented in the journal article? Note that as with the holistic critique approach, students can utilize many of the items presented in Table 5.2 as an evaluative checklist for their own writing and others’ writing. Lan66821_05_c05_p071-092.indd 78 4/24/12 10:06 AM Section 5.1 Evaluating Journal Articles: The Big Picture CHAPTER 5 Pivotal Moments in Research: Sir Cyril Burt and the Question of Fraud Scholarly critiques play a role in determining the veracity of others’ work—sometimes causing furors in the scientific community. In the early 20th century, educational psychologist Sir Cyril Burt (1883–1971) sought to empirically test the question of nature versus nurture in intelligence: whether intelligence is chiefly influenced by genetics and heritability (nature), or whether it is chiefly influenced by environment and upbringing (nurture) (Plucker, 2007; Zenderland, 2000). To do this, Burt sought out adult monozygotic, or identical, twins who had been raised apart and then administered intelligence tests to these twins—an ingenious approach. If genetics, or nature, were the overriding influence in intelligence development, then there should be a high positive correlation, or relationship, between the twins’ IQ scores. If environment, or nurture, were the overriding influence, then there should be a low positive or zero correlation between the twins’ IQ scores. Burt published his results three times—in 1943, 1955, and 1966— each time including more sets of twins and each time demonstrating high positive correlation between the twins’ scores. Here are the Sir Cyril Burt correlation results for monozygotic twins’ IQ scores by year of publication (Zenderland, 2000)—r represents the correlation score, or coefficient. (The closer r is to 1.0, the more there is a positive correlation between the twins’ IQ scores, meaning that the scores move up or down highly “in sync.”) 1943: r = .770, n = 15 twin sets 1955: r = .771, n = 21 twin sets 1966: r = .771, n = 53 twin sets On the surface, these outcomes would provide strong evidence to support the heritability, or nature, hypothesis of intellectual development. However, a couple of months after Burt’s death in 1972, Princeton University psychologist Leon Kamin suggested there were problems with the data after a careful critique of Burt’s work (Zenderland, 2000). Kamin did not accuse Burt of outright fraud but suggested the data was suspect: It is highly unusual that the correlation coefficient r would remain exactly the same from 1955 to 1966 (Plucker, 2007). Kamin and others started to dig deeper, going back to data collection records, attempting to track down former research assistants, and so on. Medical correspondent Dr. Oliver Gillie made the first public fraud accusation in 1976, after he could find no evidence that two of Burt’s key research assistants, Miss Margaret Howard and Miss Jane Conway, ever existed (Plucker, 2007; Zenderland, 2000). Others questioned Burt’s amazing ability to locate 32 additional sets of monozygotic twins raised apart in a short 11 years, given the amount of work it took to locate 21 twin sets by 1955. Even Burt’s biographer Leslie Hearnshaw, who delivered a eulogy at Burt’s funeral, concluded that there was a pattern of deliberate deceit in Burt’s work, and following the publication of the biography in 1979, the British Psychological Society declared Burt guilty of fraud. Some have stepped forward to defend Burt’s work and refute these claims of fraud (Plucker, 2007). Regardless, the data Burt contributed is suspect at best, due to a careful critiquing of published work. (continued) Lan66821_05_c05_p071-092.indd 79 4/24/12 10:06 AM CHAPTER 5 Section 5.2 Higher-Level Inquiry Skills Pivotal Moments in Research: Sir Cyril Burt and the Question of Fraud (continued) Questions for Critical Thinking and Reflection • Consider for a moment if Burt had been able to “get away” with the presentation of fraudulent data without being questioned. What would be the harm to the social sciences if fraudulent claims were to be reported without repudiation? • As a student reading journal articles and consuming information, how does one know when to question the conclusions (and the data) and when to accept the conclusions as valid? Think about the numerous claims advertisers make, especially in TV commercials: When a claim is made about the safety of a new model year automobile or the cleansing power of a new mouthwash, how does one know what to believe and what to question? 5.2 Higher-Level Inquiry Skills T he ability to critique a journal article is actually a specialty version of a higher-order skill known as evaluation. This notion of higher-order (and lower-order) skills has been around for some time but became popularized due to Bloom’s (1956) work on the taxonomy of functions in the cognitive domain. A taxonomy is an organizational scheme, and in this case, Bloom organized his taxonomy into six different dimensions, placing these cognitive processes in a hierarchical continuum from lower level to higher level. His original six cognitive dimensions (from low to high) were knowledge, comprehension, application, analysis, synthesis, and evaluation. In a revised taxonomy offered by colleagues of Bloom (Anderson & Krathwohl, 2001), the six categories of cognitive processes (from low to high) are remembering, understanding, applying, analyzing, evaluating, and creating. As educators, we strive to instill these higher-level inquiry skills in our students and graduates, and in this skill set is where journal critiquing fits into the bigger picture. A brief description of each of the higher-level inquiry skills follows. Figure 5.1: Bloom’s revised taxonomy Creating • Generating • Planning • Producing Evaluating • Checking • Critiquing Analyzing • Differentiating • Organizing • Attributing HIGHER ORDER SKILLS Applying Understanding Remembering Lan66821_05_c05_p071-092.indd 80 4/24/12 10:06 AM Section 5.2 Higher-Level Inquiry Skills CHAPTER 5 Analysis Analysis “involves breaking material down into its constituent parts and determining how the parts are related to one another and to an overall structure” (Anderson & Krathwohl, 2001, p. 79). The specific subprocesses proposed here are the abilities to differentiate, organize, and attribute. In the revised cognitive taxonomy, analysis is frequently grouped in a trio of processes: understand, analyze, and evaluate. When a student is asked to perform an analysis, that task may be to separate fact from fiction, to make an argument and support it with a citation in APA format, distinguish between evidence that is empirically based versus anecdotal, uncover the relationship between ideas or behaviors, and so on. The journal article critique method (either the holistic or question-answer approach) is one method by which a student can practice analysis skills. Evaluation Evaluation “is defined as making judgments based on criteria and standards. The criteria most often used are quality, effectiveness, efficiency, and consistency” (Anderson & Krathwohl, 2001, p. 83). The key processes applied with evaluation are checking and critiquing. Checking involves an examination of facts presented such that there are minimal inconsistencies; that is, a statement made in a student’s introduction section/review of the literature does not contradict with a statement made on a similar topic in the discussion/conclusion section of the same paper. Critiquing is the central focus of this chapter; one’s ability to make judgments about others’ work as compared to external criteria. It takes time for anyone to acquire the expertise to effectively evaluate against stated criteria—and here one can appreciate the clarity of the criteria. After graduation, some rise to management positions, and those individuals may have to conduct annual performance appraisals for fellow employees. One’s ability to effectively evaluate performance should improve over time and with practice, and when placed in this leadership situation, those responsible for performance appraisals quickly appreciate when clear workplace objectives are available. Creation Creation “involves putting elements together to form a coherent or functional whole” (Anderson & Krathwohl, 2001, p. 84). The processes associated within the creation domain include generating, planning, and producing. As students write papers of various sorts throughout college (including capstone papers), the process of creation is evident. As one reflects on the individual chapters of this book, these efforts come together to help students create an outcome worthy of senior-level social scientists in training. Thus, a student Lan66821_05_c05_p071-092.indd 81 4/24/12 10:06 AM Section 5.3 Do the Conclusions Match with the Type of Data Collected? A potter must use appropriate amounts of water and clay to create a functional piece of pottery. Similarly, students must include the proper elements to create a sound research paper. CHAPTER 5 is generating rough drafts of a research paper, planning the methodology of a proposed study, and producing written work that conforms to APA format and style. In Bloom’s revised taxonomy, creation is the highest level within the cognitive domain, and at some level, creation requires the coordination and competency in all the domains leading to it: remembering, understanding, applying, analyzing, and evaluating. In many ways, these different cognitive domains are different types of tasks and skills and require different levels of competency and practice in an effort to gain expertise. 5.3 Do the Conclusions Match with the Type of Data Collected? T his chapter is all about “levels of analysis.” The first major section covered the anatomy of a journal article and provided two examples of methods of critiquing a journal article—a valuable skill for anyone studying the social sciences. The second major section jumped into the pool a bit deeper by presenting some ideas from a revision of Bloom’s taxonomy of cognitive tasks, focusing on the higher-order types of skills necessary for success following a student’s undergraduate education—specifically, analysis, evaluation, and creation. This last major section addresses a specialized area of the critique process—does the data collected by the research appropriately match the conclusions drawn? For example, has the researcher drawn appropriate conclusions from correlational data or overreached? Is the cause-and-effect relationship language used in the journal article supported by the research design and statistics results reported? This level of analysis will take some time to acquire because like any other valuable skill, it takes time and multiple repetitions to gain confidence in using this skill appropriately. By now in the undergraduate coursework sequence, a student may have completed one or more courses in research methods, experimental design, or a similarly named course. In that course, different types of research designs and the types of conclusions that each could yield were probably discussed. For example, in some experiments, cause-and-effect conclusions are often possible, while others simply yield valuable information about the relationships or predictability between variables of interest. (A brief review of many of these topics can be found in Chapter 7.) The type of research approach used influences the strength of the conclusions from the data. Lan66821_05_c05_p071-092.indd 82 4/24/12 10:06 AM Section 5.3 Do the Conclusions Match with the Type of Data Collected? CHAPTER 5 Causality in Research Studies For many social scientists, the most powerful conclusion that can be made in science is a cause-andeffect conclusion. In theory, if we are able to determine causality, we could make positive outcomes occur more often and work to prevent negative outcomes from happening as often. For example, it would be beneficial to know what causes marital satisfaction, what causes happiness, what causes college student success, and what causes self-actualization so that we could promote those causes and help individuals strive for goals they wish to achieve. Conversely, it would be nice to know what causes Alzheimer’s disease, what causes autism, what causes clinical depression, what causes lowself esteem, and what causes suicide so that we could work to prevent antecedent (before the fact) causes that lead to these negative outcomes. However, it takes very precise methodology to arrive at any level of confidence about causality, and there are many different forms of research questions to ask. Meltzoff (1998) does a very nice job of describing the types of research questions. Table 5.3 summarizes Meltzoff’s (1998) descriptions, using generic statements but also adding realistic examples. Students can try to identify which type of research question is addressed in the journal article they are reviewing. If the journal article is written with enough clarity, a student should be able to place it in one of the types of research categories here. Lan66821_05_c05_p071-092.indd 83 What causes the change in seasons? Now we know that the revolution of the Earth around the Sun dictates the four seasons: winter, spring, summer, and fall. Social scientists also implement the cause-andeffect relationship to draw conclusions from research. 4/24/12 10:06 AM Section 5.3 Do the Conclusions Match with the Type of Data Collected? CHAPTER 5 Table 5.3: Types of research questions, with examples Types of Research Questions Generic Example Specific Example Existence Questions Does x exist? Can people have a Facebook addiction? Does sincere altruism exist? Questions of Description and Classification What is x like? To what extent does x exist? What are the best practices of master teachers? What is graduate school like? To what extent are teacher-created tests like the Graduate Record Examination (GRE)? Questions of Composition What are the components What variables lead to high student satisfacthat make up x? What are tion with college? What are the leading indithe factors that comprise x? cators that someone is clinically depressed? Statistical Relationship Questions Is there an association or relationship between x and y? Is one’s age related to GPA? Is there an association between gender and political affiliation? Descriptive-Comparative Questions Is Group x different from Group y? Are males or females more likely to stay in college? For returning adults, do parents or non-parents have a better GPA in college? Causality Questions Does x cause, lead to, or prevent changes in y? Does psychotherapy help individuals with dissociative identity disorder? Does attending tutoring sessions lead to better student test performance? Causality-Comparative Questions Does x cause more of a change in y than z does? Is Prozac better than Xanax at helping people deal with depressive symptoms? Does caffeine help with better concentration skills as compared to a placebo? Causality-Comparative Interaction Questions Does x cause more change in y than does z under certain conditions, but not under other conditions? Are male Republicans more likely to vote for a Democratic nominee, but not female Republicans? Are psychology majors more likely to be successful in their healthcare-based careers than nonmajors, but only for psychology majors who attend graduate school? Source: Adapted in part from Meltzoff (1998) For many scientists, the ultimate goal is the determination of causality—that is, understanding cause and effect relationships. However, a strong significant positive correlation between two variables does not necessarily mean that causality can be inferred. Burns (1997) summarizes three criteria for establishing causality. First, there must be clear temporal precedence. That is, for the cause to be the cause, and the effect to be the effect, the cause must come first and the effect must come second. If both cause and effect occur simultaneously, then we cannot know the cause or the effect. There must be a clear time sequence. Second, measures of cause and effect must covary. That is, if there is a cause and effect relationship, the presentation of the cause needs to yield the effect, but if there is no presentation of the cause, then there should be no effect. If the nature of the cause is changed, then the nature of the effect should also change. Finally, there should be no plausible alternative explanation. If we have adequately applied our research methods, experimental controls, methodological designs, and so forth, then Lan66821_05_c05_p071-092.indd 84 4/24/12 10:06 AM Section 5.3 Do the Conclusions Match with the Type of Data Collected? CHAPTER 5 we need to say, with confidence, that there is no other logical explanation for the effect other than the cause. Note that we do not say that we proved that the cause is the reason for the effect, but we infer the relationship when we have (a) temporal precedence, (b) covariation, and (c) ruled out plausible alternative explanations (Burns, 1997). Technically speaking, we do not “prove” anything in the social sciences, but we disprove—the notion of falsificationism. Concerns About Nonscientific Evidence In drawing conclusions, the quality of evidence and type of data is a critical factor. For instance, from a correlational study, a cause-and-effect conclusion cannot be drawn. The type or kind of evidence relates to the type or kind of conclusions possible. Evidence gathered through the method and processes of science often holds special characteristics that nonscientific evidence does not possess. Bell (2005) nicely summarized some of the problems that can occur when nonscientific evidence is used (p. 38): • • • • • • • • • An example (one observation) or the observations from a few people may be used as evidence. It may not be clear who actually made the observations and under what conditions. The observations are not repeated and carefully checked. Thus, a fluke occurrence could prematurely be accepted as fact. Observations are not written down as they occur but are recalled at some later date. Given the fallibility of memory, a specified procedure for reliably recording data is highly desired. The people being observed, usually those who are available, are rarely representative of a larger population. Representative samples in the social sciences are critical if we want to be able to generalize our results beyond the individuals studied or tested. Important terms are not precisely defined, and sometimes the definitions of key terms change. Disagreement about the meaning of key terms might mean that researchers who think they are studying the same topics/constructs are not studying the same phenomenon at all. A comparison group or comparison condition is rarely available. A hallmark of the scientific approach is the comparison of the treatment group to the control group, whenever possible (see Chapter 7 for more details). Control over important variables is impossible. It is very difficult to keep constant the relevant factors to be able to state a specific cause. Subtle factors can be overlooked. All of the evidence is not used. Evidence that is contrary to a statement is overlooked, or only part of the evidence is used. Some writers have biases that interfere with objective observation and make it difficult to separate evidence from opinions and value judgments. By relying on scholarly peer review in the publication of journal articles, subjective conclusions and the chances of outright fraud are minimized. This is not to say that scientific evidence does not have the potential for flaws—of course it does—but research methods are utilized and peer review procedures take place to ensure that conclusions drawn for research evidence is appropriate and fair. As one practices Lan66821_05_c05_p071-092.indd 85 4/24/12 10:06 AM Section 5.4 Being the Savvy Consumer of Research CHAPTER 5 reviewing journal articles, this becomes a key evaluative component—are the conclusions drawn from the research appropriate for the methodology used and the results reported? As with all skills, it will take practice to acquire this skill—at times, one may be over-critical, and other times one may not be critical enough. Over time, a sense of expertise will develop in determining if the researchers who authored a study were appropriate in the conclusions drawn from the data. The benefits from this systematic approach are numerous, but the key idea is this: having a specific method for making decisions in science allows individuals to avoid making decisions about people (or behaviors, or variables) based on common sense. 5.4 Being the Savvy Consumer of Research B y developing a honed sense of critiquing, this skill can be expanded beyond the journal article technique and utilized to examine any type of research finding or claim. Social scientists need to be aware of such potential pitfalls in order to (a) avoid making fallacious claims based on one’s own work, and (b) identify false claims made by others to avoid being duped or misled. Some of the potential pitfalls in being that good consumer of research involve the appearance of illusory correlations and the natural bias to believe what one wants to believe. Being a good researcher (generating research) or a savvy consumer of research (reading research) necessitates a fair measure of quantitative reasoning skills. Some specific examples of types of inappropriate claims are presented here. Fallacious Claims and Illusory Correlations Illusory correlations and confirmation bias are two ways that common sense can mislead us. An illusory correlation is our tendency to believe that two variables are related when they are not. For example, a student may have a friend who is convinced that her favorite sports team wins only when she watches. If she were to gather the data and look at her team’s win-loss record when she watches and when she does not watch, she is likely to find that her presence is not a deciding factor. Or perhaps another friend regularly reads the horoscope and uses this information to guide behavior. One horoscope on astrology.com stated, “A big improvement is in the works, one that you may know nothing about, and today is the day for the big unveiling.” A job-seeking recent college graduate might spot some new listings for interesting positions and decide the horoscope was right. When individuals look for an association, they often find it. Yet, over and over again, careful researchers fail to find evidence to support the accuracy of astrology (e.g., Dean & Kelly, 2003). Using quantitative reasoning helps individuals to see past illusory correlations. A confirmation bias is our tendency to “see” evidence that fits with our beliefs and to ignore evidence that does not fit with our beliefs. This tendency is the main reason that we “see” illusory correlations. For example, if someone believes that watching her favorite sports team contributes to that team’s success, then she notices the times that she watched and they won and the times that she did not watch and they lost. She underestimates the number of times that she watched ...
Purchase answer to see full attachment