Write a critical thinking comment or a question to the attached articles

User Generated

Rzn00

Health Medical

Description

we have to come up with either a question or comment on each article after reading them. the comment should reflect full understanding of the article. The question has to shows a critical thinking of the issue presented in the article.

Unformatted Attachment Preview

Forum Forum contributions present essays, opinions, and professional judgments. Forum articles speak to and about the philosophical, ethical, and practical dilemmas of our profession. By design, the “Forum” is open to diverse views, in the hope that such diversity will enhance professional dialogue. Standard citations and reference lists should be used to acknowledge and identify earlier contributions and viewpoints. Manuscripts should typically not exceed 15 double-spaced typewritten pages in length, unless the paper is invited by the editor. Consequences to Federal Programs When the Logic-Modeling Process Is Not Followed With Fidelity Ralph Renger University of Arizona Abstract: The Office of Management and Budget has recommended the termination of numerous federal programs, citing a lack of program results as the primary reason for this decision. In response to this recommendation, several federal agencies have turned to logic modeling to demonstrate that programs are on the path to results accountability. However, approaches to logic modeling in some agencies have not followed the strategies that evaluators recommend as necessary to lead to a highquality logic model. Models of poor quality are unlikely to contribute to improved program accountability or better program results. In this article, the author assesses the quality of the logic-modeling approach taken by one agency to illustrate how a flawed approach to logic modeling may lead to incorrect conclusions about programs and about the benefits of logic models. As a result of this analysis, the author questions the conditions under which capacity building should be considered and whether the field of evaluation needs to be more judicious when mainstreaming methodologies. Keywords: evaluation; federal programs; logic model In the 2005 State of the Union address, President Bush stated, America’s prosperity requires restraining the spending appetite of the federal government. I welcome the bipartisan enthusiasm for spending discipline. I will send you a budget that holds the growth of discretionary spending below inflation, makes tax relief permanent, and stays on track to cut the deficit in half by 2009. My budget substantially reduces or eliminates more than 150 government programs that are not getting results, or duplicate current efforts, or do not fulfill essential priorities. The principle here is clear: Taxpayer dollars must be spent wisely, or not at all. Ralph Renger, University of Arizona, 1295 N. Martin Avenue, Building 202, P.O. Box 245210, Tucson, AZ 85724; e-mail: renger@u.arizona.edu. American Journal of Evaluation, Vol. 27 No. 4, December 2006 452-463 DOI: 10.1177/1098214006293666 © 2006 American Evaluation Association 452 Renger / Logic-Modeling Process 453 As a part of the Bush administration’s budget reconciliation process, federal programs are to undergo extensive review to determine which programs will receive appropriations and which will be phased out. As Bush warned, those programs that cannot be held to account for good performance will be reduced or eliminated. One major factor in deciding which programs to cut is how they score on the Performance Assessment Rating Tool (PART) (Office of Management and Budget, 2004b). The Office of Management and Budget (OMB) uses the PART to score programs in the following four areas: purpose and design, strategic planning, program results and accountability, and program management. A series of questions is used to operationally define and rate each area. Purpose and design focus on matters such as whether a program addresses a specific need and the extent to which it is redundant with other federal, state, or local programs. Strategic planning concerns whether a program has clear long-term outcomes that reflect the purpose of the program and a reasonable strategy to assess performance, including baseline and annual measures. Program management scores are based on items such as whether data are collected to inform ongoing program refinements and monitor how funds are spent. Finally, program results items focus on things such as whether a program is meeting its goal and is cost effective (Office of Management and Budget, 2005). On the basis of their PART scores, programs are classified as effective, moderately effective, adequate, ineffective, or results not demonstrated (Office of Management and Budget, 2006). Obviously, programs scoring in the latter two categories are prime targets for funding termination. In response to the impending threat of funding termination, many federal programs (e.g., Area Health Educations Centers [AHEC], Health Careers Opportunity Program, Maternal and Child Health) have turned to logic modeling to demonstrate that they are responsive to the criticisms leveled against them and take a proactive approach to resolve their shortcomings. Logic modeling provides a mechanism to ensure that the important dots between underlying assumptions, strategies, and outcomes are meaningfully connected (Gale, Loux, & Coburn, 2006). The inference, of course, is that if these critical elements are in fact logically connected, the likelihood of demonstrating program results and accountability will improve and be reflected in the PART score. However, for this to hold true, the logic-modeling process must be done well enough to produce a high-quality and useful product. If the logic-modeling process is poorly carried out, then the likelihood of a model leading to improvements in program accountability can be reasoned to be negligible. There are at least two undesirable consequences that may result from a poorly conducted logic-modeling process. First, federal agencies may erroneously conclude that logic modeling is an ineffective methodology. The second consequence may be that programs that have the potential to become effective are terminated prematurely because they were unable to capitalize on the benefits that logic modeling can confer. In this article, I assess the quality of a logic-modeling process executed by the Bureau of Health Professions (BHPr). I was directly involved with the BHPr initiative to incorporate logic models in response to the impending threat of funding termination and therefore had a bird’s-eye view of how parts of the process proceeded. I compare the logic-modeling approach implemented by the BHPr to a logic-modeling process I developed, the ATM (antecedent, target, measurement) approach. I then argue that federal programs are unlikely to improve if logic modeling is approached as it was by the BHPr. The BHPr Health professions programs are part of the Title VII mandate to address the shortage of health care professionals in rural and underserved areas. Forty-two federally funded health 454 American Journal of Evaluation / December 2006 Figure 1 The Performance Assessment Rating Tool Assessment for the Bureau of Health Professions Programs professions programs are under the jurisdiction of the BHPr, which is within the Health Resources and Services Administration (HRSA) in the U.S. Department of Health and Human Services. The funding problems for health professions programs have been ongoing for many decades. Most of the recent administrations, regardless of political party, have not included various Title VII programs in their budgets; instead, the congressional appropriations committee reinstates the funding. This process has been costly to program recipients in the forms of time, energy, stress, and even funds. The funding problem reached a pinnacle with the 2002 release of the OMB report regarding BHPr programs. Using the PART, OMB concluded that within the BHPr, disagreement existed regarding the purposes of programs, the failure to use performance data to improve program outcomes, and the limited impacts of some programs on the basis of outcome data available (Office of Management and Budget, 2004a). The actual PART report is shown in Figure 1. On the basis of these findings, the “administration proposes to continue the phase-out of most health professions grants consistent with the 2003 Budget and to direct resources to activities that are more capable of placing health care providers in medically underserved communities” (p. 127). In response to the pressure of OMB’s recommendation, the BHPr engaged in a logicmodeling process designed to develop better performance measures and to more clearly delineate the relationship between program-specific, bureau-level (also called Core measures), and national performance measures (Health Resources and Services Administration, Renger / Logic-Modeling Process 455 2005). I had direct experience with this process by working with the BHPr HRSA National Performance Measures Working Group, chairing the National AHEC Committee on Research and Evaluation, contributing to program publications, and through several workshops and presentations on the logic-modeling process to BHPr officials and program constituents (Huntington & Renger, 2003; Renger, 2003a, 2003b, 2003c, 2005a, 2005b, 2005c, 2005d). Thus, my assessment of the process is informed by firsthand observation, as well as access to documents associated with the process. Before describing the logic-modeling process used by BHPr, the logic-modeling approach used as a standard of comparison in the current article is presented. The ATM Approach to Logic Modeling Numerous logic-modeling approaches have been published in the evaluation and popular literature (e.g., den Hayer, 2002; Goertzen, Fahlman, Hampton, & Jeffery, 2003; Kellogg Foundation, 2001; McLaughlin & Jordan, 1999; Millar, Simeone, & Carnevale, 2001; United Way of America, 1996). The ATM approach (Renger & Titcomb, 2002) was chosen because it is the approach with which I am most familiar and is typical of approaches in the literature. Additionally, I favor the approach because it clearly specifies how a logic-modeling process ought to proceed and integrates evidence-based knowledge into the process of thinking through the logic of programs. As with most logic-modeling methodologies, the ATM approach begins by defining the problem of interest. Most problems are influenced by behavioral, environmental, social, and biological conditions; these factors, or antecedent conditions, must be identified and understood to focus intervention efforts (Green & Kreuter, 1999). The ATM approach uses interviews with individuals who have content expertise in the area of a problem to identify antecedent conditions. Each expert is interviewed individually and is asked a series of questions using the format “Why does this condition occur?” Throughout each interview, a visual map of the relationships of antecedent conditions to the problem and to other antecedent conditions is developed. These maps are then integrated into a single map summarizing all the interviewees’ descriptions of the antecedent conditions. The purpose of the resulting visual map is to illustrate the relationships between problems and their causes; this visual representation of the problem allows decision makers to best understand where program efforts should be focused. A review of the literature provides documentation to determine the extent to which the interrelationships between antecedent conditions and the linkages between antecedent conditions and a problem can be supported by research. In those rare instances in which no supporting evidence is found, the expert interviewees are contacted to determine if they are aware of any supporting evidence and, if not, whether the antecedent conditions should remain in the evolving visual map. This step ensures that the program is based on solid research, not anecdotal evidence. The visual map produced, depicting in some cases as many as 80 antecedent conditions, can be overwhelming. Clearly, even a collaborative does not have the resources and expertise to address all the identified antecedent conditions. Agencies complete a systematic prioritization process to identify those antecedent conditions on which a program might focus. This prioritization approach allows for the engagement of stakeholders to begin identifying those outcomes held important to the agency or coalition (Renger & Bourdeau, 2004). Renger and Bourdeau (2004) published a more detailed description of the prioritization process, which is described using the theory of values inquiry. At this point, agencies can begin brainstorming potential strategies to target the prioritized antecedent conditions. As agencies decide on specific program strategies, they are challenged to (a) explain which of the prioritized antecedent conditions proposed program strategies 456 American Journal of Evaluation / December 2006 Figure 2 Comparing Immediate, Intermediate, and Long-Term Outcomes Immediate Outcome Intermediate Outcome knowledge of harmful effects of sun lowered exposure to sun Long-term Outcome reduction in skin cancer morbidity and mortality target, (b) explain how the proposed strategies are hypothesized to produce change in the prioritized antecedent conditions, and (c) provide detailed written documentation in the form of implementation protocols. Making an impact on antecedent conditions and changing the outcome are central to assessing the merit and worth (i.e., the impact and outcome) of a program (Mark, Henry, & Julnes, 2000; Renger & Bourdeau, 2004). Because the visual map produced in the first step depicts the relationships among antecedent conditions, it is relatively straightforward to define immediate, intermediate, and long-term outcomes (see Figure 2). Antecedent conditions earlier in a sequence become the immediate outcomes (e.g., lack of knowledge of harmful effects of the sun), while antecedent conditions in the middle become the intermediate outcomes (e.g., sun exposure). All the antecedent conditions eventually relate to some long-term outcomes, which are usually related to solving the problem of interest (e.g., skin cancer). Once the outcomes have been identified, a logic model table is completed that summarizes the key elements of the process. The table includes the conditions being targeted (sometimes referred to as program assumptions), a brief description of the activity designed to affect the conditions, and the immediate, intermediate, and long-term outcomes. It is important to note that the logic model table is created after the process and is simply a summary. Since its publication, the ATM approach has been used in a variety of contexts, and many lessons have been learned in its application (Renger & Hurley, 2006). The reader is referred to Renger and Hurley (2006) for a discussion of the limitations of the ATM approach. The BHPr Logic-Modeling Process The logic-modeling process completed by BHPr began by asking each of the 42 programs to generate a logic model summary table. To accomplish this, bureau staff members were assigned to develop a draft logic model summary table and to present the table to the program grantees for feedback. These one-page summary tables contained the following elements: program goal, problem statement, key strategies, program outputs, program outcomes, and performance measures. Although feedback related to all elements of the logic model was welcomed, requests for input centered primarily on what were termed program-specific measures. It is important to note that the context for discussion of program-specific measures occurred independently of any discussion regarding program activities. The question was simply, “What types of measures do you [the grantees] feel reflect the work you do?” The program-specific measures are directly related to the activities of individual programs and “were designed to capture the unique accomplishments of each BHPr program” (Health Resources and Services Administration, 2005, p. 4). On a continuum from immediate to longterm, program-specific measures are more immediate. Renger / Logic-Modeling Process 457 Table 1 Comparing the Logic-Modeling Approaches: Theory Versus Practice Step ATM Logic-Modeling Process BHPr Logic-Modeling Process Impact of Skipping Step on Possibility of Improving Program Effectiveness 1 Consult experts to develop underlying rationale. Not done The scope of possible issues affecting workforce shortage may not be understood. Important issues may be missed. 2 Support evolving underlying rationale with research. Not done Programs may not be based on research evidence. In the absence of research evidence, it is simply hit or miss as to whether what a program is trying to change is in fact important to affecting the problem. 3 Prioritize antecedent conditions. Not done Individual programs are left to decide what to target. Disjointed effort toward affecting change in a common goal. 4 Develop programs to target antecedent conditions. Legislated activities remain fixed. Activities generally do not target antecedent conditions. Change may be observed in the things being measured by individual programs, but these may not relate to changing the outcome. 5 Define outcomes. Program-specific, Core, and national measures defined independently No logical relationship between immediate, intermediate, and long-term outcomes. Data may be gathered that is easy to collect rather than appropriate to collect. 6 Create logic model summary table. Created first Elements in the table may not link "logically". Note: ATM = antecedent, target, measurement; BHPr = Bureau of Health Professions; Core = PLS. DEFINE. The logic-modeling process also centered on identifying Core and national performance measures. The Core measures “summarize accomplishments in areas common to many BHPr programs” (Health Resources and Services Administration, 2005, p. 4). Core measures are more intermediate and are designed to assess changes in primary care and public health services. It is further reasoned that such changes will lead to improving the health status of the nation. I was not directly involved in generating Core measures and so am unable to comment on the exact process by which these came to be defined. National performance measures indicate whether changes in the health status of the nation are realized and are the responsibility of HRSA to collect rather than individual grantees. A national working group of experts was convened to assist in defining the national performance measures. On the continuum of outcomes, national performance measures are quite distal. The national working group met on several occasions; initial conversations centered on whether change could be observed in distal outcomes that depended on so many factors outside the control of HRSA initiative. Despite these concerns, HRSA officials requested that such performance measures be defined, with which the working group complied. Comparing Logic-Modeling Approaches: BHPr Versus ATM Table 1 provides a summary comparing the BHPr approach to logic modeling with the ATM approach. The six steps of the ATM approach are listed in the left-hand column. The extent to which BHPr followed the approach is described in the neighboring column, followed 458 American Journal of Evaluation / December 2006 by a brief summary of the impact on improving program effectiveness. Each of these steps is now discussed in greater detail. Step 1: Make the Underlying Rationale (Program Theory) Explicit I was unable to locate in any of the BHPr documentation (e.g., requests for proposals, notices of funding availability) references to the logic or underlying theory of their supported programs. There were indirect references to the underlying theory embedded in the goal statements and some of the stated objectives, but I could find no specific statement about what was being targeted for change and why. Ensuring that the underlying rationale or program theory is made explicit is of utmost importance (McLaughlin & Jordan, 1999, Renger & Tictomb, 2002). The underlying theory forms the foundation on which to build a meaningful program and an evaluation plan (Chen & Rossi, 1983; Weiss, 1997). Program theory not only describes the conditions affecting the problem and how these conditions are interrelated but gives structure to the planning process by identifying program content most likely to facilitate change (Chen, 1989; Renger & Titcomb, 2002). The underlying program theory makes clear what conditions are most likely to lead to desired outcomes and why. Knowing the program theory is essential to ensuring that (a) objectives are related to the conditions being targeted, (b) program content is linked to the objectives, and (c) the measurement tools selected assess the conditions being targeted for change. If content is not aligned with the objectives, then the likelihood of observing change is small. Similarly, if the measurement tools do not assess the conditions targeted in the objectives, program success cannot be demonstrated. Programs that lack theory-based planning processes have little likelihood of success at achieving program goals and objectives (Weiss, 1997). As previously noted, the ATM approach recommends that experts in a given subject matter be interviewed to create the underlying rationale. This is deliberate, because it allows the underlying rationale to be established relatively quickly and cost effectively, as well as creating buy-in. The BHPr did include several subject matter experts in the logic-modeling process, but these experts were used to define outcomes (see Step 5 below) rather than to establish the underlying rationale. BHPr resources were devoted to deciding what to do rather than understanding why to do it. Step 2: Support the Evolving Underlying Rationale With Research The BHPr programs, like many federal programs, are primarily service oriented. As such, they should be founded on solid research evidence so that program activities are based on reliable methods in accomplishing social change. In only a few instances was I able to find parenthetical references in the BHPr literature to research evidence justifying the implementation of legislated activities. For example, in the AHEC legislation, there is reference to professional isolation as a reason why health professionals choose not to practice in rural settings (Azer, Simmons, & Elliott, 2001; Xu & Veloski, 1998). However, it is more generally the case that there is a paucity of such explanation. There are several consequences of not ensuring and/or providing the research evidence for the underlying rationale. First, it creates confusion for those implementing a program as to whether they must gather evaluation data to assist in decision making (i.e., a service program), knowledge development (i.e., a research program), or both (Mark et al., 2000). Being clear as to whether a program is service or research oriented has a significant impact on the evaluation plan and funding allocations. For example, in a service program, process evaluation is instituted to assist with ongoing refinements. Making changes to protocols to assist delivery is perfectly acceptable in a service program, whereas changing protocols midstream is a Renger / Logic-Modeling Process 459 questionable practice in a program of research. Also, knowing whether immediate and intermediate objectives were met (i.e., the impact evaluation) will assist program staff members in deciding whether course content and/or activities need to be altered. A lack of clarity about the purpose of a program, amplified by the failure to explicitly cite research evidence for the program, can result in an unnecessary drain of resources. For programs under severe budget restraints, like those with the BHPr, this means funds that are desperately needed to improve the likelihood of demonstrating an impact (e.g., implementation, content improvement) are being depleted and diverted to gather evaluation data for an unnecessary research agenda. Second, and perhaps the most important reason for requiring that service programs be based on solid research evidence, is the assurance that program activities are in fact targeting conditions that will produce desired outcomes. It is often the case that the funding cycles of service programs are of insufficient duration to track long-term outcomes. Thus, it is necessary to demonstrate that changes in more immediate outcomes (e.g., information about the harmful effects of smoking), will lead to changes in intermediate outcomes (e.g., smoking cessation), which will ultimately lead to changes in long-term outcomes (e.g., reduction in cancer morbidity and mortality). In the absence of an explicitly stated program theory, it is uncertain whether the legislated activities target conditions necessary to produce changes in the problem of interest. This may partially explain why there has been no discernable change in the shortage of workforce professionals in the past two decades (Office of Management and Budget, 2004a). Because developing program theory grounded in research can be a time-consuming and arduous task, it is easy to forgo making this initial investment. However, despite being a resourceintensive task, it is critical because “everything which follows depends on how well the project is initially conceptualized” (Trochim, 1989, p. 1). Simply put, you get out what you put in. The BHPr’s attempt to use front-line staff members and legislative language as a first attempt to make the program theory explicit is not uncommon and an excellent first step to improving accountability. However, it is the ability to explicitly provide the underlying program theory that is necessary in impressing the PART examiners. Step 3: Prioritize Antecedent Conditions As noted earlier, the mapping process results in myriad conditions that are beyond the scope of any single funding initiative to address. The program evaluation standards are clear in the need to engage stakeholders to identify outcomes held important to the agency or coalition (Joint Committee on Standards for Educational Evaluation, 1994). Thus, it is important to include decision makers in the process of deciding which antecedent conditions should be targeted and which agencies should target them. Prioritization ensures that antecedent conditions are targeted by the appropriate agencies with enough capability to produce change in a condition. Although more than one agency may address a given antecedent condition, this step also guarantees that an overduplication of services does not exist. The BHPr did not engage in a prioritization process. As a result, there was an uncoordinated effort between the 42 BHPr programs. Some of the programs targeted immediate conditions of less importance rather than observing change in long-term outcomes, whereas others targeted conditions completely unrelated to the problems. This made it virtually impossible to combine information from across program sites to arrive at overall conclusions regarding effectiveness. Perhaps most important, the prioritization process would enable the BHPr to realize that its coalition of programs targets only a small subset of the conditions that affect the workforce shortage problem. Knowing this would help the BHPr understand that there are numerous antecedent conditions affecting the workforce shortage problem (i.e., the long-term outcome), which the BHPr had little or no control to change, such as the low socioeconomic status of rural areas (Huntington & Renger, 2003). Consequently, the likelihood of observing change in the long-term 460 American Journal of Evaluation / December 2006 outcome on the basis of the efforts of BHPr alone is remote. Once again, this may help explain why the BHPr has been unable to demonstrate any change in the shortage of health professionals. It also brings further into question the utility of defining additional national performance measures (see Step 5 below), which are even further out of the immediate control of BHPr to change. Step 4: Develop Programs to Target Antecedent Conditions It is essential that the program content (i.e., activities or strategies) be linked to the objectives (Chen, 1989; Chen & Rossi, 1983; Weiss, 1997). If it is not, there is little likelihood of meeting the objectives (Weiss, 1997). The failure to link content to the underlying rationale has been termed an activity trap (Renger & Titcomb, 2002; Spath, 2003). Activity traps are well-intended activities that appear to address particular problems but on closer inspection do not address any of the conditions (e.g., barriers, risk factors, antecedent conditions, behavioral factors, or environmental factors) that underlie the problems (Renger & Titcomb, 2002). Activity traps become apparent in the logic-modeling process when the relationship between proposed activities and the underlying conditions of a problem are established. The BHPr did not engage in this linking exercise. Instead, legislated activities remained unchanged. Thus, it is likely that many of the legislated activities will not relate to changing immediate conditions that are essential to producing long-term changes. The problem of activity traps can be illustrated using a common legislated activity in the health professions domain: rural rotations. The rationale for rural rotations is that health professions students will choose to practice in rural areas because they have had an opportunity to experience this environment. There is some evidence (Lynch et al., 2001) that rotations can be effective. However, the problem is that the known research regarding the reasons why students choose not to practice in rural areas has not been integrated into the legislative language directing these programs. Factors such as employment opportunities for spouses, the quality of schools for children, autonomy, and so forth, are all key factors in the decision whether to practice in a rural setting (Azer et al., 2001; Xu & Veloski, 1998). Despite the availability of such research evidence, it is absent in the BHPr documentation and does not shape the activities. As such, the majority of rotations remain focused on simply providing clinical skills. Across all programs, there is a dearth of structured activities (after clinical hours) specifically designed to shape the perceptions of working in rural settings and ultimately the decision of where to establish a practice. As a result, rural rotations continue to be an activity trap because essentially no activities are included to address the underlying reasons why students choose not to establish practices in rural areas. It is reasonable to assume that the logic-modeling process may have revealed that some of the original assumptions underpinning the current legislation were either lacking or invalid. Therefore, to include these new assumptions may have required amending funding legislation, a potentially daunting task. However, many of the current legislated activities could be transitioned to focus interventions without significant rewrites to legislation. For example, including structured activities, such as visits by the chamber of commerce, meetings with the local school principal and teachers, and so forth, could be included as part of the rural rotation experience to address issues known to affect a student’s decision to practice in a rural setting. The key to this transition requires that the underlying issues be made explicit (see Steps 1 and 2 above), so that grantees are forced to provide plans to address these conditions and measures to evaluate change in these targeted conditions are collected. Step 5: Define Outcomes As discussed above, defining immediate, intermediate, and long-term outcomes is relatively simple with the ATM approach, because they are derived directly from the underlying Renger / Logic-Modeling Process 461 rationale. Prioritized conditions at the beginning of a sequence are the immediate outcomes, those in the middle are the intermediate outcomes, and those at the end are the long-term outcomes. It is important to note that through the logic-modeling process, each tier of outcomes is easy to define, and there is assurance that they are in fact related to one another. With regard to BPHr, the process of defining the three tiers of performance measures (i.e., program specific, Core, and national) occurred independently. One consequence of developing these different levels of outcome measures independently is that there is no logical connection between them. There exists no underlying rationale that states changing immediate conditions will influence the change of long-term conditions. Therefore, changes observed at one level may not necessarily result in changes at another level. The lack of an underlying program rationale limits the ability to clearly define the immediate, intermediate, and long-term outcomes. Therefore, in the absence of clearly defined outcomes, it is easier to stray from the difficult task of identifying measures appropriate for assessing change in immediate, intermediate, and long-term outcomes and opt to gather data that are relatively easy (i.e., available) to collect. This may also help explain OMB’s (2004a) conclusion that the BHPr is ineffective. That is, in completing the PART report, the OMB relies on annual data provided by HRSA. HRSA requests this information from its respective offices, in this case the BHPr, which in turn requires program recipients to collect and report to them quarterly and annually. The data relate primarily to describing the nature of the program and its participants and can be characterized as oversight and compliance (Mark et al., 2000). Examples of oversight and compliance data include the ethnicity, gender, and economic status of participants, as well as the number of events sponsored. To draw conclusions about a program’s effectiveness, as is done in the OMB report, data are needed that evaluate changes to targeted antecedent conditions. This is defined as an evaluation of merit and worth, or impact (Mark et al., 2000). Merit and worth evaluations signal the effectiveness of a program in making a difference in the lives of the participants as a result of program participation. The problem is that OMB draws conclusions about the impact a program has on the lives of participants and society from oversight and compliance data. Simply put, it is impossible to judge the difference programs made in the lives of participants from data that simply describe the nature of the program and its participants. Step 6: Create a Logic Model Summary Table BHPr staff members were assigned to develop a logic model for each of the 42 programs and were not asked systematically link the necessary elements (e.g., activities; immediate, intermediate, and long-term outcomes) found in most logic models together. Staff members essentially completed their assigned task by placing legislative language in a logic model summary table. Herein lies a major problem, in that although the BHPr had a logic model summary table for each program, these did not reflect the outcome of a process of systematically identifying the conditions that are associated with the problem of interest, linking intervention activities to these underlying conditions, and identifying outcomes that are related to these conditions, so the components of the logic models were typically not rationally and meaningfully linked together. Summary Although the BHPr developed several logic models, this article shows that the agency did not use the logic-modeling process to its best advantage and failed to observe best practices in creating these models. If logic models were to form the basis for rescuing these programs 462 American Journal of Evaluation / December 2006 from elimination, it seems to me that the approach to logic modeling taken by BHPr provides an unlikely source of salvation for these programs. It is my hope that this article will motivate those who are in control of the BHPr programs to redo the logic-modeling process to a better end. Logic modeling can be a valuable tool, if done well. From an evaluation perspective, the results of this analysis raise several important questions for our field. One question relates to the limits and conditions under which evaluators should engage in capacity building. In the current context, the agency simply did not have the resources and time to develop the capacity necessary to complete a quality logic-modeling approach. In this instance, when the agency faced these constraints, it chose to assign staff members with little or no evaluation expertise the task of completing logic models. Despite their best intentions, these staff members did not posses the experience or expertise to conduct a quality evaluation, making the likelihood of achieving better program results even more remote. Because capacity building was not an option, the agency might have considered hiring an external evaluator. The dilemma of course is that the agency did not have the resources to hire an external evaluator either. This example yet again points to the importance of integrating evaluation in the planning process. Another question that is raised from this work is the extent to which evaluation methodology should be mainstreamed. For a trained evaluator, logic modeling is a relatively simple and useful tool. When all the steps are followed, logic modeling can be completed efficiently and at minimal cost. However, if steps are not followed correctly or are bypassed all together, the result is more costly and often of lower quality. Publications such as those by the United Way of America (1996) and the Kellogg Foundation (2001) attempt to simplify the logic-modeling methodology so that agencies with minimal resources and expertise can still benefit from the process. However, an argument can be made that the context of each agency is unique and presents challenges that may require adaptation of the process. Adaptation requires a deeper understanding of the purposes of evaluation and methods. Perhaps in some instances, as illustrated in this article, it is not in the best interest of agencies to follow mainstream publications. To quote an old adage, a little knowledge is a dangerous thing. Other fields, such as psychology, are careful in not allowing the public at large access to their methodologies and assessment tools. For example, only licensed psychologists can administer the Minnesota Multiphasic Personality Inventory and other inventories. Perhaps the field of evaluation should more closely examine the extent to which it freely provides its assessment methods to the public. Perhaps the field should exert more control over the application and delivery of its methodologies to improve the likelihood of producing quality evaluations. Failing to do so could undermine the credibility and integrity of the field. References Azer, S. A., Simmons, D., & Elliott, S. L. (2001). Rural training and the state of rural health services: Effect of rural background on the perception and attitude of first-year medical students at the university of Melbourne. Australian Journal of Rural Health, 9, 178-185. Bush, G. W. (2005, February 7). State of the union address. Available at http://www.whitehouse.gov/news/releases/ 2005/02/20050202-11.html Chen, H. T. (1989). The conceptual framework of the theory-driven perspective. Evaluation and Program Planning, 12, 391-396. Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7(3), 283-302. den Hayer, M. (2002). The temporal logic model concept. Canadian Journal of Program Evaluation, 17(2), 27-47. Gale, J., Loux, S., & Coburn, A. (2006). Creating program logic models: A toolkit for state Flex programs. Portland: Renger / Logic-Modeling Process 463 University of Southern Maine, Flex Monitoring Team. Goertzen, J. R., Fahlman, S. A., Hampton, M. R., & Jeffery, B. L. (2003). Creating logic models using grounded theory: A case example demonstrating a unique approach to logic model development. Canadian Journal of Program Evaluation, 18(2), 115-138. Greene, L. W., & Kreuter, M. W. (1999). Health promotion planning: An educational and ecological approach (3rd ed.). New York: McGraw-Hill. Health Resource and Services Administration. (2005). Bureau of health professions: First all-grantee conference. Performance Measurement System. Washington, DC: Department of Health and Human Services. Huntington, C., & Renger, R. (2003, April-June). NAO’s response to the OMB—PART report. NAO News of the National AHEC Organization. Retrieved from http://www.nationalahec.org/Publications/documents/Response %20to%20the%20OMB%20Performance%20and%20Management%20Assessment%20of%20the%20Health% 20Professions%20Program%202003.pdf#search=%22huntington%20renger%22. Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage. Lynch, D. C., Teplin, S. E., Willis, S. E., Pathman, D. E., Larsen, L. C., Steiner, B. D., et al. (2001). Interim evaluation of the rural scholars program. Teaching and Learning in Medicine, 13(1), 36-42. Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass. McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program’s performance story. Evaluation and Program Planning, 22, 65-72. Millar, A., Simeone, R. S., & Carnevale, J. T. (2001). Logic models: A systems tool for performance management. Evaluation and Program Planning, 24, 73-81. Office of Management and Budget. (2004a). Program: Health professions. Available at http://www.whitehouse .gov/omb/budget/fy2004/pma/healthprofessions.pdf Office of Management and Budget. (2004b). Rating the performance of federal programs. Available at http://www .gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf Office of Management and Budget. (2005). Budget Procedures Memorandum No. 861. Available at http://www .whitehouse.gov/omb/budget/fy2005/pdf/bpm861.pdf Office of Management and Budget. (2006). Performing programs. Available at http://www.whitehouse.gov/omb/ expectmore/perform.html Renger, R. (2003a). The need for a deeper level of accountability—And how we might get there. National AHEC Bulletin, 19(2), 10-13. Renger, R. (2003b). A standardized tool (vendor) for collecting data. National AHEC Bulletin, 19(2), 16-17. Renger, R. (2003c). Strategic directions for improving documentation of AHEC/HETC accomplishments. National AHEC Bulletin, 19(2), 14-16. Renger, R. (2005a). Application of logic models to the Hawaii AHEC. Honolulu, HI. Renger, R. (2005b). Logic modeling. State Offices of Rural Health (SORH) pre-meeting. Washington, DC: Health and Resources Services Administration, Office of Rural Health Policy, all programs meeting. Renger, R. (2005c). Logic modeling for Moloka’i Substance Abuse Program. Kaunakaka, HI. Renger, R. (2005d). National Health Outcomes Working Group. Presentation at the Health Resources and Services Administration all grantee meeting, Washington, DC. Renger, R., & Bourdeau, B. (2004). Strategies for values inquiry: An exploratory case study. American Journal of Evaluation, 25, 39-49. Renger, R., & Hurley, C. (2006). From theory to practice: Lessons learned in the application of the ATM approach to developing logic models. Evaluation and Program Planning, 29(2), 106-119. Renger, R., & Titcomb, A. (2002). A three-step approach to teaching logic models. American Journal of Evaluation, 23, 493-503. Spath, P. (2003). Don’t get caught in the activity trap. Hospital Peer Review, 28(9), 130-132. Trochim, W.M.K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1-16. United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: Author. Weiss, C. H. (1997). How can theory-based evaluation make even greater headway? Evaluation Review, 21(4), 501-524. W. K. Kellogg Foundation (2001). Logic model development guide. Available at http://www.wkkf.org/Pubs/ Tools/Evaluation/Pub3669.pdf Xu, G., & Veloski, J. J. (1998). Debt and primary care physicians? career satisfaction. Academic Medicine, 73(2), 119-124. A RT I C L E Process evaluation: the new miracle ingredient in public health research? Q R 699 Qualitative Research Copyright © 2010 The Author(s) http://qrj.sagepub.com vol. 10(6) 699–713 A L I S O N M U N RO Glasgow Caledonian University MICHAEL BLOOR University of Glasgow Good evaluation practice in public health research has become equated with the inclusion of a mixed-methods ‘process evaluation’ alongside an ‘outcome evaluation’ to gather data on how and why interventions are effective or ineffective. While the incorporation of process evaluations in randomized controlled trials is to be welcomed, there is a danger that they are being oversold. The problematic position of process evaluations is illustrated by data from an evaluation of an unsuccessful schools health promotion intervention. The process evaluation data (designed to ‘explain’ the outcome evaluation results) must be collected before the outcome evaluation results are typically available: unanticipated outcomes cannot always be addressed satisfactorily from prior process data. Further, qualitative process data draw inductively general inferences from particular circumstances and the generalizability of those inferences is therefore uncertain: qualitative data can deepen our understanding of quantitative data, but the commensurability of the two classes of data remains problematic. a b s t r ac t keywords: complex interventions, drugs, mixed methods, process evaluation, public health, schools, smoking, triangulation Introduction It has been suggested that, after the long-standing ‘paradigm wars’ between qualitative and quantitative methods, ‘paradigm peace’ has broken out (Bryman, 2006). One seeming manifestation of this paradigm peace is found in public health research where qualitative methods have been increasingly incorporated into outcome studies, that may utilise a variety of study designs including the randomized controlled trial (RCT), as part of a mixed-methods ‘process evaluation’: a quantitative ‘outcome evaluation’ is conducted to determine DOI: 10.1177/1468794110380522 700 Qualitative Research 10(6) whether or not a public health initiative has been effective, while a mixed-method ‘process evaluation’ is conducted to explain why the intervention was successful or unsuccessful. Such outcome-evaluation-plus-process-evaluation RCTs correspond to what Moran-Ellis et al. (2006) have termed ‘integrated methods’, where different methods retain their paradigmatic natures but are inter-meshed with each other to deepen understanding of the phenomenon under study. Such integrated methods designs have clear advantages which need not be reiterated here, but there are also dangers that the burden of expectation on process evaluations is too great. C O M P L E X I N T E RV E N T I O N S Much social research, including evaluations of educational interventions and public health initiatives, involves the assessment of ‘complex interventions’ (Giannakaki, 2005; Oakley et al., 2006; Victora et al., 2004; Young et al., 2008), that is to say that the interventions are multifaceted, organizationally elaborate, and socially mediated. These complex interventions sit uncomfortably with the classic positivist model of health services research, with its ‘hierarchy of evidence’ and the randomized controlled trial (RCT) at its apex (Maynard and Chambers, 1997): a simple RCT of a complex intervention will be wholly lacking in explanatory power (Bradley et al., 1999). The outcome of the RCT will tell the evaluators whether or not the intervention has an effect, but they will have no idea why: the processes of the delivery of the intervention and its reception remain in a ‘black box’. This lack of explanatory power is of course particularly unfortunate when the outcome evaluation shows the intervention to have been ineffective, because not only are there no data available on why the complex intervention failed, there are also no data available to suggest how that ineffective intervention might be successfully modified. Many an experienced investigator has thus been reduced to embarrassing flights of wild speculation in accounting for his or her disappointing RCT results. P RO C E S S E VA L UAT I O N It has therefore become commonplace, and considered as part of good research practice, for a ‘process evaluation’ to be paired with an ‘outcome evaluation’ in the design of studies of complex interventions (Oakley et al., 2006; O’Cathain, 2009). Advice on the conduct of RCTs by bodies such as the United Kingdom’s Medical Research Council lays particular stress on the importance of a process evaluation component in the pilot or early stages (Phases I and II) of an RCT, both to contribute to the decision about whether to proceed to a full-scale (Phase III) trial, and to inform final changes in the design of both the intervention and the outcome evaluation. Process evaluations gather data on the social processes involved in the delivery of the intervention, the reception of the intervention and the setting of the intervention. They frequently entail mixed methods (Oakley, 2005) involving both survey questions and semistructured interviews, sometimes focus groups, sometimes direct observation, Munro and Bloor: The new miracle ingredient in public health research? and sometimes (in the collection of data on the setting of the intervention) documentary methods. The advocacy of process evaluations seems to be, if not admirable, then at least unexceptionable. Until, that is, one begins to examine in detail the explanatory burden being placed on the process evaluation components of these evaluation studies. Consider, for example, this statement from an early widely-cited paper on process evaluation design: Process evaluation complements outcome evaluation by providing data to describe how a program was implemented, how well the activities delivered fit the original design, to whom the services were delivered, the extent to which the target population was reached, and the factors external to the program that may compete with the program effects. (McGraw et al., 1994: S5) The reader may think that McGraw et al. are looking for an awful lot of bangs for their buck. But Oakley et al.’s (2006) influential ‘analysis and comment’ paper on process evaluations in the British Medical Journal is, if anything, even more demanding: [Process evaluations] … may aim to examine the views of participants on the intervention; study how the intervention is implemented; distinguish between the components of the intervention; monitor dose to assess the reach of the intervention; and study the way the effects vary in subgroups. (Oakley et al., 2006: 413) A subsequent paper from the same team argued that process evaluations were also essential to allow an assessment of the generalizability, or external validity, of complex interventions (Bonnell et al., 2006). Consider also that these complex interventions are frequently multi-site trials, involving perhaps a score or more of clinics (or communities, or schools, etc.) and perhaps an equal number of intervention delivery teams. Furthermore, the effect of the intervention may be time-limited and data may have to be collected in these various sites longitudinally. The reader will no doubt now be forming the notion that any aspiration to collect systematic data on all these different topics or themes, at all these different sites, at a number of different points in time, will inevitably lead the researcher into that nightmare study mis-design, the intensive largescale study. Enormous effort expended, mountains of complex data collected, and no earthly chance of making any sense of it all. Indeed difficulties associated with the collection of ‘too much’ data have been reported by other process evaluators (e.g. Hong et al., 2005). Consider too that funders are not just being expected to fund process evaluations: there is also the outcome evaluation to fund (does the intervention have an effect or not?). Outcome evaluations can also come with a hefty balance sheet, particularly if, as well as survey data, they involve physical measures – lung function tests, oral fluid samples for testing exposure to illicit drugs or tobacco, and so on. The funder may also be being asked to fund the intervention itself, including salaries and training for the team delivering the intervention. And there may be other costs too: for example, every trial 701 702 Qualitative Research 10(6) application to the UK’s Medical Research Council is required either to incorporate a health economics evaluation or an explanation of why such an evaluation is not required (Medical Research Council, n.d.). There are therefore budgetary reasons as well as sound study design reasons for limiting the scope of data collection for a process evaluation. Hence, the contention has been made that there is a need to prioritize both the ‘type and amount of data collected’ (Linnan and Steckler, 2002). The wise Principal Investigator therefore does not design his/her process evaluation to collect comprehensive data on all aspects of the delivery, reception and context of the intervention. Rather, the optimum design will involve some element of selective sampling in respect of the more labour-intensive qualitative data collection and analysis. A typical design might involve the following comprehensive data components: first, comprehensive survey data (cross-sectional or longitudinal) at all intervention and control sites (clinics, communities, schools, etc.), usually as part of the outcome evaluation survey instrument; second, attempted comprehensive gathering and analysis of documents concerning contextual and confounding factors relating to the possible differential reception of the intervention at the different sites and to changes in outcome measures at the control sites; third, additional interview data on the same topic collected from key respondents (leading clinician, community leader, senior teacher, etc.) at the intervention sites; and fourth, attempted comprehensive interview data, gathered from those responsible for delivering the intervention at the various sites and seeking information how variously the intervention was delivered and received. While the typical selective data components might be: observation of the delivery of the intervention at a selection of the sites; and qualitative interviews or focus groups with a selection of the recipients of the intervention at a selection of the intervention sites together with comparable data from a selection of control sites. See Parry-Langdon et al. (2003) for an extended description of one such process evaluation design. However, while the above may be a sound process evaluation design in that it blends comprehensive and selective elements in a practicable manner, it does not necessarily meet the demands of positivistic health services research in opening up the ‘why questions’ black box – it does not explain unproblematically why a complex intervention worked or did not work. This paper uses data from a process evaluation of a complex intervention that did not work as a particular case study to illustrate a number of general reasons why there should be a necessary degree of indeterminancy to process evaluation explanations of outcome evaluation results. Process evaluations are not a miracle ingredient. Methods The process evaluation reported here was conducted as part of a feasibility study (Phase I and II) for a full-scale trial (Phase III) of a schools-based, peer-led, Munro and Bloor: The new miracle ingredient in public health research? drugs prevention programme. The aims of the study were threefold: to develop and deliver a training package for S2 school pupils; to assess the feasibility of a future (Phase III) randomized controlled trial; and to conduct a process evaluation of the delivery and impact of the interventions. Based on the successful ASSIST (A Stop Smoking In Schools Trial) programme for reducing the uptake of cigarette smoking amongst adolescents (Campbell et al., 2008), the intervention aimed to train a proportion of ‘influential’ S2 school pupils (second year of Scottish secondary school) to intervene with their peers, i.e. to have conversations in informal settings such as break times, within schools to prevent cannabis smoking. The influential pupils were nominated anonymously by their peers at the first (of three) data sweep(s) and then trained to become ‘peer supporters’ (n=107). Since it was thought possible that an intervention which was already known to reduce pupil smoking prevalence could, of itself, also reduce cannabis prevalence, the feasibility study followed a three-arm design with two intervention arms and a control arm. A total of six schools participated in the study. In one intervention arm, two schools received the original ASSIST programme which involved two days training for ‘peer supporters’ at a venue away from school, plus follow-up visits, over a 10-week period. In the second intervention arm, two schools received the ASSIST training and follow-up plus an extra day’s training devoted exclusively to training on cannabis prevention. The training was delivered by experienced health promotion trainers, some of whom had been involved in the delivery of the training in the original ASSIST study. Since the ‘ASSIST’ acronym had already been used in Scotland for an adolescent suicide prevention programme, the acronym CASE (Cannabis And Smoking Education) was used instead; those schools receiving the renamed ASSIST intervention we designate here as CASE schools, while those schools receiving ASSIST and an additional third day’s training on cannabis, we designate CASE+. The two control schools were asked to continue with their usual programme of health education and therefore received no additional intervention. Survey data and saliva samples (as an encouragement to truthful reporting) were collected at pre-intervention baseline, immediately post-intervention and three-months post-intervention. From a potential 1128 pupils, 896 participated at the first data sweep (achieving a 79% response rate), and data were collected from 732 pupils at all three sweeps. Outcome evaluation data were collected on cigarette and cannabis smoking, but because it was not expected that differences between intervention and control schools in these behavioural measures would necessarily be evident within this deliberately ‘under-powered’ feasibility study (as opposed to a full-scale trial involving many more schools), additional outcome measures were piloted and used, designed to elicit from pupils their future intentions on cannabis smoking, both in six months time and aged 16. See Munro and Bloor (2009) for a full report of the outcome evaluation. 703 704 Qualitative Research 10(6) The design of the process evaluation largely followed that of the earlier ASSIST trial (Audrey et al., 2006a; Parry-Langdon et al., 2003). Survey data were collected at each of the three data collection sweeps on numbers of conversations (as reported by both peer supporters and their fellow pupils) about both cigarette smoking and cannabis. Post-intervention, focus groups (see, for example, Bloor et al., 2001) were conducted with the peer supporters (in one intervention school it only proved possible to conduct a focus group with the female peer supporters), qualitative interviews (see, for example, Gubrium and Holstein, 2002) were conducted with the trainers, and qualitative interviews were also conducted with key staff contacts in the intervention and control schools. Observational data (see, for example, Atkinson et al., 2001) were also collected on the training. Semi-structured interviews and focus groups were transcribed and systematically analysed along with the fieldnotes. Ethical approval was granted by the UK National Health Services’ local research ethics committee. The results of the feasibility study were uneven. Measures of the dose, reach and fidelity (Young et al., 2008) of the interventions were all found to be good. In other words, the training was delivered in the way it was intended, and was received well by the target group (the pupils). In addition, the peer supporter training was viewed positively by the key school personnel. Furthermore, no major harms or negative consequences were experienced by pupils: in two of the schools, the key contacts stated that the peer supporters had gained in confidence and were making positive contributions to school life; 73 per cent of the peer supporters agreed with the statement that ‘being a peer supporter made me feel more confident’. And useful information was also gained that was relevant to the design of a future full-scale trial (Munro and Bloor, 2009). However, although the expected absence of an effect on reported cigarette and cannabis smoking was confirmed, there was also no effect on intentions to smoke cannabis in the future between the intervention schools and the controls. Further, and rather alarmingly, there was actually a significant increase over time among the peer supporters in their expectations that they would be smoking cannabis when they were 16. And, in respect of the survey data on pupil conversations about cannabis, there was a significant difference (p=0.03) between the two CASE+ intervention schools: in the immediately post-intervention survey, 27 per cent of pupils in one school reported having had a conversation with a peer supporter about cannabis, but only 9 per cent of pupils reported such a conversation in the other CASE+ intervention school. This difference between the CASE+ schools was not evident in pupils’ reports of conversations with peer supporters about cigarette smoking (Munro and Bloor, 2009). In what follows, we report in detail on our attempts, using the process evaluation data, to interpret the effects of the feasibility study and – we hope – to illustrate thereby the necessary degree of indeterminacy in process evaluation findings in general. Munro and Bloor: The new miracle ingredient in public health research? Problem 1: data gathering without the benefit of hindsight The prevalence of cannabis use (in the last month) among Scottish 13 yearold pupils is only 2 per cent (Maxwell et al., 2007). The questions about pupils’ intentions or expectations about future cannabis use (e.g. ‘Do you think you will take cannabis [marijuana, dope, hash, blow, joints] when you are 16 years old?’) were therefore chosen as alternative outcome measures for the feasibility study because it was thought that questions about future intentions would produce higher proportions of would-be cannabis smokers than questions about current cannabis smoking, and thus would offer a greater chance of showing an intervention effect in this relatively small population. In the event, and despite prior piloting, the questions proved doubly problematic. In the first place, the proportion of would-be cannabis smokers (at 16 years) at baseline was low – 4.2 per cent in the four intervention (CASE and CASE+) schools, so the questions’ supposed utility as more sensitive outcome measures was undermined. And, in the second place, regression analyses of changes in intentions at 16 years between pre-intervention baseline and three-months post-intervention showed that peer supporters were 4.3 times (95% confidence interval: 1.6–11.7) more likely over the period of the study to think they would be smoking cannabis at 16. Being a peer supporter proved to be a similarly significant factor in regression analyses conducted on changes in expectations of smoking cannabis in three months time. This was an unexpected finding. It would be a matter of concern if it were to be repeated in a full-scale trial. It would not be the first time that a drugs education intervention had produced an unintended negative effect, increasing the propensity to future drug misuse (cf. Palinka et al., 1996), but the point we wish to stress here is that the regression analysis findings were unexpected. The significance of the unanticipated nature of the findings lies in the fact that study timetables do not allow the luxury of the postponement of process evaluation data collection until the results of the outcome evaluation are known. In planning the process evaluation data collection it is good practice, even necessary, to anticipate the results of the outcome evaluation. In this study (as in many similar studies), much of the process evaluation data is collected in advance of the final sweep of outcome evaluation data collection (in this case, three months after the end of the intervention being monitored in the process evaluation). By the time the regression analyses were completed, implying an increased propensity for peer supporters to expect that they would go on to smoke cannabis, the focus groups with the peer supporters had long since been completed (and indeed a new school year had started) and the opportunity to quiz the peer supporters on these unanticipated changes in their expectations had been lost. The focus groups could only be oriented towards addressing anticipated changes because of their place in the study timetable. As it was, the focus groups yielded a very positive view of the peer 705 706 Qualitative Research 10(6) supporters’ reactions to the training on cannabis and gave no hint that peer supporters might be more likely to use cannabis after the training. For what it is worth, we strongly suspect that the changes in peer supporters’ responses to the intentions/expectations questions pre- and post-intervention had nothing to do with an increased resolution to use cannabis in the future, but rather were indicative of a sense of fatalism among these adolescents about their future vulnerability to drug misuse, a sense of fatalism that may have been inadvertently increased by their increased awareness of cannabis as a result of their peer supporter training. This would be consistent with other findings on a culture of fatalism towards health risks including drug misuse (e.g. Douglas and Calvez, 1990), but we are unable to support this suspicion from focus group data because we did not anticipate the future need to examine this issue. This is not sloppiness on our part, it is a problem inherent in conventional process evaluation designs: to explain all those why and how questions arising from the outcome evaluation, the process evaluation needs to be designed with 20:20 hindsight. Problem 2: induction and generalizability All analysis of qualitative data is based upon inductive (rather than deductive) thinking, upon drawing general inferences from particular circumstances (consider for example, methodological writings on analytic induction techniques in qualitative analysis [e.g. Bloor, 1978]). And there is therefore a tension here between one of the alleged purposes of process evaluations, namely to pronounce on external validity or generalizability, and the methods of analysis used, that of inferring general statements from particular instances. This same tension can be found in our analyses in this study. As previously stated, in one of the CASE+ intervention schools only 9 per cent of the pupils (as opposed to 27% in the other CASE+ school) reported a conversation about cannabis with a peer supporter in the immediate aftermath of the intervention. When it came to conversations about cigarettes, the discrepancy between the two schools was less marked, with 23 per cent of the pupils in the first-mentioned school reporting a conversation with a peersupporter, compared with 34 per cent in the other CASE+ school. And indeed that 23 per cent of pupils reporting conversations about cigarettes was higher than in either of the two other (CASE) schools that just received peer supporter training about cigarettes and not about cannabis. So we might infer that the peer supporters in the first-mentioned school only had difficulties with their peer supporter role in respect of preventing cannabis smoking, not in respect of cigarette smoking. There is some support for this inference from the focus group data and also grounds for further development of the inference. There was general agreement, among peer supporters in all the focus groups and across the different schools that cannabis was more difficult to talk about than cigarette smoking. Munro and Bloor: The new miracle ingredient in public health research? The following extract is taken from the first-mentioned school, but similar sentiments could be found in transcripts of other groups: Researcher: Yeah and how did you decide like whether you were going to talk about cannabis? Or did you decide whether you were going to look at, talk about cannabis this day or talk about cigarettes?] Pupil 3: Smoking, because not as many people Pupil 2: [Use cannabis Researcher: So what did that mean then for cannabis, was it] Pupil 5: It was still spoken about but not as much Pupil 4: Because there’s not, not a lot of people do it, aye, like, well not the people we know anyway Researcher: Uhuh, so what did that mean? Did that mean you tended not to bring it up, or it made it very difficult to bring up, or] Pupil 3: It made it quite difficult] Pupil 2: [Quite difficult Researcher: Right. Pupil 3: Cos they might think that you thought they were taking it then. The extract begins with the group suggesting that cigarettes are easier to talk about because cigarette use is more widespread, but it concludes with the suggestion that talking about cannabis to fellow pupils is more interactionally difficult because the fellow pupil may think that the peer supporter believes them to be a cannabis smoker. So we now have some grounds for peer supporters feeling that talking to fellow pupils is more difficult, and evidence that peer supporters in one of the CASE+ schools tended to concentrate their efforts on discussions about cigarettes rather than cannabis. There is the further question of why this concentration on cigarettes should occur in one of the CASE+ schools and not the other. Some relevant evidence here is supplied by the qualitative interviews with the trainers who both commented that the girls who were peer supporters in the first-mentioned CASE+ school were likely to have difficulties in performing their roles: Trainer 1: Very, very painfully shy and they seemed to really lack self-esteem and quite a lot of the things that they were asked to do they really struggled with – like with the role play. Trainer 2: You just think that if they are not confident enough to do things in that safe environment [the training venue], you know, that’s the whole idea that you can try things there, you know, and then put them into practice when they are back in school. But they weren’t happy to try it really. Moreover, the male peer supporters in the same school expressed doubts in the focus group of the effectiveness of the cannabis component of the intervention. So we might infer that, in this school, the perception that talking about cannabis was more difficult than talking about cigarettes was wedded to lack of 707 708 Qualitative Research 10(6) confidence (girls) and pessimism about effectiveness (boys) to result in a concentration of peer supporters’ efforts on cigarettes, rather than on cannabis. Ostensibly, this seems a valuable finding to emerge from the process evaluation. It suggests that the intervention needs to be re-designed to remove the element of discretion for peer supporters that allows them to concentrate on prevention of cigarette smoking, rather than cannabis smoking. It would seem that, despite the substantial economies involved in training peer supporters for more than one task at a time, peer supporters should only be trained to prevent cigarette smoking in S2 (second year). Training to prevent cannabis uptake could follow separately in S3 (third year). While this may seem like just the kind of valuable finding that process evaluations are designed to generate, there is nevertheless a potential problem here: we are drawing a general inference about intervention design from the particular case of one school. But are we justified in generalizing from this one particular case? Possibly, a contrary argument could be constructed that we should be generalizing from the other CASE+ school (where 27% of pupils – as opposed to 9% in the first-mentioned school – reported speaking with a peer supporter about cannabis) – why focus analytic attention on one school rather than another? This particular contrary argument can be addressed by pointing out that peer supporters in the other CASE+, although they too reported that it was more difficult to talk about cannabis than cigarettes, did not appear as unconfident and pessimistic as in the first school. A second contrary argument could be constructed around the fact that, although the peer supporters in the first CASE+ school reported difficulties in talking about cannabis, the proportion of conversations which pupils reported having about cannabis in that CASE+ school (9%) was in fact almost identical with the proportion of conversations which pupils reported having about cigarettes in the two CASE schools (10%). In this argument, although the peer supporters in the first CASE+ school complained about the difficulty of talking about cannabis, and although the trainers’ judgement implied that the girls in particular might ‘struggle’ with the intervention, in fact they performed adequately in having conversations about cannabis and performed above the norm in having conversations about cigarettes: 23 per cent of pupils in the first CASE+ school reported a conversation with a peer supporter about cigarettes. We are inclined to reject this argument since, in the ASSIST trial (with 29 intervention schools), the proportion of pupils reporting a conversation about cigarettes with a peer supporter was 20 per cent (Audrey et al., 2006b: 326), indicating that the peer supporters in the first CASE+ school were in fact performing at a normative level for conversations about smoking and the peer supporters at the two CASE schools were performing below the expected norm. Nevertheless, it can be seen that our original inference (peer supporters will tend to concentrate on smoking prevention over cannabis prevention and so should be trained in each of these interventions separately rather than simultaneously), although it remains plausible, is certainly contestable. The Munro and Bloor: The new miracle ingredient in public health research? generalizability of inductively-generated findings from qualitative research is problematic. Other questions that may be worth exploring in any further feasibility work would be whether a longer training course, to tackle pupils’ (often) stereotypical notions of cannabis (and other drug) users would be more effective, and whether directly training teaching staff, along with pupils, may help pupils to feel more ‘comfortable’, and supported, discussing an illegal drug in and around the school environment. Conclusion This is not the first paper to draw attention to some of the difficulties in conducting process evaluations. While commending the inclusion of process evaluations in trial designs, Wight and Obasi (2002) have pointed to the problems that arise where intervention and evaluation functions are not carefully separated, as where members of the team delivering the intervention may be asked to also collect process evaluation data. They also point to problems of interpretation and suggest that the problems of bias in the interpretation of qualitative data are such that it is best to complete the analysis of the process data before the analysis of the outcome evaluation, so as to identify the key process factors likely to affect outcomes uninfluenced by prior knowledge of what those outcomes are. It might be objected that this is not always possible logistically. But we have raised here a second difficulty in the integration and sequencing of process and outcome evaluations, namely that the decisions on the topical foci of the process evaluation data collection need to anticipate the results from the outcome evaluation, and such anticipation is not always going to be wholly successful. As a consequence, some results of the evaluation may remain only accounted for in a speculative manner, as is the case with our tentative suggestion that the greater propensity for the peer supporters to expect that they will be using cannabis at 16 years is due to a combination of the peer supporters’ greater exposure in the training sessions to the risks of cannabis smoking alongside a sense of adolescent powerlessness and fatalism – we never anticipated the need to collect process data on the latter topic. The use of multiple methods has long been the hallmark of good research design (see, for example, Barbour, 1999). But it is a mistake to think of qualitative and quantitative findings as commensurate in some straight-forward fashion. Elsewhere, Bloor (1997) has argued that triangulation, in the sense of using findings produced by one method to validate the findings produced by a second method, is a chimera: each method will produce findings that are separately distinctive in respect of their degree of specificity/abstraction and their topical focus. Although use of multiple methods can deepen analytic understanding of a specific issue, straightforward replication is an impossibility. The use of qualitative methods in process evaluations alongside quantitative data (whether it be process or outcome data) is rarely straight-forward. Data generated by qualitative methods may be made to bear on the interpretation of 709 710 Qualitative Research 10(6) data generated by quantitative methods, they may deepen and enrich our understanding of quantitative findings, but that deepened understanding will always be nuanced and qualified and rarely determinate. The qualitative data reported earlier – on the low levels of confidence of some female peer supporters, the low level of perceived self-efficacy of some male peer supporters, the perceived greater interactional difficulty of discussing cannabis rather than tobacco, and a consequent preference for discussing tobacco rather than cannabis – these data all appear to deepen and enrich our understanding of quantitative data on numbers of reported conversations about cannabis and tobacco. But they do not necessarily explain those quantitative data in some determinate fashion: other explanations remain possibilities. Relatedly, the interpretation of qualitative data involves making general inferences from particular instances. In the case study reported above, we drew general inferences from data on pupils in a particular school where numbers of reported cannabis conversations were low. But in a second CASE+ school, reported cannabis conversations were much more numerous. It has seemed to us that a properly cautious inference to draw from this feasibility study is that the intervention should be redesigned so that peer supporters efforts are directed to address cigarette and cannabis smoking sequentially rather than simultaneously, but we have no analytic grounds for giving more weight to data from one school rather than another. The generalizability of inferences from process evaluation data remains problematic. Many public health and health promotion interventions are a far cry from the simple dose-response models which randomized controlled trials typically address. In adapting trial designs to the evaluation of complex interventions, researchers have sought to complement outcome evaluation components in their designs with process evaluations which may provide answers to many questions on which outcome evaluations are silent. Process evaluations do indeed enrich our understanding of the social processes involved in the delivery and reception of complex interventions. But process evaluations do not slot comfortably into evidence-based medicine’s ‘hierarchy of evidence’, providing interpretations to which a degree of indeterminancy is always attached that cannot be expressed in probabilistic terms. While it is possible and desirable to integrate qualitative and quantitative methods pragmatically within a research design (Moran-Ellis et al., 2006), the continuing tension between positivist and interpretative paradigms within that research design pose problems of understanding and reporting. Not so much ‘paradigm peace’ as ‘paradigm truce’. Hong et al. suggest that the development of interventions ‘is both an art and a science’ (Hong et al., 2005: 9). And Professor Sir Michael Rawlins, the chair of the UK’s National Institute of Clinical Excellence (NICE), which summarizes the evidence-base for clinical interventions and advises National Health Service managers and practitioners on services effectiveness, has recently argued that in health research ‘hierarchies of evidence should be replaced by Munro and Bloor: The new miracle ingredient in public health research? accepting – indeed embracing – a diversity of approaches’ (Rawlins, 2008: 34). Process evaluations, including qualitative data, can and should be part of that diversity, but it is important that process evaluations are not oversold: they are not a miracle ingredient. ac k n ow l e d g e m e n t s This study was funded by the Medical Research Council (G0601006) with additional support from the local Alcohol & Drugs Action Team and ASH Scotland. We wish to thank the Project Advisory Group: Lesley Armitage, David Craig, Emma Cepok, Louise Kane, Brian Pringle, Maria Reid and Mary Turley. We also wish to thank fellow project worker Sarah Welsh, and grantholders Rona Campbell, Candace Currie, James McIntosh, Neil McKeganey and Laurence Moore. We are deeply grateful to the staff and pupils of the participating schools for their help and participation. An earlier version of this paper was presented at the BSA Medical Sociology conference in Manchester 05/09/09. references Atkinson, P., Coffey, A., Delamont, S., Lofland, J. and Lofland, L. (eds) (2001) Handbook of Ethnography. London & Thousand Oaks, CA: Sage. Audrey, S., Holliday, J., Parry-Langdon, N. and Campbell, R. (2006a) ‘Meeting the Challenges of Implementing Process Evaluation within Randomised Controlled Trials: The Example of ASSIST (A Stop Smoking in Schools Trial)’, Health Education Research 21: 366–77. Audrey, S., Holliday, J. and Campbell, R. (2006b) ‘It’s Good to Talk: An Adolescent Perspective of Talking to their Friends about Being Smoke-Free’, Social Science and Medicine 63: 320–34. Barbour, R. (1999) ‘The Case for Combining Qualitative and Quantitative Approaches in Health Services Research’, Journal of Health Services Research & Policy 4: 39–43. Bloor, M. (1978) ‘On the Analysis of Observational Data: A Discussion of the Worth and Uses of Inductive Techniques and Respondent Validation’, Sociology 12: 545–52. Reprinted in A. Bryman and R. Burgess (1999) (eds) Qualitative Research. London: Sage. And reprinted also in P. Johnson and M. Clark (eds) (2006) Business and Management Research. London: Sage. Bloor, M. (1997) ‘Techniques of Validation in Qualitative Research: A Critical Commentary’, in G. Millar and R. Dingwall (eds) Strategic Qualitative Research. London: Sage. Reprinted in R. Emerson (ed.) (2001) Contemporary Field Research: Perspectives and Formulations. Prospect Heights, IL: Waveland. Bloor, M., Frankland, J., Thomas, M. and Robson, K. (2001) Focus Groups in Social Research. London: Sage. Bonnell, C., Oakley, A., Hargreaves, J., Strange, V. and Rees, R. (2006) ‘Assessment of Generalisability in Trials of Health Interventions: Suggested Framework and Systematic Review’, British Medical Journal 333: 346–9. Bradley, F., Wiles, R., Kinmonth, A-L., Mant, D. and Gantley, M. (1999) ‘Development and Evaluation of Complex Interventions in Health Services Research: Case Study of the Southampton Heart Integrated Care Project (SHIP)’, British Medical Journal 318(13): 711–15. 711 712 Qualitative Research 10(6) Bryman, A. (2006) ‘Paradigm Peace and the Implications for Quality’, International Journal of Social Research Methodology 9: 111–26. Campbell, R., Starkey, F., Holliday, J., Audrey, S., Bloor, M., Parry-Langdon, N., Hughes, R. and Moore, L. (2008) ‘An Informal School-based Peer-Led Intervention for Smoking Prevention in Adolescence (ASSIST): A Cluster Randomised Trial’, Lancet 371: 1595–602. Douglas, M. and Calvez, M. (1990) ‘The Self as Risk-Taker: A Cultural Theory of Contagion in Relation to AIDS’, The Sociological Review 38: 445–64. Giannakaki, M.-S. (2005) ‘Using Mixed-Methods to Examine Teachers’ Attitudes to Educational Change: The Case of the Skills for Life Strategy for Improving Adult Literacy and Numeracy Skills in England’, Educational Research and Evaluation 11(4): 323–48. Gubrium, J. and Holstein, J. (eds) (2002) The Handbook of Interview Research. Thousand Oaks, CA: Sage. Hong, Y., Mitchell, S.G., Peterson, J.A., Latkin, C.A., Tobin, K. and Gann, D. (2005) ‘Ethnographic Process Evaluation: Piloting an HIV Prevention Intervention Program among Injecting Drug Users’, International Journal of Qualitative Methods 4: 1–10. Linnan, L., and Steckler, A. (2002) ‘Process Evaluation for Public Health Interventions and Research – An Overview’, in A. Steckler and L. Linnan (eds) Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey Bass. McGraw, S.A., Stone, E.J., Osganian, S.K., Elder, J.P., Perry, C.L., Johnson, C.C., Parcel, G.S., Webber, L.S. and Luepker, R.V. (1994) ‘Design of Process Evaluation within the Child and Adolescent Trial for Cardiovascular Health (CATCH)’, Health Education Quarterly Suppl. 2: S5–26. Maxwell, C., Kinver, A. and Phels, A. (2007) Scottish Schools Adolescent Lifestyle and Substance Use Survey: Smoking, Drinking and Drug Use among 13 and 15 Year-Olds in Scotland in 2006. URL: http://www.drugmisuse.isdscotland.org/publications/local/ SALSUS_2006.pdf Maynard, A. and Chambers, R. (1997) Non-Random Reflections on Health Services Research. London: BMJ Publishing Group. Medical Research Council (n.d.) Trial Grant Annex. URL (consulted 10 February 2009): http://hwww.mrc.ac.uk/consumption/groups/public/documents/content/ mrc001738.pdf Moran-Ellis, J., Alexander, V., Cronin, A., Dickinson, M., Fielding, J., Sleney, J. and Thomas, H. (2006) ‘Triangulation and Integration: Processes, Claims and Implications’, Qualitative Research 61: 45–59. Munro, A. and Bloor, M. (2009) ‘A Feasibility Study for a Schools-Based, Peer-Led, Drugs Prevention Programme, Based on the ASSIST Programme: The Results’, Centre for Drug Misuse Research occasional paper. Glasgow: University of Glasgow. Oakley, A. (2005) ‘Design and Analysis of Social Intervention Studies in Health’, in A. Bowling and S. Ebrahim (eds) Handbook of Health Research Methods: Investigation, Measurement and Analysis, pp. 73–81. Maidenhead: Open University Press. Oakley, A., Strange, V., Bonell, C., Allen, E., Stephenson, J. and the RIPPLE Study Team (2006) ‘Process Evaluation in Randomised Controlled Trials of Complex Interventions’, British Medical Journal 332: 413–6. O’Cathain, A. (2009) ‘Editorial: Mixed Methods Research in the Health Sciences: A Quiet Revolution’, Journal of Mixed Methods Research 3(1): 3–6. Palinka, L., Atkins, C., Miller, C. and Ferreira, D. (1996) ‘Social Skills Training for Drug Prevention in High-Risk Female Adolescents’, Preventive Medicine 25: 692–701. Munro and Bloor: The new miracle ingredient in public health research? Parry-Langdon, N., Bloor, M., Audrey, S. and Holliday, J. (2003) ‘Process Evaluation of Health Promotion Interventions’, Policy & Politics 31(Suppl.): S25–S34. Rawlins, M. (2008) De Testimonio – on the Evidence for Decisions about the Use of Therapeutic Interventions: The Harveian Oration of 2008. London: Royal College of Physicians. Victora, C., Habicht, J-P. and Bryce, J. (2004) ‘Evidence-Based Public Health: Moving Beyond Randomized Trials’, American Journal of Public Health 94(3): 400–5. Wight, D. and Obasi, A. (2002) ‘Unpacking the Black Box: The Importance of Process Data to Explain Outcomes’, in J. Stephenson, J. Imrie and C. Bonell (eds) Effective Sexual Health Interventions: Issues in Experimental Evaluation, pp. 151–66. Oxford: Oxford University Press. Young, D.R., Steckler, A., Cohen, S., Pratt, C., Felton, G., Moe, S.G., Pickrel, J., Johnson, C.C., Grieser, M., Lytle, L.A., Lee, J.S. and Raburn, B. (2008) ‘Process Evaluation Results from a School- and Community-Linked Intervention: The Trial of Activity for Adolescent Girls (TAAG)’, Health Education Research 23(6): 976–86. ALISON MUNRO is a research fellow at Glasgow Caledonian University (School of Health). She has a PhD in the area of alcohol studies, and has conducted a number of research studies related to alcohol and drug misuse. Address: School of Health, Glasgow Caledonian University, Cowcaddens Rd, Glasgow G4 0BA, UK. [email: alison.munro@ gcal.ac.uk] MICHAEL BLOOR is a medical sociologist and is a part-time professorial research fellow at the University of Glasgow (Centre for Drug Misuse Research) and Cardiff University (Seafarers International Research Centre). Address: Centre for Drug Misuse Research, University of Glasgow, 89 Dumbarton Rd, Glasgow G11 6PW, UK. [email: m.bloor@ socsci.gla.ac.uk] 713 531773 research-article2014 HEBXXX10.1177/1090198114531773Health Education & BehaviorGolden et al. Article Process Evaluation of an Intervention to Increase Provision of Adolescent Vaccines at School Health Centers Health Education & Behavior 2014, Vol. 41(6) 625­–632 © 2014 Society for Public Health Education Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1090198114531773 heb.sagepub.com Shelley D. Golden, PhD1, Kathryn E. Moracco, PhD, MPH1, Ashley L. Feld, MPH1, Kea L. Turner, MPH1, Jessica T. DeFrank, PhD1, and Noel T. Brewer, PhD1 Abstract Background. Vaccination programs in school health centers (SHCs) may improve adolescent vaccine coverage. We conducted a process evaluation of an intervention to increase SHC-located vaccination to better understand the feasibility and challenges of such interventions. Method. Four SHCs participated in an intervention to increase provision of recommended vaccines to 2,975 adolescents. We reviewed program materials and SHC staff reports, and interviewed parents to assess implementation fidelity and reactions to materials. Results. Ten percent of parents returned forms with consent to at least one vaccine. Of these, 79% checked the box consenting for “all recommended” vaccines, rather than indicating individual vaccines. SHCs sent supplemental mailings to some parents that clarified (a) vaccination costs or (b) human papillomavirus vaccine recommendation for boys and required parents to reconsent. This process resulted in loss of initial consent, primarily due to nonresponse. In interviews, parents who consented to vaccination indicated that intervention materials were clear and persuasive, but needed greater detail about costs and clinic processes. Conclusions. With limited additional investment, it appears feasible for SHCs to achieve a modest increase in the number of vaccinated adolescents. Providing a checkbox to indicate global consent for all recommended vaccinations, and close collaboration among individuals involved in intervention development, may facilitate vaccination efforts. Keywords adolescent health, evaluation, health promotion, health protective behavior, process evaluation, school-based, school-based health care Vaccines recommended specifically for adolescents effectively prevent infections that lead to numerous diseases. United States guidelines suggest that 11- or 12-year-old adolescents should receive three vaccines: meningococcal conjugate vaccine (MCV4); tetanus, diphtheria, acellular pertussis (Tdap) vaccine; and human papillomavirus (HPV) vaccine, as well as annual seasonal influenza vaccines (Centers for Disease Control and Prevention [CDC], 2013a). Yet adolescent vaccination rates remain low. According to the 2012 National Immunization Survey–Teen, 85% of adolescents had received the Tdap vaccine, 74% had received MCV4, and 33% of females and 7% of males had received the recommended three doses of HPV vaccine (CDC, 2013b). Most of these vaccination rates are below the 80% coverage targets of Healthy People 2020 (U.S. Department of Health and Human Services, 2011). Several barriers hamper adolescent vaccination. Some of these, such as parents’ concerns about immunization side effects, lack of transportation, and cost, are similar to barriers to vaccinating younger children (Kaplan, 2010). Others, such as infrequent contact with the health care system, lower rates of insurance coverage, decreased likelihood of having up-todate immunization records, and limited awareness of new adolescent vaccines may be specific challenges for vaccinating older children (Ford, English, Davenport, & Stinnett, 2009; Irwin, Adams, Park, & Newacheck, 2009; Kaplan, 2010; Kennedy, Stokley, Curtis, & Gust, 2012; Rand et al., 2007). Schools are a promising setting for adolescent vaccination because teens spend significant time there, and schools enforce immunization requirements (Moss et al., in press; 1 The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA Corresponding Author: Shelley D. Golden, Department of Health Behavior, The University of North Carolina at Chapel Hill, CB 7440, Chapel Hill, NC 27599-7440, USA. Email: sgolden@email.unc.edu 626 Health Education & Behavior 41(6) Shah, Gilkey, Pepper, Gottlieb, & Brewer, 2014). School health centers (SHCs) provide health services to enrolled students at clinics located in schools, including vaccination. A recent systematic review found strong evidence that vaccination programs in schools can be effective in improving vaccine uptake among children (Guide to Community Preventive Services [GCPS], 2009). Moreover, many parents are receptive to school-located delivery of adolescent vaccinations, with even higher acceptability among families whose children have not recently seen a doctor (Allison et al., 2007; Clevenger et al., 2011; Kadis et al., 2011; McRee, Reiter, Pepper & Brewer, 2013; Reiter, McRee, Pepper, Chantala, & Brewer, 2012; Reiter, Stubbs, Panozzo, Whitesell, & Brewer, 2011). SHCs face two key challenges to adolescent vaccine delivery: parental consent for immunization (CooperRobbins, Ward, & Skinner, 2011; Kaplan, 2010) and expanding vaccine programs with limited staff resources. In response to these challenges, we worked with SHC staff to develop and implement an intervention to provide parents with clear, motivational messages about vaccination, while streamlining clinic processes for obtaining parental vaccination consent. This article reports on the process evaluation of this effort. Process evaluation investigates how a program is executed in order to assess and improve implementation. Documentation of implementation strategies is also important for intervention dissemination and translation from research to practice (Linnan & Steckler, 2002; Saunders, Evans, & Joshi, 2005). Our process evaluation addresses four questions: 1. 2. 3. 4. What aspects of the vaccination program did SHCs modify during implementation? To what extent did parents find the vaccination program materials and communications to be clear, informative, relevant, and persuasive? How much staff time and SHC resources did intervention implementation require? On completion of the intervention and its modification, how many parents had consented to at least one adolescent vaccination? Method Participants Researchers with Cervical Cancer-Free North Carolina (CCFNC), a statewide collaboration to foster cervical cancer prevention practices and research, partnered with an organization that managed SHCs in all four traditional high schools in one district in central North Carolina. The district student population was 63% White, 20% African American, 11% Hispanic, and 6% Other race/ethnicity, and 59% of students were eligible for free and reduced lunch (Rockingham County Schools, n.d.). Study participants were parents of 2,975 students enrolled in the four SHCs in September 2011. Procedure We created a vaccine information packet that included a cover letter signed by the SHC’s director, a promotional flyer, a consent form, two vaccine information sheets, and a preaddressed, stamped return envelope. Materials were twosided in color, with Spanish translations on the back. The cover letter emphasized the convenience of the SHC, the need for and safety of adolescent vaccines, and the importance of returning a completed consent form. The promotional flyer emphasized similar points but also added quotes and images of parents and teens. We designed the consent form to limit the amount of information parents needed to provide, removing requests for information already available in SHC files, and focusing on allergies and previous vaccination background. In the consent area, we included a check box allowing parents to globally consent to all recommended vaccines, or an option to check individual vaccinations. One information sheet explained Tdap, meningitis, HPV, and seasonal flu vaccines, and the second information sheet explained vaccines recommended earlier in childhood. SHC staff implemented several procedures to encourage consent form return. Staff mailed the packets to parents to ensure they received it. Both before and after packet mailing, staff used the school’s automated phone messaging system to call all parents with reminders about the consent forms. They sent undelivered packets to parents in students’ backpacks. Students who returned forms by a specified date had a chance to win four movie tickets. The intervention as originally designed appears in the top half of Figure 1. Instruments Materials Tracking Report. SHC staff documented how many vaccine information packets they mailed, received back undelivered, and sent home in student backpacks. Each clinic also tracked how many consent forms came back by the end of January 2012, how many of the returned forms were signed, and the specific vaccines to which parents consented. Parent Interviews. Between November 2011 and January 2012, we conducted telephone interviews with a subsample of parents who had indicated a willingness to be contacted on their returned consent form. Of 62 eligible parents, we completed phone interviews with 47 (76%). The interview contained closed-ended questions assessing the extent to which parents found the materials to be clear, informative, relevant, and persuasive. Interviewers also invited parents to elaborate on their responses, identify gaps in the materials, and provide suggestions for packet improvement through open-ended questions. Parents received $10 grocery store gift cards in appreciation. Resource Reports. CCF...
Purchase answer to see full attachment
User generated content is uploaded by users for the purposes of learning and should be used following Studypool's honor code & terms of service.

Explanation & Answer

At...


Anonymous
Just what I needed. Studypool is a lifesaver!

Studypool
4.7
Trustpilot
4.5
Sitejabber
4.4

Similar Content

Related Tags