PRINT HCM570: Critical Thinking Rubric - Module 03 Content, Research, and Analysis Meets Expectation Approaches Expectation Below Expectation Limited Evidence Requirements 15 to 13 PointsIncludes all of the required components, as specified in the assignment. 12 to 10 PointsIncludes most of the required components, as specified in the assignment. 9 to 7 PointsIncludes some of the required components, as specified in the assignment. 6 to 4 PointsIncludes few of the required components, as specified in the assignment. Content 15 to 13 PointsDemonstrates strong or adequate knowledge of HIEs or EHRs; correctly represents knowledge from the 12 to 10 PointsSome significant, but not major, errors or omissions in demonstration of knowledge HIEs or EHRs. 9 to 7 PointsMajor errors or omissions in demonstration of knowledge concerning HIEs or EHRs. 6 to 4 PointsFails to demonstrate knowledge of the materials. Content, Research, and Analysis Meets Expectation Approaches Expectation Below Expectation Limited Evidence readings and sources. Critical Analysis 25 to 21 PointsProvides a strong critical analysis and interpretation of the information given, and provides an assessment of each system. 20 to 16 PointsSome significant, but not major, errors or omissions in analysis and interpretation. 15 to 11 PointsMajor errors or omissions in analysis and interpretation. 10 to 6 PointsFails to provide critical analysis and interpretation of the information given. Synthesis and Evaluation 15 to 13 PointsDemonstrates strong or adequate synthesis and evaluation of course concepts in HIEs or EHRs. 12 to 10 PointsSome significant, but not major, errors or omissions in synthesis and evaluation. 9 to 7 PointsMajor errors or omissions in synthesis and evaluation. 6 to 4 PointsFails to demonstrate synthesis and evaluation. Content, Research, and Analysis Meets Expectation Sources / Examples 10 to 9 PointsSources or examples meet required criteria and are well chosen to provide substance and perspectives on the issue under examination. Approaches Expectation Below Expectation 8 to 7 PointsSources or examples meet required criteria, but are less than adequately chosen to provide substance and perspectives on the issue under examination. 6 to 5 PointsSources or examples meet required criteria, but are poorly chosen to provide substance and perspectives on the issue under examination. Limited Evidence 4 to 3 PointsSource or example selections and integration of knowledge from the course are clearly deficient. Mechanics and Writing Meets Expectation Approaches Expectation Below Expectation Limited Evidence Demonstrates college-level proficiency in organization, grammar and style. 10 to 9 PointsProject is clearly organized, well written, and in proper format as outlined in the assignment; strong sentence and paragraph structure; few errors in grammar and spelling. 8 to 7 PointsProject is fairly well organized and written, and in proper format as outlined in the assignment;reasonably good sentence and paragraph structure; significant number of errors in grammar and spelling. 6 to 5 PointsProject is poorly organized and does not follow proper paper format;inconsistent to inadequate sentence and paragraph development; numerous errors in grammar and spelling. 4 to 3 PointsProject is not organized or well written, and is not in proper paper format; poor quality work; unacceptable in terms of grammar and spelling. Demonstrates proper use of APA style 10 to 9 Points 8 to 7 Points 6 to 5 Points 4 to 3 Points Mechanics and Writing Meets Expectation Project contains proper APA formatting. • • • Approaches Expectation Few errors in APA formatting. Max Points for Content, Research, and Analysis80 Max Points for Mechanics and Writing20 Total Points Possible100 Below Expectation Limited Evidence Significant errors in Numerous errors APA formatting. in APA formatting. Adm Policy Ment Health (2016) 43:471–477 DOI 10.1007/s10488-015-0702-5 COMMENTARY At the Intersection of Health Information Technology and Decision Support: Measurement Feedback Systems…and Beyond Bruce F. Chorpita1 • Eric L. Daleiden2 • Adam D. Bernstein2 Published online: 24 November 2015  Springer Science+Business Media New York 2015 Abstract We select and comment on concepts and examples from the target articles in this special issue on measurement feedback systems, placing them in the context of some of our own insights and ideas about measurement feedback systems, and where those systems lie at the intersection of technology and decision making. We contend that, connected to the many implementation challenges relevant to many new technologies, there are fundamental design challenges that await a more elaborate specification of the clinical information and decision models that underlie these systems. Candidate features of such models are discussed, which include referencing multiple evidence bases, facilitating observed and expected value comparisons, fostering collaboration, and allowing translation across multiple ontological systems. We call for a new metaphor for these technologies that goes beyond measurement feedback and encourages a deeper consideration of the increasingly complex clinical decision models needed to manage the uncertainty of delivering clinical care. Keywords Measurement feedback
Dashboard
Evidence base
Treatment
Clinical decision support & Bruce F. Chorpita chorpita@ucla.edu Eric L. Daleiden e.daleiden@practicewise.com Adam D. Bernstein a.bernstein@practicewise.com 1 UCLA Department of Psychology, University of California, Los Angeles, Franz Hall 3227, Los Angeles, CA 90095, USA 2 PracticeWise, LLC, 340 Lee Avenue, Satellite Beach, FL 32937, USA This issue pulls together a valuable assortment of ideas and observations from research teams working with a variety of measurement feedback systems (MFS) to guide clinical care, while confronting and studying both the human and technological implementation challenges. Their collective insights paint a complicated picture that documents both promises and challenges associated with MFS. In our own efforts to improve service delivery in children’s mental health systems over the past 15 years, we have both used and developed such systems, and have thus encountered both the good and the bad firsthand. In the context of these papers, we would like to offer some general insights for consideration as the field moves forward. Many of these insights concern the basic relation between technology and decision making. Technology generally does what we tell it to do, and thus not surprisingly, early developments in health information technology (HIT) managed the most essential and best-understood processes in health care systems, including utilization, billing, and documentation. These were the questions the field had to ask in order to function: who was seen when for what reason, and how it was paid for? But more recently, there has been a focus on the more complex questions we want to ask to guide clinical care: What treatments approaches are we using? Are they helping? What should we do when they are not? In hindsight, many of these questions were naively ambitious on our part, which may explain why they were often met with such equivocal answers as ‘‘22 sessions of individual therapy’’ or ‘‘30 days of residential,’’ along with the associated billing codes. Thanks to a growing vanguard of thinkers, answering the clinically interesting questions is getting closer to a common reality. But precisely because technology does what we tell it to do, this function of HIT will not fully mature until we have better articulated the complex 123 472 information models that underlie it. We can think of these latter questions as belonging to the domain of clinical decision support, as shown in Fig. 1. Note that there are many strategies to support decision making that don’t involve HIT (e.g., supervisor recommendations, use of a treatment manual), and there are many functions of HIT that don’t involve supporting clinical decision making (e.g., documentation, service authorization, scheduling). At this intersection lies a tremendous set of possibilities and opportunities, within which MFS serves a specific function. As the figure implies, although we believe there is much to explore within the context of MFS development— which entails a human-technology interaction defined by a user receiving measurement information in the form of a report or an alert (e.g., Bickman 2008)—there is even more work to be done within the broader context defined by the intersection of technology and clinical decision making (Fig. 1, shaded region). In this regard, we prefer a metaphor of telecommunication (or an interactive workspace), which suggests collaboration, communication, reasoning, interaction, and even design. Such systems of course can feed back information, but they should also be able to feed forward information to guide action (e.g., setting expectations for what should happen next, exploring ‘‘what if’s’’). As Bickman et al. (2014) suggest in their ‘‘final coda,’’ although the technology may be important, it should operate in the service of decision support. In our metaphor, we know that a telephone conference call must not drop participants and must be free of background noise, but its participants will also benefit from having an agenda, speaking a common language, knowing who else is on the call, having their thoughts organized, and sharing similar goals for their meeting. Thus, better articulation of the work processes to be supported on the ‘‘decision side’’ of the figure is prerequisite to any technology that could ultimately better serve those processes. Fig. 1 Measurement feedback systems in the context of health information technology and clinical decision support. MFS measurement feedback systems 123 Adm Policy Ment Health (2016) 43:471–477 There is another matter, not represented in Fig. 1, involving the human implementation challenges that arise once appropriate technologies are developed. For example, Gleacher et al. (2015) highlighted the key facilitating role of organizational leadership in achieving widespread use of the contextualized feedback system. Without such support, providers may mute the phone calls, or agencies may disconnect the phone service altogether, metaphorically speaking. This is but one of many examples that involve technology implementation issues rather than design issues (of course the two are inevitably connected in the real world; e.g., Higa-McMillan et al. 2011; Lyon et al. 2015; and in treatment contexts, design alone has been associated with implementation success; e.g., Palinkas et al. 2013, Southam-Gerow et al. 2014). Nevertheless, in the measurement feedback context, this issue’s authors provide considerable discussion of implementation, so we focus on a brief list of ideas relevant to clinical decision support and its broad intersection with technology, followed by a simple illustration. These ideas include considerations for sources of the evidence to be displayed, the value of exposing discrepancies between what has happened and what should or could happen, the types and configuration of evidence to be displayed (e.g., beyond progress rating alerts or plots) to allow users to impose a logic model on interpretation, the importance of automated translation of relevant ontologies (e.g., DSM diagnoses vs. elevated scales on a standardized symptom measure), and the creation of structures to facilitate communication and collaboration. Multiple Evidence Bases Daleiden and Chorpita (2005) outlined a model to coordinate and inform service delivery, which among other things described four core evidence bases relevant to decisionmaking: case-specific historical information, local aggregate evidence, general services research, causal mechanism research. These are outlined in Table 1, with reference to corresponding traditions or schools of thought as well as example questions addressed by each. In the current context of MFSs, there is a predominant emphasis on only one of these four sources of evidence—the case-specific history (but see Steinfeld et al. 2015, for examples of using local aggregate evidence in the form of departmental and system level reports). We feel much can be gained from coordinating a fuller set of relevant information from all four of the evidence bases, which gives us the ability to detect, consider, and act on knowledge that might otherwise remain out of view, lost in our decision-making ‘‘blind spots.’’ This task is not possible without a significant amount of ‘‘background complexity,’’ given that each evidence base Adm Policy Ment Health (2016) 43:471–477 473 Table 1 Four evidence bases relevant to clinical decision making (‘‘Evidence-Based Services System Model’’) Evidence base Relevant tradition Example questions Case-specific historical information Individualized case conceptualization What is the primary clinical concern? Is this youth improving? Local aggregate evidence Practice-based evidence What is the most common focus of care? How is the system performing on average for that specific group? General Services Research (evidence-based practice) Evidence-based practice What are the best supported treatment approaches for a defined group? How well have they worked in research trials? Causal Mechanism Research (clinical theory) Individualized case conceptualization What would theory suggest is a promising next step? What is a logical expectation for the outcome? can have multiple indicators, which can even disagree (e.g., two independent randomized trials with discrepant findings; improvement on one measure of depression with deterioration on another). The challenge of determining which knowledge is ‘‘best’’ is likely an impossible pursuit, but fortunately, perhaps not a necessary one. Rather, a sufficient knowledge management function may be for systems to prioritize ‘‘better’’ knowledge from a number of sources through a series of rules or knowledge filters (e.g., psychometric validation for case-specific measures; strength of evidence models for literature review; e.g., Chorpita et al. 2011). This is possible because MFSs inherently create a self-correcting context, in which the real validation of the knowledge used is accomplished by observing whether the desired outcomes were achieved. This notion reflects our fundamental belief that a legitimate function of these systems can be to provide multiple promising contextualized ideas to a decision maker, rather than merely to provide a ‘‘right answer’’ or single prescribed action. In other words, ultimately an agent must prioritize, act, and test results based on the information made available, and the technology should support that process rather than replace it. from the one indicated by strong research support, then there is a discrepancy between the observed value and the expected value. Such discrepancies can motivate and guide action, perhaps in this case to consideration of a different treatment, and we expect that a core function of MFS is to assist with making these discrepancies known. Several papers in this issue provide examples of using expected values. For example, Steinfeld et al. (2015) describe reporting related to expected measure completion at each encounter. Similarly, Bickman et al. (2014) illustrate alerts related to the expectation that feedback reports be viewed by practitioners. Similarly, the model-specific implementations (e.g., Bruns et al. 2015; Nadeem et al. 2015) communicate an expectation that particular service activities occur, simply by incorporating descriptions of those activities into the workflow and visual displays. Although there is evidence that observed-value-only feedback offers advantages over no feedback (e.g., Lambert et al. 2005), it is worth considering the decision support value afforded by contextualizing these with expected values. Multiple Domains Expected and Observed Values In clinical care, expected values (e.g. Chorpita et al. Research Network on Youth Mental Health 2008) refer to information that informs our best guesses about what should happen. For instance, if one needed to select a treatment that might work for a given youth, one might consider research trials involving similar youth to inform that decision (as would be consistent with the evidencebased treatment paradigm); the treatment with the best research support is thus the expected value for the treatment to be delivered. Expected values can be contrasted with observed values, which represent information about what has happened or is happening now. Staying with this example, if the youth is receiving a treatment that differs In the same manner that MFS often prioritize the casespecific evidence base and observed values, to date many systems have also placed a heavy emphasis on the progress domain (e.g., visualizing symptom change over time). However, to facilitate technology’s relevance to clinical decision making, we think MFS platforms should help examine any type of information (not just progress) that fits within the larger decision model used to guide care. For instance, if one believes practice is related to progress, then organizing information about practices delivered over time and configuring that information to be synchronized with progress measurements might be of considerable value. More generally, any events (e.g., change of medication, change of placement, stakeholder participation, session noshows, or end of school year) deemed relevant to progress 123 474 interpretation or strategy selection may be useful for decision support display. This special issue provides two illustrations of strategies for extending beyond progress measurement. For example, Nadeem et al. (2015) show how to track practices delivered directly on the feedback reports. Using a different strategy, Bruns et al. describe integration of service activities into the workflow of the system itself. Multiple Languages In keeping with our telecommunication metaphor, in addition to a shared channel, senders and receivers must share a language and concept system to transfer knowledge. Accordingly, we see a need for translating across the diverse ontologies that are found in mental health research and services (e.g., Diagnostic and Statistical Manual, Research Domain Criteria, Standardized Instrument Scores, Evidence-Based Practices, Practice Elements, etc.; see Chorpita and Daleiden 2014). Constraints that require a single common language (e.g., a fixed set of measures; a single clinical model) are less likely to generalize to diverse contexts and to facilitate communication in the language of the local jurisdiction or system. The papers in this special issue clearly illustrate the underlying dilemma. Several of the systems describe a capacity to support multiple outcome measures (e.g., Bickman et al. 2014; Bruns et al. 2015; Nadeem et al., 2015), whereas Steinfeld et al. (2015) highlight some benefits of committing to a single measurement model even though their electronic medical record could potentially support many. On the practice metric side, Bruns et al. (2015) illustrate the construction and use of a model-specific system whereas Nadeem et al. (2015) illustrate a modelspecific configuration of a generalized platform for progress and event (practice) representation. In our work with systems, we have found tremendous value in the ability to support diversity (of models, measures, display preferences, etc.) within a single platform, but think that diversity is best supported in the context of a strong default configuration that is designed to bias users initially toward ‘‘best practices,’’ while allowing extension and adaptation as user expertise develops. For this to happen, ‘‘translator’’ functions (e.g., is an elevated score on the Children’s Depression Inventory sufficiently similar to a DSM-III-R diagnosis of Major Depression to draw an inference for this youth?) as well as diverse ontological libraries (e.g., configurable lists of practice elements, evidence-based protocols, or other metrics to represent practice delivery) are a necessary infrastructure operation for MFS. 123 Adm Policy Ment Health (2016) 43:471–477 Collaboration Another fundamental premise is that MFS should both foster and structure collaboration. Rather than serving as a substitute for human decision makers, we believe a key role of these systems is to organize and inform those involved in care. Collaboration can be an implicit feature, for example, by requiring treatment team members to select targets and measures as well as selecting benchmarks (i.e., choosing expected values from various evidence bases); or it can be a more explicit feature, for example, by contiguously displaying scores from multiple informants or practices delivered by different members of the treatment team, allowing a full view of team activities and perspectives. Dynamic configuration, such as being able to toggle elements on and off, extends this capability, allowing different views for different users (e.g., sharing progress and practice history with a family member). Bruns et al. (2015) describe features built into the workflow of the TMSWrapLogic system that prompt the type of collaboration that is central to the wraparound service model, and Lyon et al. (2015) found in their contextual assessment that communication, both internal and external, were key functions of service providers. An Illustration Nathan is a 17 year old Asian American male receiving treatment for depression. Figure 2 shows parent and selfreported depression scores on a depression scale over time in days (plots a and b). In terms of the concepts above, this panel represents the case-specific evidence base, using observed values, in a single domain (progress), in a single language (T-scores on a standardized measure). In Fig. 3, we enrich the display in a number of ways. First, the progress panel now displays two additional plots (c and d), corresponding to expected values. Both can therefore be thought of as scores representing goal states. When selecting expected values, it helps to consider all four evidence bases outlined in Table 1, keeping in mind that one or more expected value could be derived from each evidence base. For example, a case-specific expected value might be a discharge score from a previous successful treatment for Nathan. In this example, plot c is derived from the local aggregate evidence base and represents the average post-treatment score for youth in the system who had received treatment for depression. Plot d is taken from the general services research evidence base, and represents pre-post scores from a randomized clinical trial for depression that included youth of the same age and ethnicity as Nathan. Of note is that simply adding expected Adm Policy Ment Health (2016) 43:471–477 475 Progress and Practice Monitoring Tool Case ID: NZ Age (in years): 17.4 Treatment Target: Depression Gender: Male Ethnicity: Asian American 85 Progress Measures 80 Left Scale a RCADS-Depression 75 b RCADS-P-Depression 70 65 60 Right Scale 55 50 180 160 140 120 100 80 60 40 20 0 45 Fig. 2 A progress panel with observed values for depression scores over time. RCADS revised child anxiety and depression scale, RCADS-P revised child anxiety and depression scale-parent version Progress and Practice Monitoring Tool Case ID: NZ Age (in years): 17.4 Treatment Target: Depression Gender: Male Ethnicity: Asian American 85 1 Progress Measures 80 Left Scale a RCADS-Depression 1 75 b RCADS-P-Depression 1 70 c Benchmark 65 d Trend 1 60 Right Scale 0 55 0 50 45 Literature Practices Performed 180 160 140 120 100 80 60 40 20 0 0 Expert ******* FOCUS ******* Relationship/Rapport Building Child Psychoed: Depression Psychoed: Depression (CG) Activity Selection Cognitive: Depression Problem Solving Relaxation Social Skills Communication Skills (CG) Support Networking (CG) Goal Setting (CG) Maintenance (CG) ******* INTERFERENCE ******* Child Psychoed: Anxiety Cognitive: Anxiety Exposure Self-Rewards Fig. 3 A progress and practice panel showing observed and expected values over time. CG indicates a caregiver directed practice, RCADS revised child anxiety and depression scale, RCADS-P revised child anxiety and depression scale-parent version, Psychoed psychoeducation. The shaded region of the practice panel indicates a hypothetical illustration of practices supported by relevant research trials (i.e., expected values for practice) 123 476 values for treatment progress can have a dramatic effect on interpretation of progress by creating context. That is, it is quite possible to infer satisfactory progress when examining Fig. 1, but less possible when inspecting the top panel of Fig. 2, given the discrepancies between observed and expected values. This discrepancy may effectively indicate when to act, but without additional information, it may say less about how to act. Given a basic logic model that practices affect outcomes, inclusion of a practice panel can help in this regard. The bottom of Fig. 2 uses white circles plotted on the same time axis as the progress ratings to indicate that Nathan has had 15 separate treatment sessions involving 9 different clinical procedures. This panel is one place to explore to determine why he has lagged the expected rate of progress. Once again, expected values can help, and plotting practices coded from evidence-based treatments (which is in essence, a translation exercise, as noted above) indicates that the 12 practices in the ‘‘focus’’ region of the practice panel are part of an evidence-based protocol for depression in adolescents. Comparing these expected practices with those observed, we find that three sessions involved ‘‘off focus’’ practices, targeting anxiety (possible errors of commission), and only one of five caregiver-directed practices has been delivered (communication skills), occurring 150 days into treatment (possible errors of omission). One other notable observation is the increasing latency between sessions starting at about day 60. A treatment team may thus hypothesize that caregiver engagement may be an issue, and begin deeper inquiry, which could include adding additional measures to the progress panel (e.g., a caregiver assessment of barriers to treatment or treatment expectancy) or otherwise enhance caregiver services. The system remains interactive, collaborative, and exploratory. Once the system may have indicated how to act, it should ideally support the treatment team in taking the next steps, including referencing the provider analogue of the case-specific history: does this provider have experience and expertise with promoting caregiver engagement? If not, are there written materials or learning resources that could be launched from the display to facilitate action? Of course, this is but one example, restricted to two domain panels, two of the evidence bases, often with only a single expected value from each. The possibilities, however, can be as varied as conversations, consistent with our telecommunication metaphor. When everything aligns neatly, these conversations can be swift and clear, indicating next steps, but even when information does not align (e.g., the research literature indicates one set of expected values for practice, whereas an expert supervisor recommends a different set of practices), the communication and collaboration is biased toward investigating and resolving 123 Adm Policy Ment Health (2016) 43:471–477 the discrepancy prospectively using evidence. More complex examples and illustrations are available elsewhere in the literature, specifically regarding multiple provider teams (e.g., Bruns et al. 2014); observed and expected values and model integrity (Chorpita et al. 2008; Regan et al. 2013), coordination of multiple evidence bases (e.g. Daleiden et al. 2005), and complex collaboration structures (Chorpita and Daleiden 2014). Conclusion This group of authors is to be commended for their efforts to implement and evaluate MFS in a variety of real-world contexts. The promises of such systems are clear, and the design challenges, although considerable, can and will be resolved. For technology to serve our will, however, our field must continue to wrestle with models for how we wish to select and organize health information. Thus, we disagree with the notion that this burden lies with HIT developers. There is, for the moment, a significant underspecification of the general information and decision models needed to dictate the functional requirements of promising new technologies. Technology will do what we tell it to, and thus, the burden, for now, lies primarily with mental health experts. That said, if we do our jobs right, the complexity of these fully articulated models may soon become a constraint. Our theories of psychopathology have moved to multifactorial risk and protective factor models, and our intervention research has identified a multitude of interventions that, although effective when measured at the group level, involve uncertainty at the case level. To help manage that uncertainty, the current technologies struggle tremendously and with only modest success at displaying a ‘‘human readable’’ form of a ‘‘progress only’’ model, much less a basic practice-yields-progress logic model. Although we encourage practice-progress type models for current applications to help those using the technology of today, we await the truly disruptive technology needed to support the highly elaborated models necessary to help humans contextually detect and respond to abstract phenomena (e.g., behavior, interactions) in the service of their personal pursuits and in their duty to help others. References Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 1114–1119. Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., et al. (2014). Implementing a Adm Policy Ment Health (2016) 43:471–477 measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 1–16. doi:10.1007/s10488-015-0647-8. Bruns, E. J., Hyde, K. L., Sather, A., Hook, A. N., & Lyon, A. R. (2015). Applying user input to the design and testing of an electronic behavioral health information system for wraparound care coordination. Administration and Policy in Mental Health and Mental Health Services Research, 1–19. doi:10.1007/ s10488-015-0658-5. Bruns, E. J., Walker, J. S., Bernstein, A. D., Daleiden, E. L., Pullmann, M. D., & Chorpita, B. F. (2014). Family voice with informed choice: Coordinating wraparound with research-based treatment for children and adolescents. Journal of Clinical Child and Adolescent Psychology, 43, 256–269. Chorpita, B. F., Bernstein, A., Daleiden, E. L., & Research Network on Youth Mental Health. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123. Chorpita, B. F., & Daleiden, E. L. (2014). Structuring the collaboration of science and service in pursuit of a shared vision. Journal of Clinical Child and Adolescent Psychology, 43, 323–338. Chorpita, B. F., Daleiden, E. L., Ebesutani, C., Young, J., Becker, K. D., Nakamura, B. J., et al. (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18, 153–171. Daleiden, E., & Chorpita, B. F. (2005). From data to wisdom: Quality improvement strategies supporting large-scale implementation of evidence-based services. Child and Adolescent Psychiatric Clinics of North America, 14, 329–349. Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V., Bickman, L., et al. (2015). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research, 1–15. doi:10.1007/s10488-015-0642-0. 477 Higa-McMillan, C. K., Powell, C., Daleiden, E., & Mueller, C. W. (2011). Pursuing an evidence-based culture through contextualized feedback: Aligning youth outcomes and practices. Professional Psychology: Research and Practice,. doi:10.1037/a0022139. Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61, 165–174. doi:10.1002/jclp. 20113. Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., et al. (2015). The Contextualized Technology Adaptation Process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research, 1–16. doi:10.1007/s10488-015-0637-x. Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2015). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 1–15. doi:10.1007/s10488-015-0651-z. Palinkas, L. A., Weisz, J. R., Chorpita, B. F., Levine, B., Garland, A., Hoagwood, K. E., & Landsverk, J. (2013). Continued use of evidence-based treatments after a randomized controlled effectiveness trial: A qualitative study. Psychiatric Services, 64, 1110–1118. Regan, J., Daleiden, E. L., & Chorpita, B. F. (2013). Integrity in mental health systems: An expanded framework for managing uncertainty in clinical care. Clinical Psychology: Science and Practice, 20, 78–98. Southam-Gerow, M. A., Daleiden, E. L., Chorpita, B. F., Bae, C., Mitchell, C., Faye, M., & Alba, M. (2014). MAPping Los Angeles County: Taking an evidence-informed model of mental health care to scale. Journal of Clinical Child and Adolescent Psychology, 43, 190–200. Steinfeld, B., Fraynt, R., & Simon, G. (2015). Progress monitoring in an integrated health care system: Tracking behavioral health vital signs. Administration and Policy in Mental Health and Mental Health Services Research, 1–10. doi:10.1007/s10488015-0648-7. 123 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Aller et al. BMC Health Services Research (2015) 15:323 DOI 10.1186/s12913-015-0968-z RESEARCH ARTICLE Open Access Development and testing of indicators to measure coordination of clinical information and management across levels of care Marta-Beatriz Aller1*, Ingrid Vargas1, Jordi Coderch2, Sebastià Calero3, Francesc Cots4, Mercè Abizanda5, Joan Farré6, Josep Ramon Llopart7, Lluís Colomés8 and María Luisa Vázquez1 Abstract Background: Coordination across levels of care is becoming increasingly important due to rapid advances in technology, high specialisation and changes in the organization of healthcare services; to date, however, the development of indicators to evaluate coordination has been limited. The aim of this study is to develop and test a set of indicators to comprehensively evaluate clinical coordination across levels of care. Methods: A systematic review of literature was conducted to identify indicators of clinical coordination across levels of care. These indicators were analysed to identify attributes of coordination and classified accordingly. They were then discussed within an expert team and adapted or newly developed, and their relevance, scientific soundness and feasibility were examined. The indicators were tested in three healthcare areas of the Catalan health system. Results: 52 indicators were identified addressing 11 attributes of clinical coordination across levels of care. The final set consisted of 21 output indicators. Clinical information transfer is evaluated based on information flow (4) and the adequacy of shared information (3). Clinical management coordination indicators evaluate care coherence through diagnostic testing (2) and medication (1), provision of care at the most appropriate level (2), completion of diagnostic process (1), follow-up after hospital discharge (4) and accessibility across levels of care (4). The application of indicators showed differences in the degree of clinical coordination depending on the attribute and area. Conclusion: A set of rigorous and scientifically sound measures of clinical coordination across levels of care were developed based on a literature review and discussion with experts. This set of indicators comprehensively address the different attributes of clinical coordination in main transitions across levels of care. It could be employed to identify areas in which health services can be improved, as well as to measure the effect of efforts to improve clinical coordination in healthcare organizations. Keywords: Quality indicators, Coordination across levels of care, Clinical management coordination, Clinical information coordination, Health services research Background Healthcare systems are in a constant process of adaptation due to rapid advances in technology, new treatments, high specialisation and changes in the organization of health services [1]. As a consequence, patients are seen by an everexpanding array of different providers in a variety of * Correspondence: maller@consorci.org 1 Health Policy and Health Services Research Group, Health Policy Research Unit, Consortium for Health Care and Social Services of Catalonia, Avenida Tibidabo, 21, 08022 Barcelona, Spain Full list of author information is available at the end of the article locations, making coordination difficult [1, 2]. This is particularly challenging in the care of patients with chronic and multiple conditions, who tend to use healthcare services more frequently and use a greater array of services than other patients [3, 4]. Clinical coordination across levels of care should prevent wasteful duplication of diagnostic testing, perilous polypharmacy and conflicting care plans [5, 6]; thus the effects of clinical coordination extend beyond cost reduction through improving quality of care [7–9]. This study is set within a conceptual framework for analysing the performance of integrated healthcare © 2015 Aller et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Aller et al. BMC Health Services Research (2015) 15:323 networks, which is based on an extensive literature review [6, 10] and could be applied in any healthcare area that arranges to provide a coordinated continuum of services to a defined population. In this framework, clinical coordination, together with continuity of care and access to health services, is considered an intermediate objective of integrated healthcare networks and is regarded as a means by which to reach the ultimate objectives of quality of care, efficiency and equity of access [6, 10, 11]. To analyse the achievement of these objectives, both external and internal processes and contextual factors are taken into account, as well as the different perspectives (services, professionals and users) and approaches. In this conceptual framework, clinical coordination is defined as the harmonious connection of the different health services needed to provide care to a patient throughout the care continuum in order to achieve a common objective without conflicts [10, 12]. Continuity of care refers to how individual patients experience coordination of services, and it is defined as the degree to which patients experience care over time as coherent and linked [1]. Clinical coordination across levels of care consists of the coordination of both clinical information and clinical management [6, 10]. Clinical information coordination is the transfer and use of patients’ clinical information in order to harmonize activities between providers, and consists of two dimensions: transfer of clinical information and the use of this information [13]. Clinical management coordination is the provision of care in a sequential and complementary way according to a healthcare plan shared by the different services and healthcare levels involved, and consists of three dimensions: care coherence (i.e., the existence of similar approaches and treatment objectives among professionals from different levels of care), follow-up across care levels (i.e., the adequate monitoring of the patient when there are transitions from one care setting to another) and accessibility across levels (provision of care without interruption across levels of care throughout the clinical episode of the patient) [13]. The results of clinical coordination can be assessed by analysing processes aimed at coordination or their outputs (immediate results of activities related to clinical coordination) or outcomes (final expected middle-long term results of clinical coordination, such as hospital readmissions or avoidable hospital admissions), and using different perspectives (services, professionals, users (continuity)). The focus of this study relies on measures to assess the outputs of clinical coordination across levels of care (primary and secondary) by using service-based indicators. Despite the interest this subject has generated, there are still important gaps in terms of measures to assess clinical coordination across levels of care and the development of new indicators continues to be considered a priority in health policy and health services research [14, 15]. Many of the attempts to address this to date have Page 2 of 16 focused on developing indicators to measure healthcare outcomes which are attributed to improvements in clinical coordination [16]. However, the development of output indicators has been limited, and without this type of indicators it is not possible to conclude that outcomes in health care can be attributed to improvements in clinical coordination across levels of care [15]. Existing sets of indicators are usually designed to analyse a single dimension (e.g. transfer of information) or attribute (e.g. due completion of referral forms and discharge reports) of clinical coordination [17–20]. Those which address more than one dimension of clinical coordination are not exhaustive in their approach to clinical coordination and are often insufficiently operative or are not directed at the assessment of clinical coordination across levels of care [21–25]. Furthermore, the conceptual framework used to develop these measures is not generally explained in detail, so it is not obvious exactly which aspects of clinical coordination are being analysed or how measures relate to clinical coordination. As a result of these issues, there is an overrepresentation of some dimensions of clinical coordination addressed by indicators, whilst other dimensions have scarcely been investigated [26]. Studies have concentrated in particular on the transfer of clinical information [22–24, 27–30], especially in terms of completeness of information in discharge reports [22, 30–34] and to a lesser degree in emergency reports [30] and referral forms [20, 35], and on the followup of patients and accessibility across care levels [22, 24, 29, 30, 36]. Only a few studies have used indicators to measure clinical coherence between care levels [30, 37]. The aim of this study is to develop and test a set of output indicators to comprehensively evaluate clinical coordination across care levels of care, i.e. addressing both types of clinical coordination, information and management, and their dimensions and attributes. Methods The study consisted of two phases: in the first phase, a set of indicators to measure clinical coordination across levels of care was developed based on the literature review and expert discussions, and in the second phase, the set was tested in three different healthcare areas. 1. Development of a set of indicators to measure clinical coordination across levels of care Identification of indicators: literature review The study was based on the conceptual framework for analysing the performance of integrated healthcare networks [6, 10], which identifies two types of clinical coordination across levels of care (clinical information and clinical management) and five dimensions (transfer of information, use of information, care coherence, follow-up across levels and accessibility across levels). A systematic Aller et al. BMC Health Services Research (2015) 15:323 review of literature was undertaken to identify previously developed indicators. A computerised search of the following bibliographic databases was conducted: Pubmed, Social Science Citation Index, Science Citation Index, ECONLIT, CINAHL and LILACS, in addition to standard internet search engines such as Google. The search strategy included a combination of descriptors and keywords relating to clinical coordination (‘coordination of care’ or associated key terms with similar meaning), levels of care (‘primary care’, ‘secondary care’, ‘hospitalization’, ‘interface’, ‘cross-level’ or associated terms) and measurement tools (‘measure’, ‘indicator’ or associated key terms), making use of the Boolean operator ‘AND’. References from retrieved studies were also screened for possible omissions. The search was conducted in May 2011. Additional searches were conducted on the following organizations’ websites: Agency for Healthcare Research and Quality (AHRQ), World Health Organization (WHO), Pan American Health Organization (PAHO), Physician Consortium for Performance Improvement (PCPI), The Joint Commission, Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS), Observatori de Tendències de Serveis de Salut, the RAND corporation and the National Quality Forum (NQF). Studies in English, Spanish, Portuguese or Catalan which included one or more indicators of clinical coordination across levels of care were selected. Selection and adaptation of indicators according to the different types and dimensions of clinical coordination First, the indicators included in the selected studies were analyzed to identify which attributes of clinical coordination they addressed [10]. They were subsequently grouped according to type, dimension and the attribute Page 3 of 16 of clinical coordination across levels of care that they addressed (Fig. 1). Second, three meetings took place with a team of 13 experts, who were either healthcare researchers with experience in the development of performance indicators and clinical coordination assessment or managers of healthcare services. Decisions concerning the selection and adaptation of indicators were made taking the scientific literature into account and when a consensus was reached among all participating members over three sessions of roundtable discussions. During the first meeting, it became obvious that in order to be applied, most of the indicators could not be generic but rather needed to be defined in relation to a specific disease. However, in order to gain a good grasp of the degree of coordination, a number of different diseases were included which require high levels of coordination across levels of care: diabetes mellitus type II, heart failure, chronic obstructive pulmonary disease (COPD) and breast, lung, bladder and colon cancer. During the meetings, indicators were discussed and adapted based on the existing local clinical practice guidelines, which formed the basis on which standards of clinical coordination across levels of care were established [38–41] (for example, the guidelines allowed the team to determine when an urgent referral to secondary care is appropriate or to define the maximum acceptable time from discharge to a consultation in primary care). Each indicator was described in terms of numerator, denominator, target population, exclusion criteria, definition of terms involved, sources of data and bibliography [42]. For each indicator, the team discussed its relevance to clinical coordination across levels of care and its capacity Fig. 1 Attributes of clinical coordination across levels identified, according to the type and dimension of clinical coordination Aller et al. BMC Health Services Research (2015) 15:323 to measure that for which it was designed (face validity). The team also discussed whether the indicator measured an aspect of care that was susceptible to being improved by services (opportunity for improvement), as well as the formulation of the indicator in such precise terms that it could be applied consistently within and between organizations, allowing for comparability (reliability). The experts identified the best sources of data to calculate each indicator (electronic medical record audit and clinical and administrative databases) and discussed its feasibility in terms of data availability and accuracy. 2. Test of the set of indicators Design and settings A retrospective cross-sectional study was conducted applying the set of indicators in three healthcare areas of the Catalan public healthcare system. The objectives were to evaluate the feasibility of the indicators (availability of valid, reliable and consistent data across the system) and to apply the indicators in three different healthcare areas in order to assess their usefulness in describing clinical coordination across levels of care. Three healthcare areas were selected in order to represent the diversity of providers present in Catalonia: Baix Empordà (rural and semi-urban), the city of Girona (urban) and the Ciutat Vella district of Barcelona (urban), which all serve a population of 75,000-100,000. A single entity manages both primary and secondary care in Baix Empordà (Serveis de Salut Integrats Baix Empordà — SSIBE) and in Girona (Institut Català de la Salut — ICS). In Ciutat Vella, two public entities manage primary care (ICS and Institut de Prestacions d’Assistència Mèdica al Personal Municipal — PAMEM) and a different public entity manages secondary care (Parc Salut Mar). With regard to the coordination mechanisms used in these areas, patients served in Baix Empordà had a single electronic medical record for both care levels, whereas patients served in the other two areas had two shared but different electronic medical records for primary and secondary care. Several additional mechanisms have been implemented to improve clinical coordination across levels of care within the organizations, such as shared clinical guidelines, online consultations between primary care physicians and specialists, automated notification of primary care following hospital discharge and clinical case discussions between the two care levels. Page 4 of 16 Two sources of data were used: a) electronic medical record audit, to calculate seven indicators (five related to clinical information coordination across levels of care and two related to clinical management coordination across levels of care); b) clinical and administrative electronic databases (which differ from patients’ individual electronic medical records in the fact that they collate patient data), to calculate twelve indicators (all related to clinical management coordination). For indicators based on electronic medical record audits, the sample size was calculated to estimate proportions, which were expected to be around 0.50; the margin of error was ±0.15 and alpha error 0.05. The sample size required was 42 patients. A simple random sample without replacement was selected from records provided by primary care centres and hospitals. For indicators based on electronic databases, all records were selected. Data collection Instructions for the data collection procedure were developed and systematically applied. Problems during data collection and analysis were recorded. For indicators based on electronic medical record audits, data was retrieved by one researcher using standardized forms. For indicators based on databases, primary care centres and hospitals of the healthcare areas provided clinical and administrative electronic databases. Information was retrieved based on specified procedures, which had to be adapted to each information system. Data analysis In the case of dichotomous indicators, percentages were calculated, and 95 % confidence intervals were estimated when indicators were based on electronic medical record audits. Means and standard deviation were calculated for continuous indicators. Problems during data collection and analysis were discussed with the group in order to assess the applicability of the indicators and identify the main barriers to their implementation. Ethical considerations The principles of confidentiality and anonymity were upheld in the researchers’ conduct, reporting, and storage of data arising from this study, in accordance with European and Spanish legislation on ethical research [43]. The study protocol was approved by the Ethical Committee for Clinical Research ‘Parc Salut Mar (2010/4124/I)’. Study population, data source and sample The study population consisted of patients who had the selected conditions and who had used more than one care level, i.e. they were discharged from hospital, had received outpatient secondary care, were referred to secondary care or were newly diagnosed in primary care, depending on the indicator (Tables 1 and 2). Results 1. Development of a set of indicators to measure clinical coordination across levels of care Identification of indicators: literature review A total of 892 documents were identified: 863 from bibliographic databases, 11 from organizations’ websites and 18 Dimension attribute Description Formula Source of data Adapted from IT1. Percentage of hospital discharges for which a - Numerator: Discharge report available in primary care within discharge report is made available to primary care the first 24 h after hospital discharge within the first 24 h - Denominator: Hospital discharges Discharge reports in EMRs [22, 30, 34, 37, 49] IT2. Mean time to discharge report availability in primary care Discharge reports in EMRs [19] IT3. Percentage of emergency care visits for which - Numerator: Emergency care report available in primary care there is an emergency care report available in within 24 h of the emergency care visit primary care within 24 h - Denominator: Emergency care discharges Discharge reports in EMRs [22, 30, 34, 37, 49] IT4. Mean time to emergency care report availability in primary care Discharge reports in EMRs [19] EMR audit [30] EMR audit [30] EMR audit [30] Information transfer Information flow across levels - Numerator: Total hours elapsed from the time of hospital discharge to report availability in primary care - Denominator: Hospital discharges - Numerator: Total hours elapsed from the emergency care visit to report availability in primary care - Denominator: Emergency care discharges Referral forms and discharge reports duly completed IT5. Percentage of discharge reports duly completed Transfer of information on medication and tests across levels - Numerator: Hospital discharge reports which contain at least four of the following items: reason for admission, additional tests performed and pending, follow-up or monitoring for the patient after discharge, list of current medications and recommendations for the patient Aller et al. BMC Health Services Research (2015) 15:323 Table 1 Indicators related to clinical information coordination across levels of care - Denominator: Hospital discharge reports of patients discharged with a diagnosis of COPD, DM and/or HF IT6. Percentage of emergency care reports duly completed - Numerator: Emergency care reports which contain at least four of the following items: the reason for the emergency care visit, additional tests performed and pending (laboratory, radiology, etc.), follow-up or monitoring of the patient after the emergency care visit, list of current medications and recommendations for the patient - Denominator: Emergency care reports of patients with COPD, DM and/or HF IT7. Percentage of referral forms from primary care duly completed - Numerator: Patients diagnosed with HF, COPD and/or DM that have been referred to secondary care with a referral form that contains relevant background morbidity, current medical treatment, and the reason for the referral - Denominator: Patients diagnosed with HF, COPD and/or DM that have been referred to secondary care Page 5 of 16 Indicators are available at: http://www.consorci.org/coneixement/cataleg-de-publicacions/80/indicadores-de-coordinacion-asistencial-entre-niveles-documento-de-trabajo COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, EMR electronic medical record Aller et al. BMC Health Services Research (2015) 15:323 Page 6 of 16 Table 2 Indicators related to clinical management coordination across levels of care Dimension attribute Description Formula Source of data Adapted from CC1. Percentage of secondary care visits of patients diagnosed with HF in which the specialist ordered tests that were performed in the previous six months in primary care - Numerator: First secondary care visit of HF Clinical and patients referred from primary care in which administrative the specialist ordered a non-urgent, non-priority databases X-ray of the thorax, ECG or general blood test that was performed in the previous six months in primary care Care coherence Coordinated medical testing across involved care levels [50] - Denominator: Total first non-urgent, nonpriority secondary care visits of patients referred from primary care for HF CC2. Percentage of pneumology visits of patients diagnosed with COPD in which the specialist ordered a spirometry that was performed in the previous six months in primary care - Numerator: First non-urgent, non-priority Clinical and pneumology visit of COPD patients referred from administrative primary care in which the specialist ordered a databases spirometry that was performed in the previous six months in primary care [50] - Denominator: Total first non-urgent, nonpriority pneumology visits of patients referred from primary care for COPD Coordinated management of medication by involved levels CC3. Percentage of patients with DM who started insulin therapy during hospitalization and whose primary care medical record documents a follow-up within one week of discharge - Numerator: Patients with DM who started insulin therapy during hospitalization and whose primary care medical record documents a follow-up within one week of discharge Clinical and administrative databases [36, 47] EMR audit [17] EMR audit [17] Clinical and administrative databases [47] Clinical and administrative databases [21, 24, 32, 37, 44, 60] Clinical and administrative databases [21, 24, 32, 37, 44, 60] - Denominator: Patients with DM who started insulin therapy during hospitalization Care at the most appropriate level CC4. Percentage of patients with HF correctly referred from primary care to non-urgent outpatient secondary care - Numerator: Patients diagnosed with HF and correctly referred to cardiology or internal medicine - Denominator: Patients diagnosed with HF that have been referred from primary care to cardiology or internal medicine CC5. Percentage of patients with HF that have - Numerator: Patients with exacerbation of HF been correctly referred to emergency care that have been correctly referred to from primary care emergency care from primary care - Denominator: Patients that visit emergency care for decompensated HF referred by primary care Completion of diagnostic process when more than one level is involved CC6. Percentage of patients with HF diagnosed in the past year who had an echocardiogram as part of the diagnostic process - Numerator: Patients diagnosed with HF who had an echocardiogram as part of the diagnostic process FU1. Percentage of hospital discharges with contact between the hospital and primary care prior to the discharge of patients hospitalized for severe exacerbation of COPD - Numerator: Hospital discharges with principal diagnosis related to the severe exacerbation of COPD and in which the hospital has contacted primary care prior to the discharge - Denominator: Total of patients diagnosed with HF Follow-up across levels Communication between involved levels - Denominator: Hospital discharges with principal diagnosis related to severe exacerbation of COPD FU2. Percentage of hospital discharges with contact between the hospital and primary care prior to the discharge of patients hospitalized for decompensated HF - Numerator: Hospital discharges with principal diagnosis related to decompensated HF in which primary care has been contacted prior to discharge - Denominator: Hospital discharges with principal diagnosis related to decompensated HF Aller et al. BMC Health Services Research (2015) 15:323 Page 7 of 16 Table 2 Indicators related to clinical management coordination across levels of care (Continued) Follow-up visits after hospital discharge FU3. Percentage of hospital discharges of patients admitted for exacerbation of COPD who have a consultation in primary care in less than 72 h - Numerator: Hospital discharges with Clinical and principal diagnosis related to severe administrative exacerbation of COPD and with a consultation databases in primary care in less than 72 h [21, 24, 32, 37, 44, 60] - Denominator: Hospital discharges with principal diagnosis related to severe exacerbation of COPD FU4. Percentage of hospital discharges of patients admitted for decompensated HF who have a consultation in primary care in less than 7 days - Numerator: Hospital discharges with principal diagnosis related to decompensated HF and with a consultation in primary care in less than 7 days Clinical and administrative databases [21, 24, 32, 37, 44, 60] Clinical and administrative databases [61, 62] Clinical and administrative databases [61, 62] - Denominator: Patients discharged with principal diagnosis related to decompensated HF Accessibility across levels Waiting time after referral AAL1. Mean time elapsed from non-urgent, non-priority primary care referral of HF patients to cardiologist visit - Numerator: Total days elapsed from nonurgent, non-priority, primary care referral of HF patients to cardiologist visit - Denominator: Total HF patients with nonurgent, non-priority referrals from primary care to cardiology AAL2. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to the first specialist care visit - Numerator: Total days elapsed from the primary care referral of a patient with suspected cancer to the first appointment with rapid diagnosis program - Denominator: Total patients referred from primary care to specialist care for suspected cancer (lung, colorectal, breast, bladder and prostate) AAL3. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to time of cancer diagnosis - Numerator: Total days elapsed from the primary care referral of a patient with suspected cancer to the diagnosis of cancer Clinical and administrative databases [61, 62] AAL4. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to the initiation of cancer treatment (surgery and/or chemotherapy and/or radiotherapy) Clinical and - Numerator: Total days elapsed from the referral from primary care of a patient with administrative suspected cancer to the initiation of cancer databases treatment (surgery and/or chemotherapy and/ or radiotherapy) [61, 62] - Denominator: Total patients with suspected cancer (lung, colorectal, breast, bladder and prostate) first identified in primary care and with a later diagnosis of cancer - Denominator: Total patients diagnosed with cancer (lung, colorectal, breast, bladder and prostate) referred to secondary care from primary care who initiate treatment including surgery, chemotherapy and/or radiotherapy at the hospital to which they were referred from primary care Indicators are available at: http://www.consorci.org/coneixement/cataleg-de-publicacions/80/indicadores-de-coordinacion-asistencial-entre-niveles-documento-de-trabajo COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, EMR electronic medical record Aller et al. BMC Health Services Research (2015) 15:323 from references in retrieved studies (Appendix 1). Of these documents, 862 were excluded because they did not describe nor use indicators of clinical coordination across levels of care, and 30 met the inclusion criteria, containing at least one indicator. From these documents, 52 indicators were initially identified [17, 19, 21–24, 27, 28, 30–34, 36, 37, 44–50]. Selection and adaptation of indicators according to the different types and dimensions of clinical coordination The 52 indicators addressed 11 different attributes of clinical coordination across levels of care (Fig. 2): 3 related to clinical information coordination and 8 related to clinical management coordination. The dimension “use of information” was not addressed by any attribute or indicator. After two meetings, an initial set of 21 indicators was drawn up (Fig. 2), which addressed 10 of the 11 identified attributes, since it was not possible to establish an unambiguous criterion which would permit the identification of Fig. 2 Stages in the development of the set of indicators Page 8 of 16 redundant consultations. The remaining attributes were represented by at least 1 indicator. The final set of indicators was as follows (Table 1 and 2): a) Clinical information coordination across levels of care: 7 indicators measure the transfer of clinical information across care levels, addressing the availability of inpatient and emergency discharge reports in primary care (four indicators) and the completeness of inpatient and emergency discharge reports and referral forms, including the transfer of information on new medication, medical tests, reasons for referral and information for patients (three indicators). No indicator addressed the dimension “use of transferred information” since it was not possible to identify or design any indicator measuring the effective use by professionals of information generated in the other care level. Aller et al. BMC Health Services Research (2015) 15:323 b) Clinical management coordination across levels of care: 6 indicators address care coherence by measuring the coordinated management of medical testing in primary and secondary care of patients with heart failure and COPD (2 indicators), the adequacy of the referral of heart failure patients from primary care to non-urgent outpatient secondary care or emergency care (2 indicators) and the completion of the diagnostic process for heart failure, which requires coordination between the two care levels (1 indicator). Four indicators measure the follow-up of patients, addressing the communication between the hospital and the primary care centre when patients with heart failure and COPD are discharged (2 indicators) and their follow-up in primary care after being discharged (2 indicators). Finally, four indicators measure accessibility across care levels, specifically the time elapsed from the primary care referral of patients with heart failure or suspected cancer to their first specialist care appointment (2 indicators) and the time elapsed from the suspicion of cancer in primary care to cancer diagnosis or initiation of treatment (2 indicators). 2. Test of the set of indicators Clinical care information: transfer of clinical information across levels of care In the three healthcare areas, inpatient and emergency discharge reports were immediately available in primary care, since the two care levels share electronic medical records. In general, the quality of transferred clinical information was high (i.e. the clinical information required for the transfer of patients between care levels is duly registered; for example, in referral forms: background morbidity, current medical treatment and reason for referral), especially with respect to the completeness of inpatient and emergency discharge reports, although there are notable differences between areas (57.1 % of discharge reports duly completed in Baix Empordà as opposed to 95.2 % in Girona) (Table 3). In contrast, there were low percentages of duly completed referral reports in two of the healthcare areas (11.9 % and 26.2 % of reports). Feasibility All indicators were feasible in the three healthcare areas; however, in some cases the specified sample size was not reached due to an insufficient number of cases per year or due to insufficient precision in the available data to allow identification of the denominator – more than 100 records were reviewed without reaching the required sample size. Page 9 of 16 Clinical management coordination across levels of care Care coherence Indicators showed different degrees of test duplication (coordinated medical testing across levels of care) depending on the type of medical test performed: the highest level of duplication was observed in electrocardiograms for patients with heart failure (48 %) and the lowest was observed in spirometries for patients with COPD (2.5 %) (Table 4). In terms of care at the most appropriate care level, indicators showed high levels of adequate referral to non-urgent and emergency care. Finally, in two healthcare areas there were low percentages of patients (13.9 % and 22.7 %) who had had an echocardiogram performed in the year prior to the diagnosis of heart failure (completion of the diagnostic process). Follow-up across care levels In terms of communication, there were significant differences in the degree to which hospitals communicate with primary care prior to the discharge of heart failure or COPD patients (58 % and 3.2 % of COPD patient discharges in Baix Empordà and Ciutat Vella respectively). Similarly, with regard to follow-up after hospital discharge, there were marked differences between areas (follow-up of patients with COPD in primary care ranged from 26 % in Ciutat Vella to 76.7 % in Baix Empordà). Accessibility across care levels The average time waited to access secondary non-urgent care for heart failure patients referred to cardiology was higher than three weeks in all cases, with significant variations across areas. In contrast, the average time waited to access urgent care for patients with suspected cancer was lower than a week in all healthcare areas, with little variation between areas. Feasibility Five indicators were feasible in the three healthcare areas, six indicators were feasible in two healthcare areas, two indicators were feasible in only one area and lastly, one indicator was not feasible in any of the healthcare areas. Difficulties in calculating indicators were due to two types of problems. Firstly, problems related to the identification of the denominator: not possible to identify patients who had started insulin therapy (3 areas), patients referred for the first time to the secondary care level (2 areas), and patients referred to secondary care for suspected cancer (1 area). Secondly, problems related to the availability and accuracy of data needed to calculate the numerator: reason for seeking emergency care not recorded (2 areas), echocardiograms conducted in secondary care not systematically registered (1 area). Indicator Baix Empordà Girona Ciutat Vella ICS- Parc de Salut Mar PAMEM- Parc de Salut Mar IT1. Percentage of hospital discharges for which a discharge report is made available to primary care within the first 24 h % 100 % 100 % 100 % 100 % IT2. Mean time to discharge report availability in primary care hours immediate immediate immediate immediate IT3. Percentage of emergency care visits for which there is an emergency care report available in primary care within 24 h % 100 % 100 % 100 % 100 % IT4. Mean time to emergency care report availability in primary care hours immediate immediate immediate immediate IT5. Percentage of discharge reports duly completed (at least four of the five selected items) % (95 % IC) n 57.1 % (41.5-72.7) n: 42 95.2 % (88.5 -100) n: 42 83.3 % (65.6- 91.5) n: 42 83.3 % (65.6-91.5) n: 42 Reason for admission % (95 % IC) 100 % 95.2 % (88.5 - 100) 97.6 % (92.7 - 100) 100 % Additional tests performed and pending % (95 % IC) 95.2 % (88.5 - 100) 97.6 % (92.7 - 100) 95.2 % (88.5 - 100) 88.1 % (74.7- 96.7) Follow-up or monitoring of the patient after discharge % (95 % IC) 64.3 % (49.2 - 79.0) 97.6 % (92.7 - 100) 92.9 % (84.7 - 100) 88.1 % (74.7- 96.7) List of current medications % (95 % IC) 88.1 % (77.9 - 98.31) 92.9 % (84.7 - 100) 88.1 % (77.9 - 98.3) 83.3 % (65.6- 91.5) Recommendations for the patient IT6. Percentage of emergency care reports duly completed (at least four of the five selected items) Reason for admission % (95 % IC) 0% 97.6 % (92.7 - 100) 26.2 % (12.3 - 40.1) 16.7 % (6.7-31.4) % (95 % IC) n 85.4 % (74.1-96.7) n: 41 85.7 % (74.6-96.7) n: 42 86.7 % (73.8 - 100) n: 30 64.3 % (49.2-79.4) n: 42 % (95 % IC) 100 % 100 % 100 % 100 % Additional tests performed and pending % (95 % IC) 92.7 % (84.4 - 100) 100 % 90 % (78.6 - 100) 88.1 % (77.9 - 98.3) Follow-up or monitoring of the patient after discharge % (95 % IC) 90.2 % (80.7 - 99.7) 97.6 % (92.7 - 100) 90% (78.6 - 100) 76.2 % (62.8 - 89.6) List of current medications % (95 % IC) 97.6 % (92.6 - 100) 88.1 % (74.7- 96.7) 90 % (78.6 - 100) 85.71 % (74.6 - 96.7) Recommendations for the patient % (95 % IC) 19.51 % (6.8 - 32.2) 50 % (34.2 - 65.8) 43.3 % (25.5 - 62.2) 33.3 % (18.5 - 48.2) % (95 % IC) n 26.2 % (12.3- 40.1) n:42 71.4 % (57.2-85.7) n: 42 11.9 % (1.7 -22.1) n: 42 88.5 % (75.3 - 100) n:26 IT7. Percentage of referral forms from primary care duly completed % (95 % IC) 90.5 % (81.2 - 99.7) 95.2 % (88.5 - 100) 86.7 % (73.8 - 100) 100 % Current medical treatment % (95 % IC) 30.9 % (16.4 - 45.5) 90.8 % (81.2 - 99.7) 16.7 % (4.9 - 28.4) 96.1 % (88.0 - 100) Reason for the referral % (95 % IC) 90.7 % (81.2 - 99.7) 57.1 % (41.5 - 72.7) 76.2 % (62.7 - 89.6) 88.5 % (75.3 - 100) CI confidence interval, COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, SD standard deviation Page 10 of 16 Background morbidity Aller et al. BMC Health Services Research (2015) 15:323 Table 3 Application of the set of indicators related to clinical information coordination across levels of care Aller et al. BMC Health Services Research (2015) 15:323 Page 11 of 16 Table 4 Application of the set of indicators related to clinical management coordination across levels of care Indicator Baix Empordà Girona %; n 3.6 %; n:56 %; n Ciutat Vella ICS- Parc de Salut Mar ICS- Parc de Salut Mar - - - 48.2 %; n:56 - - - %; n 16.1 %; n:56 - - - CC2. Percentage of pneumology visits of patients diagnosed with COPD in which the specialist ordered a spirometry that was performed in the previous six months in primary care %; n 2.5 %; n:81 - - - CC3. Percentage of patients with DM who started insulin therapy during hospitalization and whose primary care medical record documents a follow-up within one week of discharge %; n - - - - CC4. Percentage of patients with HF correctly referred from primary care to non-urgent outpatient secondary care % 85.7 % 81.0 % 83.3 % (95 % IC) n (69.4-100) (68.9-93.3) (75.6-95.1) n:42 n:42 n:42 88.5 % (75.3-100) n:26 CC5. Percentage of patients with HF that have been correctly referred to emergency care from primary care % 97.4 % (95 % IC) n (92.0-100) n:39 - 95.2 % (88.5-100) n:42 CC6. Percentage of patients with HF diagnosed in the past year who had an echocardiogram as part of the diagnostic process %; n 22.7 %; n:203 13.9 %: n:216 - - FU1. Percentage of hospital discharges with contact between the hospital and primary care prior to the discharge of patients hospitalized for severe exacerbation of COPD %; n 58 %; n:88 36.8 %; n:407 3.2 %; n:95 0 %; n:49 FU2. Percentage of hospital discharges with contact between the hospital and primary care prior to the discharge of patients hospitalized for decompensated HF %; n 40.3 %; n:119 44.78 %; n:201 0 %; n:48 2.78 %; n:36 FU3. Percentage of hospital discharges of patients admitted for exacerbation of COPD who have a consultation in primary care in less than 72 h %; n 76.7 %; n:86 52.9 %; n:240 26.0 %; n:68 53.3 %; n:45 FU4. Percentage of hospital discharges of patients admitted for decompensated HF who have a consultation in primary care in less than 7 days %; n 70.9 %; n:110 79.6 %; n:157 55.3 %; n:38 70.6 %; n:19 AAL1. Mean time elapsed from non-urgent, non-priority primary care referral of HF patients to cardiologist visit Mean (SD); 28.2 (4.0); n n:42 - 39.6 (5.6); n:57 100.9 (9.1); n:86 AAL2. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to the first specialist visit Mean (SD); 5.3 (0.3); n n:362 - 6.5 (0.4); n:87 6.6 (1.0); n:17 AAL3. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to cancer diagnosis Mean (SD); 46.9 (9.7); n n:70 - 31.4 (4.6); n:36 39.9 (8.5); n:8 AAL4. Mean time elapsed from the referral of a patient with suspected cancer (lung, colorectal, breast, bladder and prostate) to the initiation of cancer treatment (surgery and/or chemotherapy and/or radiotherapy) Mean (SD); 71.4 (9.2); n n:64 - 48.1 (5.1); n:33 46.9 (5.8); n:8 CC1. Percentage of secondary care visits of patients Duplication of diagnosed with HF in which the specialist radiographies ordered tests that were performed in the Duplication of previous six months in primary care electrocardiograms Duplication of analytics CI confidence interval, COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, SD standard deviation Discussion Clinical coordination is considered a health policy priority, as a lack of coordination can lead to poor quality of care and inefficiencies in the use of resources [5–9]. However, its measurement is still challenging [1, 14, 15], since calculating the degree of clinical coordination in its multidimensional nature requires the availability of indicators that cover the different types and dimensions of clinical coordination. Until now, most attempts to tackle this challenge have focused on the design of indicators to measure certain outcomes which can potentially be attributed to clinical Aller et al. BMC Health Services Research (2015) 15:323 coordination [16]. However, progress must be made in the design of instruments to measure the outputs of clinical coordination in order to be able to attribute improvements in the outcomes of health care to improvements in clinical coordination [51]. With this in mind, this research constitutes a step forward by using a pre-established conceptual framework to generate a set of output indicators which address the two types of clinical coordination across levels of care (and most of their dimensions and attributes) that have been highlighted in previous studies [52]. Furthermore, in contrast with previous efforts [17, 37], the indicators presented here have been described in operative terms, thus allowing for their precise application in healthcare organizations. With regard to clinical information coordination across levels of care, seven indicators addressing the transfer of clinical information have been created. Applying these indicators has permitted the analysis of transfer of information in the three healthcare settings, taking in both evidence of information transfer between levels and the quality of the information transferred. No previous set of indicators has allowed researchers to address these two attributes jointly in main transitions between levels of care [1, 30, 34, 53], so this is one of its most significant contributions. In addition, the result of the applicability test has proven that these indicators have the accuracy and feasibility needed to make their calculation possible in different healthcare areas. Their joint application has revealed that, although there is a flow of information between the different care levels, the quality of information varies across transitions and organizations, thus leading to the identification of specific margins of improvement in each healthcare area. It is important to highlight, however, that clinical information coordination is not fully represented by the set of indicators, since we were unable to address the use of transferred information; i.e. we could not determine whether information was actually read and used by the receiving professional [54]. The lack of this type of measure of clinical coordination has been previously expressed in the literature [1, 54] and reflects the complexity of analyzing an activity which is not generally recorded but is considered central to clinical coordination. The fact that indicators are unable to systematically address all dimensions and attributes of clinical coordination points to the need to complement and enrich indicator results with those that can be obtained via different techniques, such as surveys or qualitative interviews with health professionals and patients. With regard to clinical management coordination across levels of care, the systematic review led us to identify five attributes that define care coherence, two that define follow-up and one that defines accessibility across care levels. Their operationalization has resulted in a set of indicators to measure the main attributes of Page 12 of 16 care coherence (such as coordinated medical testing across care levels or the provision of care at the most appropriate care level), follow-up (such as the existence of communication and follow-up after discharge) and accessibility across care levels (waiting time after referral). However, one of the eight identified attributes of care coherence, no redundant visits to primary and secondary care, is not represented by any indicator, since we were unable to establish an unambiguous criterion, either though the literature review or by expert consensus, which would permit the identification of redundant consultations. During the first meeting, it became obvious that in order to be applied, most indicators could not be general but needed to be defined relating to a specific disease. However, in order to gain a good grasp of the degree of coordination in the area, a number of different diseases were included, which require high levels of coordination across levels of care: diabetes mellitus type II, heart failure, chronic obstructive pulmonary disease (COPD) and breast, lung, bladder and colon cancer. The indicators to measure clinical management coordination have been adapted to several clinical conditions, due to the fact that the standards of clinical coordination upon which indicators are based need to be precise and based on what the evidence dictates, which varies according to the disease. Nevertheless, they can be adapted to other conditions as long as they have an evidence-based recommendation upon which to base the standard of clinical coordination measured by the indicator. Moreover, the use of the selected conditions (diabetes mellitus, chronic obstructive pulmonary disease, heart failure and cancer) could be considered a good strategy to identify the strengths and weakness in clinical coordination across levels of care [3, 55], since they meet the criteria to be considered adequate tracer conditions [55]: care is provided across levels and over the course of time; the care that should be provided at each care level is well defined; they are among the most prevalent diseases in the population; diagnoses are well defined; and their epidemiology is well known. The applicability test illustrates the usefulness of these indicators in describing clinical management coordination, pointing to areas for improvement, such as the coordination of medical testing in Baix Empordà and communication with primary care after discharge in Ciutat Vella. Furthermore, as they cover the main attributes of clinical management coordination across levels, they can be used to support the design of strategies to improve clinical coordination between levels of care. It is important to note, however, that several problems arose which made the calculation of some of these indicators difficult or impossible in some healthcare areas. Aller et al. BMC Health Services Research (2015) 15:323 The problems were related to non-registration of the variables in information systems and under-registration of information by professionals, which points to the need for further improvements in information systems and record-keeping skills before we can systematically measure certain relevant aspects related to clinical management coordination across levels of care in these healthcare areas [56–59]. The methodology adopted in this study provides guarantees in terms of reliability of the indicators, since they have been adequately defined and precisely specified so that they can be implemented consistently within and across organizations (48). This is also true in terms of face and content validity, as the indicators have been adapted or newly created on the basis of scientific evidence and expert consensus. Furthermore, the applicability test provided information regarding data availability and accuracy (feasibility), thus highlighting major and minor problems in calculating the indicators, which could be informative for future studies. Finally, the indicators have been shown to be able to identify differences between areas, even in small samples. Further research should provide evidence regarding other relevant characteristics of the indicators, such as test-retest reliability and discriminant validity. Certain aspects should be taken into account when applying the set of indicators in other healthcare contexts. First of all, data availability and validity should be explored. Secondly, it should be determined whether the information taken from the different health information systems is linkable, since indicators are constructed upon information generated in different levels of care. Lastly, although information recorded in digital format is desirable, most indicators could be calculated from a medical record audit, so data computerization is not a prerequisite. One limitation of this study is the possibility of a publication bias in which relevant indicators were not identified (for example, indicators that measure clinical coordination but employ different terms, indicators published in other languages or grey literature not easily accessed by standard internet search engines). Moreover, the inclusion of terms referring to certain attributes of clinical coordination, such as “follow-up” or “referral adequacy”, might have extended the range of studies obtained. However, we employed several additional strategies for the identification of studies to reduce the possibility of publication bias, such as reviewing the reference lists of eligible documents and consulting the websites of the main organizations that design indicators. Another limitation is that one dimension (the use of clinical information) and one attribute (no redundant visits to primary and secondary care) are not Page 13 of 16 represented by the set of indicators, pointing to the need to enrich the results obtained by the indicators with additional information from health professionals and patients in order to attain a more accurate evaluation of the process of coordination. Conclusions A set of rigorous and scientifically sound measures of clinical coordination were developed based on a literature review and discussion with experts. These indicators of clinical information and management coordination across levels of care could be employed to identify areas in which health care can be improved, as well as to measure the effect of efforts to improve clinical coordination. However, some relevant attributes of clinical coordination are not represented in the final set of indicators, which detracts from its comprehensiveness. In fact, clinical coordination is a multidisciplinary construct, and certain relevant dimensions and attributes of clinical coordination across levels of care such as the effective use of transferred information or redundant visits cannot be properly measured through indicators. Other approaches are therefore needed to obtain additional information, such as surveys or qualitative interviews. The indicators provided may also be useful for conducting comparative studies of clinical coordination across healthcare areas. Aspects such as the possibility of linking information from different health information systems, data availability and validity should be explored before proceeding to implement these indicators. Appendix 1 Search Strategy and number of studies retrieved in the bibliographic databases Table 5 A1: medline-pubmed; 19/05/2011 References number 1. Clinical Coordination "Coordinated care" OR " coordination of care" OR 5.573 "integrated care" OR "shared care" OR "transitional care" OR "continuity of care" 2. Levels of care "Primary care" OR "family practice" OR "Generalist" OR "GP" OR "outpatient" OR "secondary care" OR "specialized" OR ‘specialist’ OR "inpatient" OR "hospitalization" OR "hospitalisation" OR "care levels" OR "interface" OR "crosslevel" OR "referral" OR "communication" 655.466 3. Measurement tools measure OR measures OR indicator 939.670 4. (1) and (2) and (3) 466 Aller et al. BMC Health Services Research (2015) 15:323 Page 14 of 16 Table 6 A2: Isi web of knowledge; social science citation index & science citation index 19/05/2011 References number 1. Clinical coordination "Coordinated care" OR "care coordination" OR "collaborative care" OR "integrated care" OR "shared care" OR "transitional care" OR "continuity of care" OR "care continuity" OR "informational continuity" OR "managerial continuity" OR "management continuity" 5.509 References number 1. Clinical coordination AB (Coordinated care) OR (care coordination) OR (collaborative care) OR (integrated care) OR (shared care) OR (transitional care) OR (continuity of care) OR (care continuity) OR (informational continuity) OR (managerial continuity) OR (management continuity) 5.092 2. Levels of care 2. Levels of care "Primary care" OR "family practice" OR "Generalist" OR "GP" OR "outpatient" OR "secondary care" OR "specialized" OR "specialised" OR ‘specialist’ OR "inpatient" OR "hospitalization" OR "hospitalisation" OR "care levels" OR "interface" OR "cross-level" OR "referral" OR "communication" Table 8 A4: CINAHL; 19/05/2011 >100.000 AB (Primary care) OR (family practice) OR (Generalist) OR (GP) OR (outpatient) OR (secondary care) OR (specialized) OR (specialised) OR ‘specialist’ OR (inpatient) OR (hospitalization) OR (hospitalisation) OR (care levels) OR (interface) OR (cross-level) OR (referral) OR (communication) 74.288 3. Measurement tools 3. Measurement tools measure OR measures OR indicator >100.000 AB measure OR measures OR indicator 104.884 4. (1) and (2) and (3) 500 4. (1) and (2) and (3) 296 Duplicates 266 Duplicates 139 Total 234 Total 157 Table 9 A5: LILACS; 19/05/2011 References number Table 7 A3: ECONLIT; 19/05/2011 ((coordinación) OR (continuidad) OR (colaborativa) OR (transición) OR (compartido)) AND ((asistencial) OR (de la atención)) AND (indicador OR indicadores OR medida OR medición OR mediciones) [Palabras del resumen] 5 Duplicates 0 Total 5 References number Competing interests The authors declare that they have no competing interest. 1. Clinical coordination TX (Coordinated care) OR (care coordination) OR (collaborative care) OR (integrated care) OR (shared care) OR (transitional care) OR (continuity of care) OR (care continuity) OR (informational continuity) OR (managerial continuity) OR (management continuity) 129 2. Levels of care TX (Primary care) OR (family practice) OR (Generalist) OR (GP) OR (outpatient) OR (secondary care) OR (specialized) OR (specialised) OR ‘specialist’ OR (inpatient) OR (hospitalization) OR (hospitalisation) OR (care levels) OR (interface) OR (cross-level) OR (referral) OR (communication) 25.224 3. Measurement tools TX measure OR measures OR indicator 51.184 4. (1) and (2) and (3) 4 Duplicates 3 Total 1 Authors’ contribution MBA, IV and MLV were responsible for the study conception and design, for the systematic review and data analysis. MBA, IV, JC, SC, FC, MA, JF, JRL, LC and MLV participated in the selection, adaptation, development and interpretation of the indicators. JC, SC, FC, MA participated providing operational and methodological support to the fieldwork. MBA, IV and MLV drafted the manuscript. MBA, IV, JC, SC, FC, MA, JF, JRL, LC and MLV participated in data interpretation, reviewed draft versions of the paper and approved the final version. All authors read and approved the final manuscript. Acknowledgments The research leading to these results received funding from the Instituto de Salud Carlos III (PI10/00348) and Fondos FEDER. The funding source had no involvement in the study design, nor in the collection, analysis and interpretations of data, or in the writing of the article and the decision to submit it for publication. The authors are most grateful to the people that participated in the study and generously gave their time and Claudia Ortiz and Josep Maria Lisbona who contributed to data collection. We thank Nuria Martinez for her administrative support and help, Irene Garcia Subirats for her statistical Aller et al. BMC Health Services Research (2015) 15:323 support, and Kate Bartlett for her help in correcting the English version of this article. Author details 1 Health Policy and Health Services Research Group, Health Policy Research Unit, Consortium for Health Care and Social Services of Catalonia, Avenida Tibidabo, 21, 08022 Barcelona, Spain. 2Grup de Recerca en Serveis Sanitaris i Resultats en Salut, Serveis de Salut Integrats Baix Empordà, Carrer Hospital, 17-19 Edif. Fleming, 17230 Palamós, Spain. 3Catalan Health Institute, Gran Via de les Corts Catalanes, 587, 08007 Barcelona, Spain. 4IMIM - Hospital del Mar Medical Research Institute, Carrer Dr. Aiguader, 88, 08003 Barcelona, Spain. 5 Institut de Prestacions d’Assistència Mèdica al Personal Municipal, Carrer Viladomat, 127, 08015 Barcelona, Spain. 6Centre Integral de Salut Cotxers, Avinguda de Borbó, 18 - 30, 08016 Barcelona, Spain. 7Health Policy and Health Services Research Group; Division of Management, Planning and Organizational Development, Badalona Healthcare Services, Via Augusta, 9-13, 08911 Badalona, Spain. 8Health Policy and Health Services Research Group; Strategic Planning Division, SAGESSA Group, Avinguda del Dr. Josep Laporte, 2, 43204 Reus, Spain. Received: 15 January 2015 Accepted: 24 July 2015 References 1. Reid R, Haggerty J, McKendry R. Defusing the confusion: concepts and measures of continuity of healthcare. Ottawa: Canadian Health Services Research Foundation; 2002. 2. Starfield BH. Atención primaria. [Primary Care: Balancing Health Needs, Services, and Technology]. Barcelona: Masson, S.A.; 2001. 3. Nolte E, McKee M. Caring for people with chronic conditions. A health system perspective. Maidenhead: European Observatory on Health Systems and Policies Series. Mc Graw Hill, Open University Press; 2008. 4. Vogeli C, Shields AE, Lee TA, Gibson TB, Marder WD, Weiss KB, et al. Multiple chronic conditions: prevalence, health consequences, and implications for quality, care management, and costs. J Gen Intern Med. 2007;22 Suppl 3:391–5. 5. Longest B, Young G. Coordination and communication. In: Shortell SM, Kaluzny AD, editors. Health care managment. Organization design and behavior. Albany, NY: Delmar; 2000. p. 210–43. 6. Vázquez ML, Vargas I, Unger JP, Mogollon AS, da Silva MRF, De Paepe P. Integrated health care networks in Latin America: toward a conceptual framework for analysis. Rev Panam Salud Publica. 2009;26:360–7. 7. Banks P. Policy framework for integrated care for older people. London: King's Fund; 2004. 8. Øvretveit J. Does clinical coordination improve quality and save money? Vol.1 A summary review of the evidence. London: Health Foundation; 2011. 9. Moore C, Wisnivesky J, Williams S, McGinn T. Medical errors related to discontinuity of care from an inpatient to an outpatient setting. J Gen Intern Med. 2003;18:646–51. 10. Terraza-Núñez R, Vargas I, Vázquez ML. Coordination among healthcare levels: systematization of tools and measures. Gac Sanit. 2006;20:485–95. 11. Vázquez ML, Vargas I, Unger JP, De Paepe P, Mogollón-Pérez AS, Albuquerque P, et al. Evaluating the effectiveness of care integration strategies in different health care systems in Latin America: the EQUITY-LA II quasi-experimental study protocol. BMJ Open 2015;5:E007037. 12. Shortell SM, Gillies RR, Anderson DA, Morgan KL, Mitchell JB. Remaking health care in America. San Francisco: The Jossey-Bass health care series; 1996. 13. Vargas I, Mogollón-Pérez AS, De Paepe P, da Silva MRF, Unger JP, Vázquez ML. Do existing mechanisms contribute to improvements in care coordination across levels of care in health services networks? Opinions of health personnel in Colombia and Brazil. BMC Health Serv Res 2015; 15:213. 14. U.S. Department of Health and Human Services. 2012 Annual Progress Report to Congress: National Strategy for Quality Improvement in Health Care. 2012. http://www.ahrq.gov/workingforquality/nqs/nqs2012annlrpt.pdf. Accessed 28 July2015. 15. Øvretveit J. Does clinical coordination improve quality and save money?Vol.2 A detailed review of the evidence. London: Health Foundation; 2011. 16. DuGoff EH, Dy S, Giovannetti ER, Leff B, Boyd CM. Setting standards at the forefront of delivery system reform: aligning care coordination quality measures for multiple chronic conditions. J Healthc Qual. 2013;35:58–69. Page 15 of 16 17. Bird JC, Beynon GJ, Prevost AT, Baguley DM. An analysis of referral patterns for dizziness in the primary care setting. Br J Gen Pract. 1998;48:1828–32. 18. Minvielle E, Leleu H, Capuano F, Grenier C, Loirat P, Degos L. Suitab...
Purchase answer to see full attachment