PRINT
HCM570: Critical Thinking Rubric - Module 03
Content, Research, and Analysis
Meets Expectation
Approaches
Expectation
Below
Expectation
Limited Evidence
Requirements
15 to 13
PointsIncludes all
of the required
components, as
specified in the
assignment.
12 to 10
PointsIncludes
most of the
required
components, as
specified in the
assignment.
9 to 7
PointsIncludes
some of the
required
components, as
specified in the
assignment.
6 to 4
PointsIncludes few
of the required
components, as
specified in the
assignment.
Content
15 to 13
PointsDemonstrates
strong or adequate
knowledge of HIEs
or EHRs; correctly
represents
knowledge from the
12 to 10
PointsSome
significant, but not
major, errors or
omissions in
demonstration of
knowledge HIEs or
EHRs.
9 to 7 PointsMajor
errors or omissions
in demonstration of
knowledge
concerning HIEs or
EHRs.
6 to 4 PointsFails
to demonstrate
knowledge of the
materials.
Content, Research, and Analysis
Meets Expectation
Approaches
Expectation
Below
Expectation
Limited Evidence
readings and
sources.
Critical Analysis
25 to 21
PointsProvides a
strong critical
analysis and
interpretation of the
information given,
and provides an
assessment of each
system.
20 to 16
PointsSome
significant, but not
major, errors or
omissions in
analysis and
interpretation.
15 to 11
PointsMajor errors
or omissions in
analysis and
interpretation.
10 to 6 PointsFails
to provide critical
analysis and
interpretation of
the information
given.
Synthesis and
Evaluation
15 to 13
PointsDemonstrates
strong or adequate
synthesis and
evaluation of course
concepts in HIEs or
EHRs.
12 to 10
PointsSome
significant, but not
major, errors or
omissions in
synthesis and
evaluation.
9 to 7 PointsMajor
errors or omissions
in synthesis and
evaluation.
6 to 4 PointsFails
to demonstrate
synthesis and
evaluation.
Content, Research, and Analysis
Meets Expectation
Sources /
Examples
10 to 9
PointsSources or
examples meet
required criteria and
are well chosen to
provide substance
and perspectives on
the issue under
examination.
Approaches
Expectation
Below
Expectation
8 to 7
PointsSources or
examples meet
required criteria,
but are less than
adequately chosen
to provide
substance and
perspectives on the
issue under
examination.
6 to 5
PointsSources or
examples meet
required criteria,
but are poorly
chosen to provide
substance and
perspectives on the
issue under
examination.
Limited Evidence
4 to 3
PointsSource or
example selections
and integration of
knowledge from
the course are
clearly deficient.
Mechanics and Writing
Meets
Expectation
Approaches
Expectation
Below Expectation
Limited
Evidence
Demonstrates
college-level
proficiency in
organization,
grammar and
style.
10 to 9
PointsProject is
clearly
organized, well
written, and in
proper format as
outlined in the
assignment;
strong sentence
and paragraph
structure; few
errors in
grammar and
spelling.
8 to 7 PointsProject is
fairly well organized and
written, and in proper
format as outlined in the
assignment;reasonably
good sentence and
paragraph structure;
significant number of
errors in grammar and
spelling.
6 to 5 PointsProject
is poorly organized
and does not follow
proper paper
format;inconsistent
to inadequate
sentence and
paragraph
development;
numerous errors in
grammar and
spelling.
4 to 3
PointsProject is
not organized or
well written, and
is not in proper
paper format;
poor quality
work;
unacceptable in
terms of
grammar and
spelling.
Demonstrates
proper use of
APA style
10 to 9 Points
8 to 7 Points
6 to 5 Points
4 to 3 Points
Mechanics and Writing
Meets
Expectation
Project contains
proper APA
formatting.
•
•
•
Approaches
Expectation
Few errors in APA
formatting.
Max Points for Content, Research, and Analysis80
Max Points for Mechanics and Writing20
Total Points Possible100
Below Expectation
Limited
Evidence
Significant errors in Numerous errors
APA formatting.
in APA
formatting.
Adm Policy Ment Health (2016) 43:471–477
DOI 10.1007/s10488-015-0702-5
COMMENTARY
At the Intersection of Health Information Technology
and Decision Support: Measurement Feedback Systems…and
Beyond
Bruce F. Chorpita1 • Eric L. Daleiden2 • Adam D. Bernstein2
Published online: 24 November 2015
Springer Science+Business Media New York 2015
Abstract We select and comment on concepts and
examples from the target articles in this special issue on
measurement feedback systems, placing them in the context of some of our own insights and ideas about measurement feedback systems, and where those systems lie at
the intersection of technology and decision making. We
contend that, connected to the many implementation
challenges relevant to many new technologies, there are
fundamental design challenges that await a more elaborate
specification of the clinical information and decision
models that underlie these systems. Candidate features of
such models are discussed, which include referencing
multiple evidence bases, facilitating observed and expected
value comparisons, fostering collaboration, and allowing
translation across multiple ontological systems. We call for
a new metaphor for these technologies that goes beyond
measurement feedback and encourages a deeper consideration of the increasingly complex clinical decision models
needed to manage the uncertainty of delivering clinical
care.
Keywords Measurement feedback Dashboard
Evidence base Treatment Clinical decision support
& Bruce F. Chorpita
chorpita@ucla.edu
Eric L. Daleiden
e.daleiden@practicewise.com
Adam D. Bernstein
a.bernstein@practicewise.com
1
UCLA Department of Psychology, University of California,
Los Angeles, Franz Hall 3227, Los Angeles, CA 90095, USA
2
PracticeWise, LLC, 340 Lee Avenue, Satellite Beach,
FL 32937, USA
This issue pulls together a valuable assortment of ideas and
observations from research teams working with a variety of
measurement feedback systems (MFS) to guide clinical
care, while confronting and studying both the human and
technological implementation challenges. Their collective
insights paint a complicated picture that documents both
promises and challenges associated with MFS. In our own
efforts to improve service delivery in children’s mental
health systems over the past 15 years, we have both used
and developed such systems, and have thus encountered
both the good and the bad firsthand. In the context of these
papers, we would like to offer some general insights for
consideration as the field moves forward.
Many of these insights concern the basic relation
between technology and decision making. Technology
generally does what we tell it to do, and thus not surprisingly, early developments in health information technology
(HIT) managed the most essential and best-understood
processes in health care systems, including utilization,
billing, and documentation. These were the questions the
field had to ask in order to function: who was seen when for
what reason, and how it was paid for? But more recently,
there has been a focus on the more complex questions we
want to ask to guide clinical care: What treatments
approaches are we using? Are they helping? What should
we do when they are not? In hindsight, many of these
questions were naively ambitious on our part, which may
explain why they were often met with such equivocal
answers as ‘‘22 sessions of individual therapy’’ or ‘‘30 days
of residential,’’ along with the associated billing codes.
Thanks to a growing vanguard of thinkers, answering
the clinically interesting questions is getting closer to a
common reality. But precisely because technology does
what we tell it to do, this function of HIT will not fully
mature until we have better articulated the complex
123
472
information models that underlie it. We can think of these
latter questions as belonging to the domain of clinical
decision support, as shown in Fig. 1. Note that there are
many strategies to support decision making that don’t
involve HIT (e.g., supervisor recommendations, use of a
treatment manual), and there are many functions of HIT
that don’t involve supporting clinical decision making
(e.g., documentation, service authorization, scheduling).
At this intersection lies a tremendous set of possibilities
and opportunities, within which MFS serves a specific
function. As the figure implies, although we believe there is
much to explore within the context of MFS development—
which entails a human-technology interaction defined by a
user receiving measurement information in the form of a
report or an alert (e.g., Bickman 2008)—there is even more
work to be done within the broader context defined by the
intersection of technology and clinical decision making
(Fig. 1, shaded region). In this regard, we prefer a metaphor of telecommunication (or an interactive workspace),
which suggests collaboration, communication, reasoning,
interaction, and even design. Such systems of course can
feed back information, but they should also be able to feed
forward information to guide action (e.g., setting expectations for what should happen next, exploring ‘‘what if’s’’).
As Bickman et al. (2014) suggest in their ‘‘final coda,’’
although the technology may be important, it should
operate in the service of decision support. In our metaphor,
we know that a telephone conference call must not drop
participants and must be free of background noise, but its
participants will also benefit from having an agenda,
speaking a common language, knowing who else is on the
call, having their thoughts organized, and sharing similar
goals for their meeting. Thus, better articulation of the
work processes to be supported on the ‘‘decision side’’ of
the figure is prerequisite to any technology that could
ultimately better serve those processes.
Fig. 1 Measurement feedback systems in the context of health
information technology and clinical decision support. MFS measurement feedback systems
123
Adm Policy Ment Health (2016) 43:471–477
There is another matter, not represented in Fig. 1,
involving the human implementation challenges that arise
once appropriate technologies are developed. For example,
Gleacher et al. (2015) highlighted the key facilitating role of
organizational leadership in achieving widespread use of
the contextualized feedback system. Without such support,
providers may mute the phone calls, or agencies may disconnect the phone service altogether, metaphorically
speaking. This is but one of many examples that involve
technology implementation issues rather than design issues
(of course the two are inevitably connected in the real
world; e.g., Higa-McMillan et al. 2011; Lyon et al. 2015;
and in treatment contexts, design alone has been associated
with implementation success; e.g., Palinkas et al. 2013,
Southam-Gerow et al. 2014). Nevertheless, in the measurement feedback context, this issue’s authors provide
considerable discussion of implementation, so we focus on
a brief list of ideas relevant to clinical decision support and
its broad intersection with technology, followed by a simple
illustration. These ideas include considerations for sources
of the evidence to be displayed, the value of exposing discrepancies between what has happened and what should or
could happen, the types and configuration of evidence to be
displayed (e.g., beyond progress rating alerts or plots) to
allow users to impose a logic model on interpretation, the
importance of automated translation of relevant ontologies
(e.g., DSM diagnoses vs. elevated scales on a standardized
symptom measure), and the creation of structures to facilitate communication and collaboration.
Multiple Evidence Bases
Daleiden and Chorpita (2005) outlined a model to coordinate and inform service delivery, which among other things
described four core evidence bases relevant to decisionmaking: case-specific historical information, local aggregate evidence, general services research, causal mechanism
research. These are outlined in Table 1, with reference to
corresponding traditions or schools of thought as well as
example questions addressed by each. In the current context of MFSs, there is a predominant emphasis on only one
of these four sources of evidence—the case-specific history
(but see Steinfeld et al. 2015, for examples of using local
aggregate evidence in the form of departmental and system
level reports). We feel much can be gained from coordinating a fuller set of relevant information from all four of
the evidence bases, which gives us the ability to detect,
consider, and act on knowledge that might otherwise
remain out of view, lost in our decision-making ‘‘blind
spots.’’
This task is not possible without a significant amount of
‘‘background complexity,’’ given that each evidence base
Adm Policy Ment Health (2016) 43:471–477
473
Table 1 Four evidence bases relevant to clinical decision making (‘‘Evidence-Based Services System Model’’)
Evidence base
Relevant tradition
Example questions
Case-specific historical
information
Individualized case
conceptualization
What is the primary clinical concern? Is this youth improving?
Local aggregate evidence
Practice-based evidence
What is the most common focus of care? How is the system performing on
average for that specific group?
General Services Research
(evidence-based practice)
Evidence-based practice
What are the best supported treatment approaches for a defined group? How
well have they worked in research trials?
Causal Mechanism Research
(clinical theory)
Individualized case
conceptualization
What would theory suggest is a promising next step? What is a logical
expectation for the outcome?
can have multiple indicators, which can even disagree (e.g.,
two independent randomized trials with discrepant findings; improvement on one measure of depression with
deterioration on another). The challenge of determining
which knowledge is ‘‘best’’ is likely an impossible pursuit,
but fortunately, perhaps not a necessary one. Rather, a
sufficient knowledge management function may be for
systems to prioritize ‘‘better’’ knowledge from a number of
sources through a series of rules or knowledge filters (e.g.,
psychometric validation for case-specific measures;
strength of evidence models for literature review; e.g.,
Chorpita et al. 2011). This is possible because MFSs
inherently create a self-correcting context, in which the real
validation of the knowledge used is accomplished by
observing whether the desired outcomes were achieved.
This notion reflects our fundamental belief that a legitimate
function of these systems can be to provide multiple
promising contextualized ideas to a decision maker, rather
than merely to provide a ‘‘right answer’’ or single prescribed action. In other words, ultimately an agent must
prioritize, act, and test results based on the information
made available, and the technology should support that
process rather than replace it.
from the one indicated by strong research support, then
there is a discrepancy between the observed value and the
expected value. Such discrepancies can motivate and guide
action, perhaps in this case to consideration of a different
treatment, and we expect that a core function of MFS is to
assist with making these discrepancies known.
Several papers in this issue provide examples of using
expected values. For example, Steinfeld et al. (2015)
describe reporting related to expected measure completion
at each encounter. Similarly, Bickman et al. (2014) illustrate alerts related to the expectation that feedback reports
be viewed by practitioners. Similarly, the model-specific
implementations (e.g., Bruns et al. 2015; Nadeem et al.
2015) communicate an expectation that particular service
activities occur, simply by incorporating descriptions of
those activities into the workflow and visual displays.
Although there is evidence that observed-value-only feedback offers advantages over no feedback (e.g., Lambert
et al. 2005), it is worth considering the decision support
value afforded by contextualizing these with expected
values.
Multiple Domains
Expected and Observed Values
In clinical care, expected values (e.g. Chorpita et al.
Research Network on Youth Mental Health 2008) refer to
information that informs our best guesses about what
should happen. For instance, if one needed to select a
treatment that might work for a given youth, one might
consider research trials involving similar youth to inform
that decision (as would be consistent with the evidencebased treatment paradigm); the treatment with the best
research support is thus the expected value for the treatment to be delivered. Expected values can be contrasted
with observed values, which represent information about
what has happened or is happening now. Staying with this
example, if the youth is receiving a treatment that differs
In the same manner that MFS often prioritize the casespecific evidence base and observed values, to date many
systems have also placed a heavy emphasis on the progress
domain (e.g., visualizing symptom change over time).
However, to facilitate technology’s relevance to clinical
decision making, we think MFS platforms should help
examine any type of information (not just progress) that fits
within the larger decision model used to guide care. For
instance, if one believes practice is related to progress, then
organizing information about practices delivered over time
and configuring that information to be synchronized with
progress measurements might be of considerable value.
More generally, any events (e.g., change of medication,
change of placement, stakeholder participation, session noshows, or end of school year) deemed relevant to progress
123
474
interpretation or strategy selection may be useful for
decision support display. This special issue provides two
illustrations of strategies for extending beyond progress
measurement. For example, Nadeem et al. (2015) show
how to track practices delivered directly on the feedback
reports. Using a different strategy, Bruns et al. describe
integration of service activities into the workflow of the
system itself.
Multiple Languages
In keeping with our telecommunication metaphor, in
addition to a shared channel, senders and receivers must
share a language and concept system to transfer knowledge. Accordingly, we see a need for translating across
the diverse ontologies that are found in mental health
research and services (e.g., Diagnostic and Statistical
Manual, Research Domain Criteria, Standardized Instrument Scores, Evidence-Based Practices, Practice Elements, etc.; see Chorpita and Daleiden 2014). Constraints
that require a single common language (e.g., a fixed set of
measures; a single clinical model) are less likely to generalize to diverse contexts and to facilitate communication
in the language of the local jurisdiction or system. The
papers in this special issue clearly illustrate the underlying dilemma. Several of the systems describe a capacity
to support multiple outcome measures (e.g., Bickman
et al. 2014; Bruns et al. 2015; Nadeem et al., 2015),
whereas Steinfeld et al. (2015) highlight some benefits of
committing to a single measurement model even though
their electronic medical record could potentially support
many. On the practice metric side, Bruns et al. (2015)
illustrate the construction and use of a model-specific
system whereas Nadeem et al. (2015) illustrate a modelspecific configuration of a generalized platform for progress and event (practice) representation. In our work
with systems, we have found tremendous value in the
ability to support diversity (of models, measures, display
preferences, etc.) within a single platform, but think that
diversity is best supported in the context of a strong
default configuration that is designed to bias users initially
toward ‘‘best practices,’’ while allowing extension and
adaptation as user expertise develops. For this to happen,
‘‘translator’’ functions (e.g., is an elevated score on the
Children’s Depression Inventory sufficiently similar to a
DSM-III-R diagnosis of Major Depression to draw an
inference for this youth?) as well as diverse ontological
libraries (e.g., configurable lists of practice elements,
evidence-based protocols, or other metrics to represent
practice delivery) are a necessary infrastructure operation
for MFS.
123
Adm Policy Ment Health (2016) 43:471–477
Collaboration
Another fundamental premise is that MFS should both
foster and structure collaboration. Rather than serving as a
substitute for human decision makers, we believe a key
role of these systems is to organize and inform those
involved in care. Collaboration can be an implicit feature,
for example, by requiring treatment team members to
select targets and measures as well as selecting benchmarks
(i.e., choosing expected values from various evidence
bases); or it can be a more explicit feature, for example, by
contiguously displaying scores from multiple informants or
practices delivered by different members of the treatment
team, allowing a full view of team activities and perspectives. Dynamic configuration, such as being able to toggle
elements on and off, extends this capability, allowing different views for different users (e.g., sharing progress and
practice history with a family member). Bruns et al. (2015)
describe features built into the workflow of the TMSWrapLogic system that prompt the type of collaboration
that is central to the wraparound service model, and Lyon
et al. (2015) found in their contextual assessment that
communication, both internal and external, were key
functions of service providers.
An Illustration
Nathan is a 17 year old Asian American male receiving
treatment for depression. Figure 2 shows parent and selfreported depression scores on a depression scale over time
in days (plots a and b). In terms of the concepts above, this
panel represents the case-specific evidence base, using
observed values, in a single domain (progress), in a single
language (T-scores on a standardized measure).
In Fig. 3, we enrich the display in a number of ways.
First, the progress panel now displays two additional plots
(c and d), corresponding to expected values. Both can
therefore be thought of as scores representing goal states.
When selecting expected values, it helps to consider all
four evidence bases outlined in Table 1, keeping in mind
that one or more expected value could be derived from
each evidence base. For example, a case-specific expected
value might be a discharge score from a previous successful treatment for Nathan. In this example, plot c is
derived from the local aggregate evidence base and represents the average post-treatment score for youth in the
system who had received treatment for depression. Plot d is
taken from the general services research evidence base, and
represents pre-post scores from a randomized clinical trial
for depression that included youth of the same age and
ethnicity as Nathan. Of note is that simply adding expected
Adm Policy Ment Health (2016) 43:471–477
475
Progress and Practice Monitoring Tool
Case ID: NZ
Age (in years): 17.4
Treatment Target: Depression
Gender: Male
Ethnicity: Asian American
85
Progress Measures
80
Left Scale
a RCADS-Depression
75
b RCADS-P-Depression
70
65
60
Right Scale
55
50
180
160
140
120
100
80
60
40
20
0
45
Fig. 2 A progress panel with observed values for depression scores over time. RCADS revised child anxiety and depression scale, RCADS-P
revised child anxiety and depression scale-parent version
Progress and Practice Monitoring Tool
Case ID: NZ
Age (in years): 17.4
Treatment Target: Depression
Gender: Male
Ethnicity: Asian American
85
1
Progress Measures
80
Left Scale
a RCADS-Depression
1
75
b RCADS-P-Depression
1
70
c Benchmark
65
d Trend
1
60
Right Scale
0
55
0
50
45
Literature
Practices
Performed
180
160
140
120
100
80
60
40
20
0
0
Expert
******* FOCUS *******
Relationship/Rapport Building
Child Psychoed: Depression
Psychoed: Depression (CG)
Activity Selection
Cognitive: Depression
Problem Solving
Relaxation
Social Skills
Communication Skills (CG)
Support Networking (CG)
Goal Setting (CG)
Maintenance (CG)
******* INTERFERENCE *******
Child Psychoed: Anxiety
Cognitive: Anxiety
Exposure
Self-Rewards
Fig. 3 A progress and practice panel showing observed and expected
values over time. CG indicates a caregiver directed practice, RCADS
revised child anxiety and depression scale, RCADS-P revised child
anxiety
and
depression
scale-parent
version,
Psychoed
psychoeducation. The shaded region of the practice panel indicates
a hypothetical illustration of practices supported by relevant research
trials (i.e., expected values for practice)
123
476
values for treatment progress can have a dramatic effect on
interpretation of progress by creating context. That is, it is
quite possible to infer satisfactory progress when examining Fig. 1, but less possible when inspecting the top panel
of Fig. 2, given the discrepancies between observed and
expected values.
This discrepancy may effectively indicate when to act,
but without additional information, it may say less about
how to act. Given a basic logic model that practices affect
outcomes, inclusion of a practice panel can help in this
regard. The bottom of Fig. 2 uses white circles plotted on
the same time axis as the progress ratings to indicate that
Nathan has had 15 separate treatment sessions involving 9
different clinical procedures. This panel is one place to
explore to determine why he has lagged the expected rate
of progress. Once again, expected values can help, and
plotting practices coded from evidence-based treatments
(which is in essence, a translation exercise, as noted above)
indicates that the 12 practices in the ‘‘focus’’ region of the
practice panel are part of an evidence-based protocol for
depression in adolescents. Comparing these expected
practices with those observed, we find that three sessions
involved ‘‘off focus’’ practices, targeting anxiety (possible
errors of commission), and only one of five caregiver-directed practices has been delivered (communication skills),
occurring 150 days into treatment (possible errors of
omission). One other notable observation is the increasing
latency between sessions starting at about day 60.
A treatment team may thus hypothesize that caregiver
engagement may be an issue, and begin deeper inquiry,
which could include adding additional measures to the
progress panel (e.g., a caregiver assessment of barriers to
treatment or treatment expectancy) or otherwise enhance
caregiver services. The system remains interactive, collaborative, and exploratory. Once the system may have
indicated how to act, it should ideally support the treatment
team in taking the next steps, including referencing the
provider analogue of the case-specific history: does this
provider have experience and expertise with promoting
caregiver engagement? If not, are there written materials or
learning resources that could be launched from the display
to facilitate action?
Of course, this is but one example, restricted to two
domain panels, two of the evidence bases, often with only a
single expected value from each. The possibilities, however, can be as varied as conversations, consistent with our
telecommunication metaphor. When everything aligns
neatly, these conversations can be swift and clear, indicating next steps, but even when information does not align
(e.g., the research literature indicates one set of expected
values for practice, whereas an expert supervisor recommends a different set of practices), the communication and
collaboration is biased toward investigating and resolving
123
Adm Policy Ment Health (2016) 43:471–477
the discrepancy prospectively using evidence. More complex examples and illustrations are available elsewhere in
the literature, specifically regarding multiple provider
teams (e.g., Bruns et al. 2014); observed and expected
values and model integrity (Chorpita et al. 2008; Regan
et al. 2013), coordination of multiple evidence bases (e.g.
Daleiden et al. 2005), and complex collaboration structures
(Chorpita and Daleiden 2014).
Conclusion
This group of authors is to be commended for their efforts to
implement and evaluate MFS in a variety of real-world
contexts. The promises of such systems are clear, and the
design challenges, although considerable, can and will be
resolved. For technology to serve our will, however, our
field must continue to wrestle with models for how we wish
to select and organize health information. Thus, we disagree
with the notion that this burden lies with HIT developers.
There is, for the moment, a significant underspecification of
the general information and decision models needed to
dictate the functional requirements of promising new technologies. Technology will do what we tell it to, and thus, the
burden, for now, lies primarily with mental health experts.
That said, if we do our jobs right, the complexity of
these fully articulated models may soon become a constraint. Our theories of psychopathology have moved to
multifactorial risk and protective factor models, and our
intervention research has identified a multitude of interventions that, although effective when measured at the
group level, involve uncertainty at the case level. To help
manage that uncertainty, the current technologies struggle
tremendously and with only modest success at displaying a
‘‘human readable’’ form of a ‘‘progress only’’ model, much
less a basic practice-yields-progress logic model. Although
we encourage practice-progress type models for current
applications to help those using the technology of today,
we await the truly disruptive technology needed to support
the highly elaborated models necessary to help humans
contextually detect and respond to abstract phenomena
(e.g., behavior, interactions) in the service of their personal
pursuits and in their duty to help others.
References
Bickman, L. (2008). A measurement feedback system (MFS) is
necessary to improve mental health outcomes. Journal of the
American Academy of Child and Adolescent Psychiatry, 47,
1114–1119.
Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M.,
Gleacher, A., Olin, S., et al. (2014). Implementing a
Adm Policy Ment Health (2016) 43:471–477
measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services
Research, 1–16. doi:10.1007/s10488-015-0647-8.
Bruns, E. J., Hyde, K. L., Sather, A., Hook, A. N., & Lyon, A. R.
(2015). Applying user input to the design and testing of an
electronic behavioral health information system for wraparound
care coordination. Administration and Policy in Mental Health
and Mental Health Services Research, 1–19. doi:10.1007/
s10488-015-0658-5.
Bruns, E. J., Walker, J. S., Bernstein, A. D., Daleiden, E. L.,
Pullmann, M. D., & Chorpita, B. F. (2014). Family voice with
informed choice: Coordinating wraparound with research-based
treatment for children and adolescents. Journal of Clinical Child
and Adolescent Psychology, 43, 256–269.
Chorpita, B. F., Bernstein, A., Daleiden, E. L., & Research Network
on Youth Mental Health. (2008). Driving with roadmaps and
dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy
in Mental Health and Mental Health Services Research, 35(1–2),
114–123.
Chorpita, B. F., & Daleiden, E. L. (2014). Structuring the collaboration of science and service in pursuit of a shared vision.
Journal of Clinical Child and Adolescent Psychology, 43,
323–338.
Chorpita, B. F., Daleiden, E. L., Ebesutani, C., Young, J., Becker, K.
D., Nakamura, B. J., et al. (2011). Evidence-based treatments for
children and adolescents: An updated review of indicators of
efficacy and effectiveness. Clinical Psychology: Science and
Practice, 18, 153–171.
Daleiden, E., & Chorpita, B. F. (2005). From data to wisdom: Quality
improvement strategies supporting large-scale implementation of
evidence-based services. Child and Adolescent Psychiatric
Clinics of North America, 14, 329–349.
Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V.,
Bickman, L., et al. (2015). Implementing a measurement
feedback system in community mental health clinics: A case
study of multilevel barriers and facilitators. Administration and
Policy in Mental Health and Mental Health Services Research,
1–15. doi:10.1007/s10488-015-0642-0.
477
Higa-McMillan, C. K., Powell, C., Daleiden, E., & Mueller, C. W.
(2011). Pursuing an evidence-based culture through contextualized
feedback: Aligning youth outcomes and practices. Professional
Psychology: Research and Practice,. doi:10.1037/a0022139.
Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E.
J. (2005). Providing feedback to psychotherapists on their
patients’ progress: Clinical results and practice suggestions.
Journal of Clinical Psychology, 61, 165–174. doi:10.1002/jclp.
20113.
Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J.,
Unützer, J., et al. (2015). The Contextualized Technology
Adaptation Process (CTAP): Optimizing health information
technology to improve mental health systems. Administration
and Policy in Mental Health and Mental Health Services
Research, 1–16. doi:10.1007/s10488-015-0637-x.
Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G.
(2015). Development and piloting of a classroom-focused
measurement feedback system. Administration and Policy in
Mental Health and Mental Health Services Research, 1–15.
doi:10.1007/s10488-015-0651-z.
Palinkas, L. A., Weisz, J. R., Chorpita, B. F., Levine, B., Garland, A.,
Hoagwood, K. E., & Landsverk, J. (2013). Continued use of
evidence-based treatments after a randomized controlled effectiveness trial: A qualitative study. Psychiatric Services, 64,
1110–1118.
Regan, J., Daleiden, E. L., & Chorpita, B. F. (2013). Integrity in
mental health systems: An expanded framework for managing
uncertainty in clinical care. Clinical Psychology: Science and
Practice, 20, 78–98.
Southam-Gerow, M. A., Daleiden, E. L., Chorpita, B. F., Bae, C.,
Mitchell, C., Faye, M., & Alba, M. (2014). MAPping Los
Angeles County: Taking an evidence-informed model of mental
health care to scale. Journal of Clinical Child and Adolescent
Psychology, 43, 190–200.
Steinfeld, B., Fraynt, R., & Simon, G. (2015). Progress monitoring in
an integrated health care system: Tracking behavioral health
vital signs. Administration and Policy in Mental Health and
Mental Health Services Research, 1–10. doi:10.1007/s10488015-0648-7.
123
Reproduced with permission of the copyright owner. Further reproduction prohibited without
permission.
Aller et al. BMC Health Services Research (2015) 15:323
DOI 10.1186/s12913-015-0968-z
RESEARCH ARTICLE
Open Access
Development and testing of indicators to
measure coordination of clinical information
and management across levels of care
Marta-Beatriz Aller1*, Ingrid Vargas1, Jordi Coderch2, Sebastià Calero3, Francesc Cots4, Mercè Abizanda5, Joan Farré6,
Josep Ramon Llopart7, Lluís Colomés8 and María Luisa Vázquez1
Abstract
Background: Coordination across levels of care is becoming increasingly important due to rapid advances in
technology, high specialisation and changes in the organization of healthcare services; to date, however, the
development of indicators to evaluate coordination has been limited. The aim of this study is to develop and test a
set of indicators to comprehensively evaluate clinical coordination across levels of care.
Methods: A systematic review of literature was conducted to identify indicators of clinical coordination across levels of
care. These indicators were analysed to identify attributes of coordination and classified accordingly. They were then
discussed within an expert team and adapted or newly developed, and their relevance, scientific soundness and
feasibility were examined. The indicators were tested in three healthcare areas of the Catalan health system.
Results: 52 indicators were identified addressing 11 attributes of clinical coordination across levels of care. The final set
consisted of 21 output indicators. Clinical information transfer is evaluated based on information flow (4) and the
adequacy of shared information (3). Clinical management coordination indicators evaluate care coherence through
diagnostic testing (2) and medication (1), provision of care at the most appropriate level (2), completion of diagnostic
process (1), follow-up after hospital discharge (4) and accessibility across levels of care (4). The application of indicators
showed differences in the degree of clinical coordination depending on the attribute and area.
Conclusion: A set of rigorous and scientifically sound measures of clinical coordination across levels of care were
developed based on a literature review and discussion with experts. This set of indicators comprehensively address the
different attributes of clinical coordination in main transitions across levels of care. It could be employed to identify
areas in which health services can be improved, as well as to measure the effect of efforts to improve clinical
coordination in healthcare organizations.
Keywords: Quality indicators, Coordination across levels of care, Clinical management coordination, Clinical
information coordination, Health services research
Background
Healthcare systems are in a constant process of adaptation
due to rapid advances in technology, new treatments, high
specialisation and changes in the organization of health services [1]. As a consequence, patients are seen by an everexpanding array of different providers in a variety of
* Correspondence: maller@consorci.org
1
Health Policy and Health Services Research Group, Health Policy Research
Unit, Consortium for Health Care and Social Services of Catalonia, Avenida
Tibidabo, 21, 08022 Barcelona, Spain
Full list of author information is available at the end of the article
locations, making coordination difficult [1, 2]. This is particularly challenging in the care of patients with chronic and
multiple conditions, who tend to use healthcare services
more frequently and use a greater array of services than
other patients [3, 4]. Clinical coordination across levels of
care should prevent wasteful duplication of diagnostic testing, perilous polypharmacy and conflicting care plans [5, 6];
thus the effects of clinical coordination extend beyond cost
reduction through improving quality of care [7–9].
This study is set within a conceptual framework for
analysing the performance of integrated healthcare
© 2015 Aller et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a
link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain
Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this
article, unless otherwise stated.
Aller et al. BMC Health Services Research (2015) 15:323
networks, which is based on an extensive literature review [6, 10] and could be applied in any healthcare area
that arranges to provide a coordinated continuum of services to a defined population. In this framework, clinical
coordination, together with continuity of care and access
to health services, is considered an intermediate objective of integrated healthcare networks and is regarded as
a means by which to reach the ultimate objectives of
quality of care, efficiency and equity of access [6, 10, 11].
To analyse the achievement of these objectives, both external and internal processes and contextual factors are
taken into account, as well as the different perspectives
(services, professionals and users) and approaches.
In this conceptual framework, clinical coordination is defined as the harmonious connection of the different health
services needed to provide care to a patient throughout the
care continuum in order to achieve a common objective
without conflicts [10, 12]. Continuity of care refers to how
individual patients experience coordination of services, and
it is defined as the degree to which patients experience care
over time as coherent and linked [1]. Clinical coordination
across levels of care consists of the coordination of both
clinical information and clinical management [6, 10]. Clinical information coordination is the transfer and use of patients’ clinical information in order to harmonize activities
between providers, and consists of two dimensions: transfer
of clinical information and the use of this information [13].
Clinical management coordination is the provision of care
in a sequential and complementary way according to a
healthcare plan shared by the different services and healthcare levels involved, and consists of three dimensions: care
coherence (i.e., the existence of similar approaches and
treatment objectives among professionals from different
levels of care), follow-up across care levels (i.e., the adequate monitoring of the patient when there are transitions
from one care setting to another) and accessibility across
levels (provision of care without interruption across levels
of care throughout the clinical episode of the patient) [13].
The results of clinical coordination can be assessed by
analysing processes aimed at coordination or their outputs (immediate results of activities related to clinical
coordination) or outcomes (final expected middle-long
term results of clinical coordination, such as hospital readmissions or avoidable hospital admissions), and using different perspectives (services, professionals, users (continuity)).
The focus of this study relies on measures to assess the outputs of clinical coordination across levels of care (primary
and secondary) by using service-based indicators.
Despite the interest this subject has generated, there
are still important gaps in terms of measures to assess
clinical coordination across levels of care and the development of new indicators continues to be considered a
priority in health policy and health services research [14,
15]. Many of the attempts to address this to date have
Page 2 of 16
focused on developing indicators to measure healthcare
outcomes which are attributed to improvements in clinical coordination [16]. However, the development of output indicators has been limited, and without this type of
indicators it is not possible to conclude that outcomes in
health care can be attributed to improvements in clinical
coordination across levels of care [15].
Existing sets of indicators are usually designed to analyse a single dimension (e.g. transfer of information) or
attribute (e.g. due completion of referral forms and discharge reports) of clinical coordination [17–20]. Those
which address more than one dimension of clinical coordination are not exhaustive in their approach to clinical coordination and are often insufficiently operative or
are not directed at the assessment of clinical coordination across levels of care [21–25]. Furthermore, the
conceptual framework used to develop these measures is
not generally explained in detail, so it is not obvious
exactly which aspects of clinical coordination are being
analysed or how measures relate to clinical coordination.
As a result of these issues, there is an overrepresentation
of some dimensions of clinical coordination addressed by
indicators, whilst other dimensions have scarcely been investigated [26]. Studies have concentrated in particular on
the transfer of clinical information [22–24, 27–30], especially in terms of completeness of information in discharge
reports [22, 30–34] and to a lesser degree in emergency
reports [30] and referral forms [20, 35], and on the followup of patients and accessibility across care levels [22, 24, 29,
30, 36]. Only a few studies have used indicators to measure
clinical coherence between care levels [30, 37].
The aim of this study is to develop and test a set of output indicators to comprehensively evaluate clinical coordination across care levels of care, i.e. addressing both types
of clinical coordination, information and management,
and their dimensions and attributes.
Methods
The study consisted of two phases: in the first phase, a
set of indicators to measure clinical coordination across
levels of care was developed based on the literature review and expert discussions, and in the second phase,
the set was tested in three different healthcare areas.
1. Development of a set of indicators to measure clinical
coordination across levels of care
Identification of indicators: literature review
The study was based on the conceptual framework for
analysing the performance of integrated healthcare networks [6, 10], which identifies two types of clinical coordination across levels of care (clinical information and
clinical management) and five dimensions (transfer of information, use of information, care coherence, follow-up
across levels and accessibility across levels). A systematic
Aller et al. BMC Health Services Research (2015) 15:323
review of literature was undertaken to identify previously
developed indicators. A computerised search of the following bibliographic databases was conducted: Pubmed,
Social Science Citation Index, Science Citation Index,
ECONLIT, CINAHL and LILACS, in addition to standard
internet search engines such as Google. The search strategy included a combination of descriptors and keywords
relating to clinical coordination (‘coordination of care’ or
associated key terms with similar meaning), levels of care
(‘primary care’, ‘secondary care’, ‘hospitalization’, ‘interface’,
‘cross-level’ or associated terms) and measurement tools
(‘measure’, ‘indicator’ or associated key terms), making use
of the Boolean operator ‘AND’. References from retrieved
studies were also screened for possible omissions. The
search was conducted in May 2011. Additional searches
were conducted on the following organizations’ websites:
Agency for Healthcare Research and Quality (AHRQ),
World Health Organization (WHO), Pan American
Health Organization (PAHO), Physician Consortium for
Performance Improvement (PCPI), The Joint Commission, Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS), Observatori de Tendències de Serveis de
Salut, the RAND corporation and the National Quality
Forum (NQF). Studies in English, Spanish, Portuguese or
Catalan which included one or more indicators of clinical
coordination across levels of care were selected.
Selection and adaptation of indicators according to the
different types and dimensions of clinical coordination
First, the indicators included in the selected studies were
analyzed to identify which attributes of clinical coordination they addressed [10]. They were subsequently
grouped according to type, dimension and the attribute
Page 3 of 16
of clinical coordination across levels of care that they addressed (Fig. 1).
Second, three meetings took place with a team of 13
experts, who were either healthcare researchers with experience in the development of performance indicators
and clinical coordination assessment or managers of
healthcare services. Decisions concerning the selection
and adaptation of indicators were made taking the scientific literature into account and when a consensus was
reached among all participating members over three sessions of roundtable discussions. During the first meeting,
it became obvious that in order to be applied, most of
the indicators could not be generic but rather needed to
be defined in relation to a specific disease. However, in
order to gain a good grasp of the degree of coordination,
a number of different diseases were included which require high levels of coordination across levels of care:
diabetes mellitus type II, heart failure, chronic obstructive pulmonary disease (COPD) and breast, lung, bladder
and colon cancer.
During the meetings, indicators were discussed and
adapted based on the existing local clinical practice guidelines, which formed the basis on which standards of clinical coordination across levels of care were established
[38–41] (for example, the guidelines allowed the team to
determine when an urgent referral to secondary care is appropriate or to define the maximum acceptable time from
discharge to a consultation in primary care). Each indicator was described in terms of numerator, denominator,
target population, exclusion criteria, definition of terms
involved, sources of data and bibliography [42].
For each indicator, the team discussed its relevance to
clinical coordination across levels of care and its capacity
Fig. 1 Attributes of clinical coordination across levels identified, according to the type and dimension of clinical coordination
Aller et al. BMC Health Services Research (2015) 15:323
to measure that for which it was designed (face validity).
The team also discussed whether the indicator measured
an aspect of care that was susceptible to being improved
by services (opportunity for improvement), as well as
the formulation of the indicator in such precise terms
that it could be applied consistently within and between
organizations, allowing for comparability (reliability).
The experts identified the best sources of data to calculate each indicator (electronic medical record audit and
clinical and administrative databases) and discussed its
feasibility in terms of data availability and accuracy.
2. Test of the set of indicators
Design and settings
A retrospective cross-sectional study was conducted applying the set of indicators in three healthcare areas of
the Catalan public healthcare system. The objectives
were to evaluate the feasibility of the indicators (availability of valid, reliable and consistent data across the
system) and to apply the indicators in three different
healthcare areas in order to assess their usefulness in describing clinical coordination across levels of care.
Three healthcare areas were selected in order to represent the diversity of providers present in Catalonia: Baix
Empordà (rural and semi-urban), the city of Girona
(urban) and the Ciutat Vella district of Barcelona (urban),
which all serve a population of 75,000-100,000. A single
entity manages both primary and secondary care in Baix
Empordà (Serveis de Salut Integrats Baix Empordà —
SSIBE) and in Girona (Institut Català de la Salut — ICS).
In Ciutat Vella, two public entities manage primary care
(ICS and Institut de Prestacions d’Assistència Mèdica al
Personal Municipal — PAMEM) and a different public entity manages secondary care (Parc Salut Mar). With regard
to the coordination mechanisms used in these areas, patients
served in Baix Empordà had a single electronic medical record for both care levels, whereas patients served in the other
two areas had two shared but different electronic medical records for primary and secondary care. Several additional
mechanisms have been implemented to improve clinical
coordination across levels of care within the organizations,
such as shared clinical guidelines, online consultations between primary care physicians and specialists, automated notification of primary care following hospital discharge and
clinical case discussions between the two care levels.
Page 4 of 16
Two sources of data were used: a) electronic medical record audit, to calculate seven indicators (five related to clinical information coordination across levels of care and two
related to clinical management coordination across levels
of care); b) clinical and administrative electronic databases
(which differ from patients’ individual electronic medical
records in the fact that they collate patient data), to calculate twelve indicators (all related to clinical management
coordination).
For indicators based on electronic medical record audits, the sample size was calculated to estimate proportions, which were expected to be around 0.50; the margin
of error was ±0.15 and alpha error 0.05. The sample size
required was 42 patients. A simple random sample without replacement was selected from records provided by
primary care centres and hospitals. For indicators based
on electronic databases, all records were selected.
Data collection
Instructions for the data collection procedure were developed and systematically applied. Problems during data
collection and analysis were recorded.
For indicators based on electronic medical record audits, data was retrieved by one researcher using standardized forms. For indicators based on databases, primary
care centres and hospitals of the healthcare areas provided
clinical and administrative electronic databases. Information was retrieved based on specified procedures, which
had to be adapted to each information system.
Data analysis
In the case of dichotomous indicators, percentages were
calculated, and 95 % confidence intervals were estimated
when indicators were based on electronic medical record
audits. Means and standard deviation were calculated
for continuous indicators. Problems during data collection and analysis were discussed with the group in order
to assess the applicability of the indicators and identify
the main barriers to their implementation.
Ethical considerations
The principles of confidentiality and anonymity were upheld in the researchers’ conduct, reporting, and storage of
data arising from this study, in accordance with European
and Spanish legislation on ethical research [43]. The study
protocol was approved by the Ethical Committee for Clinical Research ‘Parc Salut Mar (2010/4124/I)’.
Study population, data source and sample
The study population consisted of patients who had the
selected conditions and who had used more than one
care level, i.e. they were discharged from hospital, had
received outpatient secondary care, were referred to secondary care or were newly diagnosed in primary care,
depending on the indicator (Tables 1 and 2).
Results
1. Development of a set of indicators to measure clinical
coordination across levels of care
Identification of indicators: literature review
A total of 892 documents were identified: 863 from bibliographic databases, 11 from organizations’ websites and 18
Dimension attribute
Description
Formula
Source of data
Adapted from
IT1. Percentage of hospital discharges for which a - Numerator: Discharge report available in primary care within
discharge report is made available to primary care the first 24 h after hospital discharge
within the first 24 h
- Denominator: Hospital discharges
Discharge
reports in
EMRs
[22, 30, 34, 37, 49]
IT2. Mean time to discharge report availability in
primary care
Discharge
reports in
EMRs
[19]
IT3. Percentage of emergency care visits for which - Numerator: Emergency care report available in primary care
there is an emergency care report available in
within 24 h of the emergency care visit
primary care within 24 h
- Denominator: Emergency care discharges
Discharge
reports in
EMRs
[22, 30, 34, 37, 49]
IT4. Mean time to emergency care report
availability in primary care
Discharge
reports in
EMRs
[19]
EMR audit
[30]
EMR audit
[30]
EMR audit
[30]
Information transfer
Information flow across levels
- Numerator: Total hours elapsed from the time of hospital
discharge to report availability in primary care
- Denominator: Hospital discharges
- Numerator: Total hours elapsed from the emergency care
visit to report availability in primary care
- Denominator: Emergency care discharges
Referral forms and discharge reports
duly completed
IT5. Percentage of discharge reports duly
completed
Transfer of information on medication
and tests across levels
- Numerator: Hospital discharge reports which contain at
least four of the following items: reason for admission, additional
tests performed and pending, follow-up or monitoring for the
patient after discharge, list of current medications and
recommendations for the patient
Aller et al. BMC Health Services Research (2015) 15:323
Table 1 Indicators related to clinical information coordination across levels of care
- Denominator: Hospital discharge reports of patients discharged
with a diagnosis of COPD, DM and/or HF
IT6. Percentage of emergency care reports duly
completed
- Numerator: Emergency care reports which contain at least four
of the following items: the reason
for the emergency care visit, additional tests performed and
pending (laboratory, radiology, etc.), follow-up or monitoring of
the patient after the emergency care visit, list of current
medications and recommendations for the patient
- Denominator: Emergency care reports of patients with COPD,
DM and/or HF
IT7. Percentage of referral forms from primary
care duly completed
- Numerator: Patients diagnosed with HF, COPD and/or DM
that have been referred to secondary care
with a referral form that contains relevant background morbidity,
current medical treatment, and the reason for the referral
- Denominator: Patients diagnosed with HF, COPD and/or DM that
have been referred to secondary care
Page 5 of 16
Indicators are available at: http://www.consorci.org/coneixement/cataleg-de-publicacions/80/indicadores-de-coordinacion-asistencial-entre-niveles-documento-de-trabajo
COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, EMR electronic medical record
Aller et al. BMC Health Services Research (2015) 15:323
Page 6 of 16
Table 2 Indicators related to clinical management coordination across levels of care
Dimension attribute
Description
Formula
Source of data Adapted
from
CC1. Percentage of secondary care visits
of patients diagnosed with HF in which the
specialist ordered tests that were performed
in the previous six months in primary care
- Numerator: First secondary care visit of HF
Clinical and
patients referred from primary care in which
administrative
the specialist ordered a non-urgent, non-priority databases
X-ray of the thorax, ECG or general blood test
that was performed in the previous six months
in primary care
Care coherence
Coordinated medical
testing across
involved care levels
[50]
- Denominator: Total first non-urgent, nonpriority secondary care visits of patients referred
from primary care for HF
CC2. Percentage of pneumology visits of
patients diagnosed with COPD in which the
specialist ordered a spirometry that was
performed in the previous six months in
primary care
- Numerator: First non-urgent, non-priority
Clinical and
pneumology visit of COPD patients referred from administrative
primary care in which the specialist ordered a
databases
spirometry that was performed in the previous
six months in primary care
[50]
- Denominator: Total first non-urgent, nonpriority pneumology visits of patients referred
from primary care for COPD
Coordinated
management of
medication by
involved levels
CC3. Percentage of patients with DM who
started insulin therapy during hospitalization
and whose primary care medical record
documents a follow-up within one week of
discharge
- Numerator: Patients with DM who started
insulin therapy during hospitalization and
whose primary care medical record
documents a follow-up within one week of
discharge
Clinical and
administrative
databases
[36, 47]
EMR audit
[17]
EMR audit
[17]
Clinical and
administrative
databases
[47]
Clinical and
administrative
databases
[21, 24,
32, 37,
44, 60]
Clinical and
administrative
databases
[21, 24,
32, 37,
44, 60]
- Denominator: Patients with DM who started
insulin therapy during hospitalization
Care at the most
appropriate level
CC4. Percentage of patients with HF
correctly referred from primary care to
non-urgent outpatient secondary care
- Numerator: Patients diagnosed with HF and
correctly referred to cardiology or internal
medicine
- Denominator: Patients diagnosed with HF
that have been referred from primary care to
cardiology or internal medicine
CC5. Percentage of patients with HF that have - Numerator: Patients with exacerbation of HF
been correctly referred to emergency care
that have been correctly referred to
from primary care
emergency care from primary care
- Denominator: Patients that visit emergency
care for decompensated HF referred by
primary care
Completion of
diagnostic process
when more than one
level is involved
CC6. Percentage of patients with HF
diagnosed in the past year who had an
echocardiogram as part of the diagnostic
process
- Numerator: Patients diagnosed with HF who
had an echocardiogram as part of the
diagnostic process
FU1. Percentage of hospital discharges with
contact between the hospital and primary
care prior to the discharge of patients
hospitalized for severe exacerbation of COPD
- Numerator: Hospital discharges with
principal diagnosis related to the severe
exacerbation of COPD and in which the
hospital has contacted primary care prior to
the discharge
- Denominator: Total of patients diagnosed
with HF
Follow-up across levels
Communication
between involved
levels
- Denominator: Hospital discharges with
principal diagnosis related to severe
exacerbation of COPD
FU2. Percentage of hospital discharges with
contact between the hospital and primary
care prior to the discharge of patients
hospitalized for decompensated HF
- Numerator: Hospital discharges with
principal diagnosis related to decompensated
HF in which primary care has been contacted
prior to discharge
- Denominator: Hospital discharges with
principal diagnosis related to decompensated
HF
Aller et al. BMC Health Services Research (2015) 15:323
Page 7 of 16
Table 2 Indicators related to clinical management coordination across levels of care (Continued)
Follow-up visits after
hospital discharge
FU3. Percentage of hospital discharges of
patients admitted for exacerbation of COPD
who have a consultation in primary care in
less than 72 h
- Numerator: Hospital discharges with
Clinical and
principal diagnosis related to severe
administrative
exacerbation of COPD and with a consultation databases
in primary care in less than 72 h
[21, 24,
32, 37,
44, 60]
- Denominator: Hospital discharges with
principal diagnosis related to severe
exacerbation of COPD
FU4. Percentage of hospital discharges of
patients admitted for decompensated HF who
have a consultation in primary care in less
than 7 days
- Numerator: Hospital discharges with
principal diagnosis related to decompensated
HF and with a consultation in primary care in
less than 7 days
Clinical and
administrative
databases
[21, 24,
32, 37,
44, 60]
Clinical and
administrative
databases
[61, 62]
Clinical and
administrative
databases
[61, 62]
- Denominator: Patients discharged with
principal diagnosis related to decompensated
HF
Accessibility across levels
Waiting time after
referral
AAL1. Mean time elapsed from non-urgent,
non-priority primary care referral of HF
patients to cardiologist visit
- Numerator: Total days elapsed from nonurgent, non-priority, primary care referral of HF
patients to cardiologist visit
- Denominator: Total HF patients with nonurgent, non-priority referrals from primary care
to cardiology
AAL2. Mean time elapsed from the referral of
a patient with suspected cancer (lung,
colorectal, breast, bladder and prostate) to the
first specialist care visit
- Numerator: Total days elapsed from the
primary care referral of a patient with
suspected cancer to the first appointment
with rapid diagnosis program
- Denominator: Total patients referred from
primary care to specialist care for suspected
cancer (lung, colorectal, breast, bladder and
prostate)
AAL3. Mean time elapsed from the referral of
a patient with suspected cancer (lung,
colorectal, breast, bladder and prostate) to
time of cancer diagnosis
- Numerator: Total days elapsed from the
primary care referral of a patient with
suspected cancer to the diagnosis of cancer
Clinical and
administrative
databases
[61, 62]
AAL4. Mean time elapsed from the referral of
a patient with suspected cancer (lung,
colorectal, breast, bladder and prostate) to the
initiation of cancer treatment (surgery and/or
chemotherapy and/or radiotherapy)
Clinical and
- Numerator: Total days elapsed from the
referral from primary care of a patient with
administrative
suspected cancer to the initiation of cancer
databases
treatment (surgery and/or chemotherapy and/
or radiotherapy)
[61, 62]
- Denominator: Total patients with suspected
cancer (lung, colorectal, breast, bladder and
prostate) first identified in primary care and
with a later diagnosis of cancer
- Denominator: Total patients diagnosed with
cancer (lung, colorectal, breast, bladder and
prostate) referred to secondary care from
primary care who initiate treatment including
surgery, chemotherapy and/or radiotherapy at
the hospital to which they were referred from
primary care
Indicators are available
at: http://www.consorci.org/coneixement/cataleg-de-publicacions/80/indicadores-de-coordinacion-asistencial-entre-niveles-documento-de-trabajo
COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, EMR electronic medical record
Aller et al. BMC Health Services Research (2015) 15:323
from references in retrieved studies (Appendix 1). Of
these documents, 862 were excluded because they did not
describe nor use indicators of clinical coordination across
levels of care, and 30 met the inclusion criteria, containing
at least one indicator. From these documents, 52 indicators were initially identified [17, 19, 21–24, 27, 28, 30–34,
36, 37, 44–50].
Selection and adaptation of indicators according to the
different types and dimensions of clinical coordination
The 52 indicators addressed 11 different attributes of
clinical coordination across levels of care (Fig. 2): 3 related to clinical information coordination and 8 related to
clinical management coordination. The dimension “use of
information” was not addressed by any attribute or indicator. After two meetings, an initial set of 21 indicators was
drawn up (Fig. 2), which addressed 10 of the 11 identified
attributes, since it was not possible to establish an unambiguous criterion which would permit the identification of
Fig. 2 Stages in the development of the set of indicators
Page 8 of 16
redundant consultations. The remaining attributes were
represented by at least 1 indicator.
The final set of indicators was as follows (Table 1 and 2):
a) Clinical information coordination across levels of
care: 7 indicators measure the transfer of clinical
information across care levels, addressing the
availability of inpatient and emergency discharge
reports in primary care (four indicators) and the
completeness of inpatient and emergency
discharge reports and referral forms, including the
transfer of information on new medication,
medical tests, reasons for referral and information
for patients (three indicators). No indicator
addressed the dimension “use of transferred
information” since it was not possible to identify
or design any indicator measuring the effective use
by professionals of information generated in the
other care level.
Aller et al. BMC Health Services Research (2015) 15:323
b) Clinical management coordination across levels of
care: 6 indicators address care coherence by
measuring the coordinated management of medical
testing in primary and secondary care of patients
with heart failure and COPD (2 indicators), the
adequacy of the referral of heart failure patients
from primary care to non-urgent outpatient
secondary care or emergency care (2 indicators)
and the completion of the diagnostic process for
heart failure, which requires coordination between
the two care levels (1 indicator). Four indicators
measure the follow-up of patients, addressing the
communication between the hospital and the
primary care centre when patients with heart
failure and COPD are discharged (2 indicators) and
their follow-up in primary care after being
discharged (2 indicators). Finally, four indicators
measure accessibility across care levels, specifically the
time elapsed from the primary care referral of patients
with heart failure or suspected cancer to their first
specialist care appointment (2 indicators) and the time
elapsed from the suspicion of cancer in primary care
to cancer diagnosis or initiation of treatment
(2 indicators).
2. Test of the set of indicators
Clinical care information: transfer of clinical information
across levels of care
In the three healthcare areas, inpatient and emergency
discharge reports were immediately available in primary care, since the two care levels share electronic
medical records. In general, the quality of transferred
clinical information was high (i.e. the clinical information required for the transfer of patients between care
levels is duly registered; for example, in referral forms:
background morbidity, current medical treatment and
reason for referral), especially with respect to the completeness of inpatient and emergency discharge reports, although there are notable differences between
areas (57.1 % of discharge reports duly completed in
Baix Empordà as opposed to 95.2 % in Girona)
(Table 3). In contrast, there were low percentages of
duly completed referral reports in two of the healthcare areas (11.9 % and 26.2 % of reports).
Feasibility
All indicators were feasible in the three healthcare areas;
however, in some cases the specified sample size was not
reached due to an insufficient number of cases per year
or due to insufficient precision in the available data to
allow identification of the denominator – more than 100
records were reviewed without reaching the required
sample size.
Page 9 of 16
Clinical management coordination across levels of care
Care coherence Indicators showed different degrees of
test duplication (coordinated medical testing across levels
of care) depending on the type of medical test performed:
the highest level of duplication was observed in electrocardiograms for patients with heart failure (48 %) and the lowest was observed in spirometries for patients with COPD
(2.5 %) (Table 4). In terms of care at the most appropriate
care level, indicators showed high levels of adequate referral
to non-urgent and emergency care. Finally, in two healthcare areas there were low percentages of patients (13.9 %
and 22.7 %) who had had an echocardiogram performed in
the year prior to the diagnosis of heart failure (completion
of the diagnostic process).
Follow-up across care levels
In terms of communication, there were significant differences in the degree to which hospitals communicate with
primary care prior to the discharge of heart failure or
COPD patients (58 % and 3.2 % of COPD patient discharges in Baix Empordà and Ciutat Vella respectively).
Similarly, with regard to follow-up after hospital discharge,
there were marked differences between areas (follow-up
of patients with COPD in primary care ranged from 26 %
in Ciutat Vella to 76.7 % in Baix Empordà).
Accessibility across care levels
The average time waited to access secondary non-urgent
care for heart failure patients referred to cardiology was
higher than three weeks in all cases, with significant variations across areas. In contrast, the average time waited
to access urgent care for patients with suspected cancer
was lower than a week in all healthcare areas, with little
variation between areas.
Feasibility
Five indicators were feasible in the three healthcare
areas, six indicators were feasible in two healthcare
areas, two indicators were feasible in only one area and
lastly, one indicator was not feasible in any of the
healthcare areas. Difficulties in calculating indicators
were due to two types of problems. Firstly, problems related to the identification of the denominator: not possible to identify patients who had started insulin
therapy (3 areas), patients referred for the first time to
the secondary care level (2 areas), and patients referred
to secondary care for suspected cancer (1 area). Secondly, problems related to the availability and accuracy
of data needed to calculate the numerator: reason for
seeking emergency care not recorded (2 areas), echocardiograms conducted in secondary care not systematically registered (1 area).
Indicator
Baix Empordà
Girona
Ciutat Vella
ICS- Parc de Salut Mar
PAMEM- Parc de Salut Mar
IT1. Percentage of hospital discharges for which a
discharge report is made available to primary care
within the first 24 h
%
100 %
100 %
100 %
100 %
IT2. Mean time to discharge report availability in
primary care
hours
immediate
immediate
immediate
immediate
IT3. Percentage of emergency care visits for which
there is an emergency care report available in
primary care within 24 h
%
100 %
100 %
100 %
100 %
IT4. Mean time to emergency care report availability
in primary care
hours
immediate
immediate
immediate
immediate
IT5. Percentage of discharge reports duly completed
(at least four of the five selected items)
% (95 % IC) n
57.1 % (41.5-72.7) n: 42
95.2 % (88.5 -100) n: 42
83.3 % (65.6- 91.5) n: 42
83.3 % (65.6-91.5) n: 42
Reason for admission
% (95 % IC)
100 %
95.2 % (88.5 - 100)
97.6 % (92.7 - 100)
100 %
Additional tests performed and pending
% (95 % IC)
95.2 % (88.5 - 100)
97.6 % (92.7 - 100)
95.2 % (88.5 - 100)
88.1 % (74.7- 96.7)
Follow-up or monitoring of the patient
after discharge
% (95 % IC)
64.3 % (49.2 - 79.0)
97.6 % (92.7 - 100)
92.9 % (84.7 - 100)
88.1 % (74.7- 96.7)
List of current medications
% (95 % IC)
88.1 % (77.9 - 98.31)
92.9 % (84.7 - 100)
88.1 % (77.9 - 98.3)
83.3 % (65.6- 91.5)
Recommendations for the patient
IT6. Percentage of emergency care reports duly
completed (at least four of the five selected items)
Reason for admission
% (95 % IC)
0%
97.6 % (92.7 - 100)
26.2 % (12.3 - 40.1)
16.7 % (6.7-31.4)
% (95 % IC) n
85.4 % (74.1-96.7) n: 41
85.7 % (74.6-96.7) n: 42
86.7 % (73.8 - 100) n: 30
64.3 % (49.2-79.4) n: 42
% (95 % IC)
100 %
100 %
100 %
100 %
Additional tests performed and pending
% (95 % IC)
92.7 % (84.4 - 100)
100 %
90 % (78.6 - 100)
88.1 % (77.9 - 98.3)
Follow-up or monitoring of the patient
after discharge
% (95 % IC)
90.2 % (80.7 - 99.7)
97.6 % (92.7 - 100)
90% (78.6 - 100)
76.2 % (62.8 - 89.6)
List of current medications
% (95 % IC)
97.6 % (92.6 - 100)
88.1 % (74.7- 96.7)
90 % (78.6 - 100)
85.71 % (74.6 - 96.7)
Recommendations for the patient
% (95 % IC)
19.51 % (6.8 - 32.2)
50 % (34.2 - 65.8)
43.3 % (25.5 - 62.2)
33.3 % (18.5 - 48.2)
% (95 % IC) n
26.2 % (12.3- 40.1) n:42
71.4 % (57.2-85.7) n: 42
11.9 % (1.7 -22.1) n: 42
88.5 % (75.3 - 100) n:26
IT7. Percentage of referral forms from primary care
duly completed
% (95 % IC)
90.5 % (81.2 - 99.7)
95.2 % (88.5 - 100)
86.7 % (73.8 - 100)
100 %
Current medical treatment
% (95 % IC)
30.9 % (16.4 - 45.5)
90.8 % (81.2 - 99.7)
16.7 % (4.9 - 28.4)
96.1 % (88.0 - 100)
Reason for the referral
% (95 % IC)
90.7 % (81.2 - 99.7)
57.1 % (41.5 - 72.7)
76.2 % (62.7 - 89.6)
88.5 % (75.3 - 100)
CI confidence interval, COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, SD standard deviation
Page 10 of 16
Background morbidity
Aller et al. BMC Health Services Research (2015) 15:323
Table 3 Application of the set of indicators related to clinical information coordination across levels of care
Aller et al. BMC Health Services Research (2015) 15:323
Page 11 of 16
Table 4 Application of the set of indicators related to clinical management coordination across levels of care
Indicator
Baix
Empordà
Girona
%; n
3.6 %;
n:56
%; n
Ciutat Vella
ICS- Parc de
Salut Mar
ICS- Parc de
Salut Mar
-
-
-
48.2 %;
n:56
-
-
-
%; n
16.1 %;
n:56
-
-
-
CC2. Percentage of pneumology visits of patients
diagnosed with COPD in which the specialist ordered
a spirometry that was performed in the previous six
months in primary care
%; n
2.5 %;
n:81
-
-
-
CC3. Percentage of patients with DM who started insulin
therapy during hospitalization and whose primary care medical
record documents a follow-up within one week of discharge
%; n
-
-
-
-
CC4. Percentage of patients with HF correctly referred from
primary care to non-urgent outpatient secondary care
%
85.7 %
81.0 %
83.3 %
(95 % IC) n (69.4-100) (68.9-93.3) (75.6-95.1)
n:42
n:42
n:42
88.5 %
(75.3-100)
n:26
CC5. Percentage of patients with HF that have been correctly
referred to emergency care from primary care
%
97.4 %
(95 % IC) n (92.0-100)
n:39
-
95.2 %
(88.5-100)
n:42
CC6. Percentage of patients with HF diagnosed in the past
year who had an echocardiogram as part of the diagnostic
process
%; n
22.7 %;
n:203
13.9 %:
n:216
-
-
FU1. Percentage of hospital discharges with contact between
the hospital and primary care prior to the discharge of patients
hospitalized for severe exacerbation of COPD
%; n
58 %;
n:88
36.8 %;
n:407
3.2 %; n:95
0 %; n:49
FU2. Percentage of hospital discharges with contact between
the hospital and primary care prior to the discharge of patients
hospitalized for decompensated HF
%; n
40.3 %;
n:119
44.78 %;
n:201
0 %; n:48
2.78 %; n:36
FU3. Percentage of hospital discharges of patients admitted for
exacerbation of COPD who have a consultation in primary care
in less than 72 h
%; n
76.7 %;
n:86
52.9 %;
n:240
26.0 %; n:68
53.3 %; n:45
FU4. Percentage of hospital discharges of patients admitted for
decompensated HF who have a consultation in primary care in
less than 7 days
%; n
70.9 %;
n:110
79.6 %;
n:157
55.3 %; n:38
70.6 %; n:19
AAL1. Mean time elapsed from non-urgent, non-priority primary care
referral of HF patients to cardiologist visit
Mean (SD); 28.2 (4.0);
n
n:42
-
39.6 (5.6); n:57
100.9 (9.1); n:86
AAL2. Mean time elapsed from the referral of a patient with
suspected cancer (lung, colorectal, breast, bladder and prostate)
to the first specialist visit
Mean (SD); 5.3 (0.3);
n
n:362
-
6.5 (0.4); n:87
6.6 (1.0); n:17
AAL3. Mean time elapsed from the referral of a patient with
suspected cancer (lung, colorectal, breast, bladder and prostate)
to cancer diagnosis
Mean (SD); 46.9 (9.7);
n
n:70
-
31.4 (4.6); n:36
39.9 (8.5); n:8
AAL4. Mean time elapsed from the referral of a patient with
suspected cancer (lung, colorectal, breast, bladder and prostate)
to the initiation of cancer treatment (surgery and/or chemotherapy
and/or radiotherapy)
Mean (SD); 71.4 (9.2);
n
n:64
-
48.1 (5.1); n:33
46.9 (5.8); n:8
CC1. Percentage of secondary care visits of patients Duplication of
diagnosed with HF in which the specialist
radiographies
ordered tests that were performed in the
Duplication of
previous six months in primary care
electrocardiograms
Duplication of analytics
CI confidence interval, COPD chronic obstructive pulmonary disease, DM diabetes mellitus, HF heart failure, SD standard deviation
Discussion
Clinical coordination is considered a health policy priority, as a lack of coordination can lead to poor quality of
care and inefficiencies in the use of resources [5–9].
However, its measurement is still challenging [1, 14, 15],
since calculating the degree of clinical coordination in
its multidimensional nature requires the availability of
indicators that cover the different types and dimensions
of clinical coordination.
Until now, most attempts to tackle this challenge have
focused on the design of indicators to measure certain
outcomes which can potentially be attributed to clinical
Aller et al. BMC Health Services Research (2015) 15:323
coordination [16]. However, progress must be made in the
design of instruments to measure the outputs of clinical
coordination in order to be able to attribute improvements in the outcomes of health care to improvements in
clinical coordination [51]. With this in mind, this research
constitutes a step forward by using a pre-established conceptual framework to generate a set of output indicators
which address the two types of clinical coordination
across levels of care (and most of their dimensions and
attributes) that have been highlighted in previous studies [52]. Furthermore, in contrast with previous efforts
[17, 37], the indicators presented here have been described in operative terms, thus allowing for their precise application in healthcare organizations.
With regard to clinical information coordination across
levels of care, seven indicators addressing the transfer of
clinical information have been created. Applying these indicators has permitted the analysis of transfer of information in the three healthcare settings, taking in both
evidence of information transfer between levels and the
quality of the information transferred. No previous set of
indicators has allowed researchers to address these two attributes jointly in main transitions between levels of care
[1, 30, 34, 53], so this is one of its most significant contributions. In addition, the result of the applicability test has
proven that these indicators have the accuracy and feasibility needed to make their calculation possible in different
healthcare areas. Their joint application has revealed that,
although there is a flow of information between the different care levels, the quality of information varies across transitions and organizations, thus leading to the identification
of specific margins of improvement in each healthcare area.
It is important to highlight, however, that clinical information coordination is not fully represented by the
set of indicators, since we were unable to address the
use of transferred information; i.e. we could not determine whether information was actually read and used by
the receiving professional [54]. The lack of this type of
measure of clinical coordination has been previously
expressed in the literature [1, 54] and reflects the complexity of analyzing an activity which is not generally recorded but is considered central to clinical coordination.
The fact that indicators are unable to systematically address all dimensions and attributes of clinical coordination
points to the need to complement and enrich indicator results with those that can be obtained via different techniques, such as surveys or qualitative interviews with
health professionals and patients.
With regard to clinical management coordination
across levels of care, the systematic review led us to
identify five attributes that define care coherence, two
that define follow-up and one that defines accessibility
across care levels. Their operationalization has resulted
in a set of indicators to measure the main attributes of
Page 12 of 16
care coherence (such as coordinated medical testing
across care levels or the provision of care at the most
appropriate care level), follow-up (such as the existence
of communication and follow-up after discharge) and accessibility across care levels (waiting time after referral).
However, one of the eight identified attributes of care
coherence, no redundant visits to primary and secondary
care, is not represented by any indicator, since we were unable to establish an unambiguous criterion, either though
the literature review or by expert consensus, which would
permit the identification of redundant consultations.
During the first meeting, it became obvious that in
order to be applied, most indicators could not be general
but needed to be defined relating to a specific disease.
However, in order to gain a good grasp of the degree of
coordination in the area, a number of different diseases
were included, which require high levels of coordination
across levels of care: diabetes mellitus type II, heart failure, chronic obstructive pulmonary disease (COPD) and
breast, lung, bladder and colon cancer.
The indicators to measure clinical management coordination have been adapted to several clinical conditions, due to the fact that the standards of clinical
coordination upon which indicators are based need to
be precise and based on what the evidence dictates,
which varies according to the disease. Nevertheless,
they can be adapted to other conditions as long as they
have an evidence-based recommendation upon which
to base the standard of clinical coordination measured
by the indicator. Moreover, the use of the selected conditions (diabetes mellitus, chronic obstructive pulmonary
disease, heart failure and cancer) could be considered
a good strategy to identify the strengths and weakness
in clinical coordination across levels of care [3, 55],
since they meet the criteria to be considered adequate
tracer conditions [55]: care is provided across levels
and over the course of time; the care that should be
provided at each care level is well defined; they are
among the most prevalent diseases in the population;
diagnoses are well defined; and their epidemiology is
well known.
The applicability test illustrates the usefulness of
these indicators in describing clinical management
coordination, pointing to areas for improvement, such
as the coordination of medical testing in Baix
Empordà and communication with primary care after
discharge in Ciutat Vella. Furthermore, as they cover
the main attributes of clinical management coordination across levels, they can be used to support the
design of strategies to improve clinical coordination
between levels of care.
It is important to note, however, that several problems
arose which made the calculation of some of these indicators difficult or impossible in some healthcare areas.
Aller et al. BMC Health Services Research (2015) 15:323
The problems were related to non-registration of the
variables in information systems and under-registration
of information by professionals, which points to the
need for further improvements in information systems
and record-keeping skills before we can systematically
measure certain relevant aspects related to clinical management coordination across levels of care in these healthcare areas [56–59].
The methodology adopted in this study provides
guarantees in terms of reliability of the indicators,
since they have been adequately defined and precisely
specified so that they can be implemented consistently within and across organizations (48). This is
also true in terms of face and content validity, as the
indicators have been adapted or newly created on the
basis of scientific evidence and expert consensus. Furthermore, the applicability test provided information
regarding data availability and accuracy (feasibility),
thus highlighting major and minor problems in calculating the indicators, which could be informative for
future studies. Finally, the indicators have been shown
to be able to identify differences between areas, even
in small samples. Further research should provide evidence regarding other relevant characteristics of the
indicators, such as test-retest reliability and discriminant validity.
Certain aspects should be taken into account when applying the set of indicators in other healthcare contexts.
First of all, data availability and validity should be explored. Secondly, it should be determined whether the
information taken from the different health information
systems is linkable, since indicators are constructed
upon information generated in different levels of care.
Lastly, although information recorded in digital format
is desirable, most indicators could be calculated from a
medical record audit, so data computerization is not a
prerequisite.
One limitation of this study is the possibility of a publication bias in which relevant indicators were not identified (for example, indicators that measure clinical
coordination but employ different terms, indicators published in other languages or grey literature not easily
accessed by standard internet search engines). Moreover,
the inclusion of terms referring to certain attributes of
clinical coordination, such as “follow-up” or “referral
adequacy”, might have extended the range of studies
obtained. However, we employed several additional
strategies for the identification of studies to reduce the
possibility of publication bias, such as reviewing the
reference lists of eligible documents and consulting the
websites of the main organizations that design indicators. Another limitation is that one dimension (the use
of clinical information) and one attribute (no redundant visits to primary and secondary care) are not
Page 13 of 16
represented by the set of indicators, pointing to the
need to enrich the results obtained by the indicators
with additional information from health professionals
and patients in order to attain a more accurate evaluation of the process of coordination.
Conclusions
A set of rigorous and scientifically sound measures of
clinical coordination were developed based on a literature review and discussion with experts. These indicators
of clinical information and management coordination
across levels of care could be employed to identify
areas in which health care can be improved, as well as
to measure the effect of efforts to improve clinical coordination. However, some relevant attributes of clinical coordination are not represented in the final set of
indicators, which detracts from its comprehensiveness.
In fact, clinical coordination is a multidisciplinary construct, and certain relevant dimensions and attributes
of clinical coordination across levels of care such as
the effective use of transferred information or redundant visits cannot be properly measured through indicators. Other approaches are therefore needed to
obtain additional information, such as surveys or
qualitative interviews. The indicators provided may
also be useful for conducting comparative studies of
clinical coordination across healthcare areas. Aspects
such as the possibility of linking information from different health information systems, data availability and
validity should be explored before proceeding to implement these indicators.
Appendix 1
Search Strategy and number of studies retrieved in the
bibliographic databases
Table 5 A1: medline-pubmed; 19/05/2011
References
number
1. Clinical Coordination
"Coordinated care" OR " coordination of care" OR
5.573
"integrated care" OR "shared care" OR "transitional care" OR
"continuity of care"
2. Levels of care
"Primary care" OR "family practice" OR "Generalist" OR "GP"
OR "outpatient" OR "secondary care" OR "specialized" OR
‘specialist’ OR "inpatient" OR "hospitalization" OR
"hospitalisation" OR "care levels" OR "interface" OR "crosslevel" OR "referral" OR "communication"
655.466
3. Measurement tools
measure OR measures OR indicator
939.670
4. (1) and (2) and (3)
466
Aller et al. BMC Health Services Research (2015) 15:323
Page 14 of 16
Table 6 A2: Isi web of knowledge; social science citation index
& science citation index 19/05/2011
References
number
1. Clinical coordination
"Coordinated care" OR "care coordination" OR
"collaborative care" OR "integrated care" OR "shared
care" OR "transitional care" OR "continuity of care" OR
"care continuity" OR "informational continuity" OR
"managerial continuity" OR "management continuity"
5.509
References
number
1. Clinical coordination
AB (Coordinated care) OR (care coordination)
OR (collaborative care) OR (integrated care)
OR (shared care) OR (transitional care) OR
(continuity of care) OR (care continuity) OR
(informational continuity) OR (managerial continuity)
OR (management continuity)
5.092
2. Levels of care
2. Levels of care
"Primary care" OR "family practice" OR "Generalist" OR
"GP" OR "outpatient" OR "secondary care" OR
"specialized" OR "specialised" OR ‘specialist’ OR
"inpatient" OR "hospitalization" OR "hospitalisation" OR
"care levels" OR "interface" OR "cross-level" OR "referral"
OR "communication"
Table 8 A4: CINAHL; 19/05/2011
>100.000
AB (Primary care) OR (family practice) OR (Generalist)
OR (GP) OR (outpatient) OR (secondary care) OR
(specialized) OR (specialised) OR ‘specialist’ OR
(inpatient) OR (hospitalization) OR (hospitalisation)
OR (care levels) OR (interface) OR (cross-level) OR
(referral) OR (communication)
74.288
3. Measurement tools
3. Measurement tools
measure OR measures OR indicator
>100.000
AB measure OR measures OR indicator
104.884
4. (1) and (2) and (3)
500
4. (1) and (2) and (3)
296
Duplicates
266
Duplicates
139
Total
234
Total
157
Table 9 A5: LILACS; 19/05/2011
References
number
Table 7 A3: ECONLIT; 19/05/2011
((coordinación) OR (continuidad) OR (colaborativa) OR
(transición) OR (compartido)) AND ((asistencial) OR
(de la atención)) AND (indicador OR indicadores OR
medida OR medición OR mediciones)
[Palabras del resumen]
5
Duplicates
0
Total
5
References
number
Competing interests
The authors declare that they have no competing interest.
1. Clinical coordination
TX (Coordinated care) OR (care coordination)
OR (collaborative care) OR (integrated care) OR
(shared care) OR (transitional care) OR
(continuity of care) OR (care continuity) OR
(informational continuity) OR (managerial continuity)
OR (management continuity)
129
2. Levels of care
TX (Primary care) OR (family practice) OR (Generalist)
OR (GP) OR (outpatient) OR (secondary care) OR
(specialized) OR (specialised) OR ‘specialist’ OR
(inpatient) OR (hospitalization) OR (hospitalisation)
OR (care levels) OR (interface) OR (cross-level) OR
(referral) OR (communication)
25.224
3. Measurement tools
TX measure OR measures OR indicator
51.184
4. (1) and (2) and (3)
4
Duplicates
3
Total
1
Authors’ contribution
MBA, IV and MLV were responsible for the study conception and design, for
the systematic review and data analysis. MBA, IV, JC, SC, FC, MA, JF, JRL, LC
and MLV participated in the selection, adaptation, development and
interpretation of the indicators. JC, SC, FC, MA participated providing
operational and methodological support to the fieldwork. MBA, IV and MLV
drafted the manuscript. MBA, IV, JC, SC, FC, MA, JF, JRL, LC and MLV
participated in data interpretation, reviewed draft versions of the paper and
approved the final version. All authors read and approved the final
manuscript.
Acknowledgments
The research leading to these results received funding from the Instituto de
Salud Carlos III (PI10/00348) and Fondos FEDER. The funding source had no
involvement in the study design, nor in the collection, analysis and
interpretations of data, or in the writing of the article and the decision to
submit it for publication.
The authors are most grateful to the people that participated in the study
and generously gave their time and Claudia Ortiz and Josep Maria Lisbona
who contributed to data collection. We thank Nuria Martinez for her
administrative support and help, Irene Garcia Subirats for her statistical
Aller et al. BMC Health Services Research (2015) 15:323
support, and Kate Bartlett for her help in correcting the English version of
this article.
Author details
1
Health Policy and Health Services Research Group, Health Policy Research
Unit, Consortium for Health Care and Social Services of Catalonia, Avenida
Tibidabo, 21, 08022 Barcelona, Spain. 2Grup de Recerca en Serveis Sanitaris i
Resultats en Salut, Serveis de Salut Integrats Baix Empordà, Carrer Hospital,
17-19 Edif. Fleming, 17230 Palamós, Spain. 3Catalan Health Institute, Gran Via
de les Corts Catalanes, 587, 08007 Barcelona, Spain. 4IMIM - Hospital del Mar
Medical Research Institute, Carrer Dr. Aiguader, 88, 08003 Barcelona, Spain.
5
Institut de Prestacions d’Assistència Mèdica al Personal Municipal, Carrer
Viladomat, 127, 08015 Barcelona, Spain. 6Centre Integral de Salut Cotxers,
Avinguda de Borbó, 18 - 30, 08016 Barcelona, Spain. 7Health Policy and
Health Services Research Group; Division of Management, Planning and
Organizational Development, Badalona Healthcare Services, Via Augusta,
9-13, 08911 Badalona, Spain. 8Health Policy and Health Services Research
Group; Strategic Planning Division, SAGESSA Group, Avinguda del Dr. Josep
Laporte, 2, 43204 Reus, Spain.
Received: 15 January 2015 Accepted: 24 July 2015
References
1. Reid R, Haggerty J, McKendry R. Defusing the confusion: concepts and
measures of continuity of healthcare. Ottawa: Canadian Health Services
Research Foundation; 2002.
2. Starfield BH. Atención primaria. [Primary Care: Balancing Health Needs,
Services, and Technology]. Barcelona: Masson, S.A.; 2001.
3. Nolte E, McKee M. Caring for people with chronic conditions. A health
system perspective. Maidenhead: European Observatory on Health Systems
and Policies Series. Mc Graw Hill, Open University Press; 2008.
4. Vogeli C, Shields AE, Lee TA, Gibson TB, Marder WD, Weiss KB, et al. Multiple
chronic conditions: prevalence, health consequences, and implications for
quality, care management, and costs. J Gen Intern Med. 2007;22 Suppl 3:391–5.
5. Longest B, Young G. Coordination and communication. In: Shortell SM,
Kaluzny AD, editors. Health care managment. Organization design and
behavior. Albany, NY: Delmar; 2000. p. 210–43.
6. Vázquez ML, Vargas I, Unger JP, Mogollon AS, da Silva MRF, De Paepe P.
Integrated health care networks in Latin America: toward a conceptual
framework for analysis. Rev Panam Salud Publica. 2009;26:360–7.
7. Banks P. Policy framework for integrated care for older people. London:
King's Fund; 2004.
8. Øvretveit J. Does clinical coordination improve quality and save money?
Vol.1 A summary review of the evidence. London: Health Foundation; 2011.
9. Moore C, Wisnivesky J, Williams S, McGinn T. Medical errors related to
discontinuity of care from an inpatient to an outpatient setting. J Gen
Intern Med. 2003;18:646–51.
10. Terraza-Núñez R, Vargas I, Vázquez ML. Coordination among healthcare
levels: systematization of tools and measures. Gac Sanit.
2006;20:485–95.
11. Vázquez ML, Vargas I, Unger JP, De Paepe P, Mogollón-Pérez AS,
Albuquerque P, et al. Evaluating the effectiveness of care integration
strategies in different health care systems in Latin America: the EQUITY-LA II
quasi-experimental study protocol. BMJ Open 2015;5:E007037.
12. Shortell SM, Gillies RR, Anderson DA, Morgan KL, Mitchell JB. Remaking health
care in America. San Francisco: The Jossey-Bass health care series; 1996.
13. Vargas I, Mogollón-Pérez AS, De Paepe P, da Silva MRF, Unger JP, Vázquez
ML. Do existing mechanisms contribute to improvements in care
coordination across levels of care in health services networks? Opinions of
health personnel in Colombia and Brazil. BMC Health Serv Res 2015; 15:213.
14. U.S. Department of Health and Human Services. 2012 Annual Progress
Report to Congress: National Strategy for Quality Improvement in Health
Care. 2012. http://www.ahrq.gov/workingforquality/nqs/nqs2012annlrpt.pdf.
Accessed 28 July2015.
15. Øvretveit J. Does clinical coordination improve quality and save money?Vol.2 A
detailed review of the evidence. London: Health Foundation; 2011.
16. DuGoff EH, Dy S, Giovannetti ER, Leff B, Boyd CM. Setting standards at the
forefront of delivery system reform: aligning care coordination quality
measures for multiple chronic conditions. J Healthc Qual. 2013;35:58–69.
Page 15 of 16
17. Bird JC, Beynon GJ, Prevost AT, Baguley DM. An analysis of referral patterns
for dizziness in the primary care setting. Br J Gen Pract. 1998;48:1828–32.
18. Minvielle E, Leleu H, Capuano F, Grenier C, Loirat P, Degos L. Suitab...
Purchase answer to see full
attachment