Journal of Brief Therapy Volume 5 • Number 1 • 2006 Using Formal Client Feedback to Improve Retention and Outcome: Making Ongoing, Real-time Assessment Feasible Scott D. Miller Barry L. Duncan Jeb Brown Ryan Sorrell Mary Beth Chalk Institute for the Study of Therapeutic Change Chicago, Illinois Research has found that client change occurs earlier rather than later in the treatment process, and that the client’s subject experience of meaningful change in the first few sessions is critical. If improvement in the client’s subject sense of well-being does not occur in the first few sessions then the likelihood of a positive outcome significantly decreases. Recent studies have found that there are significant improvements in both retention and outcome when therapists receive formal, realtime feedback from clients regarding the process and outcome of therapy. However, the most used instruments in these feedback studies are long and take up valuable therapy time to complete. It has been found that most therapists are not likely to use any feedback instruments if it takes more than five minutes to complete, score and interpret. This article reports the results of an evaluation of the use of two very brief instruments for monitoring the process and outcome of therapy, the Outcome Rating Scale (ORS) and the Session Rating Scale (SRS), in a study involving 75 therapists and 6,424 clients over a two year period. These two instruments were found to be valid and reliable and had a high use-rate among the therapists. The findings are discussed in light of the current emphasis on evidence-based practice. O “The proof of the pudding is in the eating.” Cervantes, Don Quixote *** utcome research indicates that the general trajectory of change in successful psychotherapy is highly predictable, with most change occurring earlier rather than later in the treatment process (Brown, Dreis, & Nace, 1999; Hansen & Lambert 2003). In their now classic article on the dose-effect relationship, Howard, Kopte,    Improving Retention and Outcome Krause, and Orlinsky (1986) found that between 60-65% of people experienced significant symptomatic relief within one to seven visits—figures that increased to 70-75% after six months, and 85% at one year. These same findings further showed, “a course of diminishing returns with more and more effort required to achieve just noticeable differences in patient improvement” as time in treatment lengthened (p. 361, Howard et al., 1986). Soon after Howard et al.’s (1986) pioneering study, researchers began using early improvement—specifically, the client’s subjective experience of meaningful change in the first few visits—to predict whether a given pairing of client and therapist or treatment system would result in a successful outcome (Haas, Hill, Lambert, & Morrell, 2002; Lambert, Whipple, Smart, Vermeersch, Nielsen, & Hawkins, 2001; Lueger, 1998; Lueger, 2001). Continuing where they had left off, Howard, Lueger, Maling, & Martinovich (1993) not only confirmed that most change takes place earlier than later, but also found that an absence of early improvement in the client’s subjective sense of well-being significantly decreased the chances of achieving symptomatic relief and healthier life functioning by the end of treatment. Similarly, in a study of more than 2000 therapists and thousands of clients, Brown, et al. (1999) found that therapeutic relationships in which no improvement occurred by the third visit did not on average result in improvement over the entire course of treatment; this study further found that clients who got worse by the third visit were twice as likely to drop out of treatment than clients who reported making progress. More telling, variables such as diagnosis, severity, family support, and type of therapy were, “not . . . as important [in predicting eventual outcome] as knowing whether or not the treatment being provided [was] actually working” (p. 404). By the mid-nineties, researchers were using data generated during treatment to improve the quality and outcome of care. In 1996, Howard, Moras, Brill, Martinovich, and Lutz showed how measures of client progress could be used to “determine the appropriateness of the current treatment…the need for further treatment…[and] prompt a clinical consultation for patients who [were] not progressing at expected rates” (p. 1063). That same year, Lambert and Brown (1996) made a similar argument using a shorter, and hence more feasible, outcome tool. Other researchers had already found that clients’ early ratings of the alliance, like progress, were “significant predictors of final treatment outcome” (Bachelor & Horvath, 1999, p. 139). Building on this knowledge, Johnson and Shaha (1996, 1997; Johnson, 1995) were among the first to document the impact of outcome and process tools on the quality and outcome of psychotherapy as well as demonstrate how such data could foster a cooperative, accountable relationship with payers. Several recent studies have documented significant improvements in both retention in and outcome from treatment when therapists have access to formal, real-time feedback from clients regarding the process and outcome of therapy (Duncan & Miller, 2000; Duncan, Miller, & Sparks, 2004). For example, Whipple, Lambert, Vermeersch, Smart, Nielsen, and Hawkins (2003), found that clients whose therapists had access to progress and alliance information were less likely to deteriorate, more likely to stay longer, and twice as likely to achieve a clinically significant change. Formal client feedback has also been shown to be particularly helpful in cases at risk for a negative or null outcome. A meta-analysis of three studies by Lambert, Whipple, Hawkins, Vermeersch, Nielsen, and Smart (2003) found that cases informed by client ratings of progress were, at the conclusion of treatment, better off than 65% of those without access to such data (Average ES = .39). Miller, Duncan, Brown, Sorrell & Chalk  The present study was designed to assess the impact of two simple and brief, clientcompleted, rating scales of alliance and outcome on retention in and outcome from therapy. Research and clinical experience indicate that the length and complexity of the measures employed in the studies to date hinder their application in real world clinical settings (Miller, Duncan, Brown, Sparks, & Claud, 2003). Indeed, Brown et al. (1999) found that the majority of practitioners are unlikely to use any measure or combinations of measures that took more than five minutes to complete, score, and interpret. Therapists, it is clear, not only require valid and reliable but also feasible tools for inviting client feedback. As Lambert, Hansen, and Finch (2001) pointed out in a special issue of the Journal of Consulting and Clinical Psychology on client feedback, “treatment systems cannot tolerate expensive and time-intensive markers of change, especially when used as a start up procedure or where patient (sic) progress is reported to therapists on a weekly schedule” (p. 160). Participants Method The participants in the study were clients of Resources for Living® (RFL), an international Employee Assistance Program (EAP) based in Austin, Texas. The company employs 75 “in-house” therapists who provide telephonic-based employee assistance, information and referral, executive coaching, individual therapy, disease management, and critical incident services to 28 different corporate and organization customers. Therapists at RFL range in age from 25 to 57, with an average age of 37.4, and are predominantly female (72.2%). Average length of employment at RFL for those included in the study was 3 years, with an average of 7 years of total clinical experience. The staff comes from a variety of professional disciplines, including clinical psychology (45%), social work (35%), and marriage and family therapy (20%), and the majority of them (92%) were licensed to practice independently by their respective discipline. The clientele of RFL is culturally and economically diverse, including people of American, European, African, Latin, and Caribbean decent. In any given year, the severity of problems presented by clients of organization is comparable to those seen in a typical mental health clinic, including anxiety, depression, alcohol and drug abuse, work and family issues, as well as chronic mental and physical health problems (Miller, Duncan, Brown et al., 2003). The sample in the present study included 6,424 clients that received telephonic based counseling between April 1, 2002 and March 31, 2004. In order to be included in the sample, the client must have received at least two sessions and completed an outcome questionnaire by the end of each. Because callers have a right to remain anonymous, limited demographic information is available. Similar to most community mental health outpatient settings, two-thirds of the participants were female, one third male. The average age of the sample was 36, with a median age of 34, mode of 20, and standard deviation of 13. The level of distress as assessed by the outcome measure administered at intake was also similar to that found in a typical community mental health outpatient sample—in fact, it was slightly greater than the figure reported by Miller, Duncan, Brown et al. 2003 (18.6 versus 19.6). Given that the services offered by RFL are employer funded, it can be safely assumed that all of the clients in the current study were either employed or were a family member   Improving Retention and Outcome of someone who was working for a covered organization. The majority of clients who utilized the service during the study period fell at the lowest end of the pay scale in their respective work settings, with 68% of the sample made up of “line workers,” 12% from middle and upper management, and 4% who had either retired or been terminated. Family members of a covered employee made up the remaining 16% of contacts. During the study period, the top five presenting problems were: (1) marital (24.7%); (2) depression (10%); (3) anxiety (5.9%); (4) issues related to grief and loss (4.8%); and (5) drug and alcohol problems (3.5%). Measures Client progress was assessed via the oral version of the Outcome Rating Scale (ORS [Miller & Duncan, 2000]), a four-item, self-report instrument (see Appendix 1). The ORS was developed as a brief alternative to the Outcome Questionnaire 45 (OQ-45)—a popular but longer measure developed by Lambert and colleagues (Lambert, Hansen, Umphress, Lunnen, Okiishi, Burlingame, Huefner & Reisinger, 1996). Both scales are designed to assess change in three areas of client functioning widely considered valid indicators of progress in treatment: individual (or symptomatic) functioning, interpersonal relationships, and social role performance (work adjustment, quality of life [Lambert & Hill, 1994]). In a recent issue of the Journal of Brief Therapy, Miller, Duncan, Brown et al. (2003) reported results of an initial investigation of the reliability and validity of the ORS. Pearson product moment correlation between the ORS and the OQ-45 yielded a concurrent validity coefficient of .58, a figure considered adequate given the brevity of the ORS. Reliability of the measure, as assessed by Cronbach’s coefficient alpha, was .93, test-retest reliability at the second session, .66. Independent confirmation of the reliability of the ORS was conducted by the Center for Clinical Informatics11 using data collected at RFL. In this sample, coefficient alpha was found to be .79 (n = 15,778), while test-retest reliability at second administration was .53 (n = 1,710). With regard to the latter, it is important to note that lower test-retest reliability is expected for measures designed to be sensitive to change from week to week as research has shown both the ORS and OQ-45 to be (Miller, Duncan, Brown et al. 2003; Vermeersch, Lambert, & Burlingame, 2000). The therapeutic alliance was assessed via the oral version of the Session Rating Scale 3.0 (SRS [Miller, Duncan, & Johnson, 2000] see Appendix 2). The SRS is a brief, four-item, client-completed measure derived from a ten-item scale originally developed by Johnson (1995). Items on this measure reflect the classical definition of the alliance first stated by Bordin (1979), and a related construct known as the client’s theory of change (Duncan & Miller, 2000). As such, the scale assesses four interacting elements, including the quality of the relational bond, as well as the degree of agreement between the client and therapist on the goals, methods, and overall approach of therapy. To test the reliability and validity of the SRS, Duncan, Miller, Reynolds, Sparks, Claud, Brown, & Johnson (2004) compared the instrument to the Revised Helping Alliance Questionnaire (HAQ-II), a widely used measure of therapeutic alliance. The reliability for the SRS compared favorably with the HAQ-II (.88 and .90, respectively). Test-retest reliability for the SRS over six administrations was .74, compared to .69 for the HAQ-II. Concurrent validity as estimated by Pearson product moment correlations averaged .48, evidence that the SRS and HAQ-II are referencing similar domains. As with Miller, Duncan, Brown, Sorrell & Chalk  the ORS, independent confirmation of the reliability of the SRS was conducted by the Center for Clinical Informatics using data collected at RFL. In a sample of nearly 15,000 administrations, coefficient alpha was found to be .96, remarkably high for a four-item measure. Test-retest reliability was .50, comparable to that of the ORS. Procedures The study was divided into four distinct phases: (1) initial training (including several site visits by the first two authors over a 6-month period); (2) baseline data collection and analysis (6 months); (3) implementation of automated feedback condition (6 months); and (4) continued evaluation (12 months). During the first phase, therapists were trained on site by the first two authors in the proper administration of the ORS and SRS. Both tools were then incorporated into RFL’s existing, computerized client tracking system, making the use of the scales a uniform and automatic process along with routine record keeping. In practice, the ORS was completed at the start of each session and the SRS at the end. During the second phase, baseline data from the ORS and SRS was collected for 1,244 clients that received two or more telephonic counseling sessions. Gathering such data was a critical step in developing the clinical norms that would form the basis for the automated feedback system known as SIGNAL (Statistical Indicators of Growth, Navigation, Alignment and Learning). As the name implies, the WindowsTM-based system used a traffic light graphic to provide “real-time” warnings to therapists when an individual client’s ratings of either the alliance or outcome fell significantly outside of the established norms. As an example of the kind of feedback a therapist would receive when a particular client’s outcomes fell outside of the expected norms, consider Figure 2. The dotted line represents the expected trajectory of change for clients at RFL whose total score at intake on the ORS is 10. Consistent with prior research and methodology (Lambert & Brown, 2002), trajectories of change were derived via linear regression, and provide a visual representation of the relationship between ORS scores at intake and at each subsequent administration. Colored bands corresponding to the 25th (yellow) and 10th (red) percentiles mark the distribution of actual scores below the expected trajectory over time. The horizontal dashed-dotted line at 25 represents the clinical cutoff score for the ORS. Scores falling above the line are characteristic of individuals not seeking treatment and scores below similar to people who are in treatment and likely to improve (Duncan, Miller, Reynolds, et al. 2003). The remaining solid line designated the client’s actual score from session to session. As can be seen in Figure 2, the client’s score at the second session falls below the 25th percentile. By session 3 the score has fallen even further, landing in the red area representing the 10th percentile in the distribution of actual scores. As a result, the therapist receives a “red” signal, warning of the potential for premature drop out and an increased risk for a negative or null outcome should therapy continue unchanged. An option button provides suggestions for addressing the problem, including: (1) talking with the client about problems in the alliance; (2) changing the type and amount of treatment being offered; and (3) recommending consultation or supervision. 10 Improving Retention and Outcome Figure 2: SIGNAL Outcome Feedback Client feedback regarding the alliance was presented in a similar fashion at the end of each visit (see Figure 3). A solid line designates a client’s actual score from session to session. Colored bands represent the 25th (yellow) and 10th (red) percentile of responders in the study sample (Duncan, Miller, Reynolds et al. 2004). In this particular example, the client scores a 34 on the SRS at the conclusion of the first visit. As can be seen, this score falls below the 25th percentile thus triggering a yellow signal. Given the relative rarity of such a score, the therapist is advised to check in with the client about their experience, express concern for their work together, and explore options for changing the interaction before ending the session. Once the normative data was collected and the SIGNAL feedback system created, the study entered its third phase. Outcome and alliance scores were entered and SIGNAL feedback given for the next 1568 clients that sought services at RFL. During this time, a handful of site visits by the first two authors, plus ongoing support from administration and management at RFL encouraged a high rate of compliance with and consistency in the use of the measures and SIGNAL system. In the fourth and final phase of the study, data was collected from an additional 3,612 clients, providing a large sample by which the effect of feedback on the retention in and outcome from clinical services could be evaluated. Data Analysis The outcome of treatment was assessed in three ways. First, a continuous variable Miller, Duncan, Brown, Sorrell & Chalk 11 Figure 3: SIGNAL Alliance Feedback gain score was calculated by subtracting the ORS score at intake from ORS score at the final session. Second, a residualized gain score was computed based on a linear regression model. Residualized gain scores are necessary whenever intake scores are correlated with change scores in order to control for the change in gain scores associated with differences in ORS scores at intake (Cohen & Cohen, 1983; Campbell & Kenny, 1999). Third, and finally, a categorical variable classification of outcome as “improved,” “unchanged,” or “deteriorated” was determined by comparing the gain score against the reliable change index of the ORS (RCI = 5; Duncan & Miller, 2004). The RCI for the ORS is 5, so cases with a gain score greater than +5 were classified as improved, -5 as worse, and those falling between + or – 4 points as unchanged. In order to facilitate interpretation of the magnitude of improvement across phases, gain scores were converted to effect sizes (Smith & Glass, 1987). The effect sizes in the present study were calculated by dividing the gain score by the standard deviation of the ORS in a non-treatment normative sample (Miller, Duncan, Brown et al. 2003). As such, the effect sizes reported can be interpreted as an indication of how much clients in the study improved relative to a normal population. Finally, the relationship between the alliance and outcome was also subjected to analysis. Given prior research showing that clients’ early ratings of the alliance are significant predictors of final treatment outcome, simple correlations were computed between SRS scores at intake and end of treatment gain scores. To determine the effect of improving alliances on the outcome of treatment, gain scores for the SRS were computed and then correlated with gain scores on ORS. Finally, the relationship between the actual use of the 12 Improving Retention and Outcome SRS and outcome from and retention in treatment the first session was also examined. Results Data gathered during the baseline phase of the study revealed that the majority of clients (56%) who received two or more sessions did not remain with the same therapist over the course of services. Analysis of the outcome data from this period further showed that clients who switched therapists fared significantly worse than those treated by the same therapist from session to session (Effect Size = .02 versus .79). Reflecting on the possible causes of the rampant switching, therapists and administrators identified official agency policy favoring immediate access over continuity of services. Prior to entering the third phase of the study during which SIGNAL was launched, the policy was changed. Thereafter, therapists were strongly encouraged to retain clients by setting aside a certain amount of time per week during which standing appointments could be scheduled. By the last phase in the study, the number of clients that switched therapists had been cut in half (~27%). The outcomes for clients who stayed with the same therapist compared to those who switched can be found in Table 1. Across phases, clients who stayed with the same therapist from session to session fared significantly better. Note, additionally, that the overall outcomes of both groups improved over time. Progress was most pronounced in the group of clients that switched therapists, going from an effect size of .02 at baseline to .40 by the end of the final evaluation phase (p < .001). Clients who remained with the same therapist also improved significantly over the course of the study, with an initial effect size of .79 at baseline increasing to .93 during the last phase of the study (p < .01). Switched therapist Time period Baseline Intervention Evaluation Sample size 695 689 993 % of Mean Mean ORS Mean sample in Residual at intake gain score time period gain score 56% 44% 27% 18.3 18.3 18.3 0.13 1.9 2.7 Effect Size -4.6 -2.9 -2* 0.02 0.28 0.40 Baseline 549 44% 18.3 5.4 0.97 Intervention 879 56% 18.9 5.9 1.5 Evaluation 2719 73% 19.2 6.3 2 * p .05). However, increases in SRS scores over the course of treatment were associated with better outcomes. For all cases, gain scores on the SRS correlated .13 with gain scores on the ORS (n=4785, p
Purchase answer to see full attachment