NIJ's Multisite Adult Drug Court
Evaluation highlights important considerations when analyzing the costs and
benefits of crime interventions.
Policymakers, philanthropists and
others interested in what works in reforming criminal justice policy and
practice are concerned traditionally with whether new approaches have better
outcomes than business as usual. But funders at all levels increasingly see
themselves as investors and are concerned not only with outcomes but also with
costs and benefits. They ask whether the investment of additional resources is
worth the added costs — and whether they will see those benefits down the road
in their budgets. A cost-benefit analysis (CBA) can help answer these
questions, but its application to criminology can be tricky.
Consider this example: A court
sentences a drug-involved offender to community-based substance abuse treatment
instead of incarcerating him. After several months of fits and starts, he is
clean and in recovery. One evening, he is standing outside a subway exit as you
leave the platform. Before going into recovery, he may have walked up to you,
punched you in the face, and grabbed your phone and purse to support his drug
habit. Now, however, he simply goes about his business.
We know how much the treatment
program that he went through cost. We can ask him about his work, income and
family life and solidly estimate how they have benefited since he went to
treatment and not prison. And although we do not know exactly what sentence he
might have received, we can use sentencing grids and past court behavior to
estimate whether he would have been locked up that evening and what those costs
would have been.
Yet we are far less certain of what
harm you would have experienced had you been beaten and robbed by the offender
outside the subway: Would you have gone to the hospital? Missed work? Suffered
from post-traumatic stress disorder? What about the harm your family and
neighbors may have experienced because of your victimization?
Criminal justice cost-benefit
researchers could argue about which of these costs and benefits even
"matter." Should the criminal justice system be concerned about
improving wages and income? Should it count the benefits you experienced
because you were not victimized, even though they do not show up in a budget?
How can a CBA even generate estimates about hypothetical events?
This article explores these and
other important considerations when analyzing the costs and benefits of crime
interventions. It also examines NIJ's recent Multisite Adult Drug Court Evaluation
to demonstrate how CBAs can:
- Include a wide range of potential costs and benefits,
including those related to crime, drug use, education, employment, family
functioning and mental health.
- Calculate — in dollar amounts — the difference or
"net benefit" between drug court participants and a comparison
group of probationers across a variety of outcomes.
- Improve the accuracy of cost and benefit estimates
while showing how variable those estimates really are.
The article closes with some general
recommendations for improving CBAs of criminal justice reforms.
Market for Crime
Basic economic theory says that the
price of a product or service will be determined by how much demand there is
for that product or service and how much manufacturers are willing to supply.
The higher the price, the less consumers will demand, but the more
manufacturers will be willing to supply.
Although this basic idea works in
theory, the market often behaves inefficiently in practice, for a variety of
reasons. For example, a monopoly might cause a price to be artificially higher
than it would be in a competitive market. CBA was traditionally used to
determine whether the benefits of correcting these inefficiencies were worth
Many government activities exist within
a clearly observable market. Public health researchers can study the direct
costs and benefits of health care reforms. Transportation planners can predict
accurately the effect new tolls have on driver behavior and thus can reliably
total the costs and benefits.
There is, however, no market for
crime. No one chooses to be victimized. And although
people can alter their chances of being victimized by changing routine activities,
all victims are unwilling participants in the exchange of the crime
"good." Not surprisingly, most cost-benefit work in law and economics
focuses on the few areas in which there is something of a more defined
"marketplace" — for example, when examining whether changes in
sentencing practices and the costs of more imprisonment are offset by crime
reduction due to incapacitation and deterrence.
Analyses in Criminology
CBAs in criminology are usually part
of an impact evaluation, which looks at how a new program affects outcomes for
participants. Most applied criminology CBAs count the costs of new
interventions, translate participant outcomes into dollars, and compare those
costs and benefits to business as usual.
When performing CBAs in criminology,
there are three important issues to consider:
Explanations, or Counterfactuals
Early CBAs in criminology simply
counted costs and benefits and compared them to each other, without considering
whether there were alternative explanations for the results. Consider our
successful treatment client. To put a value on his recovery, we need to know
whether he would have been in prison or on the street without treatment. We
also need to know how much of his recovery was due to the treatment. What else
happened in his life between sentencing and the evening at the subway that
might have affected his behavior? This process of developing an appropriate
"counterfactual" is critical to generating rigorous CBA results.
For most consumers, return on
investment is the most critical bottom line. But earlier CBAs of criminal
justice reforms make clear one inconvenient truth — most of the benefits of
reform will fall to individuals outside of the criminal justice system.
Consider our offender in treatment.
In that example, his recovery prevented a criminal victimization. Had he
committed the act, he may have been arrested, gone to court and been sentenced
to prison. Avoiding those specific events, however, did not yield direct benefits
to the criminal justice system. For costs to be recoverable, enough offenders
must succeed in treatment to drive crime rates down so low that we need fewer
police and corrections workers. In practice, reforms are rarely of this scale.
The question then is whether to
include nonrecoverable benefits — such as avoided harm to victims — as
benefits. There is a strong scientific basis to do so; however, researchers and advocates should be
judicious when discussing nonrecoverable benefits because policymakers and
those in charge of budgets tend to be skeptical of so-called "soft"
numbers. Nevertheless, including such benefits reflects a more honest account
of a reform's effect.
Researchers commonly report CBA
results as a cost-benefit ratio, which compares average costs to average
benefits. However, using a cost-benefit ratio can hide how variable an outcome
really is. People often misunderstand ratios to be facts, but they are actually
estimates of the average outcome within a broader range of plausible outcomes.
There is also
"uncertainty" to consider; this includes those important factors —
such as the recovering offender's intrinsic motivation to change — that cannot
be included in a statistical analysis. To indicate the uncertainty of an
estimate, researchers use confidence limits to reflect measurement error and
variance or they provide a range in estimates. For example, the results of a
telephone survey may be reported as 60 percent in favor of a ballot item with a
3 percent margin of error or as an estimate between 57 percent and 63 percent.
The typical criminological study
evaluates enormously complex human behavior. Efforts to quantify an intervention's
effect never will yield exact results, and studies that use seemingly precise
cost-benefit ratios can be misleading.
Multisite Adult Drug Court Evaluation
To examine these issues more closely
in an applied CBA, we turn to NIJ's Multisite Adult Drug Court Evaluation
(MADCE), an unprecedented study examining the effects of adult treatment drug
MADCE involved process and impact
evaluations as well as a CBA. To date, most drug court evaluations have relied
on recidivism as the sole measure of impact. MADCE, however, measured both
short- and long-term outcomes — for example, crime, drug use, education,
employment, family functioning and mental health — and captured the role of
court policies and practices, offender perceptions, and interim compliance with
Researchers at the Urban Institute,
the Center for Court Innovation and RTI conducted three waves of interviews
with nearly 1,800 drug court participants and comparison probationers from 29
U.S. jurisdictions. Additional data included drug tests, administrative records
on treatment and recidivism, court observation, interviews with staff and other
stakeholders, and budget and other cost information.
Researchers collected comparable
information from program participants and the group of comparison probationers.
This allowed them to control for characteristics that might offer competing
explanations — or counterfactuals — for why the behavior of the drug court
participants changed relative to that of the comparison probationers.
the MADCE Impact Evaluation Found
The impact evaluation found that
adult drug courts significantly reduce participants' drug use and criminal
offending during and after program participation. Drug court participants
reported less drug use (56 percent versus 76 percent) and were less likely to
test positive for drug use (29 percent versus 46 percent) than the comparison
probationers. Participants also reported less criminal activity (40 percent
versus 53 percent) and had fewer rearrests (52 percent versus 62 percent, but
not statistically significant difference) than the comparison probationers.
Differences in employment, schooling, community service and other outcomes were
not statistically significant.
A full description of the MADCE design and results
can be found on the NIJ website.
the Costs and Benefits of Drug Courts
When performing a CBA, researchers
can take a top-down or a bottom-up approach to estimating costs. The top-down
approach divides the total budget for the service by the number of people
served and assigns the same value to each person. In the bottom-up approach, researchers first
identify the unit cost for the service (e.g., the cost of a counseling session)
and then multiply it by the number of units an individual receives; the result
is a person's individual cost. All individual costs are summed to arrive at the
total cost. MADCE's CBA used a bottom-up approach, drawing from individual
interviews and administrative data.
As for benefits, criminal justice
reforms can lead to reductions in criminal offending and improvements in other
outcomes. This results in:
- Cost reductions associated with investigating,
arresting and supervising offenders (sometimes recoverable)
- Reductions in harm to victims (rarely recoverable)
Table 1 shows the outcomes that MADCE measured.
In addition to offending, the evaluation examined social productivity outcomes,
which include wages, educational attainment and payment of legal obligations
such as child support — all positive outcomes for society.
1: Outcomes Measured by MADCE
Services and support provided
Child support payments, community service
Probation officer meetings, drug
tests, electronic monitoring
Jail and prison (sanctions or otherwise)
Case management, administrative costs
Mental health treatment
Emergency room, detoxification,
residential care, outpatient, methadone
Hospital stays unrelated to drugs
Stays in mental health facilities unrelated to drugs
Halfway houses, public housing, homeless shelters
Welfare, disability, other
Money from family and friends
The last two categories in the table
— service use and financial support use — are more ambiguous in that they may
lead to greater cost or greater benefits. For example, an effective drug court
should lead to less acute care for participants, such as detox. However, an
effective court also could mean more use of relapse prevention treatment
services, although hopefully at a declining rate over time. Receipt of welfare
funds would decline if a program met its goals, but the potential impact of
changes in the receipt of disability payments is unclear. Nevertheless, these
outcomes represent a real use of resources resulting from a drug court program
and thus were included in the CBA.
Up the Costs and Benefits
The MADCE researchers identified all
services provided and all outcomes experienced by each person and then
converted relevant benefits into dollars. They weighted a recidivism event by
the price of crime to victims plus the price of processing the case.
Researchers repeated the same process for other outcomes, if data were
Rather than directly comparing costs
and benefits, the researchers instead summed the costs and benefits for each
participant. This included positive outcomes (such as wage increases), negative
outcomes (such as lost wages), and all costs associated with treatment and
criminal case processing. The resulting measure could have a positive or
negative value, depending on the participant's own experience.
This approach has several important
advantages. First, because new costs and benefits are hard to predict, this
approach allowed researchers to easily calculate the variance for an estimate
and show how certain or uncertain that estimate is.
Second, the researchers did not have
to determine subjectively what constitutes a cost and a benefit. In traditional
studies, events that require new spending are considered costs, and activities
that reduce spending are benefits. These designations are arbitrary and often
lead to controversy. For example, should prison or jail time for drug court
participants be counted as a cost or a benefit? What about post-disposition
jail or prison time for comparison probationers? By summing the costs and
benefits together, the MADCE researchers were able to combine all drug court
outcomes without subjectively defining them as costs or benefits.
Finally, as mentioned, the
researchers opted for a bottom-up approach, using individual rather than
aggregated data. Aggregated data — dividing total costs by the number of
participants — yield only an average, which is then assumed to be the same for
all participants. Individual data, on the other hand, allow researchers to
estimate a range of plausible values and describe variations in costs, benefits
and outcomes among drug court participants and the comparison probationers.
the MADCE CBA Found
The MADCE CBA used these
calculations to estimate the individual costs and benefits for both the drug
court participants and the comparison probationers for each of the outcomes
listed in Table 1.
Table 2 shows the net benefits for each outcome
category. Although cost-benefit analysts use the term frequently, "net
benefit" can be slightly misleading — it is the difference between total
costs and total benefits. Thus, net benefits can be positive (i.e., drug court
participants used fewer resources overall than the comparison probationers) or
negative (where the total end result is negative).
The total net benefit ($5,680) is in
the positive direction and is substantial, but there are not savings in every category,
and the overall difference is not statistically significant. Looking at
specific outcome categories, there are substantial and statistically
significant savings in crime and victimization ($11,566). However, these
savings are offset by significantly higher service use costs (−$8,135) because
drug court participants accessed more drug and alcohol treatment.
2. Net Benefits by Category for Drug Court Participants and Comparison
*Difference is statistically significant (p <
The MADCE researchers recognized
uncertainty throughout their analysis to help highlight important conclusions
that might otherwise be obscure. For example, if they had conducted a top-down
analysis of the data, they would have arrived at a statistically significant
cost-benefit ratio similar to what has been found in previous research.
However, by including uncertainty throughout their bottom-up analysis, the
researchers found that the results are not statistically significant.
This latter point is critical. The
MADCE researchers estimate that drug courts produce about $1.50 in benefits for
every dollar in costs — this is similar with much of the current research
literature. However, the researchers do not find that difference to be
statistically significant once they account for additional factors, such as the
range of victimization costs. The severity of crimes avoided ranges from
low-level misdemeanors to violent felonies, and the associated victim costs
range from very low to very high. Therefore, the confidence limits surrounding
average cost estimates are wide, making it difficult to confirm whether the
range for drug court participants differs from the range for comparison
probationers, especially for violent felonies, which are relatively rare. Thus,
prior studies that produced only a single cost-benefit estimate may overestimate
the effects of drug courts.
A careful examination of MADCE's
impact evaluation and CBA, however, paints a clear story: Drug courts prevent
many petty crimes and a few serious crimes. In fact, the CBA results showed
that those few serious crimes drive much of the drug court effect; if we remove
those outliers, the benefits of drug courts barely exceed the cost. This
finding suggests that although drug courts may reduce recidivism among many
types of offenders, drug courts that target serious criminal offenders with a
high need for substance abuse treatment will produce the most effective
interventions and a maximum return on investment.
CBAs in Criminology
The CBA performed in the MADCE study
demonstrates that criminal justice reforms can have tangible, positive
benefits, including fewer crimes and more savings in victimization costs. It
also shows that some reforms can lead to additional costs, such as increased
drug and alcohol treatment services. However, increased costs that achieve important
objectives — such as keeping drug addicts in treatment — should not be used to
argue against reform. No one would argue that we should not reduce school
truancy because more kids would go to school and require more resources. And
some of the positive outcomes shown in the MADCE study — such as improved
family life — simply cannot be measured.
Over the last decade, the
criminology field has seen a rapid increase in the use of more sophisticated
statistical analyses. But when it comes to CBAs, much work remains to be done.
More sophisticated CBAs that examine each category of spending and savings
could yield important information about a program's success or failure. For
instance, significant savings in public safety costs may require significant
investment in treatment costs, but potential benefits may be missed if you look
only at overall estimates.
An intervention can directly benefit
offenders in dramatic ways. It might prevent or ameliorate health problems,
save lives by preventing overdoses, or reduce criminal behavior and community
supervision violations that affect income and families. To really use CBA to
improve public policy, stakeholders should consider expanding the range of
included benefits. Broad measures of a program's impact — including on victims
and clients — provide much more useful information than studies focused solely
on returns on investment.
NIJ Journal No. 272, posted May 2013
John Roman is a senior fellow in the Urban Institute's Justice Policy
Back to the top.
For More Information
 There is no market for the exchange of the
crime "good" that includes voluntary participation from both a victim
and an offender. When a criminal incident occurs, the offender supplies crime,
but the victim cannot be said to demand to be victimized, so no market-based
transaction occurs. There is a market for the exchange of crime prevention, but
the offender is not part of that transaction. This is different from a health
care transaction, in which the illness is not an actor — transactions occur
between consenting patients and health care providers.
 Consider the long tradition in economics of
opportunity costs, which describe the next best use of a resource. Simply put,
if a dollar is used for one purpose, it cannot be used for another. A clear
example of the import of opportunity costs can be found in the death penalty
literature (Roman, John K., Aaron J. Chalfin and Carly R. Knight,
"Reassessing the Cost of the Death Penalty Using Quasi-Experimental
Methods: Evidence From Maryland," American Law and Economics Review
11 (Fall 2009): 530-574). In states with the death penalty, there is often
"super due process," where a state's attorneys devote substantially
more time to those cases. Prosecutors have argued that there would be no actual
savings if the death penalty were abolished because it would not change the
number of prosecutors. However, there is clearly an opportunity cost of
attorneys working on death penalty cases and not working on other cases — cases
that would receive more attention in the absence of the death penalty.
 Some studies calculate an average cost for a
specific period — for example, the average cost of treatment per month.
Researchers then assign the cost for each month a client was in the program. As
a result, total costs may vary across individuals.
 For example, if a newly committed crime
results in a new prison term, researchers cannot know in advance the type of
facility that will house the offender. Costs vary considerably across
facilities, depending on whether the offender is in a boot camp or in a
minimum- or maximum-security prison. Consequently, describing the new costs as
a range of costs rather than a precise estimate is more appropriate. Such
uncertainty exists for all costs and benefits.
 Overall, the net benefit of drug courts is an
average of $5,680 to $6,208 per participant. The researchers calculated net
benefits in two ways, based on two different assumptions about individual
earnings. The more conservative approach relied on minimum wage, probably an
underestimate, while the alternative relied on the average wage reported in the
U.S. Census, likely an overestimate. Readers are encouraged to rely on the
range of net benefits ($5,680 to $6,208) and not a single estimate.