PUAD 701
1
Government Organization Information Sources
Information on government organizations is typically the most difficult to find because there is not a
database that presents the necessary information in a consolidated fashion. Typically, you will have to
scour a particular agency’s website to find information you need. Many agencies produce annual reports
that are a good source for finding summary information students will need, particularly, understandable
presentations of financial information.
USA.gov is probably the best site for finding up-to-date links and information on the national, state,
local and tribal governments in the United States.
Also, budget or finance offices within a large agency or external administrative agencies, such as a city
budget or finance office, typically provide useful information - audit reports or comprehensive annual
financial reports (CAFRs) are often good sources for summary agency personnel and budget figures
such as revenues and expenditures.
For example, the Ohio Legislation Service Commission provides useful summaries of the purpose,
function, staffing, and budget of each state agency in Ohio. Also, the Ohio Auditor of State provides a
searchable database of financial audit reports and CAFRs for almost all state and local agencies in Ohio.
State and Local Governments
The National Conference of State Legislatures provides links to states’ legislative web sites including
legislation, government reports, state law or statutes, and state Constitutions. For example, if you want
information on governments in Ohio, look to:
U.S. Government Agencies
At the national level, the following websites have useful information:
Catalogue of Federal Domestic Assistance: Provides a full listing of all Federal programs available to
State and local governments (including the District of Columbia); federally-recognized Indian tribal
governments; Territories (and possessions) of the United States; domestic public, quasi- public, and
private profit and nonprofit organizations and institutions; specialized groups; and individuals. It
contains detailed program descriptions for 2,165 Federal assistance programs.
General Accountability Office: A source for thousands of reports summarizing and analyzing the
financial and programmatic performance of national government agencies and important public policy
issues on behalf of Congress.
Office of Management of Budget: The best centralized source for summaries of agencies activities and
current and historical budget information and analysis.
International Government Organizations
Department of State: Provides detailed, summaries on all the countries in the world.
United Nations: Provides extensive information on international issues and on the countries of the world
from an international perspective.
PUAD 701
1
Nonprofit Information Sources
Form 990
Nonprofits must regularly file information with the national government about their finances,
organizational leadership, etc. This is done on the Internal Revenue Service (IRS) Form 990.
The Form 990 is particularly helpful for finding budget information. For example, Part VIII of the form
offers a sufficiently detailed presentation of organizational program revenue sources and Part IX shows
a useful breakdown of expenses by major categories or functions. Also, Part VII and Schedule J, Part II,
which is usually filed with Form 990, lists the names and salaries of key officers and board members of
an organization.
Not-for-Profit Websites
Much of the information you will need to find about nonprofits can be found on a nonprofit website that
compiles information about and compares organizations in the not-for-profit sector:
GuideStar: One of the best sites for detailed information on non-profit organizations, including helpful
web links and links to Form 990 filings. You must register with the site to access most of the
information; however, registration is free.
Charity Navigator: Excellent source of summary information on nonprofit organizations’ finances,
mission, and top leadership and provides links to recent news stories about the organization. The site
also offers numeric ratings of each nonprofit based on multiple measures of financial performance and
on transparency and accountability. The ratings offer one method for thinking about how to evaluate the
efficiency and effectiveness of nonprofit, including comparative assessment. You must register with the
site to access most of the information; however, registration is free, particularly by reviewing the most
recent Form 990 which should be available on the website. However, you will have to register (for free)
on the site to access much of the necessary information.
Foundation Center: This is an excellent site for finding financial information about non-profits,
particularly foundations and other organizations that provide grant funding to other organizations. The
site also has summary information about nonprofits and one of the most comprehensive databases for
finding the Form 990 filings that most nonprofits must send to the IRS. The Form 990 contains detailed
financial, organizational, and other information on a nonprofit.
Internal Revenue Service: The IRS provides information on the tax code and legal requirements for
nonprofit organizations. The site also enables searches to find out the legal classification of nonprofit
charities.
Any information not available on a site like GuideStar typically can be found on the particular
nonprofit organization’s own website.
A Short Guide for Conducting Research Interviews
An interview is a great way to get information and insight into a research topic, and it can be
fun for you and the person you are interviewing. Most people enjoy sharing their expertise,
experiences, and insights if they’ve been given enough time to do so. If you plan to interview a
child, a patient, or another vulnerable individual OR will ask for medical, psychological, or legal
information, ask your professor about the ethical standards that apply to your discipline,
including those relating to confidentiality and obtaining written consent to use the material
gathered. The following is a step-‐by-‐step guide to conducting a productive interview.
Before the Interview
• Ask the person you would like to interview for an appointment as quickly as you can. This
gives you the best chance of getting a “yes” in answer to your request because the window
of opportunity for meeting with the person is the longest possible. Be courteous and as
flexible as you can in arranging the date and time of the interview. It is a good idea to
confirm the date, time, and place of the interview in an email.
• When you ask for the interview, be clear about the purpose of the interview, how the
information will be used, and how long the interview will take (30 minutes or an hour are
typical timeframes). Be clear about whether the information given will be attributed to the
person being interviewed or will be anonymous.
• For a face-‐to-‐face interview, be sure to arrange a meeting place where you and the person
being interviewed feel safe and comfortable. If you are interviewing a professional, his or
her office during regular business hours might be a good arrangement, but other public
places may also be appropriate.
• Try to arrange for a face-‐to-‐face interview, but phone or email interviews can also work.
Providing written questions to get written answers is not ideal because it is hard to ask for
clarification or to ask follow-‐up questions about answers that surprise you, but email
interviews can be effective if you give yourself enough time to follow up with additional
questions or requests for clarification.
• Research your topic and, if possible, the person you plan to interview before the interview
occurs. Then you can focus your questions and your limited interview time on getting
insights and information not available through other types of research.
writingcenter.appstate.edu 828-‐262-‐3144 writingctr@appstate.edu
Updated 03/2014
•
•
•
Prepare your questions in advance:
o Be sure your questions are clear.
o Unless you want a very clear or precise answer, questions that can’t be answered
“yes” or “no” may be the best for encouraging fuller answers, and may give you an
important explanation or interesting story.
o Try to create questions that don’t include assumptions or bias. For example, “What
do you think about this issue?” is better than “Don’t you think that (inserting your
own view on the issue)?”
The particular questions you ask will be, of course, related to the subject of your research
and should be tailored carefully to get you the information you need. For an interview of an
expert, some typical questions might include:
o Why did you get into this field?
o What education, training, or other preparation did you need to get into the field?
o What are you working on now and what is interesting about it?
o What do you spend the most time doing?
o What continues to inspire you or hold your interest in this field?
o Where is this field going in the future? What will you be working on?
o Is there anything else you think I should know?
Consider whether you want to record the interview. Generally, this is a good practice
because it will give you an exact record of the interview, frees you up to think about the
responses to your questions rather than taking careful notes (though you still might want to
make some notes), and the person you interview will be reassured that you will quote
accurately. However, in a few situations, such as an interview on a sensitive subject or if you
are interviewing a very shy person, recording may make the person interviewed uneasy and
they may be more guarded. Generally, however, it makes good sense to record research
interviews. Plan to take a your own recording device.
During the Interview
• Arrive on time and be friendly and courteous in greeting the person you are interviewing. It
may be helpful to remind them of the purpose of the interview.
writingcenter.appstate.edu 828-‐262-‐3144 writingctr@appstate.edu
Updated 03/2014
•
•
•
Make a note of (or record) the full name of the person you are interview, along with the
date, time, and place of the interview, so that you can create proper citations for the
interview in your paper.
If you are recording the interview, you must ask the person you are interviewing for
permission to record the conversation, preferably with their agreement recorded on your
recording device at the first part of the interview. (In some states, the law requires you to
get the consent of anyone you record.) If they decline, go ahead and conduct the interview
without recording it, keeping as careful notes as you can.
Relax and have fun at your interview.
o Having prepared your questions in advance and researched the background of your
topic, you will be free to listen to the responses to your questions with greater
understanding and ask important follow-‐up questions.
o Respect the time of the person you are interviewing, keeping to the agreed upon
timeframe or asking if he or she can continue a little longer.
o Thank the person for the interview.
After the Interview
• Immediately after the interview, consider transcribing the recording you’ve made because it
will help you to understand any nuances that you might have missed during the interview.
If you don’t transcribe your recording or didn’t record the interview, go over your notes to
make sure they are complete.
• Send a thank-‐you note or email even if you thanked the person you interviewed in person.
• Properly cite the information gained in the interview in your research paper according to
your documentation style.
• Consider sending the finished research paper to the person you interviewed with a final
note of thanks. As an expert, he or she will probably be interested and may become a
valuable academic or professional contact.
writingcenter.appstate.edu 828-‐262-‐3144 writingctr@appstate.edu
Updated 03/2014
Journal of Public Administration Research and Theory Advance Access published March 23, 2012
Desperately Seeking Management:
Understanding Management Quality and
Its Impact on Government Performance
Outcomes under the Clean Air Act
Alexander C. Heckman
Franklin University
This study analyzes the impact of management quality, spending, problem severity, and
political factors on states’ air pollution control outcomes and provides insights for improving
the measures and methods used in public management and government performance
research. The analysis illustrates the importance of selecting proper outcome measures and
taking into account the interaction of management and spending when conducting
empirical analysis into the causes of government performance. Additionally, the author
demonstrates the benefits of conducting comparative empirical analysis using different, but
theoretically connected, outcome measures. The findings and analysis presented should be
of particular interest to public administration scholars seeking to conduct research that
produces practical insights for public managers and policy makers on improving public
management and government performance.
THE CHALLENGE OF MEASURING MANAGEMENT
Public management studies consistently find that measures of management quality and the
use of certain management practices are positively related to government performance.
However, the results of these studies often do not provide useful, generalizable, guidance
about what practitioners should do to improve management practice. Unfortunately, management quality measures are frequently based on the subjective opinions of employees
within the organizations studied or they have been too narrowly defined for a particular
organizational or programmatic context to be of general applicability (Brudney, O’Toole,
and Rainey 2000; Donahue 2004; Meier and O’Toole 2002; Nicholson-Crotty and O’Toole
This article would not have been possible without the support and guidance of Trevor Brown at the Glenn School at
The Ohio State University. I also thank Anand Desai and the anonymous reviewers for their suggestions for
improving this study and Jenine Larrabee and Kyle Yaggi for their assistance in improving the quality of the
manuscript. Thanks also to R. Steven Brown and Mary Blakeslee of the Environmental Council of the States; Rachel
Weiss of the National Institute on Money in State Politics; Jim DeMocker and Tom McMullen of the USEPA; Neal
Johnson (now with Pearson Education) and the good people at the Pew Charitable Trust; and the helpful state and
local public servants in 47 states, for helping me to obtain and understand the data for this analysis. Address
correspondence to the author at alexander.heckman@franklin.edu.
doi:10.1093/jopart/mur068
ª The Author 2012. Published by Oxford University Press on behalf of the Journal of Public Administration Research
and Theory, Inc. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
ABSTRACT
2
Journal of Public Administration Research and Theory
STUDY OVERVIEW
This study uses the GPP management measure to test management’s impact on environmental outcomes—a wholly different context than past studies using the GPP. The specific
questions addressed in the study are:
1) Does state management quality impact air pollution control outcomes?
2) What is the impact of management quality on air pollution control outcomes relative to other
state-level factors related to problem severity, resources, and the political environment?
The study incorporates design elements and data that make it well-suited for answering these questions. The important study elements include:
Model Specification: The model specification was developed using the Mazmanian-Sabatier
implementation model (MSIM) illustrated in Figure 1, which is a well-established policy
implementation model.
Outcome Measures: The analysis uses two different measures of performance outcomes
including a unique outcome measure that estimates reductions in air pollution emissions due
to national Clean Air Act (CAA). This second measure better reflects the outcomes sought by
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
2004; O’Toole, Meier, and Nicholson-Crotty 2005; Selden and Sowa 2004). Additionally,
management measures used in the literature are often opaque as to what constitutes good
management. For instance, Meier and O’Toole (2002) used the residual from a regression
model of a superintendent’s salary as a measure of managerial quality and found that good
management had a positive impact on student learning outcomes. Although findings from
such studies may support the notion that good management produces good performance,
they shed little light on general steps for improving public management practice and policy
implementation outcomes.
The lack of concrete, yet general, measures of management quality represents a major
obstacle to producing pragmatic insights about the public management-government
performance relationship (Boyne 2003, 2004; Boyne et al. 2005). Although Meier and
O’Toole (2002) have stated that this measurement challenge is ‘‘intractable,’’ developing
more useful, general measures of management quality is essential if public administration
research is to produce practical knowledge for improving public management practice that
results in better government performance.
A significant exception to the lack of general and concrete measures of management
quality is the management grades assigned to the 50 states in 1999, 2001, 2005, and 2008
by the Government Performance Project (GPP), which is currently managed by the Pew
Center on the States. The GPP management capacity measure provides detailed, general,
criteria that clearly define good management practice. The GPP grades represent the best,
criteria-based, multidimensional measure that is regularly applied to a large number of
governments in the United States (Borins 2005; Ingraham, Joyce, and Kneedler 2003).
Studies using the GPP suggest that it is a promising way to operationalize the concept
of management quality in a general way that provides practical insights into what constitutes good management practice (Burke and Wright 2002; Coggburn and Schneider 2003;
Donahue, Selden, and Ingraham 2000; Knack 2002).
Heckman
Management Quality and Its Impact on Government Performance
3
Figure 1
Mazmanian and Sabatier Implementation Success Model
Causal Measures and Data: The analysis also uses measures of management quality and
interest group influence that more accurately capture these concepts than similar studies (e.g.,
Lester, Franke, and Bowman 1983). Also, the study includes better data on the total amounts
spent within each state on air pollution control efforts. The data were collected by the author
and provide the most accurate estimates available of these expenditures.
Comparative, Large-N Analysis: Two different measures of air pollution control outcomes
affected by the same causal variables are modeled, which enables a comparative analysis of
the empirical results. Also, air pollution outcomes for 47 states are modeled, providing
generalizable results not found in many studies examining government performance,
particularly studies on policy implementation.
Together, these elements create a unique study that sheds new light on the
management-performance connection and the causal factors affecting national air pollution
control outcomes.
CONTEXT FOR ASSESSING MANAGEMENT IMPACT
National air pollution policy provides a useful context for assessing the impact of state
management quality on performance outcomes because it provides a common policy
design in regards to air pollution control, but state-level implementation efforts and characteristics are critical determinants of the extent to which policy goals are achieved
(Woods, Konisky, and Bowman 2009). The CAA is the primary air pollution control
legislation in the United States.
Under the CAA, the US Environmental Protection Agency (USEPA) sets national
ambient air quality standards to ensure basic protection of human health and natural resources, but state and local governments are primarily responsible for enforcement of the
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
policy makers and achieved by policy implementers, than similar studies using only aggregate
emission outcome measures (e.g., Carson, Jeon, and McCubbin 1997; Lowry 1992).
4
Journal of Public Administration Research and Theory
THEORETICAL MODEL AND MANAGEMENT QUALITY MEASURE
The MSIM is one of the most comprehensive and empirically tested policy implementation
theories and was developed to model intergovernmental implementation situations like the
one examined in this study—see Figure 1 (McFarlane and Gruebel 2006; Zheng 2000).
The MSIM states that government performance is the result of three categories of
factors (Mazmanian and Sabatier 1989):
Statutory: The statutory structure of a policy is set in law by elected officials to direct and
constrain policy implementation. According to the MSIM, policy implementation should be
more successful when goals are clearly understood, there is a sound understanding of the causes
of the problem, and sufficient resources are provided to achieve policy goals. Under the CAA,
the policy design and goals are set by the national government and is the same for all states. Total
spending on implementing air pollution control policies is the main statutory variation across the
states. The combined decisions of national, state, and certain local governments determine the
level of resources spent on air pollution control efforts within each state.
Nonstatutory: Nonstatutory factors are not in direct control of national policy makers but impact
the policy outcomes achieved by officials implementing the policy. Relevant nonstatutory
factors include the support or opposition of the general public and influential interest groups for
the policy and the skill of implementing officials or management quality within each state.
1
Appendix 1 summarizes the changes made to the CAA in 1990 that are most relevant to this analysis.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
policy. All 50 states have been delegated responsibility for implementing the CAA provisions and states face financial and other sanctions from the USEPA if they fail to adequately carry out their duties (ECOS 2007; USEPA 2007). States and certain local
environmental agencies implement the CAA by monitoring air quality, inspecting facilities
and enforcing air pollution regulations. Amendments to the CAA enacted in 1990 further
increased the regulatory role states play and strengthened the ability of the states to meet
national air pollution standards (Rabe 2006).1
Evidence of the prominent role states play in carrying the CAA can be seen in USEPA
statistics. Nearly all the air pollution data in the USEPA’s databases are collected and
reported by the states (Brown and Green 2001; ECOS and USEPA 1999). States also bear
most of the financial responsibility for enforcing air pollution control programs. On average, more than $1 billion per year is spent on air pollution control activities within each of
the 47 states included in this study. However, according to the National Association of
Clean Air Agencies (NACAA 2009), less than 25% of all the revenue for state and local
air pollution control activities comes from the national government.
Complying with CAA requirements and avoiding federal sanctions is an important
goal of each state government, including the state environmental agency, the legislature,
and top executive officials. Researchers generally agree that states have been active at carrying out their duties under the CAA, although the level of commitment and competency
varies by state (Rabe 2007). Therefore, differences in states’ spending, management quality, and political environments should determine the air pollution control outcomes they
achieve (Lester 1995; Rabe 2006). This is particularly true for industrial source nitrogen
dioxide (NOX) emissions, which are the outcomes modeled in this study.
Heckman
Management Quality and Its Impact on Government Performance
5
Under the CAA, the ability of management to effectively achieve policy goals depends on good
management practices, sufficient funding for environmental efforts, and political support that
enables states to take the actions necessary to enforce environmental standards that negatively
impact, or are perceived to negatively impact, economic output in a state and may be seen as
overly intrusive policy interventions.
Problem Tractability: The idea that the technology available to deal with the problem situation,
the severity of the problem, and the desired change in the problem state will impact the effectiveness of any policy intervention. No policy can accomplish unrealistic goals that we do not
have the knowledge or methods for achieving.
Empirical Support for the MSIM
Empirical analysis has typically supported the MSIM (Bullock 1981; Goodwin and Moen
1981; Lester and Bowman 1989; McFarlane 1989; Meier and McFarlane 1995; Sabatier
and Klosterman 1981; Zheng 2000). However, these studies typically have not incorporated
measures of all three types of factors in their analysis. Also, the impact of implementer skill
or management quality has not been tested in prior research because a suitable measure was
not available (Mazmanian and Sabatier 1989; McFarlane and Gruebel 2006). Additionally,
implementation research has typically been limited in regards to the generalizable insights
it has produced because such research often analyzes case studies of failures or disasters
rather than conducting quantitative analyses of a large number of similar government entities implementing similar policies (Fox 1987; O’Toole 2000, 2004). This study provides
generalizable results about factors affecting policy implementation by analyzing nearly all
the American states’ implementation of air pollution policy and incorporating state-level
measures for all the major categories defined in the MSIM, including management quality.
The GPP Management Measure
According to Ingraham, Joyce, and Donahue (2003), the GPP grading process is ‘‘the largest and most systematic effort to define and measure good management using rigorous and
consistent techniques.’’ The GPP grades are used by public administration scholars,
policy makers, and the media as indicators of state management quality because they
are based upon transparent, concrete, and detailed criteria developed by expert panels
comprised of academics and practitioners from all levels of government (Borins 2005;
Clifton 2000; Coggburn and Schneider 2003; Donahue, Selden, and Ingraham 2000;
Lewis 2005). The original GPP criteria for each management system are summarized
in Table 1.
The management systems presented in Table 1 were selected ‘‘because they were
judged to be common systems in all levels of government, they have common characteristics that are amenable to comparison, and they compromise a major part of government
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
In this context, states vary widely on both the level of industrial NOX pollution they face and the
extent of change required for bringing them into compliance with the CAA standards. The
technology and techniques for reducing air pollution are available and well understood, so the
level of industrial output is the major difference in problem tractability between states that will
impact industrial NOX emissions and each state’s ability to reduce those emissions.
6
Journal of Public Administration Research and Theory
Table 1
GPP Criteria for Assessing State Management Quality
Management System
Financial
Human Resources
Capital Infrastructure
Managing for Results
Source: Adapted from GPP (2002, Appendix E).
managers’ and organizations’ management activities’’ (GPP 2002). The criteria represent
a synthesis of academic research and measure the application of general management principles that can be accomplished using different practices, depending on the context (e.g.,
Williams 2005). Overall, the GPP criteria emphasize rational decision making, operational
transparency, public accountability, and formal planning for and monitoring of the performance of each management system. According to the GPP, sustained, long-term performance at a high level requires effective maintenance, ongoing coordination, continual
monitoring, and timely improvement to the management systems.
The final GPP evaluation relies primarily on three main data sources: (a) a management survey completed by each state, (b) documents about each state’s management
systems (e.g., budget documents, strategic plans, and human resources policies and procedures), and (c) interviews conducted with state government officials and external stakeholders, such as reporters and representatives from citizen watchdog groups. The GPP
graders obtain and review similar documents and interview people in similar positions
within each state to maximize the comparability of the information obtained and enable
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Information Technology
Quality Criteria/Principles
Multiyear perspective on budgeting
Mechanisms in place that preserve fiscal stability and health
Sufficient financial information available to policy makers,
managers, and citizens
Appropriate control over financial operations
Conduct strategic analysis of present and future HR needs
Able to obtain necessary employees
Able to maintain an appropriately skilled workforce
Able to motivate employees to perform effectively in support of
goals
Civil structure supports ability to achieve workforce goals
Provide information that supports managers’ needs and strategic
goals
Systems form a coherent structure
Conduct meaningful, multiyear planning
Perform adequate training
Can evaluate and validate return on investments
Can procure necessary systems in a timely manner
Systems support the ability to communicate with and serve citizens
Conduct thorough analysis of future needs
Monitor and evaluate projects during implementation
Conduct appropriate asset maintenance
Engage in results-oriented strategic planning
Develop indicators and evaluative data to measure progress toward
accomplishments and results
Use data for policy making, management, and progress evaluation
Clearly communicate results to stakeholders
Heckman
Management Quality and Its Impact on Government Performance
7
an assessment on all the criteria (Ingraham, Joyce, and Kneedler 2003; N. Johnson, personal
communication, April 22, 2010).2
The final grades are based on an assessment of the management survey responses and
documentary evidence by academic experts and an assessment of the interview responses
by the journalists who conduct the interviews. In both cases, multiple academics and journalists individually review the data and assign grades for each state. Then, a team of the
academics and a team of the journalists separately agree on a grade for each state. Finally,
the academic and journalistic groups collaborate to arrive at a final grade. The approach
used to assign the GPP grades seems sound given the ambitious and complicated nature of
assigning management quality grades to all fifty states (Borins 2005; N. Johnson, personal
communication, April 22, 2010).
Studies using the GPP have typically found the expected relationships with other management quality measures and government performance variables. The GPP has been found to
be related to another measure of management quality (Burke and Wright 2002), to various
measures of state quality of life (Coggburn and Schneider 2003), to certain measures
of social capital (Knack 2002), and to local government human resources effectiveness
(Donahue, Selden, and Ingraham 2000).
This study uses the overall state GPP grades, instead of the management system
grades, to focus the analysis on the GPP as a holistic measure of management quality. Also,
the management system grades for each year are not comparable for all the years included
in this analysis because the management system categories were consolidated after 2001
from the five presented in Table 1 to four—Money, People, Infrastructure, and Information.
However, the GPP still applied similar criteria, followed a similar grading process, and
reviewed substantially the same evidence in assigning the overall grades used in this study.
The 2005 changes to the GPP present a challenge for comparing the subcategory grades to
past grades but are only of minimal concern when comparing the overall grades to each
other (Pew Center for the Sates, 2008; N. Johnson, personal communication, April 22,
2010).3
METHODS, DATA, AND HYPOTHESES
The foundation for my empirical examination is a regression analysis, using a model
specification based upon the MSIM, to analyze the impact of management quality and other
key state-level factors on two different measures of state industrial NOX emissions.
Modeling different dependent variables determined by the same causal factors facilitates
comparative analysis of the two models’ regression results, including some assessment of
the robustness of the empirical findings (Pawson 1989).
More detailed information on the grading process can be found at http://www.governing.com/gpp/index.htm, http://
www.pewcenteronthestates.org/initiatives_detail.aspx?initiativeID536072, and in Ammar, Wright, and Selden
(2000).
3
There is a Pearson correlation of .84 between the 1999 and 2001 overall grades and correlation of .81 between the
2001 and 2005 overall grades, suggesting relative stability and continuity in the overall grades over time.
2
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Empirical Studies Using the GPP
8
Journal of Public Administration Research and Theory
4
A similar program was initiated in the late 1990s for NOX emissions trading, but the implementation was too late to
impact the outcomes examined in this analysis. Also, the program focused primarily on NOX emissions from electric
utilities, which are excluded from this analysis.
5
External assessments of the EPA’s effort to measure the costs and benefits of the CAA have not questioned the
accuracy of its emission estimates. However, scholars have criticized the projected costs and benefits resulting from
these estimated emissions reductions (see Freeman 2002, Lutter and Belzer 2000, and Portney 2000).
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Industrial NOX emissions, excluding emissions from power plants, are analyzed
because these pollution levels are most likely to be impacted by state implementation efforts.
In contrast, pollutants, such as sulfur dioxide, are more likely to be affected by federal cap and
trade programs and utility regulation.4 Similarly, mobile source pollutants are mostly
affected by federal enforcement of national regulations, such as compliance with automobile fuel efficiency standards. Focusing on industrial NOX emissions also minimizes the
problem of cross-state pollution affecting the findings (USEPA 2001). Therefore, analyzing
NOX industrial point source emissions should provide the most favorable context for discerning the impact of state-level factors on these air pollution control outcomes.
Additionally, NOX pollution is an important policy problem because it negatively
impacts public health by contributing to the formation of ozone and particulate matter.
Reducing ozone levels and particulate matter was a priority of the 1990 CAA because
of the severity of the problems caused by these pollutants (USEPA 1999).
The first dependent variable is tons of industrial point source NOX emissions per state.
An aggregate emissions measure such as this, often standardized by population or some
other factor, is one of the most common air pollution control outcome measures used in
this type of analysis (e.g., Carson, Jeon, and McCubbin 1997; Lowry 1992; Stern 1998).
Table 2, which presents the descriptive statistics for the dependent and independent variables, shows that most states produce industrial, point-source NOX emissions of less than
115,000 tons.
Also, the distribution of NOX emissions is skewed above the mean, so the log of the
emission variable is used in the analysis to bring the distribution close to normal, which
helps to ensure more accurate coefficient estimates (Stern 1998).
Although an aggregate emissions measure is a commonly used dependent variable in
analyses examining state and national environmental performance, such variables are not
necessarily the best measure of the policy goals because environmental policy is ultimately
focused on reducing emissions. Therefore, a change in emissions measure is a better outcome variable to use when analyzing the impacts of pollution control efforts because it
better reflects policy goals (Ringquist 1993).
To address this issue, the second dependent variable is emissions reductions per state
as a result of the 1990 CAA Amendments (ERNE). This variable measures the estimated
reductions in state-level emissions between 1990 and 2000 after the CAA was amended in
1990.
The 1990 CAA law required the USEPA to issues a series of reports on the costs and
benefits of the policy (USEPA 1997). The second of these reports was issued in 1999 and
estimated the national impact of the 1990 CAA Amendments on pollution emission between the years 1990 and 2000. These estimates are the foundation for deriving the second
dependent variable used in this study.5
In the report on the benefits and costs of the 1990 CAA Amendments, the USEPA
(1999) forecasted pollutant emission levels for NOX emissions under two scenarios.
Heckman
Management Quality and Its Impact on Government Performance
9
Table 2
Descriptive Statistics for Model Variables
Variable
Mean
SD
Minimum
Maximum
2.6
0.5
1.0
3.7
50
15
15
86
0.14
2.20
0.00
0.96
$26.1
$7.3
$2.1
$594.2
$27,802
$30,712
$628
$169,395
$7,824
$13,093
$22,837
$77,615
51,918
61,789
386
452,007
25,453
26,148
278,344
45,221
One scenario assumed that the CAA Amendments were not passed in 1990 and that no
additional control requirements would be passed by the national, state, or local governments between 1990 and 2010. The second scenario estimated emissions based on the forecasted impact of the CAA with the 1990 Amendments. In short, USEPA estimated the
impact of the actual policy and the impact under the counterfactual.
I used the USEPA emission estimates and actual data on NOX industrial point source
emissions for the years 1990 and 2001 to calculate each state’s reduction in emissions as a
result of the CAA. A key assumption of the ERNE calculation is that the each state’s relative share of national emissions in 1990 would remain the same if more stringent standards
and enforcement mechanisms had not been introduced by the 1990 CAA Amendments.
Based on this assumption, the actual 1990 relative shares can be used to apportion the
USEPA national projection for 2000 under the non-CAA Amendments scenario. Similarly,
each state’s share of actual 2001 emissions is used to apportion out its share of the USEPA
NOX emissions projections for industrial point sources in 2000 under the CAA Amendments scenario.6
As shown in Table 2, most states were estimated to have reduced NOX industrial
emissions by less than 21,000 tons. Five states were projected to have had increases in
NOX emissions after implementation of the 1990 CAA Amendments—Arkansas, Georgia,
Hawaii, Kentucky, and Mississippi. The distribution of the ERNE variable is approximately
normal, so no transformation of this variable was necessary.
The dependent variables are not standardized by state population or some other factor
because it is the aggregate pollution levels that are of concern for public officials. It is the
6
More detail on the ERNE calculation and an example state calculation is presented in Appendix 2.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Nonstatutory
GPP Management Grade
(1999, 2001, 2005)
Level of Citizen Liberal Ideology
(1999, 2001, 2004)
Environmental Interest Group Influence
(1999–2000, 2000–01, 2003–04)
Statutory
Air Pollution Control Spending
(in millions) (1999, 2001, 2004)
Problem Tractability
Real Manufacturing GDP
(in millions) (1999, 2001, 2004)
Increase in Value of Manufacturing
GDP from 1990 to 2000 (in millions)
Dependent
NOX Emissions
(in tons) (2000, 2002, 2005)
Estimated Reduction in
NOX Emissions
10
Journal of Public Administration Research and Theory
Table 3
Model Variables, Research Hypotheses, and Data Sources
Variable
Lag of Dependent
Variable
Data Source
Negative
http://www.pewcenteronthestates.org
Negative
Negative
Berry et al. (1998)
http://www.followthemoney.org
Negative
Author’s original data collection and
Environmental Council of the States
Positive
http://www.bea.gov/regional/gsp/
Negative
http://www.bea.gov/regional/gsp/
Positive in AE Model
Negative in ERNE Model
USEPA National Emissions Inventory,
http://epa.gov/airdata/neidb.html
Note: AE, Aggregate Emission model.
overall levels that create the health and safety impact and the goal of policy is to reduce
aggregate levels of pollutants not per capita reductions. However, the analysis does account
for size effects on the dependent variables by including a lag of the dependent variable in
the model for each outcome measure. I mitigate simultaneity issues in the emissions model
analysis by using prior years’ values for all the independent variables, which should produce more accurate coefficient estimates. Simultaneity is not a concern with the ERNE
model because it is a measure of change (Ringquist 1993; Stern 1998).
Causal Variables and Expected Relationships
Table 3 shows the independent variables used in the analysis and the hypothesized relationships with the dependent variables.
Nonstatutory Variables
1) The GPP grades are used to measure management quality. As shown in Table 2, almost all
states have C-level management or better, according to the GPP grades. The average GPP
grade is 2.6 or just below a B-level. The grade distribution is essentially bimodal with 95% of
states rating a B-level (26 states) or C-level (21 states) grade.
GPP Hypotheses: The GPP variable should have a negative relationship with both
emissions levels and the ERNE because states with better management quality should have
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Nonstatutory
GPP Management
Quality Grade
Citizen Ideology Index
Environmental Interest
Group Influence
Statutory
Air Pollution
Control Spending
Problem Tractability
Real Value of State
Manufacturing GDP
(AE model)
Change in Value of State
Manufacturing GDP
(ERNE model)
Expected Relationship
with Dependent
Variables
Heckman
Management Quality and Its Impact on Government Performance
11
lower emissions and achieve larger reductions, which are measured by larger negative
values.
2) A state-level citizen ideology index developed by Berry et al. (1998) is used to capture the
concept of citizen support for CAA implementation.7 The index determines how liberal
citizens are in each state by calculating the ideology of each state’s Congresspeople based
upon ratings from the liberal group Americans for Democratic Action and the conservative
group Americans for Constitutional Action. The ideology index can range from 0 (extremely
conservative) to 100 (extremely liberal).
3) The MSIM concept of interest group support is captured with a measure of environmental
interest group influence. The variable is the proportion of pro-environmental interest group
campaign contributions to successful state-level candidates relative to total interest group
campaign contributions to successful candidates from both pro-environmental and promanufacturing industry groups. The larger the value of the spending ratio variable the more
influential environmental interest groups are in the state relative to industry interests. This is
a better measure of the actual relative influence of interest groups than prior studies, which
have typically only examined potential influence of environmental groups based upon state
membership in certain environmental groups (Lester 1995).
Table 2 shows that environmental interest groups are not very influential relative to
industry groups, although there is wide variation across states. Environmental groups’ contributions to winning state-level candidates represented an average of only 14% relative to
total campaign contributions made during the three election cycles used in the analysis. In
most states, environmental contributions did not exceed 34% of total contributions.
Interest Group Hypotheses: States with relatively more influential pro-environment
interest groups should achieve lower emission levels and greater emission reductions,
so this variable should have a negative relationship with both dependent variables.
7
Berry et al. reported on the measure in 1998 and subsequently made available updated measures for states through
the year 2004 in the revised 1960–2004 citizen ideology series. Details on the derivation of the ideology index and the
updated figures used here can be found at http://www.icpsr.umich.edu/icpsrweb/ICPSR/.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Berry et al. have demonstrated the reliability and validity of the measure through
empirical testing of many of the assumptions upon which the measure is based and testing
expected relationships with other relevant phenomenon. This variable avoids the pitfall of
measures commonly used in implementation research that merely quantify the partisan
composition of state or national officials as a measure of state ideology (e.g., Lester et al.
1983). This is problematic because the meaning of partisan ideology is not consistent
across states (Berry et al. 1998). For example, Democrats in Texas are more likely to
be ideologically similar to Republicans in Massachusetts rather than Democrats.
As shown in Table 2, states are, on average, somewhat moderate ideologically with
a mean of 47 out of a possible 100 on the ideology index. Nearly 60% of states have a score
in the 40s or 50s.
Citizen Ideology Hypotheses: More liberal state populations are expected to demand
more stringent environmental enforcement, and so there should be a negative relationship
between the citizen ideology variable and the dependent variables.
12
Journal of Public Administration Research and Theory
Statutory Resources Variable
Problem Tractability Variables
The model includes two problem tractability variables. The first variable is a measure of the
value of each state’s manufacturing gross domestic product (GDP). The value of production
in the manufacturing sector in each state should have a direct impact on emission levels and
reductions that can be achieved. In the aggregate emissions model, the variable is the total
real value of each state’s manufacturing GDP. In the ERNE models, the manufacturing
GDP variable is the change in manufacturing value between 1990 and 2000.
The second variable is a lag measure of the dependent variable, which also captures
the extent of each state’s pollution problem. In the emissions model, the variable is tons of
NOX emissions from the prior year, and in the ERNE model, it is the tons of NOX emissions in each state in 1990. The lag measures should help account for size effects that can
explain variation arising from the use of the aggregated dependent variables.
With all of the problem tractability variables, the aggregate levels of manufacturing
output are used, rather than standardizing by population or some other basis, because it is
the total amounts that create the negative environmental impacts.
Two-thirds of states having manufacturing output valued at less than $60 billion. The
average increase in manufacturing GDP is more than $7.8 billion with two thirds of states
generating less than $21 billion in increased manufacturing output.
Manufacturing Output Hypotheses: More manufacturing should mean higher
emissions, and so there should be a positive relationship with state emissions levels. The
greater the increase in manufacturing, the more difficult it would be to reduce emissions.
Therefore, manufacturing GDP should have a positive relationship the ERNE variable.
Lag Variables Hypotheses: The prior year emissions lag measure should have a positive relationship with the aggregate emissions outcome variable, but the ERNE outcome
8
Appendix 3 provides more detail on the process used to collect the spending data used in this analysis.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Total state spending on air pollution control activities is included as a measure of the
financial resources used to achieve policy goals.8 An aggregate measure of spending, rather
than a per capita measure, is used because aggregate spending is a better measure of
capacity. Following the logic of Ringquist (1993), pollution control is an ‘‘aggregate problem’’ that requires a certain minimum level of spending to effectively generate results,
regardless of the amount of pollution to be reduced. It is the absolute level of spending
that best captures the fiscal capacity of a state to carry out the activities required to effectively reduce pollution rather than some standardized measure such as spending per capita.
For example, a small state may have quite high per capita spending, but its overall spending
may not provide sufficient capacity to carry out an effective enforcement program. States
that spend more on air pollution control should have higher capacity to implement the CAA
and achieve greater emission reductions.
Table 2 shows that within each state, an average of more than $26 million is spent
on air pollution control with two thirds of states spending more than $18.5 million and less
than $33.5 million.
Spending Hypotheses: The air quality spending variable should have a negative
relationship with both dependent variables.
Heckman
Management Quality and Its Impact on Government Performance
13
measure should have a negative relationship with the lag measure because states starting
with higher levels of emissions in 1990 should be able to achieve greater reductions.9
FINDINGS AND ANALYSIS
Table 4 presents the results of the regression analysis with the log of NOX emissions as the
dependent variable and Table 5 shows the results of the regression analyses with the ERNE
measure as the dependent variable.10
Management Quality and Other Nonstatutory Variables
Table 4
Regression Results with Log of Industrial Source NOX Emissions as Dependent Variable (Panel Data
for Years 2000, 2002, and 2005)
Independent Variable
Nonstatutory
GPP Grade
Level of Citizen Liberal
Ideology
Environmental Interest
Group Influence
Statutory
Air Pollution Control Spending
Problem Tractability
Value of State Manufacturing
Lag of Aggregate NOX Emissions
Other
2002 Dummy Variable
2005 Dummy Variable
Constant
Model Statistics
Coefficient
SE
p Value
2.008
2.026
0.13
0.01
.95
.00
.022
2.35
.95
0.00
.02
0.00
0.00
.00
.00
0.16
0.16
0.43
.36
.55
.00
2.000000004
.00002
.000008
2.149
2.097
10.78
Adjusted R2 5 .62
N 5 141
Note: All F tests were significant at 99% confidence level.
The models also include dummy variables for relevant years to account for temporal effects.
The NOX emission analysis was also performed using per capita measures to control for differences in population.
The direction and magnitude of the GPP coefficient was similar in these models; however, the coefficient sign for
citizen ideology, spending, and the value of manufacturing was the opposite of these results. Accounting for a change
in population in the ERNE analysis produced similar results to those presented here.
11
The importance of statistical significance is debatable here since the data represent almost the entire population
rather than a sample. As a result, the analysis focuses on direction and magnitude of the variables and less on statistical
significance. However, variable p values are presented and discussed to allow readers to make their own judgment.
9
10
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
In both the aggregate emissions model and ERNE Model 1, the GPP management grade has
the expected relationship with emissions but is not statistically significant.11 However,
there is a notable difference in the magnitudes of the GPP coefficients between the two
models. Even if one ignores the lack of statistical significance, the GPP coefficient in
the aggregate NOX emissions model is close to zero, suggesting no substantive impact
on overall emissions. In contrast, the GPP coefficient in the ERNE Model 1 indicates
14
Journal of Public Administration Research and Theory
Table 5
Regression Results with Dependent Variable of Projected Emission Reductions for Industrial Source
NOX Emissions from 1990 to 2000 Attributable to the 1990 CAA Amendments
Independent
Variable
Model Statistics
SE
p Value
Model 2
SE
27,743
2550
4,968
212
.13
.01
213,495
2608
6,106
211
.03
.01
214,067
15,563
.37
214,496
15,289
.35
2.0000000003
0.00
.04
2.0000000009
.00
.05
.60
0.49
.23
.57
.49
.24
2.19
0.04
.00
2.24
.05
.00
—
—
.0000000003
.00
.13
19,443
.01
72,872
20,176
.00
—
53,145
2
Adjusted R 5 .52
N 5 47
p Value
2
Adjusted R 5 .53
N 5 47
Note: All F-tests were significant at 99% confidence level.
a substantive impact of management quality on the level of emission reductions achieved.
The GPP coefficient in Model 1 is nearly 15% of the mean NOX emissions, suggesting
a substantive impact of management quality on the emission reduction outcomes.
The citizen ideology coefficient is statistically significant and has the expected
negative relationship with emissions in both the aggregate emissions model and ERNE
Model 1; however, the magnitude of the coefficient is notably different in the two
models. A one-point increase in the citizen ideology measure is associated with
a .02% decrease in NOX emissions. This means that a 10-point increase in the ideology
index (10% on the 100-point scale) would be associated with a 2% decrease in NOX
emissions. The ideology coefficient magnitude in the ERNE model is notably greater
at about 1% of the mean of total NOX emissions. Therefore, a 10-point increase in
the citizen ideology index would be associated with a reduction equivalent to 10% of
total NOX emissions. This indicates a substantive difference between the two variables’
impacts on emissions.
The coefficient for the interest group influence measure is positive and not statistically significant in the emissions model but is negative and statistically significant in
ERNE Model 1. Additionally, there is a notable difference in the magnitude of the interest
group influence coefficient between the two models. Even if one ignores the lack of
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Nonstatutory
GPP Grade
Citizen Liberal
Ideology
Environmental
Interest Group
Influence
Statutory
Air Pollution
Control Spending
Problem Tractability
Change in State
Manufacturing
Value
NOX Emissions
(1990)
Other
Management
Spending Per
Capita
Constant
Model 1
Heckman
Management Quality and Its Impact on Government Performance
15
statistical significance, the interest group coefficient in the aggregate NOX emissions
model is close to zero, suggesting no substantive impact on overall emissions. Conversely, a 10% increase in interest group influence is associated with an emission reduction equal to more than 25% of the mean in aggregate emissions. One plausible
explanation for the difference in the coefficient sign between the two models (ignoring
the lack of statistical significance) is that pro-environment interest groups are stronger in
states with worse emission problems, but they influence states to make greater NOX pollution reductions.
Lastly, the coefficient of the interest group influence variable in the ERNE Model 1
is nearly double the magnitude of the GPP coefficient, suggesting politics affect outcomes
more than management.
The spending coefficient is statistically significant and has the expected negative relationship with both outcome measures. However, the impact of spending is modest or negligible
in both the aggregate emissions model and ERNE Model 1. A 1 standard deviation increase
in spending is associated with a 3% lower level of NOX emissions and a 0.02% reduction in
emission levels.
One plausible explanation for the limited impact of spending increases is that the
federal government policy design and USEPA oversight causes states to spend at high
enough levels that spending differences between states have negligible impact on pollution
outcomes. Another plausible explanation for the negligible impact of spending is that
higher spending by itself does not reduce emissions. Researchers such as Boyne (2003)
have theorized that spending must be coupled with good management to have a substantive
and positive impact on performance.
I examined this possibility by including a management-spending interaction variable
in the ERNE model. The interaction variable is calculated by multiplying the GPP and air
quality spending per capita for each state, therefore weighting spending more heavily when
it is done in well-managed states. As shown in Table 5, the management-spending interaction variable is significant, associated with greater emissions, but has a negligible impact
on emissions based upon the magnitude of the coefficient.
The inclusion of the management-spending interaction variable also does not impact
the total variation explained relative to ERNE Model 1 and does not have any notable impact on the citizen ideology, the environmental interest group influence or the problem
tractability variables, nor on the signs of the spending and GPP coefficients. However,
the magnitude of the spending coefficient in Model 2 is three times greater than in Model
1, though the impact is still quite modest.
Most interesting is the impact on the significance and magnitude of the GPP
coefficient as a result of including the management-spending interaction variable.
The GPP measure is not statistically significant in Model 1, but it is significant in
Model 2 and the magnitude of the coefficient is nearly double that of the GPP coefficient
in Model 1.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Statutory Resources—Spending
16
Journal of Public Administration Research and Theory
Problem Tractability
The problem tractability variables have the expected relationship with NOX emissions in
all the models; however, problem tractability is not statistically significant in the ERNE
models. Nonetheless, the magnitude of the coefficients in all the models is quite small,
suggesting a negligible impact of problem tractability on these outcome measures. This
is not too surprising given that the methods for reducing pollution from industrial point
sources are well understood and the technology for doing so is also readily available. Therefore, the problem is quite tractable and achieving policy goals is technically feasible, but
whether the methods will be applied and the technology used effectively depend largely
upon good management and the financial and political support for implementation.
Overall, the findings indicate a consistent yet modest impact of citizen ideology and air
pollution control spending on NOX emissions. More liberal citizens and more spending
seem to contribute to modestly lower NOX emissions and greater reductions in emissions
under the CAA. The problem tractability variables were not consistently significant
and their impact was negligible in all models based upon the coefficient magnitudes.
This suggests that problem tractability is not a major factor in determining these
outcomes.
How one interprets the impact of management quality and environmental interest
group influence variables depends, in large part, on one’s views on the importance of
the statistical significance of the variables, the proper model specification, and the best
outcome measure. If statistical significance is critical, then neither management quality
nor interest group influence help explain NOX emission outcomes, except that management
quality is significant and has a substantive impact if the best model specification is ERNE
Model 2. This change in significance and magnitude suggests that analyses designed to
examine relationship between management and performance may need to account for
the interaction of management quality and spending.
The ERNE Model 2 would seem to be the best model because it includes both the best
outcome measure and the management-spending interaction variable, which captures the
critical insight that good management is required to make effective use of available financial resources (Boyne 2003).
Similarly, if one is not concerned about statistical significance and considers the
ERNE variable to be a better measure of the policy outcomes sought, then environmental
interest groups appear to a have a rather substantive impact on the ability to achieve emission reductions in a state. Otherwise, interest groups seem to have little or no impact on
NOX emissions.
RESEARCH AND POLICY IMPLICATIONS
The specific findings of this analysis have broader research and policy implications when
examined within the context of prior research into public management and government
performance, particularly if one emphasizes the results arising from the comparative analysis of the ERNE models. These findings have notable implications for public administration research and policy.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Summary of the Regression Results
Heckman
Management Quality and Its Impact on Government Performance
17
Good Management and Government Performance
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
The findings of this analysis add to the management, government performance, and implementation literature that provide evidence that management quality and the political environment have a substantive impact on government performance outcomes (e.g., Boyne
2003; Boyne et al. 2005; Lester and Bowman 1989). The impact of management on performance outcomes is important and can exceed the impact of spending and the constraints
created by the severity of the problem. Political factors such as citizen ideology and interest
group influence seem to have an even greater impact on performance outcomes than management quality. Still, improving management is a viable strategy for practitioners and
policy makers seeking to improve government performance.
Because this analysis used the GPP grades as the measure of management quality, the
study also sheds light on the management practices that constitute good management. The
GPP criteria define relatively specific practices that constitute good management including
rational decision making, operational transparency, public accountability, and formal planning for and monitoring of the performance of each management system. This transparent
and concrete definition of management should be helpful to policy makers and practitioners
seeking to improve public management.
Additionally, the detailed and transparent nature of the GPP measure can facilitate
useful dialogue about the nature of good management, which can improve both research
and practice. Whatever one thinks of the validity of the GPP, measures like it provide a starting point for professional dialogue about how to define and measure good management.
Discussions about what exactly constitutes good management practice should help scholars
to better understand the impact of management and help practitioners be more reflective
professionals. The result should be management research that makes a more practical contribution to improving management practice and government performance.
Although the GPP is a promising measure, it has limitations that relate to the key
limitations of this study. First, the practical guidance based on this study is not as specific
as it could have been if the GPP subsystem grades had been able to be incorporated in the
study. Certainly, a subsystem analysis would be worthwhile in future research.
More fundamentally, some inconsistencies in the empirical findings in this study may,
in part, be explained by the limitations of the GPP as a measure management quality. The
GPP is primarily a measure of management capacity, and it does not effectively capture the
other major aspect of management as defined in the literature—the quality of the daily
execution of policy by states and the execution of agency-level policies within the state
environmental agencies (Kotter 1990, Mintzberg 1975; Tsoukas 1994).
However, the inability to account for execution is not limited to the GPP or this study,
but is one of the fundamental obstacles to developing any quantitative measure of management quality that seeks to define good management practice. Developing formalized
policies, procedures, and structures is the part of good management practice that can
be relatively easily observed and evaluated in retrospect, but determining how well structures are utilized, policies are carried out, and procedures are executed is not amenable to
after-the-fact examination.
This challenge suggests both opportunities for improving the measurement of
management quality and limitations on such quantification efforts. The GPP project
and other scholarly efforts to quantify management quality need to better account for
18
Journal of Public Administration Research and Theory
management execution in order to develop better measures and a better understanding of
the management-performance connection.
However, because management execution is action oriented, communication based,
and ephemeral, there are limits to our ability to measure and quantify it (Kotter 1990;
Mintzberg 1975; Sull and Spinosa 2007). As such, the study of management execution
is more conducive to qualitative research approaches that rely on direct observation
and interpretation rather than on after-the-fact quantification using formal documents or
secondary data. Therefore, a mixed-methods approach may be the best way to understand
good management and its impact on performance (O’Toole 2004).
Analyzing Government Performance
Accounting for Management-Spending Interaction
Boyne (2003) has detailed the contradictory findings in the government performance literature as to the impact of spending on performance and concluded that this discrepancy
may be a result of differences in measurement. Specifically, some studies operationalize
resources using aggregate measures of spending while others examine real resources
actually used to carry out activities designed to achieve outcomes (e.g. people and equipment). Boyne theorizes that spending variables measure financial capacity, which does not
directly determine results. Instead, variables that measure real resources capture the actual
use of that financial capacity, which should directly impact outcomes.
This study provided evidence for Boyne’s claim by including in one of the model
specifications a management-spending interaction variable to account for the notion that
it is the financial capacity available for achieving a policy goal and how effectively these
resources are used that determines performance outcomes. This model produced results
most aligned with the MSIM hypotheses and other studies into the impact of management
on performance outcomes.
Selecting Outcome Measures
The findings of this study were also notably impacted by the selection of the outcome
measure. For example, the model that used aggregate pollution levels as the measure
of performance outcomes did not show a significant or substantive impact of management
quality on pollution. Conversely, the model that used the estimated reduction in pollution
emissions as the performance outcome measure did show substantive impact of management on performance.
This result may suggest that the study findings are unreliable, but I contend that they
illustrate the misleading findings that can result from selecting an outcome measure not
closely related to policy goals. For example, studies seeking to explain the impact of policy
and politics on pollution have often measured performance outcomes using a per capita
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
This study also provides insights into how different measures and model specifications may
help explain the contradictory findings often found in public administration research into
the causes of government performance (Boyne 2003). For example, this analysis provided
evidence that inclusion of a management-spending interaction variable and the selection of
the performance outcome measure have a notable effect on the empirical findings.
Heckman
Management Quality and Its Impact on Government Performance
19
emissions variable. However, governments are not trying to reduce per capita emissions but
seek to reduce total pollution from existing levels (Ringquist 1993). Therefore, a measure
of pollution reduction outcomes better captures the goals of air pollution control efforts.
In this study, the model using the emissions reductions achieved as a result of national
air pollution policy produced results more in line with theory and other studies into the
impact of management on performance outcomes. This finding supports the notion that
better outcome measures produce more accurate results.
Applying a Comparative Approach
CONCLUSION
This study incorporated three key elements: (1) a management quality measure based upon
criteria that clearly define good management practice, (2) a performance outcome measure
that accurately captured policy goals, and (3) a comparative analytical approach based upon
two different outcome measures that should be impacted by the same causal factors.
Together, these elements shed new light on the management-performance connection
and the causal factors affecting policy outcomes. For example, the comparative analysis
illustrated the importance of accounting for the interaction of management and spending
when modeling government performance outcomes.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
The findings discussed above were arrived at by conducting a comparative analysis using
two different performance outcomes that were expected to be impacted by the same causal
factors. The comparative approach facilitated an assessment, albeit limited, of the robustness of empirical findings and produced inconsistencies and discrepancies in the empirical
results, which spurred additional questions and analysis (Pawson 1989).
For instance, the citizen ideology and spending variables were consistent across all
models in terms of their sign, statistical significance, and magnitude. This provides a high
level of confidence in the findings related to these variables.
Conversely, inconsistent results in the statistical significance, direction, or magnitude
of other variables’ coefficients raised questions that required explanation or additional analysis. For example, interest group influence was associated with higher overall emission
levels but also greater reductions in emissions. This discrepancy demanded an explanation
and spurred the insight that pro-environmental interest groups become more influential in
states with more severe pollution problems, and in turn, these groups are more effective at
causing reductions in emissions.
This insight could not have been produced by an analysis that modeled emissions as
the only outcome variable. Instead such an analysis would have suggested an opposite finding—stronger pro-environment groups are ineffectual because they are associated with
higher emission levels. In fact, this is not an uncommon finding in studies looking at
the effect of interest groups on policy outcomes (e.g., Lester and Bowman 1989).
Overall, these examples illustrate how applying a comparative approach can improve
public administration research by providing a way to assess the robustness of empirical
findings and by generating discrepancies that spur additional analysis. In this case, the result was a clearer and more sophisticated understanding of the causal factors and relationships that affect government performance.
20
Journal of Public Administration Research and Theory
The study also examined the benefits that can be gained by including these elements in
public management and government performance research. For example, using management measures that are based upon assessments of clearly defined management practices
should produce more accurate and useful research results. Additionally, developing measures based upon transparent and concrete criteria will facilitate constructive dialogue about
what constitutes good management practice and inform academics’ and practitioners’
endeavors.
In short, future research could benefit from building upon the types of measures
and methods used in this study because such research would generate more accurate and
practical insights into good management practice and its impact on policy implementation
outcomes.
Trevor Brown at the Ohio State University Glenn School of Public Affairs.
Appendix 1
Amendments enacted in 1990 to Titles I, V, and VII of the CAA enhanced the responsibility and
authority of states to reduce air pollution from industrial point sources (McCarthy, et al. 2007).
Changes most relevant for this analysis include:
1. All states had to develop implementation plans for meeting air pollution standards both in areas
in nonattainment with the USEPA standards and for preventing significant deterioration of air
quality in areas that were in attainment. New classifications were created for metropolitan areas
deemed to be in ‘‘nonattainment’’ with one or more air quality standards, and states’ implementation plans were required to outline strategies for bringing such areas into attainment
within specific time frames established by the law.
2. In nonattainment areas, new facilities and major modifications to existing facilities were only
to be approved by states if offsetting emission reductions could be achieved.
3. All major air pollution sources, and certain nonmajor sources in nonattainment areas, were
required to obtain annual operating permits that specified the level and type of pollutants a
source may emit. States were charged with implementing the permit program including
monitoring and enforcing compliance with permit requirements.
4. Penalties for purposeful violations of the CAA were increased, a grace period for ceasing
violations without penalties was removed, and environmental agencies were authorized to
assess administrative penalties.
Appendix 2
The basic formula used by the USEPA to forecast the impact of the 1990 CAA Amendments on
pollution levels was 2000 NOX Emissions 5 1990 Industrial Point Source NOX Emissions
Economic Growth Factor Pollution Reduction Factor.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
FUNDING
Heckman
Management Quality and Its Impact on Government Performance
21
Both scenarios employed an economic growth factor that is based upon federal government
projections of the growth in production from each industrial facility and growth in state
populations. The projections assumed that the geographic distribution of population and
economic activity would be the same under both scenarios. The USEPA also estimated total
national NOX emissions in a counterfactual scenario in which all national and state policies in
existence prior to the 1990 CAA Amendments remained unchanged (the non-1990 CAA
Amendments projection).
An example calculation of the ERNE for Pennsylvania is presented below to illustrate how this
variable was derived for each state:
1. Pennsylvania’s Estimated 2000 Emissions without the 1990 CAA Amendments:
B. Actual 1990 national industrial NOX emissions = 3,157,647 tons
C. Pennsylvania’s share of actual 1990 industrial NOX emissions = 188,599 tons or 6.0%
D. Estimated emissions in year 2000 without CAAA: 3,173,300 .06 = 190,398
2. State’s Estimated 2000 Emissions with 1990 CAA Amendments:
A. USEPA national NOX emissions estimate for year 2000 with CAAA = 2,060,400 tons
B. Actual 2001 national industrial NOX emissions = 2,959,829 tons
C. Pennsylvania’s share of actual 2001 emissions = 114,888 tons or 3.8%
D. Estimated emissions in year 2000 with CAAA: 2,060,400 .038 = 78,295
3. Pennsylvania’s ERNE: 78,295 2 190,398 = 2112,103
Appendix 3
The total air quality spending figure includes spending within a state by both state and local
agencies using revenue from all sources (i.e., national, state, and local). Local agencies in 27
states have significant CAA enforcement responsibilities that have been delegated to them by the
state or directly by the USEPA.
I collected data for the air quality spending measure from different sources depending upon the
state and local air agency. Initially, state and local Web sites were reviewed for annual reports
from the state environmental agency and relevant budget and financial documents. When these
documents did not provide the necessary data or the data did not clearly break out air pollution
control spending, the author, with some assistance from NACAA, contacted officials in the state
or local environmental agency and relevant budget agencies to obtain spending data or clarifications regarding published data.
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
A. USEPA national NOX emissions estimate for year 2000 without CAAA = 3,173,300
tons
22
Journal of Public Administration Research and Theory
For a few states, I also relied on spending data from the Environmental Council of the States
after assessing its likely accuracy. In some cases, data for a particular year had to be estimated,
based upon historical data and historical rates of growth in air quality spending, because I could
not obtain actual figures.
Ultimately, the total state and local spending by state were determined for 47 states. Illinois,
Montana, and New York are excluded from the analysis because data to calculate accurate air
pollution control spending figures could not be obtained.
REFERENCES
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Ammar, Salwa, Ronald Wright, and Sally Selden. 2000. Rating state financial management: A multilevel
fuzzy-based system. Decision Sciences 31:449–81.
Berry, William D., Evan J. Ringquist, Richard C. Fording, and Russell L. Hanson. 1998. Measuring citizen
and government ideology in the American states, 1960–93. American Journal of Political Science
42:327–48.
Borins, Sandford. 2005. Grading the graders. [Review of the book Government performance: Why
management matters]. Journal of Public Administration Research and Theory 15 (3):463–66.
Boyne, George A. 2003. Sources of public service improvement: A critical review and research agenda.
Journal of Public Administration Research and Theory 13:367–94.
———. 2004. Explaining public service performance: Does management matter? Public Policy and
Administration 19:100–17.
Boyne, George A., Kenneth J. Meier, Laurence J. O’Toole, and Richard M. Walker. 2005. Where next?
Research directions on performance in public organizations. Journal of Public Administration
Research and Theory 15:633–39.
Brown, R. Steven, and Valerie Green. 2001. State environmental agency contributions to enforcement and
compliance. Environmental Council of the States.
Brudney, Jeffrey L., Laurence J. O’Toole, and Hal G. Rainey. 2000. Chapter 14: Concluding perspectives.
In Advancing public management: new developments in theory, methods, and practice, ed. Jeffrey L.
Brudney, Laurence J. O’Toole Jr., and Hal G. Rainey, 235–52. Washington, DC: Georgetown Univ.
Press.
Bullock, Charles S. 1981. Implementation of the equal education opportunity act: A comparative analysis.
In Effective policy implementation, ed. Daniel Mazmanian and Paul Sabatier. Lexington, MA:
Lexington Books.
Burke, Brendan F., and Deil S. Wright. 2002. Reassessing and reconciling reinvention in the American
states: exploring state administrative performance. State and Local Government Review 34:7–19.
Carson, Richard T., Yongil Jeon, and Donald R. McCubbin. 1997. The relationship between air pollution
emissions and income: US data. Environment and Development Economics 2:433–50.
Clifton, Diane B. 2000. AU outreach report challenges state government’s poor grade. Auburn University
News, October 24. http://www.auburn.edu/administration/univrel/news/archive (accessed July 9,
2007).
Coggburn, Jerrell D., and Saundra K. Schneider. 2003. The relationship between state government
performance and quality of life. International Journal of Public Administration 26:1337–54.
Donahue, Amy Kneedler. 2004. The influence of management on the cost of fire protection. Journal of
Policy Analysis and Management 23:71–92.
Donahue, Amy Kneedler., Sally Coleman Selden, and Patricia W. Ingraham. 2000. Measuring government
management capacity: A comparative analysis of city human resources management systems.
Journal of Public Administration Research and Theory 10:381–411.
ECOS and USEPA. 1999. State environmental data in EPA’s National System. Washington, DC: ECOS
and USEPA.
Environmental Council of the States (ECOS). 2007. Delegation by environmental act. http://ecos.org/
section/states/enviro_actlist (accessed May 21, 2010).
Heckman
Management Quality and Its Impact on Government Performance
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Fox, Charles J. 1987. Biases in public policy implementation evaluation. Policy Studies Review 7:128–41.
Freeman, Myrick, III. 2002. Environmental policy since earth day: What do we know about the benefits
and costs? Agricultural and Resource Economics Review 31 (1): 1–14.
Goodwin, Leonard, and Phyllis Moen. 1981. The evolution and implementation of family welfare policy.
In Effective policy implementation, ed. Daniel Mazmanian and Paul Sabatier, 147–68. Lexington,
MA: Lexington Books.
Government Performance Project (GPP). 2002. Paths to performance in state and local government: A
final assessment from the Maxwell School of Citizenship and Public Affairs. In ed. A. Colonna and
J. Puma. Syracuse, NY: Campbell Public Affairs Institute.
———. 2005. State Grade Report. http://www.pewcenteronthestates.org/report_detail.aspx?id532742
(accessed October 22, 2007).
Ingraham, Patricia W., Philip G. Joyce, and Amy E. Kneedler Donahue. 2003. Government performance:
Why management matters. Baltimore, MD: Johns Hopkins Univ. Press.
Knack, Stephen. 2002. Social capital and the quality of government: Evidence from the States. American
Journal of Political Science 46:772–85.
Kotter, John P. 1990. What leaders really do. Harvard Business Review 68:103–11.
Lester, James P., 1995. Federalism and state environmental policy. In Environmental politics and policy:
Theories and evidence, ed. James P. Lester, 39–60. Durham, NC: Duke Univ. Press.
Lester, James P., and Ann O’M Bowman. 1989. Implementing environmental policy in a federal system: A
test of the Sabatier-Mazmanian model. Polity 21:731–53.
Lester, James P., James L. Franke, Ann O’M Bowman, and Kenneth W. Kramer. 1983. Hazardous waste,
politics, and public policy: A comparative state analysis. Western Political Quarterly 36:257–85.
Lewis, Bob. 2005. Virginia gets nation’s top grade in management. WTOPnews.com, March 31. http://
www.wtopnews.com/ (accessed July 9, 2007).
Lowry, William R. 1992. Dimensions of federalism: State governments and pollution control policies.
Durham, NC: Duke Univ. Press.
Lutter, Randall, and Richard B. Belzer. 2000. EPA pats itself on the back. Regulation 23 (3): 23–38.
Mazmanian, Daniel A., and Paul A. Sabatier. 1989. Implementation and public policy. With a new
postscript. Lanham, MD: The Univ. Press of America.
McCarthy, Jemes E., Claudia Copeland, Larry Parker, and Linda-Jo Schierow. 2007. Clean Air Act:
A summary of the Act and its major requirements. Report No. RL30853. Washington, DC:
Congressional Research Service.
McFarlane, Deborah R. 1989. Testing the statutory coherence hypothesis: The implementation of federal
family planning policy in the states. Administration and Society 20:395–422.
McFarlane, Deborah R., and Marilyn M. Gruebel. 2006. Public management and policy implementation:
Intersection, subset, or neither? Paper presented at the Fall Conference of the Association of Public
Policy Analysis and Management, Madison, WI, November.
Meier, Kenneth J., and Deborah R. McFarlane. 1995. Statutory coherence and policy implementation: The
case of family planning. Journal of Public Policy 15:281–98.
Meier, Kenneth J., and Laurence J. O’Toole. 2002. Public management and organizational performance:
The effect of managerial quality. Journal of Policy Analysis and Management 21:629–43.
Mintzberg, Henry. 1975. The manager’ job: Folklore and fact. Harvard Business Review 53:49–61.
NACAA. 2009. Investing in clean air and public health. http://www.4cleanair.org/Documents/Reportneedssurvey042709.pdf (accessed April 6, 2010).
Nicholson-Crotty, Sean, and Laurence J. O’Toole. 2004. Public management and organizational performance: The case of law enforcement agencies. Journal of Public Administration Research and
Theory 14:1–18.
O’Toole, Laurence J. 2000. Research on policy implementation: Assessment and prospects. Journal of
Public Administration Research and Theory 10:263–88.
———. 2004. The theory-practice issue in policy implementation research. Public Administration
82:209–329.
O’Toole, Laurence J., Kenneth J. Meier, and Sean Nicholson-Crotty. 2005. Managing upward, downward,
and outward. Public Management Review 7:45–68.
23
24
Journal of Public Administration Research and Theory
Downloaded from http://jpart.oxfordjournals.org/ at Franklin University on April 5, 2012
Pawson, Ray. 1989. A measure for measures: A manifesto for empirical sociology. New York, NY:
Routledge.
Portney, Paul R. 2000. An economic evaluation of the Clean Air Act. In Public policies for environmental
protection, ed. Paul R. Portney and Robert N. Stavins. New York, NY: Resources for the Future.
Rabe, Barry G. 2006. Power to the states: Promise and pitfalls of decentralization. In Environmental
policy: New directions for the twenty-first century, 6th ed., ed. Norman J. Vig and Michael E. Kraft,
34–56. Washington, DC: CQ Press.
———. 2007. Racing to the top, bottom, or middle of the pack? In Environmental policy: New directions
for the twenty-first century, 7th ed., ed. Norman J. Vig and Michael E. Kraft, 27–50. Washington,
DC: CQ Press.
Ringquist, Evan J. 1993. Environmental protection at the state level. New York, NY: M. E. Sharpe.
Sabatier, Paul, and P. J. Klosterman. 1981. A comparative analysis of policy implementation under
different statutory regimes. In Effective policy implementation, ed. Daniel Mazmanian and Paul
Sabatier, 127–46. Lexington, MA: Lexington Books.
Selden, Sally C., and Jessica E. Sowa. 2004. Testing a multi-dimensional model of organizational performance: Prospects and problems. Journal of Public Administration Research and Theory
14:395–416.
Stern, David L. 1998. Progress on the environmental Kuznets curve? Environment and Development
Economics 3:173–96.
Sull, Donald N., and Charles Spinosa. 2007. Promised-based management: The essence of execution.
Harvard Business Review, April. http://hbr.org/products/R0704E/R0704Ep4.pdf.
Tsoukas, Haridimos. 1994. What is management? An outline of a metatheory. British Journal of
Management 5:289–301.
US Environmental Protection Agency (USEPA). 1997. The benefits and costs of the Clean Air Act, 1970 to
1990. http://www.epa.gov/air/sect812 (accessed July 29, 2008).
———. 1999. The benefits and costs of the Clean Air Act 1990 to 2010.
———. 2001. National air quality and emissions trend report, 2001. http://www.epa.gov/airtrends/
reports.html (accessed October 22, 2007).
———. 2007. The plain English guide to the Clean Air Act. http://www.epa.gov/air/oaq_caa.html/peg/
index.html (accessed October 22, 2007).
Williams, Chuck. 2005. Management, 3rd ed. Mason, OH: Southwestern.
Woods, Neal D., David M. Konisky, and Ann O’M Bowman. 2009. You get what you pay for: Environmental policy and public health. Publius: The Journal of Federalism 39:95–116.
Zheng, Henry Yisheng. 2000. Exploring problem intractability in public policy implementation: The cases
of superfund policy and low level radioactive waste management policy. Ph.D diss., The Ohio State
Univ., 1999, Dissertation Abstracts International, 60/11, 4181.
Purchase answer to see full
attachment