Defense Management: Tools for Measuring and Managing Defense
Agency Performance Could Be Strengthened (13-SEP-04, GAO-04-919).
GAO was mandated to assess the effectiveness of defense agency
performance contracts as management tools. As agreed, GAO also
reviewed other tools (performance plans and balanced scorecards)
and focused on three defense agencies--the Defense Logistics
Agency (DLA), the Defense Information Systems Agency (DISA), and
the Department of Defense Education Activity (DODEA). GAO
addressed (1) the extent that the defense agencies initially used
performance contracts, including whether this tool addressed
attributes associated with results-oriented management; (2)
defense agencies' efforts to implement performance plans using
lessons learned from the initial contracts; and (3) the extent
DOD established mechanisms to share lessons learned. GAO reviewed
the content of these tools, but not the actual or reported
performance. DISA has not yet finalized its scorecard, thus this
report discusses only DISA's plans for its scorecard.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-919
ACCNO: A12273
TITLE: Defense Management: Tools for Measuring and Managing
Defense Agency Performance Could Be Strengthened
DATE: 09/13/2004
SUBJECT: Agency missions
Internal controls
Performance measures
Strategic planning
Performance plans
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-919
United States Government Accountability Office
GAO Report to the Committee on Armed Services, U.S. Senate
September 2004
DEFENSE MANAGEMENT
Tools for Measuring and Managing Defense Agency Performance Could Be
Strengthened
a
GAO-04-919
[IMG]
September 2004
DEFENSE MANAGEMENT
Tools for Measuring and Managing Defense Agency Performance Could Be
Strengthened
What GAO Found
Since fiscal year 1998, the Department of Defense (DOD) has implemented
various tools to help manage and oversee the performance of defense
agencies. Between fiscal year 1999 and 2003, DLA, DISA, and DODEA
initially used "performance contracts"-internal management agreements- to
bring specific problems to the attention of senior DOD and agency
leadership. While the contracts produced some useful information for
decision makers, this tool would have been more effective for assessing
performance, making resource allocation decisions, and taking corrective
actions if DOD had required the agencies to include certain attributes
associated with results-oriented management. Such attributes include
aligning agency performance goals and measures with agency strategic plans
and departmentwide goals; identifying individuals accountable for
achieving results; providing a comprehensive view of organizational
performance; linking resource needs to performance; discussing data
quality; and providing contextual information, including external factors
that affect reported performance.
Beginning in fiscal year 2003, DOD renamed the performance contracts as
"performance plans" and gave the defense agencies the option to use a
"balanced scorecard" approach, a tool used in the public and private
sectors to assess organizational performance. Based on experiences using
the initial contracts, DOD took steps to strengthen performance plans and
scorecards by revising the oversight and review process, requiring
performance measures to align with agency and departmentwide goals, and
requiring measures to provide a more comprehensive view of agency
performance. DLA's scorecard, DODEA's performance plan, and DISA's plans
for the agency's scorecard incorporated these changes and other attributes
to varying degrees. While these tools have the potential to provide
information useful to decision makers, they would be strengthened if DOD
had required the agencies to include additional attributes such as
designating specific individuals responsible for achieving results;
identifying the relationship between resource needs and performance;
reporting on data quality; and providing contextual information to allow
top leaders to understand the extent of progress made, take corrective
actions to achieve goals, and establish realistic performance goals for
future years. With these attributes, decision makers would potentially
gain additional insights into agency performance and areas needing greater
management attention.
DOD has developed mechanisms, such as a performance management Web site
and roundtables, to help agencies share lessons learned from implementing
performance plans and scorecards. In response to GAO's suggestions during
this review, DOD recognized the need to continue to hold roundtables more
frequently. DLA and DISA have also proactively shared their experiences
with each other.
United States Government Accountability Office
Contents
Letter 1
Results in Brief 4 Background 7 While Used to Varying Degrees, Performance
Contracts That We
Reviewed Could Have Been More Effective Had They Included
Certain Attributes Associated with Results-Oriented
Management 10 The Plan and Scorecard We Reviewed Show Improvement, but
Could Be Strengthened by Including Certain Key Attributes 19 DOD Captures
and Shares Lessons Learned 35 Conclusions 35 Recommendations for Executive
Action 36 Agency Comments and Our Evaluation 37
Appendix I Scope and Methodology
Appendix II Department of Defense's Risk Management Framework
Appendix III Fiscal Year When Defense Agencies Implemented
Their First Performance Contract, Plan, or Scorecard 42
Appendix IV Certain Key Attributes Associated with Results-Oriented
Management
Appendix V Comments from the Department of Defense 48
Appendix VI Key Contact and Staff Acknowledgments 51
Related GAO Products 52
Tables
Table 1: Time Frames for DLA's, DISA's, and DODEA's Use ofPerformance
Contracts, Performance Plans, or Balanced Scorecards
Table 2: Extent To Which DLA's, DISA's, and DODEA'sPerformance Contracts
Incorporated Certain Key Attributes Associated with Results-Oriented
Management
Table 3: Extent To Which DLA's Scorecard and DODEA's Plan Incorporated
Certain Key Attributes Associated withResults-Oriented Management
3 13 21
Figures
Figure 1: Depiction of How DLA's May 2004 Scorecard Supports
Departmentwide Goals and Cascades to Lower Organizational Levels
Figure 2: Example of the Type of Information in DODEA's Performance Plan
That OSD Officials Monitor, as of May 2004
Figure 3: Detailed Data on DODEA's Pupil to Teacher Ratio as of May 2004
Figure 4: Format of a Sample Measure from DISA's Draft September 2003
Balanced Scorecard
Figure 5: DOD's Risk Management Framework
Figure 6: Defense Agencies
Figure 7: Defense Field Activities
23
25
26
34 41 43 44
Abbreviations
DISA Defense Information Systems Agency DLA Defense Logistics AgencyDOD
Department of Defense DODEA Department of Defense Education ActivityOSD
Office of the Secretary of Defense PA&E Program Analysis and Evaluation
USD/P&R Under Secretary of Defense for Personnel and Readiness
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
September 13, 2004
The Honorable John W. WarnerChairman The Honorable Carl Levin Ranking
Minority MemberCommittee on Armed Services United States Senate
Department of Defense (DOD) agencies and field activities, such as
theDefense Logistics Agency, are a key part of DOD's business operations
and provide numerous support services, ranging from information technology
and education of servicemembers' dependents to logistics support for the
department.1 The agencies' customers include the military services, other
agencies, and military personnel and their families. During the past2
decades, service officials have expressed concern about inadequate
oversight and performance of the defense agencies; specifically, the lack
of customer responsiveness and inefficient business processes in light
ofgrowing defense agency budgets.
As part of a departmentwide effort, known as the Defense ReformInitiative,
aimed at improving DOD's business operations, in 1998, the Deputy
Secretary of Defense directed the defense agencies to developagreements
between the defense agencies and the Under and Assistant Secretaries to
which they report. Known as "performance contracts" at the time, these
agreements were intended to help improve the Office of the Secretary of
Defense's (OSD) oversight of these agencies as well as their performance.
According to a DOD official, these performance contractswere not intended
to be contracts in the legal sense, but rather were one of a number of
management tools-such as strategic plans-used by the defense agencies to
improve performance by setting performance goals, focusing the attention
of OSD and agency leaders on customer concerns, and bringing management
visibility over key areas of the agencies'
1 Defense agencies and DOD field activities perform similar support
functions. However, the field activities are smaller and serve a more
limited portion of DOD than do defense agencies. For purposes of this
report, we use the term "defense agencies" to refer collectively to
defense agencies and field activities.
performance.2 In fiscal year 2003, DOD refined its approach to performance
management and renamed the contracts as "performance plans" in order to
more accurately reflect the intent of the agreements. According to DOD,
these plans are intended to provide a more comprehensive view of agency
goals and performance and align with agency strategic plans and
departmentwide goals. DOD gave the defense agencies the option to use a
"balanced scorecard"-a form of performance plan.3 Balanced scorecards
organize performance measures by organizational drivers-including
financial, customer, and internal business processes as well as workforce
learning and growth-to help measure performance, make improvements, and
assess how well organizations are positioned to perform in the future.
Although the plans and scorecards themselves are not intended to resolve
or prevent problems, OSD and defense agency officials intend to use these
tools to monitor agency performance, identify developing problems and
corrective actions to improve business practices, meet customer needs, and
set funding priorities.
We have conducted an extensive body of work over the last decade
identifying leading practices in results-oriented management, including
performance measurement and reporting. In our prior work on government
performance, we identified attributes associated with results oriented
management that help inform management decision making.4 For this report,
we identified seven key attributes that we feel are the most important for
accurately assessing the strengths and weaknesses of programs and making
improvements. These attributes include aligning performance measures with
strategic goals, assigning accountability for achieving results,
developing measures that demonstrate results, developing measures that
provide a comprehensive view of agency
2 For purposes of this report, we use the term "performance contracts" to
include both performance contracts as well as accompanying annual
performance contract reports that discuss agencies' progress toward
meeting performance contract goals.
3 The balanced scorecard is a private-sector concept introduced by Robert
Kaplan and David Norton in 1992 to assess organizational performance and
is used by several government agencies.
4 Some of these products include GAO, Results-Oriented Government: GPRA
Has Established a Solid Foundation for Achieving Greater Results,
GAO-04-38 (Washington, D.C.: Mar. 10, 2004); The Results Act: An
Evaluator's Guide to Assessing Agency Annual Performance Plans,
GAO/GGD-10.1.20 (Washington, D.C.: Apr. 1, 1998); and Executive Guide:
Effectively Implementing the Government Performance and Results Act,
GAO/GGD-96-118 (Washington, D.C.: June 1, 1996).
performance, linking resource needs to performance, discussing data
quality, and including contextual information.
The Senate Committee on Armed Services report accompanying the National
Defense Authorization Act for Fiscal Year 2004 directed us to assess the
effectiveness of defense agency performance contracts as management
tools.5 Because of the evolution from performance contracts to plans or
balanced scorecards, this report evaluates all of these tools. Of the 14
defense agencies that presently use these tools, we focused our work on
3-the Defense Logistics Agency (DLA), the Defense Information Systems
Agency (DISA), and the Department of Defense Education Activity
(DODEA)-based on their levels of experience using performance contracts,
different types of missions, and types of services and customers.
Table 1 shows the time frames during which these agencies used performance
contracts, performance plans, or balanced scorecards.
Table 1: Time Frames for DLA's, DISA's, and DODEA's Use of Performance
Contracts, Performance Plans, or Balanced Scorecards
Fiscal year
Agency Performance contract Performance plan Balanced scorecard
DLA 1999-2001 Not applicable 2003-2004
DISA 2000-2002 2003 2004
DODEA 2000-2003 2004 Not applicable
Sources: DLA, DISA, and DODEA.
Note: Because DLA was transitioning to a balanced scorecard, with DOD's
approval, the agency did not develop a performance contract or plan during
fiscal year 2002. DODEA officials decided to use a performance plan rather
than exercise their option to develop a balanced scorecard for fiscal year
2004. DISA submitted a sample draft balanced scorecard to the Office of
the Secretary of Defense, Program Analysis and Evaluation in September
2003 in order to meet reporting requirements.
Although we obtained DISA's draft balanced scorecard dated September 2003,
we did not evaluate it against certain key attributes associated with
results-oriented management that we discuss in this reportbecause the
scorecard had not been finalized by the time we completed our review.
However, we do discuss DISA's plans for using a scorecard
5 S. Rep. No. 108-46, at 347 (2003).
and addressing the attributes, as applicable. Our objectives were to
evaluate
(1) the extent to which the defense agencies initially used performance
contracts to manage and measure agency performance, including whether the
contracts addressed certain key attributes associated with
results oriented management, such as performance measurement and
reporting;
(2) the status of efforts by the defense agencies to implement performance
plans, including the extent to which these plans reflect any lessons
learned from DOD's experience with the initial performance contracts and
address certain key attributes associated with results-oriented
management; and
(3) the extent to which DOD has established mechanisms to capture and
share lessons learned from the implementation of performance plans.
To address these objectives, we collected data from and interviewed
knowledgeable OSD, defense agency, and service officials about the use of
performance contracts, plans, and balanced scorecards, as applicable, at
DLA, DISA, and DODEA. We obtained these agencies' performance contracts,
plans, and scorecards, as applicable, and analyzed them against certain
key attributes associated with results-oriented management that we
identified in our prior reports. We did not review actual or reported
agency performance, only the content of the tools and how they were used.
We also did not validate procedures that the defense agencies have in
place to ascertain the reliability of data used in their performance plans
and scorecards. We conducted our review between July 2003 and May 2004 in
accordance with generally accepted government auditing standards. Details
about our scope and methodology appear in appendix I.
Between fiscal year 1999 and 2003, DLA, DISA, and DODEA-the three agencies
we reviewed-initially used performance contracts to varying degrees to
identify performance issues, elevate management challenges and customer
concerns to the attention of senior OSD and defense agency leaders, and
take corrective action. The contracts were an important step toward more
results-oriented management practices for the defense agencies. However,
the contracts would have been more effective tools for informing decision
making and managing and measuring agency performance had they included
additional attributes associated with results-oriented management. The
contracts showed progress made and encouraged discussion among top agency
and OSD leaders about actions needed to achieve targeted results,
particularly in terms of quality,
Results in Brief
quantity, cost, and timeliness of services. For example, DODEA officials
told us that the agency's fiscal year 2003 performance contract included a
measure to achieve optimum pupil to teacher ratios that influenced DOD to
fund the hiring of 200 full-time teachers. However, DOD guidance did not
require performance contracts to include certain key attributes that we
have identified in prior reports that are associated with results-oriented
management. While other management documents may reflect some of these
attributes, it is important that they also be reflected in the contracts
to provide a comprehensive picture of goals and performance. Consequently,
the performance contracts could have been strengthened by including the
following attributes:
o
Aligns goals and measures with agency and departmentwide goals.
Performance goals and measures for some of the agencies we reviewed only
partially aligned with agency and departmentwide goals and none of the
contracts cascaded goals and measures to lower organizational levels.
o
Assigns accountability for achieving results. Individuals at lower
organizational levels accountable for accomplishing specific goals and
taking corrective actions were not identified.
o
Demonstrates results. Contracts were inconsistent in using a combination
of clearly defined output and outcome goals accompanied by trend data.
o
Provides a comprehensive view of agency performance. With the exception of
DODEA, the contracts we reviewed lacked some measures that could have
provided officials with a comprehensive view of factors that drive
organizational performance, such as financial, customer, and internal
business processes. The contracts, to some extent, did reflect the
different concerns of customers and stakeholders.
o
Links resource needs to performance. Contracts did not discuss the
relationship between resource needs and performance outcomes.
o
Discusses data quality. Contracts did not discuss whether data were
reliable, valid, or verifiable, or identify data sources.
o
Provides contextual information. Contracts lacked some contextual
information to help officials assess the significance of factors that
affected reported performance, such as how performance measures
help achieve mission and departmentwide goals or the impact of contingency
operations on achieving performance goals.
Based on experiences using the initial performance contracts, DOD took
steps to strengthen performance plans and scorecards by revising the
oversight and review process as well as guidance to require that,
beginning with fiscal year 2004, performance measures align with agency
and departmentwide goals and provide a more comprehensive view of agency
performance. However, these tools could have been strengthened had DOD
required the defense agencies to include certain key attributes associated
with results-oriented management. For example, while DLA's scorecard
included a combination of output and outcome measures accompanied by trend
data, it did not include clearly defined measures. DODEA's performance
plan measures also were not consistently clearly defined and not always
accompanied by trend data. The scorecard and plan we reviewed also lacked
other attributes-not required by DOD-that could be useful to officials for
discussing and understanding competing priorities and resource allocation
constraints, assessing and understanding the extent of progress made
against existing performance targets, making improvements, and
establishing realistic performance goals for future years. Such attributes
include
o
identification of specific individuals who are responsible for achieving
results;
o linkage of resource needs to performance;
o
discussion of data quality, including data sources, as well as whether
data were reliable, valid, and verifiable; and
o
inclusion of contextual information such as external factors that could
affect performance and relevant information about the agencies' missions,
business lines, and customers.
DOD developed some mechanisms-such as a performance management Web site
and roundtable discussions-to help the defense agencies capture and share
experiences and lessons learned from developing performance plans and
balanced scorecards, and some agencies are proactively sharing such
information with each other. Based on our suggestions during this review,
DOD recognized the need to continue to hold roundtables to discuss
balanced scorecards. A DOD official also told us that DOD plans to revise
and update its performance management Web site to include defense
agencies' performance plans and balanced scorecards as well as guidance.
Some defense agencies, such as DLA and DISA, are sharing their experiences
with each other to leverage lessons learned.
This report contains recommendations to DOD aimed at making performance
plans and scorecards more informative for decision making and further
strengthening the potential of these tools for measuring and managing
agency performance. In commenting on a draft of this report, DOD generally
concurred with our recommendations.
Background
Evolution of Performance Contracts
In 1997, the Secretary of Defense issued the Defense Reform Initiative
report outlining a plan for reforming the department's business
operations. DOD established a management oversight structure to help
sustain the direction and emphasis of these efforts. As part of its
efforts to improve its business operations, DOD began requiring selected
defense agencies to develop annual performance contracts for fiscal year
1999. These contracts were internal management agreements between the
principal staff assistants-also known as Under and Assistant
Secretaries-in the Office of the Secretary of Defense, the Deputy
Secretary, and the defense agencies. DOD intended the performance
contracts to improve oversight of the defense agencies by (1) identifying
programming and budgeting issues, (2) providing agency directors with
clear objectives and performance targets, (3) establishing open and direct
communications with customers to, among other things, demonstrate how
their actions affect agency performance, and (4) changing the way DOD does
business. The contracts included improvement goals for the agencies in
terms of cost, productivity, quality, and responsiveness to customers. DOD
also required the defense agencies to assess their progress toward
achieving performance contract goals in annual performance reports. A
Defense Management Council-chaired by the Deputy Secretary and consisting
of key civilian and military leaders-was created to oversee Defense Reform
Initiative efforts, and a Defense Agency Task Force-chaired by the Deputy
Director of Resource Analysis in the Office of the Secretary of Defense,
Program Analysis and Evaluation (OSD/PA&E) and consisting of senior-level
representatives from the Under Secretaries of Defense, the service
secretaries, and the Joint Staff-was formed to oversee the development and
review of performance contracts.
DOD initially selected 4 defense agencies to implement performance
contracts during fiscal year 1999, and gradually expanded the requirement
to a total of 10 agencies by fiscal year 2002. At that time, DOD issued
guidelines that established a standard format for performance contracts
and recommended that certain attributes associated with results-oriented
management be included. DOD initially excluded defense agencies with
intelligence-or research-related missions from the requirement because
officials believed it would be more appropriate to focus primarily on
certain agencies with businesslike missions and develop lessons learned
based on their experiences. The 4 agencies DOD initially selected were
o the Defense Logistics Agency, a revolving fund activity;
o the Defense Finance and Accounting Service, a businesslike agency;
o
the Defense Contract Audit Agency, a small and appropriated fund agency;
and
o the Defense Health Program.
As part of the current administration's focus on business transformation,
DOD issued guidance in February and August 2003 that refined its approach
to performance management and renamed the contracts as performance plans.
These plans, first implemented in September 2003, build on the performance
goals and measures established in the performance contracts and are
intended to include a more comprehensive set of performance measures that
align with agency strategic plans and departmentwide goals set forth in
DOD's risk management framework. DOD uses the risk management framework,
established in the 2001 Quadrennial Defense Review Report, to consider
trade-offs among defense objectives and resource constraints. The
framework consists of four dimensions of risk:
o
Force management-the ability to recruit, retain, train, and equip
sufficient numbers of quality personnel and sustain the readiness of the
force while accomplishing its many operational tasks;
o
Operational-the ability to achieve military objectives in a near-term
conflict or other contingency;
o
Future challenges-the ability to invest in new capabilities and develop
new operational concepts to dissuade or defeat mid-to long-term military
challenges; and
o
Institutional-the ability to develop management practices and controls
that use resources efficiently and promote the effective operation of the
defense establishment.
These risk areas form the basis for DOD's annual performance goals and for
tracking associated performance results. Appendix II depicts DOD's risk
management framework.
DOD gave the defense agencies the option to use a balanced scorecard to
develop performance goals and report on progress. Currently, according to
an official in OSD/PA&E, DOD requires 10 of 27 defense agencies with
businesslike missions to have either performance plans or balanced
scorecards. Five defense agencies use performance plans, and the remainder
have exercised the option to develop balanced scorecards. In addition,
this official told us that 3 intelligence agencies have developed
performance plans in order to demonstrate how they are coordinating with
DOD on strategic and budgetary planning in the post-September 2001
environment. Appendix III depicts the first fiscal year when each defense
agency began using performance contracts, plans, or scorecards.
DOD Organizations Responsible for Development of Performance Contracts and
Plans
The Director, PA&E, has responsibility for overseeing the development of
defense agency performance plans, including reviewing the progress of each
defense agency. Specifically, this official oversees the development of
defense agency performance plans and balanced scorecards, provides
guidance for interpreting and applying DOD's risk management framework,
and, in conjunction with the relevant Under or Assistant Secretary of
Defense, reviews the progress of each agency in building its performance
plan or scorecard and reporting on its overall performance. The Under and
Assistant Secretaries of Defense are responsible for overseeing the
defense agencies and implementing performance plans and balanced
scorecards.
Our Prior Work Identifies Attributes Associated with Results-Oriented
Management
Measuring performance allows organizations to track the progress they are
making toward their goals and gives managers critical information on which
to base decisions for improving their programs. Our prior work on
results-oriented management in the federal government indicates that
agencies that are successful in measuring performance and achieving
business transformation goals strive to establish goals and measures at
all levels of the agency that
o
align goals and measures with the agency's strategic plan and
departmentwide goals;
o assign accountability for achieving results;
o demonstrate results;
o provide a comprehensive view of agency performance;
o link resource needs to performance;
o discuss data quality, including reliability and data sources; and
o
provide contextual information to help officials evaluate the significance
of underlying factors that may affect reported performance.
While these may not cover all attributes associated with results-oriented
management, we feel they are some of the most important for
accuratelyassessing the strengths and weaknesses of programs and making
improvements. Without these attributes, decision makers may not have
ascomplete information for measuring and managing an agency's performance
as needed. Each of these attributes is discussed in appendix IV.
Between fiscal year 1999 and 2003, DLA, DISA, and DODEA-the three agencies
we reviewed-used performance contracts to varying degrees to identify
performance issues, demonstrate progress made, and encourage discussions
to improve agency performance. The contracts were an important step toward
more results-oriented management practices for thedefense agencies.
However, the contracts could have been more effective had DOD guidance
required the defense agencies to include certainattributes associated with
results-oriented management. For example, the contracts addressed key
business lines, but did not consistently include information that could
have helped demonstrate results or provide somecontextual information that
could help officials assess progress and understand the significance of
underlying factors that may affect reported performance.
While Used to Varying Degrees, Performance Contracts That We Reviewed
Could Have Been More Effective Had They Included Certain Attributes
Associated with Results-Oriented Management
Defense Agencies Used Performance Contracts to Varying Degrees
Officials at the three defense agencies we reviewed used performance
contracts to varying degrees as part of their strategic planning systems
to demonstrate progress made and increase visibility over the quality,
quantity, cost, and timeliness of products and services for their key
business lines. In some instances, the contracts helped OSD and defense
agency officials to identify developing problems and assess the
effectiveness of corrective actions, such as justifying additional funding
to achieve performance goals and implementing or revising internal
processes to meet customer expectations. We did not verify the actual or
reported performance data included in the performance contracts or in
these examples. The agencies we reviewed provided the following examples
showing how they used performance contracts.
DLA officials told us that they used their performance contracts to focus
management attention and monitor efforts to improve the agency's response
time for providing logistics commodities, such as clothing and textile
items, medical and pharmaceutical supplies, and weapon system spare parts.
In its fiscal year 2001 performance contract, DLA established a
performance measure for its supply centers to process a greater percentage
of requisitions within shorter time frames. DLA reported that by
increasing emphasis on logistics response time goals to officials in its
Ohio-based Defense Supply Center and dedicating more trucks to its Defense
Distribution Center, the agency had processed 96 percent of requisitions
of commodities in 5.8 days during fiscal year 2001 rather than the
targeted 9.3 days.
A DISA official told us that the performance contracts helped DISA reduce
the cost of providing services in response to customer concerns. For
example, DISA developed cost-related performance measures for delivering
computing services, such as "Provide mainframe information processing
services while incurring no more than the unit costs listed." As a result
of increased management attention, DISA reported in its fiscal year 2002
annual performance report, which assessed progress against contract goals,
that it had reduced the cost per unit for providing OS/390 mainframe
processing services by 24 percent, from $38.26 in fiscal year 2001 to
$29.04 in fiscal year 2002. DISA officials told us that the high-level
attention made possible by the performance contracts helped ensure that
cost reduction goals were established and pursued.
DODEA officials told us that in order to meet OSD requirements, they
developed and submitted performance contracts that included measures taken
directly from their community strategic plan. However, they primarily used
their community strategic plan-containing strategic goals, objectives, and
performance measures-rather than the performance contract, which mirrored
the strategic plan, to manage and measure systemwide school performance.
We determined that DODEA's contract did not show the relationship between
resource needs and performance, but performance data helped bring
attention to initiatives and programs needing additional funding and
provided support for funding requests. For instance, DODEA officials noted
that a measure in DODEA's fiscal year 2003 performance contract influenced
DOD's decision to provide $114 million to hire 200 full-time teachers so
DODEA could achieve lower pupil to teacher ratios and, thereby, further
improve student performance.
Contracts Did Not Fully Incorporate Certain Key Attributes
Although useful in some respects, the performance contracts for the
agencies we reviewed would have been stronger tools if they had fully
incorporated certain key attributes that are associated with
results oriented management and that contribute to effective government
performance. In previous reports, we identified seven such attributes.
However, the contracts we reviewed incorporated aspects of these
attributes to varying degrees. We found that the contracts did not
identify individuals accountable for achieving results or clearly
demonstrate relationships between resource needs and performance. We also
determined that the contracts, with the exception of DODEA, did not
provide officials with a comprehensive view of agency performance,
including factors that drive organizational performance, such as
financial, customer, and internal business processes. Furthermore, defense
agency contracts were inconsistent in demonstrating results by providing
trend data to show agency progress in meeting established goals over time,
and in discussing data quality issues and providing contextual information
to help officials understand the measures used. We based our analysis on
the most recent performance contracts developed by DLA, DISA, and DODEA
because DOD officials told us that these would reflect lessons learned
from prior iterations. Our analysis did not include other internal
management documents, such as strategic plans, that may have addressed
these attributes. We also did not evaluate the usefulness or
appropriateness of the measures themselves. Table 2 summarizes the extent
to which the performance contracts we reviewed included the key attributes
we have identified.
Table 2: Extent To Which DLA's, DISA's, and DODEA's Performance Contracts
Incorporated Certain Key Attributes Associated with Results-Oriented
Management
DLA DODEADISA
Attribute
FY 2001 FY 2002 FY 2003
Aligns goals and measures with agency and departmentwide goals
Each of the agency's strategic plan goals is supported by performance measures
Measures align with departmentwide goals
measures
Identifies individuals accountable for achieving results
Demonstrates results
Includes a combination of output- and outcome-oriented measures Measures are
clearly defined Provides trend data to demonstrate progress against targeted
performance
Provides comprehensive view of agency performance
Measures address factors that drive organizational performance (financial,
customer, and internal business processes and workforce learning and
growth)
Measures address cost, quality, quantity, and timeliness
Measures address key business lines
Incorporates customer and stakeholder priorities
Links resource needs to performance
Measures identify resources (e.g., human capital and information
technology) needed to achieve performance goals
Discusses data quality
Discusses data verification and validation procedures, including
reliability Identifies sources of data for each measure
Provides contextual information
Provides context or explanation for understanding measures and reported
performance
Legend: Fully incorporated; Partially incorporated; Not incorporated
Source: GAO analysis of DLA's, DISA's, and DODEA's performance contracts.
Notes: "Fully incorporated" indicates that all measures exhibited that
particular attribute. "Partially incorporated" indicates that some of the
measures, but not all, exhibited that attribute, whereas "not
incorporated" means that none of the measures did.
a Performance contract measures did not cascade to lower organizational
levels. However, agency officials told us that they shared strategic goals
and objectives from their strategic plans with all organizational levels
and made these available to customers and stakeholders.
Performance Contracts Did Not Always Align with Agency and Departmentwide
Goals
Contracts Did Not Assign Accountability for Achieving Results to
Individuals at Lower Organizational Levels
b According to a DLA official, performance appraisals for top agency
leadership reflected performance contract measures. However, DLA officials
were not able to provide a standardized performance appraisal template to
demonstrate this.
Additional information about the extent to which the performance contracts
for the agencies we reviewed included these attributes is discussed below.
DOD did not require performance contract measures to align with agencies'
strategic plans or departmentwide goals until fiscal year 2004. As a
result, some of the agencies' performance contract measures did align,
while other measures did not. Each agency's performance contract included
statements conceptually linking performance goals with strategic plans and
departmentwide goals. For example, DISA's fiscal year 2002 performance
contract stated that "The performance measures used in this contract
directly support the goals and objectives in the strategic plan," and
DISA's strategic plan directly related to DOD strategic goals and
objectives. We found that all of DODEA's performance measures aligned with
the agency's strategic plan goals and could be linked to departmentwide
goals. With the exception of the following example, all of DLA's fiscal
year 2001 performance contract measures aligned with the agency's
strategic plan goals, but not all aligned with departmentwide goals. DLA's
contract lacked performance measures that addressed its strategic plan
goal to ensure that the agency's workforce is able to deliver and sustain
world-class performance. Consequently, the performance contract did not
provide DLA and DOD officials with information to help assess recruiting
needs or workforce skills and training requirements that contribute to
meeting customer needs.
Furthermore, performance goals and measures used in the contracts we
reviewed did not cascade from the highest to the lowest levels of the
agencies to reinforce accountability throughout the organization.
While defense agency performance contracts identified individuals
accountable for achieving performance results at the most senior levels of
DOD and the defense agencies, they did not identify individuals
responsible for tracking performance or making improvements at lower
organizational levels.
Defense Agency Task Force officials were directed to review the defense
agencies' performance contracts and make recommendations to the Deputy
Secretary of Defense on whether the performance contracts satisfied DOD
guidance. The Defense Management Council was directed to
Contracts Did Not Consistently Provide Information to Help Assess Progress
and Demonstrate Results
recommend ways to reduce costs incurred by the defense agencies while
improving customer service. DOD officials told us that after the events of
September 11, 2001, OSD's attention on the performance contracts
temporarily diminished, less strict oversight occurred, and PA&E officials
did not hold agencies to meeting reporting requirements. However,
beginning in April 2002, the Deputy Secretary of Defense began requiring
the Under and Assistant Secretaries of Defense to provide quarterly
updates concerning the defense agencies' performance on key performance
contract measures and the corrective actions to be taken if performance
goals were not expected to be met. DOD officials were not able to provide
historical information concerning whether these reviews occurred as
frequently as required or whether corrective actions were taken.
Although top leaders had oversight of the performance contracts, none of
the performance contracts we reviewed identified individuals, such as
business managers, responsible for achieving results at lower
organizational levels as a way to reinforce accountability for achieving
results and maintain focus on organizational priorities. However, in a
related internal management report called the Chief Financial Executive
4th Quarter Fiscal Year 2002 Performance Contract Report-which DISA used
internally to report progress against performance contract goals- DISA
identified business managers responsible for monitoring each performance
measure.
Defense agency performance contracts did not consistently provide
information to help assess progress and demonstrate results. Although the
agencies' performance contracts included milestones or time frames for
completing specific initiatives, they differed in the degree to which
measures were clearly defined and outcome-oriented and the extent to which
they included trend data. For instance, we found that some measures in
DODEA's performance contract were not clearly defined. One example is:
"improve student achievement and education quality consistent with
Presidential and national educational goals. Per pupil costs will not
increase more than 7% over the previous year."
This measure is actually two measures. Furthermore, the measure does not
define the specific presidential and national education goals-such as
Contracts Did Not Provide a Comprehensive View of Organizational
Performance
the No Child Left Behind Act of 2001-against which the agency measured its
progress.6
In addition, DODEA and DISA did not consistently report trend data for
each measure in their performance contracts to show progress made over
time, and DLA limited trend data to the prior year. For example, in its
fiscal year 2003 performance contract results, DODEA reported that it had
met goals for teacher certification and school accreditation, but the
agency did not identify the baseline or percentage of teachers certified
or schools accredited.
Each agency's performance contract included measures that addressed key
business lines as well as cost, quality, quantity, and timeliness of
services, but, with the exception of DODEA, did not provide as
comprehensive a view of agency performance as possible because the
measures did not address all drivers of organizational performance,
including customer, financial, and internal business processes, and
workforce learning and growth. Doing so could have helped officials assess
how well the agencies were positioned to perform in the future.
We found that each agency's most recent performance contract included
measures assessing aspects of each key business line. For example, DLA's
fiscal year 2001 contract included 26 measures that supported its 5
business lines, including Defense Automated Printing Service, Defense
Distribution, Defense National Stockpile Center, Defense Reutilization and
Marketing Service, and Supply (Energy and Non-Energy). DISA's fiscal year
2002 contract included 29 measures that addressed 4 business lines,
including Computing Services, Enterprise Acquisition Services, Joint
Warfighting and DOD-wide Enterprise Capabilities, and Telecommunication
Services. DODEA's fiscal year 2003 performance contract included 15
measures that addressed its 2 business lines- domestic and overseas
education.
While measures addressed the agencies' key business lines as well as cost,
quality, and timeliness of services provided, they did not all address all
drivers of organizational performance. For example, the contracts did not
always reflect the different concerns of customers and stakeholders.
According to defense agency officials, PA&E officials prescribed certain
6 The No Child Left Behind Act of 2001, among other things, focused
attention on closing the achievement gaps among various groups of
students.
Linkage between Resource Needs and Performance Outcomes Was Not Clearly
Established
Quality of Data Was Not Discussed
measures, although agency officials believed these were not value added
and did not align with agency strategic goals. For instance, PA&E
officials required DODEA officials to identify Management Headquarters
Costs- and specifically the "cost per school"-in DODEA's performance
contract. Agency officials told us that this measure had limited value
because it provided aggregated data rather than specific information for
individual school's costs. Furthermore, according to a former senior
official in the Office of the Secretary of Defense for Acquisition,
Technology, and Logistics, the performance contracts did not reflect the
needs of the chiefs of staff of the services and the Secretary of Defense,
who were key customers and stakeholders. This official believed that the
measures were too technical and "in the weeds" to help inform strategic
management decisions and were more appropriate for lower-level managers to
use when monitoring program performance.
None of the contracts we reviewed identified the resources needed to meet
performance goals or explained how additional resources could have
contributed to achieving expected levels of performance. Such information
could have helped officials to better understand how trade-offs among
different investments-such as human capital, information technology, and
workforce training-affected performance outcomes and drove costs. For
instance, DLA established a fiscal year 2001 goal of improving inventory
accuracy by 4.2 percent over the previous year's performance, but it did
not provide additional data in its performance contract that would have
helped officials to determine the resources needed to achieve this goal,
if any. DOD officials stated that they began to establish links between
resource needs and performance outcomes by requiring the defense agencies
to submit their performance contracts at the same time as their Program
Objectives Memorandum.7
None of the performance contracts we reviewed discussed the quality of
performance data, including reliability, data sources, or data
verification and validation procedures. Consequently, when reviewing the
performance contracts, top officials in DLA, DISA, and DODEA may not have
had information needed to determine whether reported data and performance
results were accurate, reliable, or current enough to use in decision
making and that corrective actions would produce the desired
7 Program Objectives Memorandum is the final product of the programming
process within DOD. The Program Objectives Memorandum displays the
resource allocation decisions of the DOD components in response to, and in
accordance with, Defense Planning Guidance.
Contextual Information Could Have Better Informed Decision Making
outcomes. For example, an October 2000 internal audit initiated by the
DISA Inspector General concluded that data reported in DISA's
1st Quarter Fiscal Year 2000 Chief Financial Executive Report contained
errors or bias and lacked sufficient supporting documentation.8 The
Inspector General reported that these conditions occurred because DISA had
not established a systematic approach for collecting and reporting
performance data. The Inspector General concluded that, as a result,
reported data provided limited assurance that DISA had valid and reliable
information to manage agencywide performance. The Inspector General
concluded that DISA should establish a data collection and reporting
process and definitive procedures to ensure performance information was
sufficiently complete, accurate, and consistent to document actual
performance and support decision making. Senior DISA officials told us
that the agency did not initially implement related recommendations
because DISA discontinued use of performance contracts. However, according
to these officials, DISA is currently taking steps to address these
recommendations by setting the expectation that senior officials will be
personally accountable for the quality of data included in DISA's balanced
scorecard and establishing internal focal points for data collection and
reporting. In addition, DISA officials are reviewing commercial products
to track data electronically.
The performance contracts provided some contextual information about some
external factors outside of the agencies' control that may have impacted
the achievement of performance goals and provided general information
about the agencies' mission, business lines, and customers. However, the
performance contracts did not always provide some contextual information
to help officials understand each agency's accomplishments for that fiscal
year; determine how measures related to agency and departmentwide goals;
assess agency progress; understand the impact that external factors, such
as contingency operations, could have on the agency's ability to meet its
performance targets; and determine how resource needs should be
prioritized. For example, DISA reported in its fiscal year 2002
performance contract under the Joint Warfighting and DOD-wide Enterprise
Capabilities business line that it planned to, "Successfully complete
fielding of a DOD standard spectrum management information system
(Spectrum XXI) software version 3.0 by December 2001." However, it is
unclear how this linked specifically to DISA's
8Defense Information Systems Agency Inspector General. Audit of DISA's
Performance Contract. Final Report. Project No. 2000-H-601. (Washington,
D.C.: October 24, 2000).
strategic plan or DOD's goals or what external factors could have impacted
DISA's intended performance.
The Plan and Scorecard We Reviewed Show Improvement, but Could Be
Strengthened by Including Certain Key Attributes
In fiscal year 2003, DOD renamed the performance contracts as performance
plans and, based on the agencies' experiences using performance contracts,
revised the oversight and review process and guidance. DOD required
performance measures, beginning in fiscal year 2004, to align with agency
and departmentwide goals and provide a more comprehensive view of agency
performance. However, the plan and scorecard we reviewed could be stronger
tools for informing officials about agency performance if they
incorporated some additional key attributes associated with
results-oriented management such as linking resource needs to performance,
discussing data quality, and providing contextual information for
understanding the measures and reported performance.
DOD Strengthens Oversight Process and Requires Inclusion of Additional Key
Attributes, but Some Are Still Lacking
Six years after implementation of the performance contracts, the original
intent-to assist senior leaders at the defense agencies and within OSD to
oversee agency performance and to encourage the agencies to become more
responsive to customer needs-remains unchanged. However, beginning in
fiscal year 2004, DOD renamed the performance contracts as performance
plans and gave the defense agencies the option to use either performance
plans or balanced scorecards. These tools were to build on the performance
contracts, and to include additional attributes associated with
results-oriented management that were previously lacking under the
contracts. These changes centered on (1) revising the oversight and review
process to strengthen accountability for achieving results, (2) requiring
performance measures to align with agency strategic goals and DOD's risk
management framework, and (3) requiring the measures to be more
comprehensive in scope to better demonstrate results.
The defense agencies we reviewed are at different stages of developing and
implementing performance plans and balanced scorecards, and DOD has
required these tools to be in place for less than a year. Consequently, it
is too soon to determine how useful they will be for managing and
measuring agency performance. However, DLA's balanced scorecard and
DODEA's performance plan showed progress in incorporating many of the
attributes associated with results-oriented management that we have
identified, but because DOD did not require the defense agencies to
include all of the attributes discussed in this report, the plan and
scorecard we reviewed continue to lack certain information-such as the
relationship between resource needs and performance, discussions of data
quality, and some contextual information-that could strengthen these tools
for managing defense agency performance. Although we obtained DISA's draft
balanced scorecard dated September 2003, we did not evaluate it against
the key attributes discussed in this report because it had not been
finalized by the time we completed our review. However, we do discuss
DISA's plans for addressing several of the key attributes in the balanced
scorecard that the agency is developing.
Table 3 shows the extent to which DLA's balanced scorecard and DODEA's
performance plan incorporated certain key attributes associated with
results-oriented management. We based our analysis specifically on the
performance plan and balanced scorecard, although other agency internal
management documents may have addressed these attributes. We also did not
evaluate the usefulness or appropriateness of the measures themselves.
Table 3: Extent To Which DLA's Scorecard and DODEA's Plan Incorporated
Certain Key Attributes Associated with Results-Oriented Management
DLA DODEA Attribute FY 2004 FY 2004
Aligns goals and measures with agency and departmentwide goals
Each of the agency's strategic plan goals is supported by performance measures
Measures align with departmentwide goals
a
Goals and measures cascade from the corporate level to the lowest level of
the agency
Assigns accountability for achieving results
Establishes a foundation to hold top leadership accountable for achieving
results
Performance appraisals for top agency leadership reflect performance goals
and measures
Identifies individuals accountable for achieving results
Demonstrates results
Includes a combination of output- and outcome-oriented measures Measures are
clearly defined Provides trend data to demonstrate progress against targeted
performance
Provides comprehensive view of agency performance
Measures address factors that drive organizational performance (financial,
customer, and internal business processes and workforce learning and
growth)
Measures address cost, quality, quantity, and timeliness
Measures address key business lines
Incorporates customer and stakeholder priorities
Links resources needs to performance
Measures identify resources (e.g., human capital and information
technology) needed to achieve performance goals
Discusses data quality
Discusses data verification and validation procedures, including
reliability
Legend: Fully incorporated; Partially incorporated; Not incorporated
Source: GAO analysis of DLA's balanced scorecard and DODEA's performance
plan.
Notes: "Fully incorporated" indicates that all measures exhibited that
particular attribute. "Partially incorporated" indicates that some of the
measures, but not all, exhibited that attribute, whereas "not
incorporated" means that none of the measures did.
a Performance measures did not cascade to lower organizational levels.
However, DODEA officials told us that they shared strategic goals and
objectives from their strategic plans with all organizational levels and
made these available to customers and stakeholders.
Varying Progress Made in Aligning Performance Goals and Measures with
Agency and Departmentwide Strategic Goals and Cascading These to Lower
Organizational Levels
Additional information about the extent to which DLA's balanced scorecard
and DODEA's performance plan include these attributes is discussed on the
following pages.
The balanced scorecard and performance plan we reviewed showed varying
progress in meeting DOD's requirement to align performance goals and
measures with agency strategic plan goals and DOD's risk management
framework beginning in fiscal year 2004. However, DLA and DODEA fulfilled
DOD's requirement that every organization's performance goals and measures
be consistent with those of its next higher and lower organizations. DLA
and DODEA have adopted different strategies for cascading their scorecard
and plan, respectively. DLA's corporate-level balanced scorecard aligns
with its strategic plan goals to transform DLA's operations over the next
2 to 5 years to better support its customers and departmentwide
performance measures outlined in DOD's risk management framework. DLA has
also cascaded its corporate scorecard to each of its Defense Supply
Centers located in Richmond, Virginia, Columbus, Ohio, and Philadelphia,
Pennsylvania, as well as other field activities including the Defense
Energy Support Center, Defense Distribution Center, and Defense
Reutilization and Marketing Service. Each center's or field activity's
scorecard identifies "how and to what extent" their strategies support
DLA's corporate-level balanced scorecard and is expected to reflect DLA's
overall vision of "Right Item, Right Time, Right Place, Right Price, Every
Time...Best Value Solutions for America's Warfighters." According to DLA
officials, in this way, DLA provides a consistent vision to its customers,
stakeholders, and employees. They told us that this process also allows
customers, stakeholders, and employees to understand how corporate-level
strategies are supported by functional, day-to-day measures.
Figure 1 depicts how DLA's corporate-level scorecard aligns with
departmentwide goals and measures and how the Defense Supply Center
Richmond's internal scorecard supports these goals. DLA officials told us
that the goals within the institutional risk area of DOD's risk management
framework cascade down to the Under Secretary of Defense for Acquisition,
Technology, and Logistics' balanced scorecard. This scorecard has four
strategies that cascade down and inform DLA's scorecard. In turn, the four
strategies in DLA's corporate-level scorecard cascade down to its defense
supply centers. Figure 1 depicts specifically how the three strategies
within the customer portion of DLA's scorecard cascade down to the Defense
Supply Center Richmond. The supply center has developed four measures that
it uses to assess its performance in terms of customer service on a
monthly basis.
Figure 1: Depiction of How DLA's May 2004 Scorecard Supports
Departmentwide Goals and Cascades to Lower Organizational Levels
Source: GAO presentation of DOD and DLA information.
DODEA's fiscal year 2004 performance plan does not include goals and
measures that support all of its community strategic plan goals and
objectives such as (1) developing, promoting, and maintaining a network
Oversight and Review Process Strengthened to Reinforce Accountability
of meaningful partnerships and alliances to enhance social, emotional, and
academic growth and (2) maximizing resources or developing safe,
well-managed, and disciplined environments conducive to learning. However,
DODEA has linked each performance measure in its performance plan with the
risk management framework area that it supports. For example, DODEA linked
the measure "In fiscal years 20042009, no less than 95 percent of the
Department of Defense Domestic Dependent Elementary and Secondary Schools
teachers will be professionally certified in the subject area and grade
level to which they are assigned within three years of employment" with
force management risk, and specifically the departmentwide goal to
maintain a quality workforce. DODEA officials told us that they continue
to communicate these goals and performance measures to employees and
customers systemwide through their community strategic plan, and school
improvement plans reflect these goals, as relevant.
DOD has strengthened the accountability of OSD and top agency leadership
for achieving performance results through the development of OSD-level
scorecards and semiannual reviews of defense agency performance, and the
agencies we reviewed are taking steps to strengthen accountability at
lower organizational levels. However, DODEA's performance plan does not
identify specific individuals who are responsible for making improvements.
The Under and Assistant Secretaries of Defense continue to have
responsibility for reviewing the defense agencies' performance plans or
scorecards and providing direct feedback. However, some have developed
balanced scorecards-such as the Under Secretary of Defense for Personnel
and Readiness (USD/P&R)-and have taken steps to incorporate data from
quarterly reviews of agency performance measures in their balanced
scorecards while continuing to provide feedback to agency officials on a
regular basis. For example, prior to approving DODEA's fiscal year 2004
performance plan, the Deputy Under Secretary of Defense for Military
Community and Family Policy within USD/P&R asked the agency to incorporate
updated information on its customer satisfaction survey and to explain
annual variations in the cost-per-pupil index. OSD and agency officials
also review performance data during periodic internal meetings. For
example, figure 2 shows the high-level DODEA performance measures that
officials within USD/P&R monitor on a quarterly basis in their balanced
scorecard called Monitoring the Status of the Force. Part A: Balanced
Scorecard. USD/P&R officials are responsible for monitoring the force
management risk area, and within this area, performance strategies related
to "maintain workforce satisfaction." Specifically, these officials
monitor DODEA's progress in meeting three performance contract measures.
Figure 2: Example of the Type of Information in DODEA's Performance Plan
That OSD Officials Monitor, as of May 2004
Source: GAO presentation of DOD data.
Additional details about each performance measure are also available to
USD/P&R officials, as depicted in figure 3.
Figure 3: Detailed Data on DODEA's Pupil to Teacher Ratio as of May 2004
Number of pupils 18
17 FY 2000 FY 2001 FY 2002 FY 2003
Actual performance
Department of Defense Elementary and Secondary Schools
Department of Defense Dependent Schools
Source: DOD.
DOD officials told us that the Office of the Under Secretary of Defense
for Acquisition, Technology, and Logistics that oversees DLA and the
Assistant Secretary of Defense for Network Information and Integration
that oversees DISA are developing balanced scorecards that are expected to
incorporate certain performance measures from DLA's and DISA's balanced
scorecards, respectively, in order to help maintain oversight and monitor
the agencies' progress toward achieving specific performance goals.
DOD has also implemented semiannual performance reviews at which time the
defense agencies brief PA&E officials, respective Under and Assistant
Secretaries of Defense, and agency officials on progress to date against
performance goals. DOD began holding midyear reviews during May 2004, and
officials expect to complete these reviews by August. A DOD official told
us that in the future, the first cycle reviewing the prior year's
performance will occur between September and December and the second
cycle, the midyear review, will occur between May and August. DOD
officials expect results from these meetings to be reflected in the
agencies' fiscal year 2005 performance plans and scorecards. During a
Agencies' Progress in Demonstrating Results Varies
May 2004 midyear review of DLA's balanced scorecard, the Director, PA&E,
recommended that the agency include more specific information on its
customer surveys, including the population surveyed and the way customer
feedback is obtained.
By assigning responsibility for each performance measure to specific
individuals, some defense agencies are reinforcing accountability for
achieving performance goals at all levels of the organization. For
example, in addition to implementing performance appraisals for Senior
Executive Service staff that are linked to strategic goals and objectives
against which ratings are assessed, DLA identified individuals in its
balanced scorecard and charged them with monitoring, reporting on, and
implementing corrective actions for each scorecard measure. A senior DISA
official told us that the agency recently approved linking balanced
scorecard outcomes with Senior Executive Service members' and supervisory
employees' performance appraisals beginning in fiscal year 2005 in order
to institutionalize its balanced scorecard and improve accountability for
results. DISA also is reinforcing accountability by requiring its senior
leaders (scorecard owners), who are accountable for achieving results, to
provide status updates during corporate-board meetings of senior agency
officials. In contrast, DODEA's performance plan does not identify
specific individuals accountable for monitoring, reporting on, and
implementing corrective actions for each performance measure. However,
DODEA officials told us that the agency includes performance plan
objectives in each manager's performance appraisal, and both monetary and
nonmonetary awards are used as incentives to meet or exceed goals and
measures.
DLA's balanced scorecard and DODEA's performance plan showed varying
progress in the extent to which they included outcome-oriented measures
and trend data to help demonstrate results, and the clarity of performance
measures continues to be an issue. We determined that DLA's 2004 scorecard
includes outcome-oriented measures and trend data that help to demonstrate
progress made. For example, one of DLA's outcome-oriented financial
measures-the achievement of accuracy in non-energy sales
forecasting-includes baseline and trend data for the 3 preceding fiscal
years. However, DLA's 2004 scorecard measures include terms that are
generally not clearly defined, as demonstrated by the measure "Percentage
of reengineered functionality implemented in BSM." In contrast, DODEA's
performance plan includes almost all of the same measures as did its
fiscal year 2003 performance contract, and these measures are not always
clearly defined or accompanied by trend data. As we previously mentioned,
DODEA does not clearly define the specific
Agencies' Scorecard and Plan Show Progress in Providing a Comprehensive
View of Agency Performance
presidential and national education goals against which it measures
progress. Furthermore, DODEA's performance plan does not consistently
include baseline information or trend data to help officials gauge
progress. DODEA did not indicate the percentage of teachers who are
certified in the subject area and grade level to which they are assigned
within 3 years of employment, although the agency set a goal of no less
than 95 percent. The plan also does not indicate the percentage of schools
that have full high-speed connectivity to the Internet, although it
establishes a goal of 100 percent.
Our analysis showed that DLA's scorecard and DODEA's performance plan
included measures for key business lines that provide officials with
information about agency performance and, to varying degrees, address the
four drivers of organizational performance as well as cost, quality, and
timeliness of services provided, and that all of the agencies we reviewed
are taking steps to better reflect customers' and stakeholders'
priorities.
DLA organizes its corporate-level scorecard by drivers of performance-
including learning and growth and customer, financial, and internal
business processes-and includes strategies and supporting measures that
are cascaded to its defense supply centers and field activities. As would
be expected, lower organizational level scorecards used by the defense
supply centers include performance measures that address the key business
lines for which they are responsible and provide more specific assessments
of cost, quality, timeliness, and customer satisfaction. For example, the
Defense Supply Center Richmond's scorecard includes performance measures
focusing on aviation weapons systems and environmental logistics support,
whereas the Defense Supply Center Philadelphia's scorecard focuses on the
provision of clothing, textiles, and medical equipment, among other items.
The learning and growth quadrant of DLA's corporate scorecard contains a
measure to identify gaps between workforce competencies and skills
required to meet mission requirements for 100 percent of DLA positions by
the end of fiscal year 2007. The Defense Supply Center Richmond's
scorecard includes several measures that address employee certification,
including acquisition training certification to ensure that the supply
center's workforce is fully proficient in the acquisition process. This
measure, in turn, aligns with the departmentwide policy of establishing
education and training for each defense acquisition position. Similarly,
DODEA's plan includes measures that address its two key business
lines-domestic and overseas education-and are cascaded to schools
worldwide through its community strategic plan.
Unlike DLA's balanced scorecard, DODEA's performance plan does not
specifically categorize performance measures by the factors that drive
organizational performance, but we identified at least one measure or
initiative that can be linked with each organizational driver. For
example, DODEA's performance plan includes a measure to limit increases in
per pupil cost to no more than 7 percent over the prior fiscal year. This
measure addresses both financial and internal business processes that
affect organizational performance. DODEA addresses workforce learning and
growth by establishing a goal to certify no less than 95 percent of
teachers in the subject and grade levels to which they are assigned within
3 years of employment. Finally, DODEA includes measures that address
customer satisfaction such as students' performance and parents'
satisfaction with local schools.
Our analysis showed that the defense agencies we reviewed are also taking
steps to better address specific customer and stakeholder concerns about
performance. Both DODEA and DLA conduct periodic reviews of agency
performance and make adjustments to meet customers' and stakeholders'
priorities. For instance, DLA's balanced scorecard Executive Board,
comprised of DLA's Vice Director, designated leaders for each area of
organizational performance (quadrant leaders), the Chief Information
Officer, and a Lead Center representative, conducts monthly reviews of
enterprisewide operational measures. Each quarter, DLA's Corporate Board,
comprised of DLA's Director, Vice Director, and the heads of DLA's major
organizational components, meets with commanders from the field to review
the agency's performance from the corporate level down to the supply
centers. The defense supply centers conduct weekly reviews to monitor
progress, identify developing problems, and take corrective actions.
According to DLA officials, to make information more timely, accessible,
and relevant for measuring and managing the agency, DLA posts the agency's
balanced scorecard on its internal Web site to allow all employees,
including agency managers and leaders, to share performance data. This Web
site is also available to all of DLA's military customers and
stakeholders. DLA also continues to obtain customer and stakeholder
feedback through a variety of mechanisms, such as customer surveys,
reports from customer service liaisons embedded with deployed military
service units, and meetings between the agency's Director and leaders of
the military services. Moreover, DLA is taking steps to address the unique
concerns of each military service through the development of
performance-based agreements that include specific performance measures
that are based on collaborative, direct negotiations between DLA and each
military service. These agreements are intended to create and sustain
end-to-end customer support by aligning DLA's resources with
Funding Not Yet Linked with Performance
the logistics solutions that will meet the services' needs and maximize
their effectiveness.
In addition, we determined that the defense combat support agencies- such
as DLA and DISA-are taking steps to incorporate additional strategies and
measures in their balanced scorecards that respond to customers' changing
needs during contingencies and war. For example, DLA's scorecard assesses
readiness by measuring the percentage of items that the military services
identify as requiring surge and sustainment support (e.g., hardware items,
clothing and textiles, and medical supplies) to ensure appropriate
inventory levels to accommodate short production lead times. DLA officials
told us that to reduce procurement lead time and decrease delivery delays
to the agency's primary customer, the warfighter, the agency has developed
22 long-term contracts, called strategic supplier alliances, with its
major contractors. According to these officials, having these agreements
in place enhances customer support through the inclusion of better
performance metrics, such as reduced administrative lead times and
increased stock availability. DISA officials told us that as a combat
support agency, DISA's mission is to support the warfighter and that the
strategies and performance measures DISA is developing focus on wartime
requirements but also are applicable to peacetime environments. For
example, DISA is developing measures that will support its customer
strategy to eliminate bandwidth constraints in response to customer
complaints that limited bandwidth is a problem, especially during
contingency operations. According to DISA officials, removing current
bandwidth limitations can provide DISA's customers with better and
timelier information during combat and peacetime.
A major goal of DOD is to strategically link major decisions to program
and budget development, and the defense agencies we reviewed are in the
initial stages of attempting to do this. DOD has directed that the Under
Secretary of Defense, Comptroller, and the Director, PA&E, review program
performance and assess the degree to which budget estimates sustain and
improve programmatic results and, as appropriate, recommend alternate
solutions or funding adjustments to correct resource needs if performance
goals are not met. Senior DOD officials told us that this continues to be
a challenging undertaking. The defense agencies we reviewed are using
their performance plans and scorecards to bring attention to priority
management projects. Although these tools still do not discuss resource
implications, DODEA included goals and measures from its fiscal year 2004
performance plan in its fiscal year 2004-2005 operation and maintenance,
defensewide, biennial budget estimates. We believe that
Quality of Data Is Not Discussed
Some Contextual Information Is Lacking
inclusion of the agency's priorities can help officials better consider
appropriate funding levels and trade-offs.
Although DISA is still developing its balanced scorecard, senior agency
officials told us that the agency plans to use the scorecard to bring
visibility and management attention to priority projects in order to fund
them appropriately and to improve collaboration internally. These
officials told us that DISA is in the process of aligning its investment
and scorecard strategies and expects initiatives in its scorecard to be
assigned funding priority. To this end, DISA is developing a Program
Objectives Memorandum that aligns investments with its scorecard. During
corporate-board meetings when senior agency officials review the agency's
performance, DISA requires each measure owner to discuss the context and
status of initiatives, including the need for additional resources.
According to DISA officials, the agency has already experienced some
benefits when testing this approach. DISA officials identified a
disconnect between customer expectations, performance levels, and funding
for an initiative to develop a Web-based portal to share information about
DISA's business activities internally. The goal of the portal is to help
employees perform critical operations faster and more effectively.
Officials told us that although internal customers identified this
initiative as a priority, the portal did not receive dedicated funding
and, as a result, was neither as robust as planned nor on target to meet
development time frames. DISA officials told us that inclusion of the
project in its balanced scorecard is expected to raise these challenges to
the attention of senior leaders to encourage corrective action and
appropriate funding levels.
The balanced scorecard and performance plan we reviewed do not provide
decision makers with confidence that performance data are credible because
they do not include specific information on the procedures to verify and
validate performance information. In addition, no information on the
sources of data is included. Thus, decision makers may not have complete
information needed to determine whether reported data and performance
results are accurate.
The scorecard and plan we reviewed did not include some contextual
information that could have helped to inform decision makers because DOD
guidance did not require the defense agencies to do so. However, a sample
measure from DISA's draft September 2003 balanced scorecard depicts the
type and amount of contextual information that agency officials have
considered incorporating.
DLA's balanced scorecard lacks information about the agency's mission and
customers that had been included in its performance contract. In contrast,
DODEA's performance plan provides general information about the agency's
mission, business lines, and customers. However, neither agency clearly
articulates how its performance strategies and measures help achieve
mission or departmentwide goals or improve customer responsiveness. Also,
information about external factors, such as recent contingency operations,
that can affect achievement of performance goals is limited or absent. For
example, DODEA's performance plan includes a measure to maintain pupil to
teacher ratios for kindergarten through grade 12 between 18:1 and 24:1
during fiscal years 2004-2009. However, DODEA's performance plan does not
explain that studies have demonstrated that lower pupil to teacher ratios
contribute to improved student performance or how this measure supports
presidential and national goals against which the agency evaluates itself.
Providing this contextual information could assist officials with
realizing how human capital-related strategies, such as staffing, affect
the agency's goals to continue to improve student performance in reading,
math, and languages, and help establish links between resource needs and
performance outcomes.
In September 2003, DISA developed a draft balanced scorecard with a sample
measure that depicts the type and amount of contextual information that
agency officials have considered incorporating. The sample measure, as
currently structured, incorporates most of the attributes associated with
results-oriented management that we have discussed in this report,
including contextual information. However, the sample measure does not
include some aspects of data quality such as reliability, validity, and
verification, and it is unclear whether DISA plans to discuss external
factors that could affect the achievement of performance goals or the link
between resource needs and performance beyond funding problems. For
example, as currently structured, the draft scorecard would
(1) align measures with agency and departmentwide goals. The scorecard is
expected to identify the quadrant of DISA's scorecard that the measure
supports (e.g., customer) and the Assistant Secretary of Defense's
balanced scorecard goal and risk management framework area with which it
aligns.
(2) assign accountability for achieving results. The scorecard is
anticipated to identify individuals responsible for monitoring and
reporting on performance and making corrections, as needed for
each measure.
(3) demonstrate results. The scorecard is expected to define the measure
and present trend data showing performance improvements over 4 fiscal
years.
(4) discuss data quality. The scorecard is expected to identify data
sources but does not include information about the way data are collected
or the verification and validation procedures.
(5) provide contextual information. The scorecard is expected to include
historical data, and an assessment of the measure's status compared to
prior years, and identify any challenges that need attention (e.g., hot
issues) such as funding problems.
Figure 4 shows the content that DISA officials told us that they expect to
include and the way this information addresses the attributes we have
identified in this report.
Figure 4: Format of a Sample Measure from DISA's Draft September 2003 Balanced
Scorecard
Source: GAO analysis of DISA data.
Notes: PSA=Principal Staff Assistant. TBD=To Be Determined.
DOD Captures and Shares Lessons Learned
OSD/PA&E officials have developed some mechanisms to help the defense
agencies capture and share experiences and lessons learned from developing
performance plans and balanced scorecards. Some defense agencies, such as
DLA and DISA, are sharing their experiences with each other to leverage
lessons learned.
OSD/PA&E has taken some steps to help the defense agencies share their
experiences and lessons learned with developing performance plans and
balanced scorecards, and DOD plans additional initiatives. In response to
our suggestions during this review, an OSD/PA&E official decided to
continue to hold semiannual roundtables during which defense agency
officials could voluntarily share their experiences developing and using
performance plans or balanced scorecards. OSD/PA&E officials also told us
that they plan to revise and update a departmentwide Web site originally
used to disseminate agencies' performance contracts and related guidance
and to incorporate this information, as well as defense agencies'
performance plans and scorecards, into DOD's performance management Web
site. The original site was not updated as frequently as it could have
been, and firewalls used to protect information from unauthorized access
precluded defense agencies with domain names that did not end in ".mil"
from gaining access. For example, DODEA officials were not able to access
the original Web site because the agency's World Wide Web domain address
ends in ".edu." Consequently, DODEA could not leverage other agencies'
contracts and performance measures when developing its own. DODEA
officials told us that they were not even aware that the Web site existed,
but that it would have been helpful for identifying some performance
measures.
In addition, some defense agencies that we reviewed are proactively
sharing lessons learned from developing and implementing balanced
scorecards. For example, DISA officials told us that they contacted DLA to
learn how it had developed and implemented its balanced scorecard. In
addition, officials at both agencies told us that they have or are
developing internal Web sites to facilitate performance measurement.
DISA's internal Web site serves as a collaborative forum and repository
for balanced scorecard material, and provides related guidance on
developing balanced scorecards.
Conclusions DOD has improved its process for measuring and managing
defense agency performance by taking steps to include certain attributes
associated with results-oriented management in defense agencies'
performance plans and scorecards that were not included in the
performance contracts. Specifically, changes made to DOD's process for
overseeing and reviewing defense agencies' balanced scorecards and
performance plans have the potential to further strengthen the
accountability of top leadership for overseeing the defense agencies and
achieving business transformation goals. While it is too early to
determine the extent to which these plans and scorecards will be fully
useful, these tools still do not incorporate some specific attributes that
would strengthen them. Until all performance plans and scorecards identify
individuals responsible for monitoring progress and taking corrective
actions, DOD's ability to ensure accountability and continuous assessment
at lower organizational levels is hindered. Furthermore, without
performance measures that are clearly defined and accompanied by trend
data and absent a discussion of data quality and contextual information-
such as information about the agency's missions, business lines, and
customers as well as external limitations that could affect achievement of
performance goals-senior OSD and agency leaders will not have the
necessary information for fully assessing the extent to which progress has
been made against existing performance targets and determining realistic
performance goals for future years. Finally, without demonstrating the
relationship between resource needs and performance outcomes, DOD
officials do not have all of the information needed to make improvements
and consider trade-offs among competing priorities.
To make performance plans and scorecards more informative and useful for
decision making, and further strengthen these tools' potential for
measuring and managing defense agency performance, we recommend that the
Secretary of Defense direct the Director, Program Analysis and Evaluation,
to revise guidance to require all defense agencies to address the
following five recommendations in their plans and scorecards:
o identify individuals accountable for achieving results at lower
organizational levels;
o include measures that are clearly defined and include trend data for
at least the past fiscal year's performance to help assess progress;
o identify resources needed to achieve performance goals and inform
budget decisions;
o discuss data quality, including the reliability, validity, and
limitations of performance measures as well as data sources; and
o provide contextual information to better understand how performance
measures support the agency's mission.
Recommendations for Executive Action
Agency Comments
and Our Evaluation
In written comments on a draft of this report, DOD concurred or partially
concurred with all of our recommendations. DOD's comments are reprinted in
their entirety in appendix V. DOD also provided technical comments that we
incorporated as appropriate.
Specifically, DOD concurred with our recommendations to revise guidance to
require defense agencies to include information on individuals accountable
for achieving results at lower organizational levels; include measures
that are clearly defined and include trend data for at least the past
fiscal year's performance; discuss data quality, including the
reliability, validity, and limitations of performance measures as well as
data sources; and provide contextual information to better understand how
performance measures support the agency's mission. DOD stated that it
would revise guidance to either implement or clarify these
recommendations.
DOD partially concurred with our recommendation to revise guidance to
require all defense agencies to identify resources needed to achieve
performance goals and inform budget decisions in their performance plans
or balanced scorecards. DOD stated that it may not be possible to fully
cost performance goals given the complex nature of defense operations and
that the appropriate process for proposing resource alternatives is the
Planning, Programming, Budgeting, and Execution System. However, DOD did
acknowledge that the defense agencies' performance plans should be
synchronized with this process by demonstrating how Planning, Programming,
Budgeting, and Execution System decisions are being implemented and by
providing performance information and metrics that can be used to inform
the department's annual budget and execution review and the allocation of
resources. We believe DOD is moving in the right direction by trying to
synchronize defense agency performance management with the budget process.
However, we continue to believe that the defense agencies' performance
plans and balanced scorecards would be an appropriate vehicle to discuss
resource implications. Full cost information may not be available, but the
performance plans and balanced scorecards should discuss resource
implications and include available cost information. Inclusion of this
information would help decision makers understand the connection between
performance goals and funding levels and evaluate competing needs at all
phases of the planning and budgeting process.
We are sending copies of this report to other interested congressional
committees; the Secretary of Defense; the Secretaries of the Army, the
Navy, and the Air Force; the Commandant of the Marine Corps; and the
Director, Office of Management and Budget. We will make copies available
to others upon request. In addition, the report will be available at no
charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please call
Sharon Pickup at (202) 512-4300 or Patricia Dalton at (202) 512-6806.
Additional contacts and staff acknowledgments are provided in appendix
VI.
Sharon L. Pickup
Director, Defense Capabilities and Management
Patricia A. Dalton
Director, Strategic Issues
Appendix I: Scope and Methodology
To meet our first two reporting objectives, we interviewed knowledgeable
Department of Defense (DOD), defense agency, and service officials.
Specifically, we interviewed officials in the offices of the Under and
Assistant Secretaries of Defense who act as principal staff assistants for
the three defense agencies reviewed. We also collected data from and
interviewed officials in the Office of the Secretary of Defense, Program
Analysis and Evaluation and the Office of the Comptroller. We also
reviewed our extensive prior work on results-oriented management and
identified seven attributes that contribute to effective government
performance. While these do not cover all attributes associated with
results-oriented management, we believe they are the most important for
accurately assessing the strengths and weaknesses of programs and making
improvements. These include (1) aligning goals and measures with agency
and departmentwide goals; (2) assigning accountability for achieving
results; (3) demonstrating results; (4) providing a comprehensive view of
agency performance by addressing factors that drive organizational
performance; (5) linking resource needs to performance; (6) discussing
data quality; and (7) providing contextual information. We systematically
analyzed the measures contained in selected performance contracts,
performance plans, and balanced scorecards against these attributes. We
focused our analysis on the Defense Logistics Agency (DLA), the Defense
Information Systems Agency (DISA), and the Department of Defense Education
Activity (DODEA). We judgmentally selected these three defense agencies to
review based on (1) their experience developing and implementing
performance contracts, plans, and balanced scorecards; (2) different types
of missions; and (3) types of services and types of customers. We analyzed
the most recent performance contracts used by DLA, DISA, and DODEA, and we
limited our analysis of balanced scorecards and performance plans to DLA's
fiscal year 2004 balanced scorecard and DODEA's fiscal year 2004
performance plan. Although we obtained DISA's draft balanced scorecard
dated September 2003, we did not evaluate it against the attributes we
discuss in this report because DISA had not yet finalized its scorecard by
the time we completed our review. However, we did discuss DISA officials'
plans for developing a balanced scorecard. We did not evaluate the
usefulness or appropriateness of the measures contained in the contracts,
plans, and balanced scorecards or verify actual or reported performance.
In addition to analyzing the performance contracts, performance plans, and
balanced scorecards, we also reviewed the defense agencies' strategic
plans and annual performance contract reports. Further, we reviewed the
cascaded-balanced scorecards that are used at lower organizational levels,
specifically, the fiscal year 2004 scorecards for DLA's supply centers
(Richmond, Virginia, Philadelphia, Pennsylvania, and Columbus, Ohio) and
Appendix I: Scope and Methodology
for its Defense Energy Support Center, Defense Distribution Center, and
Defense Reutilization and Marketing Service. We also reviewed school
improvement plans for three elementary and middle schools located at Fort
Bragg, North Carolina, to determine whether they aligned with DODEA's
community strategic plan and performance contract goals and measures.
To assess the extent to which DOD has established mechanisms to capture
and share lessons learned from the implementation of performance plans and
balanced scorecards, we interviewed knowledgeable DOD, defense agency, and
service officials and reviewed pertinent documents.
We performed our work between July 2003 and May 2004 in accordance with
generally accepted government auditing standards. We obtained comments on
a draft of this report from DOD and incorporated comments where
appropriate.
Appendix II: Department of Defense's Risk Management Framework
In 2001, the Department of Defense (DOD) established a risk management
framework to use in considering trade-offs among defense objectives and
resource constraints. The framework consists of four dimensions of risk
with supporting performance measures that are being cascaded to the
defense agencies and activities as well as departmentwide. Although not
depicted, financial considerations underlay each of DOD's risk areas.
Figure 5: DOD's Risk Management Framework
Source: DOD.
Appendix III: Fiscal Year When Defense Agencies Implemented Their First
Performance Contract, Plan, or Scorecard
This appendix shows the fiscal year during which each defense agency and
activity implemented its first performance contract or performance plan.
The defense agencies and activities that implemented performance contracts
later converted these to performance plans or scorecards. We use the
generic term "performance plan" in figures 6 and 7 to include both
performance plans and balanced scorecards.
Appendix III: Fiscal Year When Defense Agencies Implemented Their First
Performance Contract, Plan, or Scorecard
Figure 6: Defense Agencies
First Performance Contract/Plan Implemented
Fiscal Year 1999
Fiscal Year 2000
Fiscal Year 2001
Fiscal Year 2002
Fiscal Year 2004
DOD agencies without performance plans as of fiscal year 2004
Source: GAO analysis and presentation of DOD data.
ASD ATSD DUSD PDUSD USD
Assistant Secretary of Defense
Assistant to the Secretary of Defense
Deputy Under Secretary of Defense
Principal Deputy Under Secretary of Defense
Under Secretary of Defense
Appendix III: Fiscal Year When Defense Agencies Implemented Their First
Performance Contract, Plan, or Scorecard
Figure 7: Defense Field Activities
First Performance Contract/Plan Implemented ASD Assistant Secretary of
Defense
ATSD Assistant to the Secretary of DefenseFiscal Year 1999
DUSD Deputy Under Secretary of Defense
Fiscal Year 2000 PDUSD Principal Deputy Under Secretary of Defense
DOD field activities without performance USD Under Secretary of
Defenseplans as of fiscal year 2004
Source: GAO analysis and presentation of DOD data.
Appendix IV: Certain Key Attributes Associated with Results-Oriented
Management
Establishing goals and measuring performance are essential to successful
results-oriented management practices. Measuring performance allows
organizations to track the progress they are making toward their goals and
gives managers critical information on which to base decisions for
improving their programs. Our body of work on results-oriented management
practices has identified key attributes of success. This work indicates
that agencies that are successful in achieving business transformation
goals, such as those being focused on by the Department of Defense in its
performance contracts and plans, strive to establish practices and
performance systems at all levels of the agency that include the seven key
attributes described in this appendix.
o Aligns goals and measures with agency and departmentwide goals.
Performance goals and measures should align with an agency's long-term
strategic goals and mission as well as with higher-level departmentwide
priorities, with the relationship clearly articulated. Such linkage is
important in ensuring that agency efforts are properly aligned with goals
(and thus contribute to their accomplishment) and in assessing progress
toward achieving these goals. Goals and measures also should cascade from
the corporate level of the agency to the operational level to provide
managers and staff with a road map that shows how their day-to-day
activities contribute to achieving agency and departmentwide performance
goals. In addition, measures used at the lowest levels of the agency to
manage specific programs should directly relate to unit results and
upwards to the corporate level of the agency.
o Assigns accountability for achieving results. We have previously
reported that the single most important element of successful management
improvement initiatives is the demonstrated commitment of top leaders in
developing and directing reform efforts. Top leadership must play a
critical role in setting results-oriented goals and quantifiable measures
that are cascaded to lower organizational levels and used to develop and
reinforce accountability for achieving results, maintain focus on the most
pressing issues confronting the organization, and sustain improvement
programs and performance, especially during times of leadership
transition. One way to reinforce accountability is through the use of
employee performance appraisals that reflect an organization's goals.
o Demonstrates results. A key element of any major program is its
ability to establish clearly defined goals and performance measures to
monitor and report progress to management, identify corrective actions,
set priorities, and allocate resources. Performance measures
Appendix IV: Certain Key Attributes Associated with Results-Oriented
Management
should show an organization's progress toward achieving an intended level
of performance or results and include quantifiable targets or other
measurable values to help assess progress and trend data toward
accomplishing goals. Using a combination of output measures that assess
the actual level of work accomplished or services provided and outcome
measures that assess the actual results, effects, or impacts of a program
or service compared to the intended purpose can also help determine
progress made.
o Provides a comprehensive view of agency performance. For each key
business line, performance goals and measures should provide a
comprehensive view of performance, including customers' and stakeholders'
priorities. Goals and measures should address key performance dimensions
such as (1) factors that drive organizational performance-including
financial, customer, and internal business processes, and workforce
learning and growth-and (2) aspects of customer satisfaction, including
timeliness, quality, quantity, and cost of services provided. Doing so can
allow managers and other stakeholders to assess accomplishments, make
decisions, realign processes, and assign accountability without having an
excess of data that could obscure rather than clarify performance issues.
o Links resource needs to performance. One of the ways that performance
management can be promoted is if this information becomes relevant for (1)
identifying resources (e.g., human capital, information technology, and
funding) needed to achieve performance goals; (2) measuring cost; and (3)
informing budget decisions. When resource allocation decisions are linked
to performance, decision makers can gain a better understanding of the
potential effect of budget increases and decreases on results.
o Discusses data quality. A key factor that contributes to the
usefulness of performance data is the degree to which officials are
confident that the performance information is credible. Useful practices
for helping decision makers to assess the quality and value of performance
data include discussion of data sources and standards and methods-
including data collection and reliability, verification and validation
procedures, and proposals to review these procedures-to assess the quality
of performance data. Reliability refers to whether standard procedures for
collecting and calculating results can be applied to measures so that they
produce the same results if applied repeatedly to the same situation.
Validity is the extent to which data adequately represent actual
performance. Verification is the assessment of data
Appendix IV: Certain Key Attributes Associated with Results-Oriented
Management
completeness, accuracy, consistency, timeliness, and related quality
control practices.
o Provides contextual information. Performance reporting systems should
include information to help clarify aspects of performance that are
difficult to quantify or to provide explanatory information such as
factors that were within or outside the control of the agency. This
information is critical to identifying and understanding the factors that
contributed to a particular result and can help officials measure, assess,
and evaluate the significance of underlying factors that may affect
reported performance. In addition, this information can provide context
for decision makers to establish funding priorities and adjust performance
targets and assess means and strategies for accomplishing an
organization's goals and objectives.
Appendix V: Comments from the Department of Defense
Now on p. 36.
Now on p. 36.
Now on p. 36.
Appendix V: Comments from the Department of Defense
Now on p. 36.
Now on p. 36.
Appendix VI: Key Contact and Staff Acknowledgments
GAO Contact Marilyn K. Wasleski (202) 512-8436
Acknowledgments Gabrielle M. Anderson, Krislin M. Bolling, Elizabeth H.
Curda, Benjamin T. Licht, William A. McKelligott, Elizabeth G. Mead,
Marjorie L. Pratt, and Robert K. Wild also made significant contributions
to the report.
Related GAO Products
Future Years Defense Program: Actions Needed to Improve
Transparency of DOD's Projected Resource Needs. GAO-04-514.
Washington, D.C.: May 7, 2004.
Department of Defense: Further Actions Needed to Establish and
Implement a Framework for Successful Business Transformation.
GAO-04-626T. Washington, D.C.: March 31, 2004.
Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.:
March 10, 2004.
Satellite Communications: Strategic Approach Needed for DOD's
Procurement of Commercial Satellite Bandwidth. GAO-04-206.
Washington, D.C.: December 10, 2003.
Defense Inventory: Several Actions Needed to Further DLA's Efforts to
Mitigate Shortages of Critical Spare Parts. GAO-03-709. Washington,
D.C.: August 1, 2003.
Results-Oriented Cultures: Creating a Clear Linkage between Individual
Performance and Organizational Success. GAO-03-488. Washington, D.C.:
March 14, 2003.
Defense Management: New Management Reform Program Still Evolving.
GAO-03-58. Washington, D.C.: December 12, 2002.
Tax Administration: IRS Needs to Further Refine Its Tax Filing Season
Performance Measures. GAO-03-143. Washington, D.C.:
November 22, 2002.
Performance Budgeting: Opportunities and Challenges. GAO-02-1106T.
Washington, D.C.: September 19, 2002.
Information Technology: Defense Information Systems Agency Can
Improve Investment Planning and Management Controls. GAO-02-50.
Washington, D.C.: March 15, 2002.
Defense Management: Actions Needed to Sustain Reform Initiatives and
Achieve Greater Results. GAO/NSIAD-00-72. Washington, D.C.:
July 25, 2000.
Related GAO Products
Management Reform: Elements of Successful Improvement Initiatives
GAO/T-GGD-00-26. Washington, D.C.: October 15, 1999.
Performance Plans: Selected Approaches for Verification and Validation of
Agency Performance Information. GAO/GGD-99-139. Washington, D.C.: July 30,
1999.
The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1, 1998.
Executive Guide: Effectively Implementing the Government Performance and
Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Gloria Jarmon, Managing Director, [email protected](202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
PRINTED ON RECYCLED PAPER
Presorted Standard Postage & Fees Paid GAO Permit No. GI00
United States
Government Accountability Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***