Defense Management: Additional Actions Needed to Enhance DOD's
Risk-Based Approach for Making Resource Decisions (15-NOV-05,
GAO-06-13).
The Department of Defense (DOD) is simultaneously conducting
costly military operations and transforming its forces and
business practices while it is also competing for resources in an
increasingly constrained fiscal environment. As a result, GAO has
advocated that DOD adopt a comprehensive threat or risk
management approach as a framework for decision making. In its
2001 strategic plan, the Quadrennial Defense Review (QDR), DOD
stated its intent to establish an approach--the risk management
framework--to balance priorities against risk over time and
monitor results against its strategic goals. GAO was asked to (1)
assess the extent to which DOD has implemented the framework,
including using it to make investment decisions, and (2) identify
the most significant challenges DOD faces in implementing the
framework, or a similar approach.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-13
ACCNO: A41497
TITLE: Defense Management: Additional Actions Needed to Enhance
DOD's Risk-Based Approach for Making Resource Decisions
DATE: 11/15/2005
SUBJECT: Accountability
Decision making
Defense budgets
Defense capabilities
Internal controls
Performance measures
Policy evaluation
Risk management
Strategic planning
Transparency
DOD Quadrennial Defense Review
OMB Program Assessment Rating Tool
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-13
Defense Management Defense Management Defense Management Defense
Management Defense Management Defense Management Defense Management
Defense Management Defense Management Defense Management Defense
Management Defense Management Defense Management Defense Management
Defense Management Defense Management Defense Management Defense
Management Defense Management Defense Management Defense Management
Defense Management Defense Management Defense Management Defense
Management Defense Management Defense Management Defense Management
Defense Management Defense Management Defense Management Defense
Management Defense Management Defense Management Defense Management
Defense Management
Contents
Letter 1
Results in Brief 3
Background 5
Despite Positive Steps, Additional Actions Needed to Fully Implement the
Risk Management Framework 9
Cultural Resistance, Combined with the Lack of Leadership, Implementation
Goals, and Process Integration, Affects DOD's Implementation of the Risk
Management Framework 18
Conclusions 24
Recommendations for Executive Action 25
Agency Comments and Our Evaluation 25
Appendix I Scope and Methodology 29
Appendix II Comments from the Department of Defense 31
Appendix III GAO Contact and Staff Acknowledgments 34
Tables
Table 1: Definitions and Examples of DOD Department-Level Measures (as of
November 2004) 11
Table 2: The Number of Activity and Performance Measures for Each Quadrant
12
Table 3: Military Service and Defense-Wide Percentage of the 2005 and 2006
Future Years Defense Programs 16
Table 4: Select Initiatives to Improve Investment Decision Making 22
Figures
Figure 1: The Risk Management Cycle 6
Figure 2: Comparison of the Balanced Scorecard and the Risk Management
Framework 9
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
Abbreviations
CBO Congressional Budget Office
CMO chief management official
DOD Department of Defense
FYDP Future Years Defense Program
GAO Government Accountability Office
GPRA Government Performance and Results Act
JCIDS Joint Capabilities Integration and Development System
OSD Office of the Secretary of Defense
PA&E Program Analysis and Evaluation (PA&E)
PPBE Planning, Programming, Budgeting, and Execution
P&R Personnel and Readiness
PART Program Assessment Rating Tool
QDR Quadrennial Defense Review
United States Government Accountability Office
Washington, DC 20548
November 15, 2005
The Honorable John Ensign Chairman The Honorable Daniel K. Akaka Ranking
Minority Member Subcommittee on Readiness and Management Support Committee
on Armed Services United States Senate
Among the 21st century challenges facing the Department of Defense (DOD)
and the nation as a whole are difficult decisions concerning how to strike
an affordable balance between current and future national security needs
and between national security and domestic needs.1 For example, DOD is
simultaneously maintaining a high pace of military operations for
combating terrorism and transforming its military forces and business
operations for the 21st century while it is also competing for federal
resources in an increasingly fiscally constrained environment. We have
advocated that DOD-as well as the rest of the federal government-adopt a
comprehensive threat or risk management approach as a framework for
decision making.2 This approach would fully link strategic goals to plans
and budgets; assess the values and risks of various courses of actions as
a tool for reexamining defense programs, setting priorities, and
allocating resources; and use performance measures to assess outcomes.
To its credit, DOD introduced a balanced scorecard for risk management,
commonly known as the risk management framework, in its strategic plan,
the 2001 Quadrennial Defense Review (QDR) report. The 2001 strategic plan
articulated the new administration's emphasis on transforming military
forces and defense business practices to meet the emerging challenges
facing our nation. DOD intended the framework to be used as a management
tool to focus DOD's efforts on implementing the defense program as
outlined in the strategic plan. In particular, DOD's senior leadership
intended the risk management framework to assist decision makers in
formulating top-down strategy, balancing investment priorities against
risk over time, measuring near- and midterm outputs against strategic
goals, and focusing on actual performance results. According to DOD
officials, the risk management framework also was intended to increase
transparency within the department over the decision-making process.
During the ongoing 2005 QDR, DOD plans to refine the risk management
framework.
1See GAO, 21st Century Challenges: Reexamining the Base of the Federal
Government, GAO-05-325SP (Washington, D.C.: February 2005) for a
comprehensive compendium of areas throughout the federal government that
could be considered for reexamination and review by Congress.
2GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January
2005).
You asked us to examine the status of DOD's efforts to adopt a risk-based
approach to decision making, given the emphasis that DOD was placing on
the risk management framework. In response, we (1) assessed the extent to
which DOD has implemented its risk management framework, including the
extent to which DOD has used the framework to make investment decisions;
and (2) identified the most significant challenges DOD faced in
implementing the risk management framework or a similar risk-based and
results-oriented management approach.
To assess the extent to which DOD has implemented the risk management
framework, we analyzed key documents, policy guidance, data, and interview
results, and compared the analysis to the principles for managing risk and
results identified in prior GAO reports. In addition, we conducted
interviews with DOD and service officials, and members of the Joint Staff.
We discussed the department's progress in implementing the risk management
framework with members of the Defense Business Board. We also analyzed
DOD's department-level performance goals and measures that are associated
with the risk management framework and assessed how DOD reported that
information externally. We did not validate the appropriateness of the
risk management framework's risk quadrants or the procedures that DOD has
in place to ascertain the reliability of performance data and we also did
not assess the basis for DOD's investment decisions. To identify the most
significant challenges DOD faced in implementing the risk management
framework, we analyzed documents, data, and interview results, and
compared the results of this analysis to the key practices to assist
mergers and organizational transformation identified in prior GAO
reports.3 A more detailed discussion of our scope and methodology is
presented in appendix I.
Our work was performed from October 2004 through September 2005 in
accordance with generally accepted government auditing standards.
Results in Brief
DOD has taken positive steps toward implementing the risk management
framework; however, additional actions are needed before the framework is
fully implemented and DOD can demonstrate real and sustainable progress in
using a risk-based and results-oriented approach to strategically allocate
resources across the spectrum of its investment priorities. For example,
while DOD established four risk areas, or quadrants, and developed
performance goals and measures of two types-activity measures (measures to
track initiatives) and performance measures-the majority of these measures
do not provide sufficient information to monitor performance against the
risk quadrants' goals. Specifically, and contrary to results-oriented
management principles, the risk management framework's measures (1) do not
clearly demonstrate results, (2) do not provide a well-rounded depiction
of performance across the department, and (3) are not being systemically
monitored across all quadrants, except for the force management quadrant.
In addition, the framework's performance goals and measures are not
clearly linked to DOD's current strategic plan and strategic goals.
Lacking measures that follow results-oriented management principles and
clear linkages to strategic goals, DOD may be unable to provide a clear
roadmap of how its activities at all levels contribute to meeting DOD's
strategic goals. Finally, although DOD officials stated that risk was
considered in the fiscal year 2006 budget cycle, the fiscal year 2006
budget submission does not include any specific information on how DOD
systematically identified or assessed departmental risks to establish
DOD-wide investment priorities. Therefore, the linkages between the risk
management framework and the budget are unclear. Without better measures,
clear linkages, and greater transparency, DOD will be unable to fully
measure progress in achieving strategic goals or demonstrate to Congress
and others how it considered risks and made trade-off decisions, balancing
needs and costs for weapon system programs and other investment
priorities.
3GAO, Highlights of a GAO Forum: Mergers and Transformation: Lessons
Learned for a Department of Homeland Security and Other Federal Agencies,
GAO-03-293SP (Washington, D.C.: Nov. 14, 2002), and Results-Oriented
Cultures: Implementation Steps to Assist Mergers and Organizational
Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003).
DOD faces four key challenges that affect its ability to fully implement
the risk management framework, or a similar risk-based and
results-oriented management approach: (1) overcoming cultural resistance
to the transformational change represented by such an approach in a
department as massive, complex, and decentralized as DOD; (2) maintaining
sustained leadership and clear accountability for this cultural
transformation; (3) providing implementation goals and timelines to gauge
progress in transforming the culture; and (4) integrating the risk
management framework with decision support processes and related reform
initiatives into a coherent, unified management approach for the
department. Our prior work on results-oriented management and
organizational transformation and mergers has shown that addressing these
challenges is at the center of successful change management efforts in
leading organizations. DOD is having difficulties implementing the
framework because it has not addressed these four challenges. With respect
to the first challenge, DOD's size and complexity result in a culture that
makes developing department-level approaches to priority setting and
investment decision making difficult. For example, the allocation of
budgets on a proportional, rather than a strategic, basis among the
services is a long-standing budgetary problem that we have reported about
for years. Second, the lack of sustained leadership and clear
accountability for the framework's implementation has resulted in a lack
of emphasis and understanding of its status and purpose within the
department. Because of the lack of sustained leadership for other
management reform efforts, we have supported legislation to create a chief
management official (CMO) at DOD to provide this leadership.4 Third, DOD
did not establish implementation goals or timelines with which to
establish accountability, measure progress, and build momentum. Finally,
integrating the risk management framework with other decision support
processes and related reform initiatives into a coherent, unified
management approach is a challenge that DOD intends to address in the
ongoing 2005 QDR. Illustrating this challenge, DOD is attempting to
implement the risk management framework while it is also shifting to
biennial budgeting and reforming defense planning. Our work has shown that
if risk-based and results-oriented management approaches are to be
successfully implemented, they must be integrated into the usual cycle of
agency decision making. Unless DOD addresses these challenges and
successfully implements the risk management framework, or a similar
approach, it may continue to experience (1) a mismatch between programs
and budgets, and (2) the proportional, rather than strategic, allocation
of resources to the services. Therefore, Congress may have insufficient
transparency into how DOD has identified and assessed risks and made
trade-offs in its investment decision making.
4S. 780, 109th Cong. S:1 (2005).
In this report, we recommend that DOD take various actions to increase its
chances of successfully implementing a risk-based approach for investment
decision making. This includes developing results-oriented measures and
assigning clear leadership with appropriate accountability and authority
to implement and sustain the risk management framework, or a similar
approach. In written comments on a draft of this report, DOD partially
concurred with our recommendations. DOD's comments and our evaluation of
them are on page 25 of this report.
Background
In our report, High-Risk Series: An Update,5 we identified agencies' lack
of comprehensive risk management strategies as an emerging challenge for
the federal government. Increasingly limited fiscal resources across the
federal government, coupled with the emerging requirements from the
changing security environment, emphasize the need for DOD to develop a
risk-based strategic investment approach. For this reason, we have
advocated that DOD adopt a comprehensive risk management approach for
decision making.6 Furthermore, DOD and other federal agencies are required
by statute to develop a results-oriented management approach to
strategically allocate resources on the basis of performance.7 The
balanced scorecard-a concept to balance an organization's focus across
financial, customer, internal business, and learning and growth management
areas-is one approach for developing results-oriented management that
government agencies have recently started to adopt.8 At the direction of
the Secretary of Defense, DOD developed a risk management framework that
DOD later aligned with its results-oriented management activities through
a DOD balanced scorecard.
5GAO-05-207.
6GAO-05-207.
7The Government Performance and Results Act of 1993 (Pub. L. No. 103-62).
8The balanced scorecard approach was advocated by Professor Robert Kaplan
and Dr. David Norton in the November/December 1992 Harvard Business
Review.
Risk Management Is an Emerging 21st Century Challenge
An emerging challenge for the federal government involves the need for the
completion of comprehensive national threat and risk assessments in a
variety of areas. For example, emerging requirements from the changing
security environment, coupled with increasingly limited fiscal resources
across the federal government, emphasize the need for agencies to adopt a
sound approach to establishing resource decisions.9 We have advocated that
the federal government, including DOD, adopt a comprehensive threat or
risk management approach as a framework for decision making that fully
links strategic goals to plans and budgets, assesses values and risks of
various courses of actions as a tool for setting priorities and allocating
resources, and provides for the use of performance measures to assess
outcomes. Based on our review of the literature,10 as shown in figure 1,
the goal of risk management is to integrate systematic concern for risk
into the usual cycle of agency decision making and implementation.
Figure 1: The Risk Management Cycle
9GAO-05-325SP.
10See for example, Committee of Sponsoring Organizations of the Treadway
Commission, Enterprise Risk Management-Integrated Framework: Executive
Summary (New York, N.Y.: September 2004).
A risk management cycle represents a series of analytical and managerial
steps, basically sequential, that can be used to assess risk, evaluate
alternatives for reducing risks, choose among those alternatives,
implement the alternatives, monitor their implementation, and continually
use new information to adjust and revise the assessments and actions, as
needed. Adoption of a risk management cycle such as this can aid in
assessing risk by determining which vulnerabilities should be addressed,
and how they should be addressed, within available resources. For the
purposes of this report, we focused on the stages of the risk management
cycle that involve DOD's actions to set strategic goals and objectives,
establish investment priorities based on risk assessments, and
implementation and monitoring.
Risk management's objectives are essentially the same as those of good
management, and they are consistent with the broad economy and efficiency
objectives of good government-namely, to provide better outcomes for the
same amount of money, or to provide the same outcomes with less money.
Therefore, risk management's objectives are also compatible with those of
the federal government's results-oriented management approach, which was
enacted in the Government Performance and Results Act (GPRA) of 1993,11
and the balanced scorecard approach. Congress enacted GPRA to focus the
federal government on achieving results through the creation of clear
links between the process of allocating scarce resources and an agency's
strategic goals, or the expected results to be achieved with those
resources. Building on GPRA's foundation, the current administration has
taken steps to strengthen the integration of budget, cost, and performance
information by including budget and performance integration as one of its
management initiatives under the umbrella of the President's Management
Agenda.12 The Budget and Performance Integration initiative includes
efforts such as the Program Assessment Rating Tool (PART), improving
outcome measures, and improving monitoring of program performance.13 The
balanced scorecard approach is a management tool that some federal
agencies have adopted to help them translate the strategy set forth in a
results-oriented management approach into the operational objectives that
drive both behavior and performance. The balanced scorecard consists of
four management areas that organizations should focus on-financial,
customer, internal business, and learning and growth.
11Pub. L. No. 103-62 (1993).
12The President's Management Agenda, by focusing on a number of targeted
areas, seeks to improve the performance management of the federal
government.
DOD's 2001 Strategic Plan Outlines a New Risk Management Framework
DOD introduced the risk management framework in its strategic plan, the
2001 QDR report. The 2001 strategic plan articulated the new
administration's emphasis on transforming military forces and defense
business practices to meet the changing threats facing our nation. In his
guidance to the department for the 2001 QDR strategic planning process,
the Secretary of Defense stated the need for DOD to use a risk mitigation
approach for balancing force, resource, and modernization requirements
across defense planning timelines. This guidance also stated that DOD must
include the identification of output-based measures to reduce
inefficiencies through the department in any approach to risk management.
Building on the guidance, the 2001 QDR outlined DOD's risk management
framework. According to the QDR, the framework would enable DOD to address
the tension between preparing for future threats and meeting the demands
of the present with finite resources. It was also intended to ensure that
DOD was sized, shaped, postured, committed, and managed with a view toward
accomplishing the strategic plan's defense policy goals.
DOD adapted the balanced scorecard concept to the risk management
framework by substituting the four dimensions of risk-force management,
operational, future challenges, and institutional-for the scorecard's four
management areas. The risk management framework was to be a
transformational tool that would provide a balanced perspective of the
organization's execution of strategy and ensure a top-down approach. The
2002 policy guidance also designated four preliminary performance goals
for each of the four risk quadrants. In addition, the guidance required
that performance goals and measures were to be cascaded to the services
and defense agencies. Figure 2 shows a comparison, as provided by DOD.
13For further information see: GAO, Performance Budgeting: PART Focuses
Attention on Program Performance, but More Can Be Done to Engage Congress,
GAO-06-28 (Washington, D.C.: Oct. 28, 2005); Management Reform: Assessing
the President's Management Agenda, GAO-05-574T (Washington, D.C.: Apr. 21,
2005); Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.:
Mar. 10, 2004); and Performance Budgeting: Observations on the Use of
OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget,
GAO-04-174 (Washington, D.C.: Jan. 30, 2004).
Figure 2: Comparison of the Balanced Scorecard and the Risk Management
Framework
Despite Positive Steps, Additional Actions Needed to Fully Implement the Risk
Management Framework
Despite positive steps, DOD needs to take additional actions before the
risk management framework is fully implemented and DOD can demonstrate
real and sustainable progress in using a risk-based and results-oriented
approach to strategically allocate resources across the spectrum of its
investment priorities. For example, DOD is still in the process of
developing department-level measures for the framework that address
results-based management principles, such as linking performance
information to strategic goals so that this information can be used to
monitor performance results and determine how well the department is doing
in achieving its strategy. Without more results-oriented performance
measures, DOD may be unable to provide the services and other defense
components with clear roadmaps of how their activities contribute to
meeting DOD's strategic goals. In addition, the framework's performance
goals and measures are not clearly linked to DOD's current strategic plan
and strategic goals. Furthermore, the extent to which the risk management
framework is linked to the budget cycle is unclear. Without better
measures, clear linkages, and greater transparency, DOD will be unable to
fully measure progress in achieving strategic goals or demonstrate to
Congress and others how it considered risks and made trade-offs in making
investment decisions.
Developing a Set of Measures That Can Be Used to Monitor Performance Is a Work
in Progress
DOD has taken positive steps toward developing measures for each of the
performance goals under the framework's four risk quadrants; however,
developing a set of measures that can be used to monitor performance
results is still a work in progress. Based on GAO's prior work on
results-based management principles, we found that leading organizations'
performance measures are: (1) designed to demonstrate results, or provide
information on how well the organization is achieving its goals; (2)
limited to a vital few, and balanced across priorities; and (3) used by
management to improve performance.14 However, the set of measures DOD has
developed for the risk management framework do not adequately address
these principles. While DOD established four risk quadrants and developed
performance goals and measures of two types-activity measures (measures to
track initiatives) and performance measures-the majority of its measures
do not provide sufficient information to monitor performance against the
risk quadrants' goals.
First, DOD officials acknowledge that establishing department-level
measures for the framework that demonstrate results is still a work in
progress, as the majority of the risk management framework's measures
require further development or refinement. In fact, as shown in table 1,
44 of the 77 department-level measures for all four quadrants, or over 50
percent, are activity measures. According to DOD sources, activity
measures are to result in a new performance measure, a new baseline or
benchmark, or define a new capability, rather than monitor a specific
annual performance target. Once these activities are completed, DOD
officials stated that the department will be better able to monitor
department-level performance against strategic goals. However, our
analysis found that the activity measures, as defined in DOD's external
reports, typically do not provide sufficient information to monitor the
department's progress in achieving the stated goal they are to measure,
such as developing a new performance measure or baseline. The desired
outcomes for activity measures generally state that a task was or will be
completed by a certain date but they do not provide sufficient information
on whether the activity is on schedule, the interdependencies among tasks,
or the contribution toward enhancing the department's performance.
Therefore, Congress and other external stakeholders lack information and
adequate assurances that DOD is making progress in implementing a
risk-based and results-oriented management approach to making investment
decisions.
14GAO, Executive Guide: Effectively Implementing the Government
Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996)
and Managing for Results: Enhancing Agency Use of Performance Information
for Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9,
2005).
Table 1: Definitions and Examples of DOD Department-Level Measures (as of
November 2004)
Description of
desired outcome
monitored by
Type Number Definition Examples measure
Activity 44a Activity measures track Deny enemy Roadmap will be
measures developmental advantages and complete by the
activities, are usually exploit end of fiscal
qualitative, and track weaknesses year 2005
key milestones or events Enhance Strategy will
in lieu of a specific homeland be complete by
annual performance defense and the first
target consequence quarter of
management fiscal year
2005
Performance 33a Performance measures Reserve Target > 90% of
measures track current outputs component recruits
and set quantitative enlisted holding high
annual targets for recruiting school diplomas
performance that are quality
measurable Actual 88% of
recruits
holding high
school diplomas
Reduce Target 15 days
customer wait from order to
time (in days) receipt for
material goods
Actual 24 days
from order to
receipt for
material goods
Source: GAO analysis of the Risk Management Framework's performance
measures.
a We have recoded five performance measures as activity measures as these
measures tracked milestones and events, which corresponds with DOD's
definition of an activity measure.
Second, DOD's department-level performance measures are still a work in
progress in that these measures do not provide a well-rounded depiction of
DOD's performance. In our previous work, we have found that performance
measurement efforts that are not balanced across priorities may skew an
agency's performance and keep its senior leadership from seeing the whole
picture.15 For example, in developing department-level measures for the
risk management framework, DOD appears to have overemphasized its force
management priorities at the expense of operational risk. As illustrated
in table 2, the operational risk quadrant has no performance measures,
while the force management risk quadrant has a total of 36 measures,
including 15 activity measures and 21 performance measures.
Table 2: The Number of Activity and Performance Measures for Each Quadrant
Activity
measures Performance measures Total measures
Force Management 15 21 36a
Operational 9 0 9
Institutional 11 10 21b
Future Challenges 9 2 11c
Source: GAO analysis of DOD data.
a We have recoded two performance measures as activity measures as these
measures tracked milestones and events, which corresponds with DOD's
definition of an activity measure.
b We have recoded one performance measure as an activity measure as this
measure tracked milestones and events, which corresponds with DOD's
definition of an activity measure.
c We have recoded two performance measures as activity measures as these
measures tracked milestones and events, which corresponds with DOD's
definition of an activity measure.
In providing technical comments to a draft of this report, DOD objected to
our recoding of five department-level performance measures as activity
measures. We recoded these measures because they tracked milestones and
events, which corresponded to DOD's definition of an activity measure. The
measures we recoded addressed the following:
o a civilian human resources strategic plan,
o a military human resources strategic plan,
o monitor the status of defense technology objectives,
o strategic transformation appraisal, and
o support acquisition excellence goals.
15GAO/GGD-96-118.
Finally, DOD officials indicated that DOD is systematically using
performance measures to monitor progress and improve performance for only
one risk quadrant, although individual measures under the other three risk
quadrants may be monitored. We have found that leading organizations use
performance information to improve organizational performance and identify
performance gaps, and to provide incentives that reinforce a
results-oriented management approach.16 According to DOD officials, the
force management quadrant is the only quadrant that is managed by one
individual and one office-the Under Secretary of Defense for Personnel and
Readiness and his office. These officials stated that this situation is a
critical factor in the progress DOD has made in systematically monitoring
performance across the force management quadrant on a routine basis. For
example, officials stated that the Under Secretary of Defense personally
leads quarterly monitoring sessions on the force management quadrant's
performance. DOD officials also told us that the Under Secretary of
Defense for Personnel and Readiness has greatly facilitated this
monitoring by developing a centralized database to capture the performance
data used to track DOD's performance in meeting the quadrant's goals.
Unless all of the risk management framework's quadrants are systematically
monitored, implementation of the framework may be hindered and the
framework risks becoming a paper-driven, compliance exercise. Indeed, one
DOD official told us that he views the risk management framework and its
measures as a "reporting drill" and, in addition, his office would not
change its processes if DOD was to no longer use the framework.
Cascading the Risk Management Framework's Goals and Measures Is an Ongoing
Effort
DOD is still in the process of cascading the risk management framework's
goals and measures to the services. We have found that leading
organizations seek to establish clear hierarchies of goals and measures
that cascade down so that subordinate units have straightforward roadmaps
to demonstrate how their activities contribute to meeting the
organization's strategy.17 According to DOD officials, all of the services
are attempting to align their existing performance measures with the
department-level performance goals and measures. However, service
officials said that it is challenging to cascade the department-level
activity measures, because these measures represent very broad initiatives
that may not be applicable at all DOD levels. Officials from one service
said they have had to develop new measures to align with the
department-level measures, because they had been assessing performance
with fewer measures than the Office of the Secretary of Defense had
developed.
16GAO/GGD-96-118 and GAO-05-927.
17GAO/GGD-96-118.
Developing a Strategic Plan with Clear Linkages between the Risk Management
Framework and Strategic Goals Is a Critical Next Step
The risk management framework's performance goals and measures are not
clearly linked-a key principle of results-oriented management-to a
coherent strategic plan.18 The development of such a strategic plan is a
critical next step in using a risk-based and results-oriented approach to
making investment decisions. Without these linkages, DOD cannot easily
demonstrate how achievement of a performance goal or measure contributes
to the achievement of strategic goals and ultimately the organization's
mission. Our previous work indicated that DOD's strategic plan, the 2001
QDR, did not provide a sound foundation for the risk management
framework.19 We reported that the usefulness of the 2001 QDR was limited
by the lack of focus on longer-term threats and requirements for critical
support capabilities, and provided few insights into how future threats
and planned technical advances could affect future force requirements. In
turn, this lack of focus and insight limited the QDR's usefulness as a
foundation for fundamentally reassessing U.S. defense plans and programs
and for balancing resources across near- and midterm risks.
DOD officials indicated that DOD has not yet defined the linkages between
the risk management framework's performance goals and the strategic goals
in the 2001 QDR. Furthermore, the Defense Business Board's official
minutes for its July 28, 2005, meeting contained a recommendation that the
Secretary of Defense define department-level objectives, which should then
be cascaded down the department.20 In discussing the ongoing 2005 QDR, DOD
stated that although the department would continue its efforts to do so,
establishing these linkages was very challenging because of the size and
scope of DOD's operations. However, as suggested by the Defense Business
Board and our previous work, if DOD's strategic plan is to drive the
department's operations, a straightforward linkage is needed among
strategic goals, annual performance goals, and day-to-day activities.21
The ongoing 2005 QDR offers DOD the opportunity to strengthen its
strategic planning.
18GAO/GGD-96-118.
19GAO, Quadrennial Defense Review: Future Reviews Can Benefit from Better
Analysis and Changes in Timing and Scope, GAO-03-13 (Washington, D.C.:
Nov. 4, 2002).
20The Defense Business Board was established in 2001 by the Secretary of
Defense to provide DOD's senior leadership with leading-edge, actionable
advice on management improvements.
Although Risk Considered, Linkages Between the Risk Management Framework and
Budget Are Unclear
According to DOD officials, the department has begun to consider risk in
its investment decision making; however, the full extent to which the
framework's risk-based and results-oriented approach has been linked to
the fiscal year 2006 budget cycle is unclear. Our work indicates that
leading organizations link strategy to the budget process through
results-oriented management to evaluate potential investments or
initiatives.22
DOD sources indicated that the department has begun to consider risk
during its usual cycle of investment decision making. For example,
according to DOD sources, the Secretary of Defense articulated broad areas
for increasing or decreasing risk under each quadrant in the fiscal years
2006-2011 planning guidance, leaving it up to the defense components to
decide how to structure their investment decisions within those broad
areas consistent with the Secretary's risk guidance. In addition, DOD
officials stated that the framework has increased awareness within the
department on the need to balance risk over time. For example, when DOD
reduced the fiscal years 2006-2011 defense program by $30 billion, DOD
officials stated that the department did not take the traditional
budgetary approach of cutting each defense component's budget by a certain
percentage. Instead, DOD officials stated that the Secretary of Defense
used a collaborative approach with service participation to discuss where
to take the budget reductions and how these cuts would affect risk,
although DOD officials offered various views on how extensively the
framework was used to make those decisions.
Second, DOD required that the services and other defense components offset
any funding increase in one area with a funding decrease in another area
for the fiscal years 2006-2007 budget submission. According to DOD
officials, risk-whether on the basis of "professional judgment" or
analysis-was considered in these deliberations. For example, the Army's
plan for fiscal years 2006-2023 articulated areas for increasing risks so
that it could decrease risk in the operational risk dimension by investing
in current capacity.
21GAO, Managing for Results: Critical Issues for Improving Agencies'
Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept. 16, 1997).
22See GAO/GGD-96-118.
However, the fiscal year 2006 budget submission does not include any
specific information on how DOD systematically identified or assessed
departmental risks to establish DOD-wide investment priorities. For
example, the military services' share of the Future Years Defense Program
(FYDP) remained relatively unchanged from fiscal year 2005 to fiscal year
2006 (see table 3),23 providing one indication that the risk management
framework may not yet be a useful tool for balancing departmental risks
across the services.
Table 3: Military Service and Defense-Wide Percentage of the 2005 and 2006
Future Years Defense Programs
2005 Percentage 2006 Percentage Percentage change by
of FYDP of FYDP department
Department of the 24.23 24.63 0.40
Army
Department of the 29.75 29.47 -0.28
Navy
Department of the 29.80 29.82 0.02
Air Force
Defense-wide 16.22 16.08 -0.14
Total 100.00 100.00
Source: GAO analysis of DOD FYDP data.
Note: Totals may not add due to rounding.
DOD has reported on the risk management framework in the department's GPRA
and other reporting requirements. For example, the fiscal year 2004
Performance and Accountability Report describes what DOD is doing, or
plans to do, to define, measure, and monitor performance goals in the four
risk quadrants but does not discuss the implementation status of the risk
management framework. Furthermore, the fiscal year 2004 report, the most
recent available, provided insufficient information to assist Congress in
overseeing how DOD plans to prioritize investment decisions within or
across the risk quadrants. Without more detailed information, Congress may
have insufficient transparency into how DOD has identified and assessed
risks and made trade-offs in its investment decision making. In addition,
we reported in May 2004 that congressional visibility over investment
decision making also was limited by the absence of linkages between the
risk management framework and military capabilities planning and the
FYDP.24 Because the FYDP lacked these linkages, we concluded that decision
makers could not use it to determine how a proposed increase in capability
would affect the risk management framework.
23The Future Years Defense Program provides information on DOD's current
and planned outyear budget requests.
Our work also has shown that the FYDP may understate the costs of weapon
system programs; therefore, DOD may be starting more programs than it can
afford. For example, our assessment of 54 major programs, representing an
investment of over $800 billion, found that the majority of these programs
were costing more and taking longer to develop than planned.25 Problems
occurred because of DOD's overly optimistic planning assumptions about the
long-term costs of weapon system programs and its failure to capture early
on the requisite knowledge that is needed to efficiently and effectively
manage program risks. When DOD has too many programs competing for funding
and approves programs with low levels of knowledge, it is accepting the
attendant likely adverse cost and schedule risks. As a result, it will
probably get fewer quantities for the same investment or face difficult
choices on which investments it cannot afford to pursue. The findings of
our work suggest that having a departmentwide investment strategy for
weapon systems, to allocate resources across investment priorities, would
help reduce these risks.
24GAO, Future Years Defense Program: Actions Needed to Improve
Transparency of DOD's Projected Resource Needs, GAO-04-514 (Washington,
D.C.: May 7, 2004).
25GAO, Defense Acquisitions: Assessments of Selected Major Weapon
Programs, GAO-05-301 (Washington, D.C.: Mar. 31, 2005).
Cultural Resistance, Combined with the Lack of Leadership, Implementation Goals,
and Process Integration, Affects DOD's Implementation of the Risk Management
Framework
Four key challenges impede DOD's progress toward implementing the risk
management framework. The first implementation challenge facing DOD is
overcoming cultural resistance to change in a department as massive,
complex, and decentralized as DOD. The second challenge is the lack of
sustained leadership, and the third challenge is the absence of
implementation goals and timelines. These challenges relate to DOD's
failure to follow crucial transformational steps. The fourth
challenge-integrating the risk management framework with decision support
processes and related reform initiatives, into a coherent, unified
management approach for the department-relates to key results-oriented
management practices. Unless DOD addresses these challenges and
successfully implements the risk management framework, or a similar
approach, it may continue to experience (1) a mismatch between programs
and budgets, and (2) the proportional, rather than strategic, allocation
of resources to the services.
Transforming DOD's Organizational Culture Is a Significant Challenge
Transforming DOD's organizational culture-from a focus on inputs and
programs to strategically balancing investment risks and monitoring
outcomes across the department-through the implementation of the risk
management framework is a significant challenge for the department for
several reasons. First, as we noted in our 21st Century Challenges report,
to successfully transform, DOD needs to overcome the inertia of various
organizations, policies, and practices that became rooted in the Cold War
era.26 The department's expense, size, and complexity, however, make
overcoming this resistance and inertia difficult. In fiscal year 2004, DOD
reported that its operations involved $1.2 trillion in assets, $1.7
trillion in liabilities, over $605 billion in net cost of operations, and
over 3.3 million military and civilian personnel. For fiscal year 2005,
DOD received appropriations of about $417 billion. Moreover, execution of
its operations spans a wide range of defense organizations, including the
military services and their respective major commands and functional
activities, numerous large defense agencies and field activities, and
various combatant and joint operation commands, which are responsible for
military operations for specific geographic regions or theaters of
operations.
Second, DOD's highly decentralized management structure is another
contributing factor that makes cultural change difficult. Although under
the authority, direction, and control of the Secretary of Defense, the
military services have the legislative authority to organize, equip, and
train the nation's armed forces for combat under Title 10 of the U.S.
Code. Furthermore, Congress directly appropriates funds to the services
for programs and activities that support these purposes. In the opinion of
knowledgeable DOD officials, this legislative authority has resulted in a
culture that makes it difficult to develop department-level, or joint,
management approaches. For example, the allocation of budgets on a
proportional, rather than a strategic basis, among the military services
is a long-standing budgetary problem that we have identified as a major
management challenge for the department.27 In addition, the Joint Defense
Capabilities Study, chartered by the Secretary of Defense in March 2003,
made the following observations on how DOD's organizational culture does
not reinforce a departmental or joint approach to investment decision
making and results management:28
26GAO-05-325SP.
o DOD's bottom-up strategic planning process did not support
early senior leadership involvement and did not provide integrated
departmentwide objectives, priorities, and roles as a framework
for planning joint capabilities.
o Service-centric focus on programs and weapons platforms
resulted in a process that did not provide an accurate picture of
joint needs, nor did it provide a consistent view of priorities
and acceptable risks across the department.
o The resulting budget did not optimize capabilities at either
the department or the service level.
o Accountability and feedback focused on monetary input rather
than output; therefore, much of the information provided did not
support the senior leaders' decision making as it did not tell how
well the department was being resourced to meet current and future
mission requirements.
The lack of sustained leadership attention and appropriate
accountability has challenged DOD's progress in implementing the
risk management framework. Our work has indicated that sustained
leadership is a key transformational, or change management,
practice.29 However, knowledgeable DOD officials indicated that
DOD's senior leadership did not provide sustained attention to the
framework's implementation. For example, a DOD official actively
involved in the framework's implementation stated that meetings
with senior leadership that were to provide oversight of the
framework's implementation have not been regularly scheduled. DOD
officials indicated that as a result of this lack of sustained
leadership, DOD has not placed much emphasis on implementing the
risk management framework at the department level. In addition,
other DOD officials stated that changes in leadership have made it
difficult to implement the risk management framework or develop
performance measures. For example, since October 2004, DOD has
experienced turnover in the following senior level positions,
including the Deputy Secretary of Defense; the Under Secretary of
Defense for Acquisition, Technology and Logistics; and the
Director of Program Analysis and Evaluation (PA&E). Lacking
sustained leadership attention, DOD officials offered conflicting
perspectives on the status of the risk management framework with
some officials suggesting that the framework had been overtaken by
other performance-based or risk-based management initiatives while
another suggested that the framework was primarily a compliance
exercise. DOD officials also held differing perspectives on the
purpose of the framework, including the beliefs that it was
developed to monitor the Secretary of Defense's priority areas or
that it was a programming and budgeting tool.
Implementation of the risk management framework has also been
challenged by the lack of clear lines of authority and appropriate
accountability. No single individual or organization has been
given overarching leadership responsibilities, authority, or the
accountability for achieving the framework's implementation.
Instead, the responsibility for various tasks and performance
measures have been spread among several organizations, including
the Director, PA&E; the Under Secretary of Defense for Personnel
and Readiness (P&R); and the Under Secretary of Defense,
Comptroller/Chief Financial Officer.
We testified in April 2005 that as DOD embarks on large-scale
change initiatives, the complexity and long-term nature of these
initiatives require the development of an executive position
capable of providing strong and sustained leadership-over a number
of years and various administrations.30 For this reason, we have
supported legislation to create a CMO at DOD to provide such
sustained leadership.31 A CMO could also provide the leadership
needed to successfully develop a risk-based and results-oriented
management approach at DOD, such as the risk management framework.
Accountability for implementation of the risk management framework
also has been hindered by the absence of implementation goals and
timelines with which to gauge progress. As we have previously
reported, successful change management efforts use implementation
goals and timelines to pinpoint performance shortfalls and gaps,
suggest midcourse corrections, and build momentum by demonstrating
progress.32 However, DOD's limited guidance on the risk management
framework did not establish implementation goals and timelines,
nor did it require that implementation goals and timelines be
developed. According to knowledgeable DOD officials, DOD did not
see the need for implementation goals or timelines because the
framework was not meant to change processes or create new ones,
but rather was a management tool to improve upon investment
decision-making processes. Regardless of how DOD classifies the
risk management framework, we have found that implementation goals
and timelines are essential to any transformational change, such
as that envisioned by the Secretary of Defense with the risk
management framework, because of the number of years it can take
to complete the change.33 Moreover, the absence of implementation
goals and timelines makes it difficult to determine whether
progress has been made in implementing the framework over the last
2 1/2 years, and whether DOD's revisiting of the framework during
the 2005 QDR represents an evolutionary progression or
implementation delays.
DOD faces a significant challenge integrating the risk management
framework with decision support processes for planning,
programming, and budgeting and with related reform initiatives
into a coherent, unified management approach. The goal of both
risk management and results-oriented management is to integrate
the systematic concern for risk and performance into the usual
cycle of agency decision making and implementation. DOD's
challenge in meeting these goals is demonstrated by the number of
initiatives, as shown in table 4, that DOD has put in place to
improve investment decision making and manage performance results.
For example, both capabilities planning and the risk management
framework are to define risks and develop performance measures
but, according to DOD officials, the department is still
determining how to align capabilities planning with the risk
management framework. Other initiatives, including GPRA and PART,
are also to develop performance measures and DOD is still working
on integrating these initiatives with the risk management
framework and individual performance monitoring approaches of the
services and other defense components into a single, integrated
system. In December 2002, the Deputy Secretary of Defense issued a
memorandum to correct this situation by requiring the alignment of
the risk management framework and the President's Management
Agenda with DOD's results-oriented management activities,
including those associated with GPRA.
27GAO-05-325SP.
28Joint Defense Capabilities Study Team, Joint Defense Capabilities Study:
Final Report (Washington, D.C.: December 2003).
Lack of Sustained Leadership and Appropriate Accountability Has Challenged DOD's
Implementation of the Risk Management Framework
29GAO, Management Reform: Elements of Successful Improvement Initiatives,
GAO/T-GGD-00-26 (Washington, D.C.: Oct. 15, 1999).
Lack of Implementation Goals and Timelines Further Challenges DOD's
Implementation of Risk Management Framework
30GAO-05-520T and GAO-05-629T.
31S. 780, 109th Cong. S:1 (2005).
32GAO-03-669.
33GAO-03-669.
Integrating the Risk Management Framework with Decision Support Processes and
Related Reform Initiatives Is a Significant Challenge
Table 4: Select Initiatives to Improve Investment Decision Making
Initiative Description
Two-Year Planning, In 2003, DOD implemented a 2-year cycle for its
Programming, Budgeting, strategic planning, program development, and
and Execution Process resource determination process. DOD stated that
(PPBE) this change was needed to integrate DOD's
processes for strategic planning, identification
of needs for military capabilities, systems
development and acquisition, and program and
budget development. During the second year of the
biennial budget, DOD is to focus on budget
execution and program performance.
Enhanced Planning In fiscal year 2004, DOD initiated a reform of
Process defense planning to make it more responsive and
adaptive to the needs of senior decision makers.
The process is to result in fiscally constrained
guidance and priorities-for military forces,
modernization, readiness and sustainability, and
supporting business processes and infrastructure
activities-for program development. The enhanced
planning process is to integrate the outcomes of
operational, enterprise, and capabilities planning
efforts in a document called the Joint Programming
Guidance. The Joint Programming Guidance is to
provide a link between planning and programming,
and it is to provide guidance to the DOD
components for the development of their program
proposals.
Capabilities Planning The 2001 QDR announced a defense strategy built
around the concept of shifting to a
"capabilities-based" approach to defense.
According to the 2001 QDR, while DOD cannot know
with confidence what nation, group of nations, or
nonstate actor might pose a threat to U.S. vital
interests, it is possible to anticipate the
capabilities an adversary might employ.
Capabilities planning is to provide a top-down,
competitive approach to weigh options against
resource constraints across a spectrum of
challenges and to apportion risk against those
challenges. It is also to enable risk assessments
and trade-off decisions across DOD organizational
stovepipes. The new concept stresses joint
solutions to problems, requires the identification
of risk trade-offs within and across mission
areas, and treats uncertainty explicitly.
Program/Budget As part of the financial management enterprise
Framework Initiative initiatives of the Business Management
Modernization Program, this initiative is to
provide a foundation for a new program and budget
data structure using a common language that
enables senior level DOD decision makers to weigh
options versus resource constraints across a
spectrum of challenges. The framework is to
consist of a number of related data transparency
initiatives that span across all portions of the
PPBE process, including creating department-level
definitions for the four risk quadrants. One of
the stated benefits is establishing an ability to
view programs and resources based on the risk
management framework.
Joint Capabilities A system for the Joint Staff to assess gaps in
Integration and military joint warfighting capabilities and
Development System recommend solutions to resolve those gaps. This
(JCIDS) system is replacing DOD's requirements-generation
process for major acquisitions in an effort to
shift the focus to a more capabilities-based
approach for determining joint warfighting needs
rather than a threat-based approach focused on
individual systems and platforms. Under this
system, boards comprised of high-level DOD
civilians and military officials are to identify
future capabilities needed around key functional
concepts and areas, such as command and control,
force application, and battlespace awareness, and
to make trade-offs among air, space, land, and sea
platforms in doing so.
President's Management The President's Management Agenda contains five
Agenda initiatives aimed at improving federal agency
management and performance: (1) strategic human
capital management, (2) competitive sourcing, (3)
improved financial performance, (4) expand
electronic government, and (5) budget and
performance integration. The President cited our
work on high-risk areas and major management
challenges in developing his initiatives, and
implementation of the agenda has reinforced the
need to focus agencies' efforts on achieving key
management and performance improvements.
Budget and Performance The budget and performance integration initiatives
Integration of the President's Management Agenda include
elements such as the PART used to review programs,
an emphasis on improving outcome measures, and
improving monitoring of program performance. PART
is the central element in the performance
budgeting piece of the President's Management
Agenda. PART builds on GPRA by actively promoting
the use of results-oriented information to assess
programs in the budget.
Source: GAO analysis.
We note that these reform initiatives address key business processes
within the department and that we have placed DOD's overall business
transformation on our list of federal programs and activities at high risk
of waste, fraud, abuse, and mismanagement.34
34GAO-05-207.
The Under Secretary of Defense for Acquisition, Technology and Logistics
indicated that DOD plans to address the challenge associated with the
integration of DOD's planning, resourcing, and execution processes and
initiatives, including the risk management framework. The Under Secretary
stated that one task of the ongoing 2005 QDR was "strategic process
integration." The Under Secretary also stated that the department is
planning to provide a roadmap with performance goals and timelines on how
it will implement initiatives to improve strategic process integration.
This roadmap is to be submitted with the 2005 QDR report to Congress in
early 2006 with the fiscal year 2007 budget.
Conclusions
DOD has made some progress in implementing the risk management framework,
including establishing risk quadrants and performance goals. However, more
work will be required for DOD to be able to put in place a management
tool, such as the risk management framework, to strategically balance the
allocation of resources across the spectrum of its investment priorities
against risk over time and to monitor performance. The development of
performance measures that clearly demonstrate results and that are
cascaded down throughout the department would enable DOD to provide a
clear roadmap of how its activities at all levels contribute to meeting
its strategic goals and would assist the department in aligning the core
processes and resources of its four military services and multiple defense
agencies to better support a departmental or joint approach to national
security. Furthermore, the risk management framework cannot be fully
implemented until its performance goals are clearly linked to DOD's
strategic planning goals. Unless a cause and effect relationship can be
demonstrated between the department's performance measures and strategic
goals, the framework's usefulness as a tool for monitoring DOD's execution
of its strategic plan and identifying performance goals will be severely
restricted, if not eliminated. Furthermore, the fiscal year 2006 budget
submission does not provide sufficient information on how DOD identified
or assessed departmental risks to establish DOD-wide investment
priorities; thus, the linkages between the framework and the budget are
unclear. Without better measures, clear linkages, and greater
transparency, DOD will be unable to fully measure progress in achieving
strategic goals or demonstrate to Congress and others how it considered
risks and made trade-off decisions, balancing needs and costs for weapon
programs and other investment priorities.
The efforts of DOD's senior leadership to establish a risk-based and
results-oriented management approach have been impeded by some key
challenges. The lack of sustained leadership and clear lines of
accountability has hampered implementation of the risk management
framework and the establishment and achievement of implementation goals
and timelines. Strong and sustained leadership could enable DOD to
overcome resistance to change that exists in a department as massive and
complex as DOD. In addition, the establishment of implementation goals and
timelines could enable DOD to determine what progress has been made in
implementing the risk management framework. Furthermore, the successful
integration of the risk management framework into DOD's investment
decision-making processes, including recent reform initiatives, could
assist DOD in its overall transformation efforts. Until DOD develops a
risk-based and results-oriented management approach for making investment
decisions, it will likely continue to experience a mismatch between
programs and budgets, and the proportional, rather than strategic,
allocation of resources to the services.
Recommendations for Executive Action
To address the challenges associated with implementing the risk management
framework, or a similar risk-based management approach, we recommend that
the Secretary of Defense take the following four actions:
o develop or refine department-level performance measures so that
they clearly demonstrate performance results and cascade those
measures down throughout the department,
o assign clear leadership with accountability and authority to
implement and sustain the risk management framework,
o develop implementation goals and timelines, and
o demonstrate the integration of the risk management framework
with DOD's decision support processes and related reform
initiatives to improve investment decision making and manage
performance results.
In written comments on a draft of this report, DOD partially
concurred with our four recommendations. DOD's written comments
are reprinted in their entirety in appendix II. DOD also provided
technical comments, which we incorporated as appropriate.
DOD partially concurred with our first recommendation. DOD stated
that it concurred with our recommendation that the Secretary of
Defense refine department-level performance measures so that they
clearly demonstrate results, but that it did not concur with the
notion that effectively cascading the risk management framework
has been inhibited by the current suite of performance measures.
DOD noted that that a number of defense components-including the
Army, DOD Comptroller, the Defense Logistics Agency, and the
Defense Information Systems Agency-have successfully cascaded
departmentwide strategic goals and implemented frameworks to
measure their organization's performance. DOD also believes that
empowering the leadership at the component level to develop
measures, while ensuring strategic alignment, is the most
effective way of encouraging performance management and increasing
its utility. In our report, we acknowledge that DOD has taken
positive steps toward developing a performance monitoring system
and cascading the framework's goals and measures to defense
components. However, our recommendation addresses limitations in
those measures that currently hinder DOD's ability to use the risk
management framework as a management tool for aligning the
components' performance goals and measures with the risk
management framework, or for strategic balancing investment
decisions across the risk quadrants. For example, the majority of
the risk management framework's measures are activity measures, or
initiatives, that do not monitor a specific annual performance
target, nor do these measures provide sufficient information to
determine whether the activity is on schedule or contributes to
enhancing the department's overall performance. Finally, our
recommendation is not intended to suggest that DOD not empower the
components to develop performance measures, but rather that DOD
establish a clear hierarchy of goals and measures that provide
straightforward roadmaps to demonstrate how the components'
activities contribute to meeting DOD's strategic goals.
DOD partially concurred with our second recommendation that the
Secretary of Defense assign clear leadership with accountability
and authority to implement and sustain the risk management
framework. DOD stated that, although it agrees that such
leadership is key to any successful performance management system,
the department's senior executives provide sufficient leadership
and accountability for implementing and sustaining the risk
management framework. DOD also stated that it did not agree that a
new organization or bureaucratic structure is needed to ensure
successful implementation and sustainment of the risk management
framework. We agree that DOD has assigned specific roles and
responsibilities for goals and measures associated with the risk
management framework to various high-level DOD officials. However,
we based our recommendation on the fact that no single individual,
with appropriate authority, was held responsible for ensuring that
the risk management framework was implemented across the
department. Further, our recommendation does not propose that DOD
set up a new organization or bureaucratic structure, but, as
stated in this report, we continue to believe that one way to
provide strong and sustained leadership for change initiatives,
such as the risk management framework, over a number of years and
various administrations is to legislatively establish a CMO.
In partially concurring with our third recommendation to develop
implementation goals and timelines, DOD agreed that tracking
progress in implementing the risk management framework is a good
management practice. DOD stated that it has established goals and
timelines for the risk management framework that are unique to the
individual metrics, or measures, and that because the risk
management framework continually evolves over time, new metrics
will be developed while others may be retired. As we stated in the
report, successful change management efforts use implementation
goals-such as, for example, linking the risk management framework
to the budget-and timelines for meeting those goals, to pinpoint
shortfalls and gaps, suggest midcourse corrections, and build
momentum by demonstrating progress. Therefore, while DOD may
continually refine the individual goals and measures associated
with the framework's risk quadrants, we believe that goals and
timelines for the overall implementation of the framework across
the department are essential for keeping this reform initiative on
track.
DOD partially concurred with our fourth recommendation that the
Secretary of Defense demonstrate the integration of the risk
management framework with DOD's decision support processes and
related reform initiatives to improve investment decision making
and manage performance results. DOD stated that the department is
currently studying ways to further integrate the risk management
framework with other decision support processes, but no single
framework or decision model can provide all the necessary
information or flexibility needed by the Secretary of Defense and
his senior leadership team. We recognize that DOD's senior
leadership needs reliable information from a variety of sources
and flexibility to make decisions among alternative actions or
solutions. However, if the risk management framework is to
successfully serve as a management tool to assist decision makers
in formulating top-down strategy, balancing investment priorities
against risk over time, measuring near- and midterm outputs
against strategic goals, and focusing on actual performance
results-as intended by DOD's senior leadership-it is crucial that
it be successfully integrated with DOD's investment
decision-making processes, including recent reform initiatives.
We are sending copies of this report to interested congressional
committees; the Secretaries of Defense, Army, Navy, and Air Force;
the Commandant of the Marine Corps; and the Director, Office of
Management and Budget. We will also make copies available to
others upon request. In addition, the report will be available at
no charge on GAO's Web site at http://www/gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-9619 or [email protected]. Contact points
for our offices of Congressional Relations and Public Affairs may
be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix III.
Sharon L. Pickup Director, Defense Capabilities and Management
To assess to what extent the Department of Defense (DOD) has
implemented the risk management framework, we obtained and
analyzed DOD directives, briefings, and other documents that
described the risk management framework's purpose, implementation
status, and performance measures. We also obtained and analyzed
DOD's 2001 Quadrennial Defense Review and annual strategic
planning and budget documents. Moreover, we interviewed
knowledgeable DOD and service officials involved with the
implementation of the risk management framework. Specifically, we
obtained testimonial evidence from officials representing the
Office of the Secretary of Defense (OSD) offices-such as Program
Analysis and Evaluation; Comptroller; Policy; Acquisition,
Technology and Logistics; and Personnel and Readiness-the Joint
Staff, the military services, and the Defense Business Board. To
identify key risk-based and results-oriented management
principles, we reviewed our prior reports and other relevant
literature, including information on the balanced scorecard
concept. For example, we identified characteristics of
results-oriented performance measures. These characteristics
focused on performance measures that are (1) designed to
demonstrate results by providing information on how well the
organization is achieving its goals; (2) limited to a vital few,
and balanced across priorities; and (3) used by management to
improve performance. As another example, risk-based and
results-oriented management principles indicate that leading
organizations seek to establish clear hierarchies of goals and
measures that cascade down so that subordinate units have
straightforward roadmaps to demonstrate how their activities
contribute to meeting the organization's strategy. We
systematically analyzed and compared the risk management
framework's department-level performance measures with these
characteristics. However, we did not validate the procedures that
DOD has in place to ascertain the reliability of the data used to
support the performance measures. Regarding strategic planning,
these principles focused on (1) establishing clear linkages among
strategic planning goals, resources, performance goals and
measures and (2) integrating the consideration of risk into the
usual cycle of agency decision making and implementation. While
these principles do not cover all attributes associated with
risk-based and results-oriented management approaches, we believe
that they are the most important ones for assessing DOD's progress
in implementing the risk management framework.
To identify the most significant challenges, we reviewed our
previous work on change management principles. We then compared
DOD's implementation of the risk management framework to sound
change management principles and interviewed knowledgeable DOD
officials about the challenges that faced the department in
implementing the risk management framework. In addition, we
reviewed our previous work to determine to what extent
deficiencies in DOD's overall business transformation efforts
might influence the implementation of the risk management
framework.
Our work was performed from October 2004 through September 2005 in
accordance with generally accepted government auditing standards.
Sharon Pickup, (202) 512-9619 or [email protected]
In addition to the contact named above, David Moser, Assistant
Director; Donna Byers; Gina Flacco; and Renee S. Brown made key
contributions to this report.
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site (www.gao.gov). Each weekday, GAO
posts newly released reports, testimony, and correspondence on its
Web site. To have GAO e-mail you a list of newly posted products
every afternoon, go to www.gao.gov and select "Subscribe to
Updates."
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
Agency Comments and Our Evaluation
Appendix I: Scope and Methodology Appendix I: Scope and Methodology
Appendix II: Comments from the Department of Defense Appendix II:
Comments from the Department of Defense
Appendix III: GAOA Appendix III: GAO Contact and Staff Acknowledgments
GAO Contact
Acknowledgments
(350611)
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone
To Report Fraud, Waste, and Abuse in Federal Programs
Congressional Relations
Public Affairs
www.gao.gov/cgi-bin/getrpt?GAO-06-13.
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Sharon Pickup at (202) 512-9619 or
[email protected].
Highlights of GAO-06-13, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate
November 2005
DEFENSE MANAGEMENT
Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making
Resource Decisions
The Department of Defense (DOD) is simultaneously conducting costly
military operations and transforming its forces and business practices
while it is also competing for resources in an increasingly constrained
fiscal environment. As a result, GAO has advocated that DOD adopt a
comprehensive threat or risk management approach as a framework for
decision making. In its 2001 strategic plan, the Quadrennial Defense
Review (QDR), DOD stated its intent to establish an approach-the risk
management framework-to balance priorities against risk over time and
monitor results against its strategic goals.
GAO was asked to (1) assess the extent to which DOD has implemented the
framework, including using it to make investment decisions, and
(2) identify the most significant challenges DOD faces in implementing the
framework, or a similar approach.
What GAO Recommends
GAO recommends that DOD take various actions to increase its chances of
successfully implementing a risk-based approach for investment decision
making, such as developing results-oriented measures and assigning clear
leadership with appropriate accountability and authority to implement the
framework. DOD partially concurred with our recommendations.
DOD has taken some positive steps to implement the framework, but
additional actions are needed before DOD can show real and sustainable
progress in using a risk-based and results-oriented approach to
strategically allocate resources across the spectrum of its investment
priorities. For example, DOD defined four risk areas, and developed
performance goals and department-level measures, but it needs to, among
other things, further develop and refine the measures so that they clearly
demonstrate results and provide a well-rounded depiction of departmental
performance. DOD's current strategic plan and goals also are not clearly
linked to the framework's performance goals and measures, and linkages
between the framework and budget are also unclear. While DOD officials
stated that risk was considered during the fiscal year 2006 budget cycle,
DOD's budget submission does not specifically discuss how DOD identified
or assessed risks to establish DOD-wide investment priorities. Without
better measures, clear linkages, and greater transparency, DOD will be
unable to fully measure progress in achieving strategic goals or
demonstrate to Congress and others how it considered risks, and made
trade-off decisions, balancing needs and costs for weapon programs and
other investment priorities.
DOD's Risk Management Framework
Force Management Risk Operational Risk
Definition: Challenge of sustaining Definition: Challenge of deterring
personnel, infrastructure, and or defeating near-term threats
equipment
Future Challenges Risk Institutional Risk
Definition: Challenge of dissuading, Definition: Challenge of improving
deterring, defeating longer-term efficiency (includes financial
threats management)
Source: DOD.
DOD faces four challenges that have affected the implementation of the
framework. First, DOD's organizational culture resists department-level
approaches to priority setting and investment decisions. Second, sustained
leadership, adequate transparency, and appropriate accountability are
lacking. Further, no one individual or office has been assigned overall
responsibility or sufficient authority for the framework's implementation.
DOD also has not developed implementation goals or timelines with which to
establish accountability, or measure progress. Finally, integrating the
risk management framework with decision support processes and related
reform initiatives into a coherent, unified management approach for the
department is a challenge that DOD plans to address during the 2005 QDR.
However, GAO has concerns about DOD's ability to follow through on this
integration, because of its limited success in implementing other
management reforms. Unless DOD successfully addresses these challenges and
effectively implements the framework, or a similar approach, it will
likely continue to experience (1) a mismatch between programs and budgets,
and (2) a proportional, rather than strategic, allocation of resources to
the services.
*** End of document. ***