Defense Acquisitions: DOD's Revised Policy Emphasizes Best
Practices, but More Controls Are Needed (10-NOV-03, GAO-04-53).
The Department of Defense's (DOD) investment in new weapon
systems is expected to exceed $1 trillion from fiscal years 2003
to 2009. To reduce the risk of cost and schedule overruns, DOD
revamped its acquisition policy in May 2003. The policy provides
detailed guidance on how weapon systems acquisitions should be
managed. The Senate report accompanying the National Defense
Authorization Act for Fiscal Year 2004 required GAO to determine
whether DOD's policy supports knowledge-based, evolutionary
acquisitions and whether the policy provides the necessary
controls for DOD to ensure successful outcomes, such as meeting
cost and schedule goals. The report also required GAO to assess
whether the policy is responsive to certain requirements in the
Bob Stump National Defense Authorization Act for Fiscal Year 2003
concerning DOD's management of the acquisition process.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-53
ACCNO: A08843
TITLE: Defense Acquisitions: DOD's Revised Policy Emphasizes
Best Practices, but More Controls Are Needed
DATE: 11/10/2003
SUBJECT: Best practices
Decision making
Defense procurement
Internal controls
Procurement policy
Weapons research and development
Policy evaluation
Reporting requirements
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-53
United States General Accounting Office
GAO
Report to the Senate and House
Committees on Armed Services
November 2003
DEFENSE ACQUISITIONS
DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed
GAO-04-53
November 2003
DEFENSE ACQUISITIONS
DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed
DOD's new policy supports knowledge-based, evolutionary acquisitions by
adopting lessons learned from successful commercial companies. One of
those lessons is a knowledge-based approach, which requires program
managers to attain the right knowledge at critical junctures-also known as
knowledge points-so they can make informed investment decisions throughout
the acquisition process. The policy also embraces an evolutionary or
phased development approach, which sets up a more manageable environment
for attaining knowledge. The customer may not get the ultimate capability
right away, but the product is available sooner and at a lower cost.
Leading firms have used these approaches-which form the backbone of what
GAO calls the best practices model-to determine whether a project can be
accomplished with the time and money available and to reduce risks before
moving a product to the next stage of development.
By adopting best practices in the acquisition policy, DOD's leadership has
taken a significant step forward. The next step is to provide the
necessary controls to ensure a knowledge-based, evolutionary approach.
Implementing the necessary controls at all three knowledge points along
the acquisition process helps decision makers ensure a knowledge-based
approach is followed. Without controls in the form of measurable criteria
that decision makers must consider, DOD runs the risk of making decisions
based on overly optimistic assumptions. Each successive knowledge point
builds on the preceding one, and having clearly established controls helps
decision makers gauge progress in meeting goals and ensuring successful
outcomes.
DOD Policy Incorporates Best Practices but Does Not Have Sufficient
Controls Best practices' intent in Sufficient controls in Knowledge point
DOD policy? DOD policy?
Highlights of GAO-04-53, a report to the Senate and House Committees on
Armed Services
The Department of Defense's (DOD) investment in new weapon systems is
expected to exceed $1 trillion from fiscal years 2003 to 2009. To reduce
the risk of cost and schedule overruns, DOD revamped its acquisition
policy in May 2003. The policy provides detailed guidance on how weapon
systems acquisitions should be managed.
The Senate report accompanying the National Defense Authorization Act for
Fiscal Year 2004 required GAO to determine whether DOD's policy supports
knowledge-based, evolutionary acquisitions and whether the policy provides
the necessary controls for DOD to ensure successful outcomes, such as
meeting cost and schedule goals.
The report also required GAO to assess whether the policy is responsive to
certain requirements in the Bob Stump National Defense Authorization Act
for Fiscal Year 2003 concerning DOD's management of the acquisition
process.
GAO recommends that the Secretary of Defense strengthen DOD's acquisition
policy by requiring additional controls to ensure decision makers will
follow a knowledge-based, evolutionary approach. DOD partially concurred
with GAO's recommendations.
Knowledge point 1 (at program Yes No
launch): Technologies, time,
funding, and other resources
match customer needs
Knowledge point 2
(between integration and Yes No
demonstration): Design
performs as expected
Knowledge point 3 Yes No
(at production commitment):
Production meets cost, schedule, and quality targets
Sources: DOD (data), GAO (analysis).
DOD was responsive to the requirements in the Bob Stump National Defense
Authorization Act for Fiscal Year 2003. DOD's responses reflected the
www.gao.gov/cgi-bin/getrpt?GAO-04-53. committee's specific concerns about
the application of certain statutory and
To view the full product, including the scope regulatory requirements to
the new evolutionary acquisition process, for
and methodology, click on the link above. more guidance for implementing
spiral development, and about technologyFor more information, contact
Katherine V. readiness at program initiations.
Schinasi at (202) 512-4841 or [email protected].
Contents
Letter
Results in Brief
Background
DOD's Revised Policy Provides a Framework for
Knowledge-Based, Evolutionary Acquisitions Controls Are Needed to Ensure
Knowledge-Based Approach Is Followed DOD Is Responsive to Congressional
Requirements about New
Acquisition Process Conclusions Recommendations for Executive Action
Agency Comments and Our Evaluation Scope and Methodology
1
2 3
5
8
10 14 14 15 16
Appendix I Section 802, Evolutionary Acquisition
Appendix II Section 803, Spiral Development
Appendix III Section 804, Technology Maturity, and Section 822,
Independent Technology Readiness Assessments
Appendix IV Comments from the Department of Defense
Related GAO Products
Tables
Table 1: Excerpts of Best Practices Contained in DOD's New Policy 6 Table
2: Comparison of Controls Used in Best Practices Model and DOD Policy 8
Table 3: How DOD Policy Addresses Section 802's Statutory and Regulatory
Requirements 19
Table 4: How DOD Responded to Section 802's Requirements Regarding
Specific Matters for Each Increment of an Evolutionary Acquisition Process
21
Table 5: How DOD Policy Addresses Section 803's Requirements 24 Table 6:
How DOD Responded to Section 804 and Section 822 Requirements 26
Figures
Figure 1: Distribution of Development Funds for Fiscal Year 2003 5 Figure
2: Comparison of DOD's Framework and Commercial Best Practices Model 7
Abbreviations
DOD Department of Defense
DODD DOD Directive
DODI DOD Instruction
MDA Milestone Decision Authority
TEMP Test and Evaluation Master Plan
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States General Accounting Office Washington, DC 20548
November 10, 2003
The Honorable John W. Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Duncan Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives
The Department of Defense (DOD) plans to spend more than $1 trillion
from fiscal years 2003 to 2009 for developing and procuring weapon
systems. To get the most out of its investment, DOD has set goals to
develop and procure weapons faster and at less cost. Guiding this effort
is
DOD's newly revised acquisition policy. Issued in May 2003, the policy
seeks to foster greater efficiency while building flexibility in the
acquisition process. The policy embraces a knowledge-based, evolutionary
framework that emphasizes shorter development times.
DOD intends to use its policy to improve its record for meeting cost and
schedule estimates and for delivering capabilities as promised. Congress
has expressed support for DOD's efforts to revamp its acquisition process
and has established requirements that DOD must take to ensure a
disciplined approach. Recognizing that a consistent and disciplined
application of policies will be key to achieving desired outcomes, the
Senate Armed Services Committee directed us to assess DOD's
current acquisition policies to determine whether they (1) support
knowledge-based, evolutionary acquisitions, (2) provide the necessary
controls to ensure DOD's policy intent is followed, and (3) respond to
specific requirements in the Bob Stump National Defense Authorization
Results in Brief
Act for Fiscal Year 2003 (Public Law 107-314) regarding DOD plans to
manage its new acquisition process.1
DOD has made major improvements to its acquisition policy by adopting
knowledge-based, evolutionary practices used by successful commercial
companies. If properly applied, these best practices can put DOD's
decision makers in a better position to deliver high-quality products on
time and within budget. The policy requires decision makers to have the
knowledge they need before moving to the next phase of development. To
ensure that the acquisition environment is conducive to a knowledge-based
approach, the policy embraces evolutionary development, which allows
managers to develop a product in increments rather than trying to
incorporate all the desired capabilities in the first version that comes
off the production line. Leading companies who use the knowledge-based
evolutionary approach have shown that these best practices help reduce
risks at critical junctures during a product's development and help ensure
that decision makers get the most out of their investments.
DOD's recent policy changes are a significant step forward. The next step
is for DOD to provide the necessary controls to ensure a knowledge-based,
evolutionary approach is followed. For example, the policy does not
establish measures to gauge design and manufacturing knowledge at critical
junctures in the product development process. Without specific
requirements to demonstrate knowledge at key points, the policy allows
significant unknowns to be judged as acceptable risks, leaving an opening
for decision makers to make uninformed decisions about continuing product
development.
DOD was responsive to the requirements in the Defense Authorization Act
for Fiscal Year 2003. DOD's responses reflected the committee's specific
concerns about the application of certain statutory and regulatory
requirements2 to the new evolutionary acquisition process, for more
guidance for implementing spiral development, and about technology
readiness at program initiations.
1 S. Rept. No. 108-46, Report of the Senate Armed Services Committee
accompanying S. 1050, National Defense Authorization Act for Fiscal Year
2004, at page 346 (May 13, 2003).
2 As used here, the term "regulatory requirements" refers to policies
governing DOD's acquisition system.
Background
This report makes recommendations that the Secretary of Defense strengthen
DOD's acquisition policy by requiring additional controls to ensure
decision makers will follow a knowledge-based, evolutionary approach. DOD
partially concurred with our recommendations. DOD believes the current
acquisition framework includes the controls necessary to achieve effective
results, but department officials will continue to monitor the process to
determine whether other controls are needed to achieve the best possible
outcomes. DOD agreed it should record and justify program decisions for
moving from one stage of development to next but did not agree with the
need to issue a report outside of the department.
Traditionally, DOD's strategy for acquiring major weapon systems has been
to plan programs that would achieve a big leap forward in capability.
However, because the needed technologies often are not yet mature,
programs stay in development for years until the technologies are
demonstrated. As a result, weapon systems have frequently been
characterized by poor cost, schedule, and performance outcomes. This has
slowed modernization efforts, reduced the buying power of the defense
dollar, delayed capabilities for the warfighter, and forced unplanned-and
possibly unnecessary-trade-offs among programs.
Our extensive body of work shows that leading companies use a product
development model that helps reduce risks and increase knowledge when
developing new products. This best practices model enables decision makers
to be reasonably certain about their products at critical junctures during
development and helps them make informed investment decisions. This
knowledge-based process can be broken down into three cumulative knowledge
points.
o Knowledge point 1: A match must be made between the customer's needs
and the developer's available resources-technology, engineering knowledge,
time, and funding-before a program starts.
o Knowledge point 2: The product's design must be stable and must meet
performance requirements before initial manufacturing begins.
o Knowledge point 3: The product must be producible within cost,
schedule, and quality targets and demonstrated to be reliable before
production begins.
To bolster the knowledge-based process, leading companies use evolutionary
product development, an incremental approach that enables developers to
rely more on available resources rather than making
promises about unproven technologies. While the user may not initially
receive the ultimate capability under this approach, the initial product
is available sooner and at a lower, more predictable cost. Also, leading
companies know that invention cannot be scheduled and its cost is
difficult to estimate. They do not bring technology into new product
development unless that technology has been demonstrated to meet the
user's requirements. Allowing technology development to spill over into
product development puts an extra burden on decision makers and provides a
weak foundation for making product development estimates.
DOD understands that it must improve acquisition process outcomes if it is
to modernize its forces within currently projected resources. To help
achieve this goal, DOD has revised its acquisition policy, called the 5000
series, to reflect best practices from successful commercial and DOD
programs.3 The policy covers most-but not all-major acquisitions. The
Secretary of Defense has delegated authority to the Missile Defense Agency
and to the National Security Space Team to develop separate guidance for
missile defense and space systems, respectively.4 Approximately 35 percent
of DOD's development funds in 2003 went to these systems. (Figure 1 shows
how $43.1 billion in development funds were distributed across space,
missile defense, and systems covered by the 5000 series.) This report
addresses policy for the defense programs covered exclusively under the
5000 series.
3 DOD Directive 5000.1, The Defense Acquisition System, describes the
management principles for DOD's acquisition programs. DOD Instruction
5000.2, The Operation of the Defense Acquisition System, outlines a
framework for managing acquisition programs. Collectively, these are known
as the 5000 series.
4 Program elements of the ballistic missile defense program enter the
formal DOD acquisition cycle at milestone C (production commitment) and
are subject to the 5000 series from that point on. To use the streamlined
process of the National Security Space Acquisition Policy, a Space System
Program Director/Program Manager must request that the DOD Space Milestone
Decision Authority (the Under Secretary of the Air Force) grant a waiver
and an exemption to the processes and procedures described in DOD
Instruction 5000.2 (the waiver authority does not include DOD Directive
5000.1). Notwithstanding these policy exceptions, statutory requirements
for major defense acquisition programs continue to apply to missile
defense and space programs.
Figure 1: Distribution of Development Funds for Fiscal Year 2003
Note: Includes research, development, test, and evaluation budget
activities for advanced component development and prototypes; system
development and demonstration; and operational systems development.
DOD's leaders have made significant improvements to DOD's acquisition
policy by adopting the knowledge-based, evolutionary approach used by
leading commercial companies. The revised policy has the potential to
transform DOD's acquisition process by reducing risks and increasing the
chances for successful outcomes. The policy provides a framework for
developers to ask themselves at key decision points whether they have the
knowledge they need to move to the next phase of acquisition. If
rigorously applied, this knowledge-based framework can help managers gain
the confidence they need to make significant and sound investment
decisions for major weapon systems. In placing greater emphasis on
evolutionary product development, the policy sets up a more manageable
environment for achieving knowledge. Another best practice reflected in
the policy's framework is separating technology development from product
development, which reduces technological risk at the start of a program.
DOD's Revised Policy Provides a Framework for Knowledge-Based, Evolutionary
Acquisitions
As shown in table 1, DOD's policy emphasizes best practices used by
leading companies.
Table 1: Excerpts of Best Practices Contained in DOD's New Policy
Best practices DOD policy
Knowledge-based acquisition DOD Directive 5000.1 (enclosure 1, p. 5):
Program managers "shall provide knowledge about key aspects of a system at
key points in the acquisition process. ... shall reduce technology risk,
demonstrate technologies in a relevant environment ... prior to program
initiation. They shall reduce integration risk and demonstrate product
design prior to the design readiness review. They shall reduce
manufacturing risk and demonstrate producibility prior to full-rate
production."
Evolutionary acquisition DOD Instruction 5000.2 (p. 3): "Evolutionary
acquisition is the preferred DOD strategy for rapid acquisition of mature
technology for the user. ... The objective is to balance needs and
available capability with resources, and to put capability into the hands
of the user quickly. The success of the strategy depends on consistent and
continuous definition of requirements, and the maturation of technologies
... that provide increasing capability towards a materiel concept."
Separating technology development from DOD Instruction 5000.2 (p. 6): "The
project shall exit Technology Development when an
product development
affordable increment of militarily-useful capability has been identified,
the technology for that increment has been demonstrated in a relevant
environment, and a system can be developed for production within a short
timeframe (normally less than five years)."
Source: DOD.
Similar to the best practices model, DOD's policy divides its acquisition
process into phases, as shown in figure 2. Key decisions are aligned with
the three critical junctures of a product's development, or knowledge
points.
Figure 2: Comparison of DOD's Framework and Commercial Best Practices
Model
In other similarities, DOD's framework pinpoints program start at
milestone B, about the same point as program start on the best practices
model. At the midway point on both approaches, a stable product design
should be demonstrated. With DOD's framework, managers are required to
know-by the time full-rate production decision review occurs-whether the
product can be produced within cost, schedule, and quality targets. This
requirement occurs earlier in the best practices model, before production
begins, or at knowledge point 3. Leading companies have used this approach
to reduce risks and to make costs and delivery dates more predictable.
Controls Are
Needed to Ensure
Knowledge-Based
Approach Is Followed
While DOD has strengthened its acquisition policy with a knowledge-based,
evolutionary framework, the policy does not include many of the same
controls that leading companies rely on to attain a high level of
knowledge before making additional significant investments. Controls are
considered effective if they are backed by measurable criteria and if
decision makers are required to consider them before deciding to advance a
program to the next level. Controls used by leading companies help
decision makers gauge progress in meeting cost, schedule, and performance
goals and ensure that managers will (1) conduct activities to capture
relevant product development knowledge, (2) provide evidence that
knowledge was captured, and (3) hold decision reviews to determine that
appropriate knowledge was captured to move to the next phase.
To determine if DOD has the necessary controls, we compared controls in
DOD's policy with those used in the best practices model at three critical
junctures. Table 2 shows the presence or absence of controls for various
versions of DOD policy since 1996, including the May 2003 revision.
Table 2: Comparison of Controls Used in Best Practices Model and DOD Policy
Controls used in DOD policy
March 1996 October 2000 October 2002 May 2003 Controls used in commercial
best practices model policy policy interim policy policy
Knowledge point 1: Occurs at program launch. Match exists between
requirements and resources. Technologies needed to meet essential product
requirements have been demonstrated to work in their intended environment
and the producer has completed a preliminary design of the product.
Demonstrate technologies to high readiness levels X X X
Ensure that requirements for the product are informed by
the systems engineering process X
Establish cost and schedule estimates for product based on knowledge from
preliminary design using systems engineering tools Conduct decision review
for program launch X X X X
Knowledge point 2: Occurs between integration and demonstration. Design is
stable and has been demonstrated through prototype testing. Ninety percent
of engineering drawings are releasable to manufacturing organizations.
Complete 90 percent of design drawings
Complete subsystem and system design reviews
Demonstrate with prototype that design meets
requirements X X X
Obtain stakeholders concurrence that drawings are complete and producible
Complete failure modes and effects analysis
Identify key system characteristics
Controls used in DOD policy
March 1996 October 2000 October 2002 May 2003 policy policy interim policy
policy
Controls used in commercial best practices model
Identify critical manufacturing processes
Establish reliability targets and growth plan based on demonstrated
reliability rates of components and subsystems
Conduct decision review to enter system demonstration X X
Knowledge point 3: Occurs at production commitment. Product is ready to be
manufactured within cost, schedule, and quality targets. All key
manufacturing processes have come under statistical control and product
reliability has been demonstrated.
Demonstrate manufacturing processes
Build production representative prototypes
Test production representative prototypes to achieve reliability goal
Test production representative prototypes to demonstrate product in
operational environment
Collect statistical process control data
Demonstrate critical processes capable and in statistical control
Conduct decision review to begin production X X X X
Sources: GAO and DOD.
At all three knowledge points, DOD's policy does not provide all the
necessary controls used by commercial companies. For example, at program
launch (milestone B) or when knowledge point 1 should be reached, the
policy requires decision makers to identify and validate a weapon system's
key performance requirements and to have a technical solution for the
system before program start. This information is then used to form cost
and schedule estimates for the product's development. However, the policy
does not emphasize the use of a disciplined systems engineering process
for balancing a customer's needs with resources to deliver a preliminary
design. The lack of effective controls at knowledge point 1 could result
in gaps between requirements and resources being discovered later in
development.
At the design readiness review or when knowledge point 2 should be
reached, DOD's policy does not require specific controls to document that
a product is ready for initial manufacturing and demonstration. DOD's
policy suggests appropriate criteria, such as number of subsystem and
system design reviews completed, percentage of drawings completed, planned
corrective actions to hardware and software deficiencies, adequate
development testing, completed failure modes and effects
analysis, identification of key system characteristics and critical
manufacturing processes, and availability of reliability targets and
growth plans. However, these criteria are not required. For example, we
found that a key indicator of a product's design stability is the
completion of 90 percent of the engineering drawings supported by design
reviews. DOD's policy does not require that a certain percentage of
drawings or design reviews be completed to ensure the design is mature
enough to enter the system demonstration phase. As a result, a decision
maker has no benchmark to consider when deciding to advance a program to
the next level of development.
Finally, at production commitment or when knowledge point 3 should be
reached, DOD's policy does not require specific controls to document that
a product can be manufactured to meet cost, schedule, and quality targets
before moving into production. For example, the policy states there should
be "no significant manufacturing risks" at the start of low-rate
production but does not define what this means or how it is to be
measured. DOD's policy does not require the demonstrated control of
manufacturing processes and the collection of statistical process control
data until full-rate production begins but even then fails to specify a
measurable control. Given that low-rate production can last several years,
a significant number of products can be manufactured before processes are
brought under control, creating a higher probability of poor cost and
schedule outcomes.
While supporting efforts to build more flexibility into the DOD
acquisition process and to develop weapon systems using an evolutionary
approach, Congress asked DOD to be more disciplined in its approach. The
Defense Authorization Act for Fiscal Year 2003 required DOD to address (1)
the way it plans to meet certain statutory and regulatory requirements for
managing its major acquisition programs, (2) needed guidance for
implementing spiral developments, and (3) technology readiness (at
acquisition program initiation). DOD was responsive to all three
requirements. With regard to the second requirement, a description of the
process that would be used to independently validate that measurable exit
DOD Is Responsive to Congressional Requirements about New Acquisition Process
criteria for applying a spiral development process5 have been met was
unclear. DOD stated that the milestone decision authority6 provides that
independent validation as part of DOD's milestone approval process. DOD's
responses to the relevant sections of the act are summarized below. More
detailed comparisons are provided in appendixes I, II, and III.
Section 802: Evolutionary Acquisition
Requirements: This section directed DOD to report on its plan to meet
certain statutory and regulatory requirements for managing its major
acquisition programs applying an evolutionary acquisition process. These
include establishing and approving operational requirements and cost and
schedule goals for each increment, meeting requirements for operational
and live fire testing for each increment, and optimizing total system
performance and minimizing total ownership costs.
DOD response: In April 2003, DOD submitted its report reflecting how these
requirements are addressed in its acquisition policy. According to the
report, the policy addresses the statutory and other requirements
applicable to all major defense acquisition programs, including each
increment of evolutionary acquisition programs. For example, the policy
requires that each program or increment of an evolutionary acquisition
have a milestone B decision to approve program initiation and to permit
entry into system development and demonstration. The policy specifies the
statutory and regulatory information necessary to support the decision.
Section 803: Spiral Development
Requirements: This section authorizes DOD to conduct a research and
development program for a major defense acquisition program using spiral
development only if approved by the Secretary of Defense or authorized
5 A spiral development program is defined in section 803 of the Defense
Authorization Act for Fiscal Year 2003 as a research and development
program conducted in discrete phases or blocks, each of which will result
in the development of fieldable prototypes and will not proceed into
acquisition until specific performance parameters, including measurable
exit criteria, have been met.
6 The milestone decision authority is typically the DOD component
acquisition executive or designee, or for certain large programs, the head
of the component or the Under Secretary of Defense for Acquisition,
Technology, and Logistics.
high-level designee.7 A program cannot be conducted as a spiral
development unless the Secretary of Defense or designee approves a plan
that describes such things as the program strategy, test plans,
performance parameters, and measurable exit criteria. The section also
requires the Secretary of Defense to issue guidance addressing the
appropriate processes for an independent validation that exit criteria
have been met, the operational assessment of fieldable prototypes, and the
management of these types of programs. It further requires the Secretary
to report to Congress on the status of each program applying spiral
development by September 30 of each year from 2003 to 2008.
DOD response: DOD established a technology development strategy in the new
policy to address this requirement. The strategy must be completed before
a program can enter the technology development phase. The strategy also
documents the cost and schedule goals, the test plans, the number of
prototypes, and a program strategy for the total research and development
program. The strategy requires a test plan to ensure the goals and exit
criteria for the first technology spiral demonstration are met, and the
policy requires an independent operational assessment for the release of
each product increment to the user. What is unclear in DOD's guidance is
the process that will be used for independently validating whether
measurable cost, schedule, and performance exit criteria have been met.
However, DOD stated that the milestone decision authority provides
independent validation that exit criteria have been met as part of DOD's
milestone approval process. As of October 23, 2003, DOD's report on the
status of each program applying spiral development was still in draft and
not yet submitted. DOD's current draft report states that there are no
research and development programs that have been approved as spiral
development programs as of September 30, 2003. Section 803 requirements
were implemented in DOD Instruction 5000.2, which was effective in May
2003. DOD anticipates that there will be approved spiral development
programs to report in 2004.
7 Section 803 authorized the Secretary of Defense to delegate authority to
approve a spiral development plan to the Under Secretary of Defense for
Acquisition, Technology, and Logistics or to the senior acquisition
executive of the military department or defense agency concerned. The
authority may not be delegated further.
Section 822: Independent Technology Readiness Assessments
Requirements: This section added a requirement to section 804 of the
National Defense Authorization Act for Fiscal Year 2002 (Public Law
107-107) that directed DOD to report by March of each year between 2003
and 2006 on the maturity of technology at the initiation of major defense
acquisition programs. Each report is required to (1) identify any major
acquisition program that entered system development and demonstration
during the preceding calendar year with immature key technology that was
not demonstrated in, at minimum, a relevant environment, as required by
the new policy; (2) justify the incorporation of any key technology on an
acquisition program that does not meet that requirement; (3) identify any
instances that the Deputy Under Secretary of Defense for Science and
Technology did not concur with the technology assessment and explain how
the issue has been or will be resolved; (4) identify each case in which a
decision was made not to conduct an independent technology readiness
assessment for a critical technology on a major defense acquisition; and
(5) explain the reasons for the decision each year through 2006.
DOD response: In March 2003, DOD reported that two programs entered system
development and demonstration in 2002 with critical technologies that did
not meet demonstration requirements and provided justification for them.8
DOD did not identify or report any cases where an independent technology
readiness assessment was not conducted or where the Under Secretary
disagreed with assessment findings.
8 The two programs were the Composite Health Care System II and the Joint
Tactical Radio System, Cluster 1.
Conclusions
Recommendations for Executive Action
DOD can maximize its $1 trillion investment in new weapons over the next 6
years by ensuring effective implementation of the new acquisition policy.
DOD's leaders have taken noteworthy steps by incorporating into the policy
a framework that supports a knowledge-based, evolutionary acquisition
process, similar to one used by leading commercial companies to get
successful outcomes. A framework is an important and significant step. DOD
must now turn its attention to establishing controls. As leading companies
have found, having clearly established controls to capture and use
appropriate knowledge to make decisions at critical junctures is crucial
for delivering affordable products as planned. DOD's policy addresses
specific congressional requirements and includes some controls that
leading companies use to capture knowledge at the start of a program.
However, additional controls are needed to ensure that decisions made
throughout product development are informed by demonstrated knowledge.
DOD must design and implement necessary controls to ensure that
appropriate knowledge is captured and used at critical junctures to make
decisions about moving a program forward and investing more money. We
recommend that the Secretary of Defense require additional controls for
capturing knowledge at three key points-program launch, design readiness
review for transitioning from system integration to system demonstration,
and production commitment. The additional controls for program launch
(milestone B) should ensure the capture of knowledge about the following:
o Cost and schedule estimates based on knowledge from a preliminary
design using systems engineering tools.
The additional controls for transitioning from system integration to
system demonstration (design readiness review) should ensure the capture
of knowledge about the following:
o Completion of 90 percent of engineering drawings.
o Completion of subsystem and system design reviews.
o Agreement from all stakeholders that drawings are complete and the
design is producible.
o Completion of failure modes and effects analysis.
o Identification of key system characteristics.
o Identification of critical manufacturing processes.
o Reliability targets and a reliability growth plan based on
demonstrated reliability rates of components and subsystems.
The additional controls for the production commitment (milestone C) should
ensure the capture of knowledge about the following:
o Completion of production representative prototypes.
o Availability of production representative prototypes to achieve
reliability goal and demonstrate the product in an operational
environment.
o Collection of statistical process control data.
o Demonstration that critical manufacturing processes are capable and in
statistical control.
Because knowledge about technology, design, and manufacturing at critical
junctures can lower DOD's investment risk, decisions that do not satisfy
knowledge-based criteria should be visible and justified. Therefore, we
also recommend that the Secretary of Defense document the rationale for
any decision to move a program to the next stage of development without
meeting the knowledge-based criteria, including those listed in the first
recommendation. The responsible milestone decision authority should
justify the decision in the program's acquisition decision memorandum and
in a report to Congress.
Agency Comments
and Our Evaluation
DOD provided us with written comments on a draft of this report. The
comments appear in appendix IV.
DOD partially concurred with our recommendation that the Secretary require
additional controls for capturing knowledge at three key points: program
launch, design readiness review for transitioning from system integration
to system demonstration, and production. DOD stated that it agrees in
principle with the advantages of using knowledge-based controls at key
points in the acquisition process to assess risk and ensure readiness to
proceed into the next phase of the acquisition process. DOD believes the
current acquisition framework includes the controls necessary to achieve
effective results, but it will continue to monitor the process to
determine whether others are necessary to achieve the best possible
outcomes.
While we believe DOD's effort to establish a solid framework for
evolutionary acquisitions is a giant step forward, our work has shown that
a disciplined application of controls in the process is needed to
implement the framework if better acquisition outcomes are to be achieved.
DOD's policy does not include all the necessary controls to ensure a high
level of product knowledge is attained and used for making decisions to
move a
program forward in the product development process. Leading product
developers use additional controls, as listed in our first recommendation,
to achieve the knowledge necessary to reduce risk to reasonable levels at
critical junctures before making additional significant investments in
product development. Simply monitoring the process may not be enough for
DOD to achieve the best outcomes. Therefore, we are retaining our
recommendation that the Secretary require additional controls at three
critical points in the acquisition process.
DOD also partially concurred with our recommendation that the Secretary
document in each program's acquisition decision memorandum and in a report
to Congress the rationale for any decision to move a program to the next
stage of development without meeting the knowledge-based criteria,
including those described in the first recommendation. DOD agreed that it
should record and be accountable for program decisions. Decision makers
will continue to use the acquisition decision memorandum to document
program decisions and the rationale for them. DOD did not concur with the
need for a report outside the department. Because we believe strongly that
knowledge-based criteria used to gauge a product's development progress at
critical junctures can lower DOD's investment risks, we think it is
important that decisions made without satisfying knowledge-based criteria
be justified in a visible and transparent way to hold managers accountable
for moving a program forward absent this knowledge. Therefore, we are
retaining our recommendation for reporting the basis for decisions to move
forward in a report to Congress.
We reviewed DOD's revised and past acquisition policies, DOD Directive
5000.1, DOD Instruction 5000.2, and DOD 5000.2-R,9 which provide
management principles and mandatory policies and procedures for managing
acquisitions programs. We contacted an official in the Office of the Under
Secretary of Defense for Acquisition, Technology, and Logistics who is
responsible for the development of the policy to better understand its
content. We also reviewed information from the Defense Acquisition
University that provided educational material on the policies.
We reviewed the relevant sections of the Bob Stump National Defense
Authorization Act for Fiscal Year 2003 and the accompanying Senate
Scope and Methodology
9 DOD 5000.2-R was canceled and replaced with the Interim Defense
Acquisition Guidebook.
Armed Services Committee report to identify the requirements applicable to
DOD's acquisition policy. We compared these requirements with DOD's
responses to determine whether they have been addressed.
Finally, we used information from more than 10 GAO products that examine
how commercial best practices can improve outcomes for various DOD
programs. During the past 6 years, we have gathered information based on
discussions and visits with the following companies:
o 3M o Boeing Commercial Airplane Group
o Chrysler o Bombardier Aerospace
o Ford Motor o Hughes Space and Communication
o Motorola o Xerox
o Hewlett-Packard o Caterpillar
o Cummins o General Electric Aircraft Engines
o Toyota o Harris Semiconductor
o Honda o Texas Instruments
o John Deere o Varian Oncology Systems
o Ethicon-Endo Surgery (division of Johnson & Johnson)
Although the approaches varied, these companies consistently applied the
basic processes and standards in use. We compared this information with
the acquisition framework and controls established by DOD's policy. We
concentrated on whether the policy provides a framework for a
knowledge-based, evolutionary process and the controls necessary to carry
out this intent.
We conducted our review from April 2003 to September 2003 in accordance
with generally accepted government auditing standards.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will also provide copies to others on
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov.
Please contact me at (202) 512-4841 if you have any questions
concerning this report. Other key contributors to this report were
Lily Chin, Chris DePerro, Matt Lea, Mike Sullivan, and Adam Vodraska.
Katherine V. Schinasi
Director, Acquisition and Sourcing Management
Appendix I: Section 802, Evolutionary Acquisition
Section 802 of the Defense Authorization Act for Fiscal Year 2003 required
the Secretary of Defense to submit a report to Congress explaining how the
Department of Defense (DOD) plans to meet certain statutory and regulatory
requirements for acquisition programs following an evolutionary approach.
In April 2003, the Secretary reported how these requirements were
addressed in DOD's policy (such as in tables of statutory and regulatory
information requirements contained in enclosure 3 of Instruction 5000.2).
According to the report, DOD's policy requires that each program-including
an increment of an evolutionary acquisition-have a milestone B decision to
approve program initiation and to permit entry into systems development
and demonstration. DOD's policy specifies the statutory and regulatory
information necessary to support the decision. We examined the policy to
ensure the statutes and regulations identified in section 802 were
addressed. Table 3 provides a list of the statutory and regulatory
requirements identified in section 802, a corresponding document and page
number where the requirement appears in DOD's policy, and a description of
the requirement from the policy.
Table 3: How DOD Policy Addresses Section 802's Statutory and Regulatory
Requirements
Section 802 requirements DOD policy reference examples Brief description
from policy
Requirements of chapter 144 of title 10, United States Code
Sec. 2430, major defense acquisition
Department of Defense Instruction
Table E.2.T1 provides a description (criteria) and the program defined (DODI)
5000.2, enclosure 2, p. 16 decision authority by acquisition program category.
Sec. 2431, weapons development Not referenceda and procurement schedules
Sec. 2432, Selected DODI 5000.2, Required at milestone B or
Acquisition enclosure 3, p.19 program initiation and
Reports annually thereafter; end of
quarter following
milestone C decision; full-rate
production decision.
Also required when there is a
baseline breach.
Sec. 2433, unit cost DODI 5000.2,
reports enclosure 3, p. 19 Required on quarterly basis.
Sec. 2434, independent cost estimates; operational manpower requirements
DODI 5000.2, enclosure 6, p. 30 DODI 5000.2, enclosure 3, p. 19
Required at program initiation for ships (cost assessment only). Required
at milestones B and C and full-rate production decision.
Sec. 2435, baseline description DODI 5000.2, enclosure 3, p. 19 Required at
program initiation for ships. Required at DODI 5000.2, p. 8
milestones B and C and full-rate production decision. Program deviation
report required immediately upon program deviation.
Sec 2440, technology and industrial DODI 5000.2, enclosure 3, p. 19 Required at
milestones B and C (part of acquisition
base plans
strategy).
Appendix I: Section 802, Evolutionary Acquisition
Section 802 requirements DOD policy reference examples Brief description
from policy
Sec. 139, Director of Operational Department of Defense Directive Cites Director
of Operational Test and Evaluation as Test and Evaluation
(DODD) 5000.1, p. 3
DODI 5000.2, enclosure 3
DODI 5000.2, enclosure 5, pp. 27-29
key official of defense acquisition system. Director responsible for
assessing adequacy of operational tests and live fire tests and evaluating
operational effectiveness, suitability, and survivability of systems.
Sec. 181, Joint Requirements DODD 5000.1, p. 3 Chairman of the Joint
Chiefs of Staff, with assistance
Oversight Council DODI 5000.2, p. 4 of Joint Requirements Oversight
Council, responsible for assessing and providing advice regarding
capability needs for defense acquisition programs. Chairman also
responsible for validating and approving capabilities documents.
Sec. 2366, major systems and DODI 5000.2, enclosure 3, p. 19 Live fire
testing and reporting required for all covered munitions programs:
survivability DODI 5000.2, enclosure 5, p. 29 systems. Strategy or live
fire waiver and alternate testing and lethality testing required DODI
5000.2, p. 10 plan required at milestone B. Live fire test and before
full-scale production evaluation report required for full-rate production
decision.
Sec. 2399, operational test and DODI 5000.2, enclosure 3, p. 18 Director
of Operational Test and Evaluation shall evaluation of defense acquisition
DODI 5000.2, pp. 9-10 determine operational effectiveness and suitability
of programs DODI 5000.2, enclosure 5, pp. 27-28 system under realistic
conditions. Beyond Low Rate
Initial Production Report required at full-rate production decision. DOD
may not conduct operational testing until Director of Operational Test and
Evaluation approves test plan.
Sec. 2400, low-rate initial production DODI 5000.2, enclosure 3, p. 19
Low-rate initial production quantities will be of new systems DODI 5000.2,
pp. 9-10 determined by milestone B.
DODD 5000.1 DODD 5000.1, pp. Policies in directive apply to
1, 2 all acquisition programs.
Evolutionary acquisition
strategies are preferred
approach to satisfying
operational needs.
DODI 5000.2 Instruction applies to all
DODI 5000.2, p. 1 defense technology projects
and acquisition projects.
Chairman of the Joint Chairman of Joint Chiefs of Staff
Chiefs of Staff DODI 5000.2, p. 4 provides advice
Instruction 3170.01Bb through Instruction 3170.01.
Capability documents
required at concept decision and
at
milestones A, B, and C.
Other provisions of law and regulations DODI 5000.2, enclosure 3, pp.
18-22 Several other statutory, regulatory, and contract (including
successor documents) that requirements are addressed in the policy. are
applicable to such programs
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
aAlthough the statutory requirement of 10 U.S.C., section 2431 is not
specifically cited in the policy, the same summarized information is
submitted to Congress in the Selected Acquisition Reports for the first
quarter of a fiscal year as required by 10 U.S.C., section 2432. The
requirement for Selected Acquisition Reports is addressed by DOD's
acquisition policy. The statute concerning weapons development and
procurement schedules requires the Secretary of Defense to submit budget
justification documents regarding this information for each weapon system
for which fund authorization is required (and for which procurement funds
are requested in the budget) not later than 45 days after the President
submits the budget to Congress.
bChairman of the Joint Chiefs of Staff Instruction 3170.01B was revised
and reissued June 24, 2003, as Joint Chiefs of Staff Instruction 3170.01C
with an accompanying manual.
Appendix I: Section 802, Evolutionary Acquisition
Section 802 also required DOD to report on its plans for addressing
certain acquisition process issues regarding each increment of an
evolutionary process. DOD reported on how it plans to establish and
approve operational requirements and cost and schedule goals; meet
requirements for operational and live fire testing; monitor cost and
schedule performance; achieve interoperability; and consider total system
performance and total ownership costs. We compared DOD's response with
section 802's reporting requirements. As shown in table 4, DOD was
responsive to the 802 requirements.
Table 4: How DOD Responded to Section 802's Requirements Regarding
Specific Matters for Each Increment of an Evolutionary Acquisition Process
Section 802 requirements DOD response Policy reference
"The manner in which the Secretary plans to establish and approve, for
each increment of an evolutionary acquisition process-"
"operational requirements; and" "Operational Requirements: Each program
is required to have documented, approved operational requirements in
accordance with authorized Joint Staff procedures. For evolutionary
acquisition programs, the requirements documents are typically time-phased
and specify the capability expected of each increment."
"cost and schedule goals." "Cost and Schedule Goals: At program
initiation, each program and program increment is required to have an
Acquisition Program Baseline approved by the Milestone Decision Authority
(MDA). The Acquisition Program Baseline includes cost and schedule goals."
Capability development document, including key performance parameters,
required at milestone B or program initiation (DODI 5000.2, p. 7, and
enclosure 3, p. 20).
Acquisition program baseline required at milestone B or program initiation
(DODI 5000.2, p. 8, and enclosure 3, p. 19).
Appendix I: Section 802, Evolutionary Acquisition
Section 802 requirements DOD response Policy reference
"The manner in which the Secretary plans, for each increment of an
evolutionary acquisition process-"
"to meet requirements for operational testing and live fire testing;"
"to monitor cost and schedule performance; and"
"to comply with laws requiring reports to Congress on results testing and
on cost and schedule performance."
"At program initiation, each program or program increment must have a Test
and Evaluation Master Plan (TEMP) approved by the Director, Operational
Test and Evaluation. The TEMP includes requirements for operational and
live fire testing."
"Progress against cost and schedule goals [for each increment] is
addressed via automated reporting systems at both the Office of Secretary
of Defense staff level and at the Service staff level, and by reviews
conducted in the context of the acquisition oversight model."
"DoD acquisition policy requires independent assessment by the operational
test authority prior to release of each successive increment to the user.
All reports by statute will be submitted."
Test and Evaluation Master Plan required at milestone B or program
initiation (DODI 5000.2, enclosure 3, p. 21, and enclosure 5, p. 25).
DOD's policy contains an enclosure on resource estimation (DODI 5000.2,
enclosure 6, pp. 30-31). Also, several reporting requirements address
monitoring cost and schedule performance such as acquisition program
baselines and selected acquisition reports (DODI 5000.2, enclosure 3, pp.
18-21).
The service shall perform an independent operational assessment prior to
release of each successive increment to the user (DODI 5000.2, enclosure
5, p. 29). Also, several statutory reports are listed in the policy
addressing testing and cost and schedule performance (DODI 5000.2,
enclosure 3, pp. 18-21).
Appendix I: Section 802, Evolutionary Acquisition
Section 802 requirements DOD response Policy reference
"The manner in which the Secretary plans to ensure that each increment of
an evolutionary acquisition process is designed-"
"to achieve interoperability within and among United States forces and
United States coalition partners; and"
"to optimize total system performance and minimize total ownership costs
by giving appropriate consideration to-"
o "logistics planning;"
o "manpower, personnel, and training;"
o "human, environmental, safety, occupational health, accessibility,
survivability, operational continuity and security factors;"
o "protection of critical program information; and"
o "spectrum management" "Each increment of an evolutionary acquisition
program is required to have an Acquisition Program Baseline, approved by
the MDA, which includes an Interoperability key performance parameter."
"Each increment of an evolutionary acquisition program is required to have
an acquisition strategy, approved by the MDA, that addresses [where
applicable]: logistics planning; manpower, personnel and training; human,
environmental, safety, occupational health; accessibility (human),
survivability, operational continuity (as required by requirements
document), security factors; critical program information; and spectrum
management. These factors are taken into account as the Department of
Defense considers total system performance and total ownership costs."
Acquisition program baseline required at milestone B or program initiation
(DODI 5000.2, p. 8, and enclosure 3, p. 19).
Acquisition strategy required at milestone B or program initiation (DODI
5000.2, p. 7, and enclosure 3, p. 20).
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
Appendix II: Section 803, Spiral Development
Section 803 of the Defense Authorization Act for Fiscal Year 2003
authorized the Secretary of Defense to conduct major defense acquisition
programs as spiral development programs. However, the section placed a
limitation on these programs. It stated that a research and development
program for a major acquisition may not be conducted as a spiral
development program unless the Secretary of Defense or authorized
high-level designee gives approval. The section requires the Secretary of
Defense to issue guidance for the implementation of such programs to
address appropriate processes for ensuring the independent validation of
exit criteria being met, the operational assessment of fieldable
prototypes, and the management of these types of programs.
DOD responded to these requirements principally by incorporating into the
acquisition policy the requirement for a technology development strategy.
This strategy is a prerequisite for a project to enter the technology
development phase of the acquisition process, or milestone A. Table 5
compares the spiral development plan requirements in the act with the
technology development strategy requirements in DOD's May 2003 acquisition
policy.
Table 5: How DOD Policy Addresses Section 803's Requirements
Section 803 requirements DOD policy
"A spiral development plan for research and development Section 3.5.4 of
DOD Instruction 5000.2 states that the technology
program for a major defense acquisition program shall, at a minimum,
include the following matters:"
development strategy document the following:
"A rationale for dividing the research and development program The
rationale for adopting an evolutionary strategy. For an into separate
spirals, together with a preliminary identification of evolutionary
acquisition, either spiral or incremental, the the spirals to be
included." strategy shall include a preliminary description of how the
program will be divided into technology spirals and
development increments.
"A program strategy, including overall cost, schedule and A program
strategy, including overall cost, schedule, and performance goals for the
total research and development." performance goals for the total research
and development program.
"Specific cost, schedule, and performance parameters, Specific cost,
schedule, and performance goals, including exit including measurable exit
criteria, for the first spiral to be criteria, for the first technology
spiral demonstration. conducted."
"A testing plan to ensure that performance goals, parameters, A test plan
to ensure that the goals and exit criteria for the first and exit criteria
are met." technology spiral demonstration are met.
"An appropriate limitation on the number of prototype units that An
appropriate limitation on the number of prototypes units that
may be produced under the research and development may be produced and
deployed during technology
program." development.
Appendix II: Section 803, Spiral Development
Section 803 requirements DOD policy
"Specific performance parameters, including measurable exit Specific performance
goals and exit criteria that must be met
criteria, that must be met before the major defense acquisition program
proceeds into production units in excess of the limitation on the number
of prototype units."
before exceeding the number of prototypes that may be produced under the
research and development program.
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
As shown in the table, DOD's policy generally responded to the
requirements in the act concerning guidance for implementation of spiral
development programs. While the policy includes a technology development
strategy that requires a test plan to ensure the goals and exit criteria
for the first technology spiral demonstration are met and an independent
operational assessment for the release of each product increment to the
user, it is unclear what the process is for independently validating that
cost, schedule, and performance exit criteria have been. However, DOD
stated that the milestone decision authority provides independent
validation that exit criteria have been met as part of DOD's milestone
approval process. Section 803 also requires that a spiral development plan
include "[s]pecific cost, schedule, and performance parameters, including
measurable exit criteria, for the first spiral to be conducted." DOD's
policy substituted "parameters" for "goals" and did not use the term
"measurable" in describing the required exit criteria.
Finally, section 803 requires the Secretary of Defense to submit to
Congress by September 30 yearly from 2003 through 2008 a status report on
each spiral development program. The report is to include information on
unit costs for the projected prototypes. As of October 23, 2003, DOD's
report on the status of each program applying spiral development was still
in draft and not yet submitted. DOD's current draft report states that
there are no research and development programs that have been approved as
spiral development programs as of September 30, 2003. Section 803
requirements were implemented in DOD Instruction 5000.2, which was
effective in May 2003. DOD anticipates that there will be approved spiral
development programs to report in 2004.
Appendix III: Section 804, Technology Maturity, and Section 822, Independent
Technology Readiness Assessments
Section 804 of the Defense Authorization Act for Fiscal Year 2002 required
DOD to report on the maturity of technology at the initiation of major
defense acquisition programs. The act directed DOD to report by March 1 of
each year between 2003 and 2006 on a requirement in DOD's policy that
technology must have been demonstrated in a relevant environment (or,
preferably, in an operational environment) to be considered mature enough
to use for product development in systems integration. Each report is
required to (1) identify any major acquisition program that entered system
development and demonstration during the preceding calendar year with
immature key technology that was not demonstrated in, at minimum, a
relevant environment, as required by the new policy; (2) justify the
incorporation of any key technology on an acquisition program that does
not meet that requirement; (3) and identify any instances that the Deputy
Under Secretary of Defense for Science and Technology did not concur and
explain how the issue has been or will be resolved, including information
on the use of independent readiness assessments. Section 822 of the
Defense Authorization Act for Fiscal Year 2003 amended section 804 by
adding a requirement that the Secretary of Defense identify each case in
which an authoritative decision has been made within DOD not to conduct an
independent technology readiness assessment for a critical technology on a
major defense acquisition program and explain the reasons for the
decision. On March 18, 2003, DOD submitted its first report. Table 6 shows
the specific requirements for the report and DOD's response.
Table 6: How DOD Responded to Section 804 and Section 822 Requirements
Reporting requirements DOD's response
Section 804: "identify each case in which a major defense DOD reported two
programs (Joint Tactical Radio System Cluster acquisition program entered
system development and 1 and Composite Health Care System II) that entered
system demonstration during the preceding calendar year and into which
development and demonstration with key technologies that did not key
technology has been incorporated that does not meet the meet the
technological maturity requirement. While DOD did not technological
maturity requirement ... and provide a justification specifically identify
all the technologies for these programs that did for why such key
technology was incorporated."a not meet the requirements, it did provide
its justification for why
the technologies were incorporated.
Section 804: "identify any determination of technological maturity DOD
reported that, in all cases, the Deputy Under Secretary of with which the
Deputy Under Secretary of Defense for Science Defense for Science and
Technology concurred with the and Technology did not concur and explain
how the issue has technology readiness assessment of the program manager
and been or will be resolved." the milestone decision authority.
Appendix III: Section 804, Technology Maturity, and Section 822, Independent
Technology Readiness Assessments
Reporting requirements DOD's response
Section 822: "identify each case in which an authoritative decision DOD
did not report on or identify any cases.
has been made within the Department of Defense not to conduct
an independent technology readiness assessment for a critical
technology on a major defense acquisition program and explain
the reasons for the decision."
Sources: National Defense Authorization Acts for Fiscal Years 2002 and
2003, DOD, and GAO.
aTechnology must have been demonstrated in a relevant environment (or,
preferably in an operational environment) to be considered mature enough
to use for product development in systems integration.
Appendix IV: Comments from the Department of Defense
Appendix IV: Comments from the Department of Defense
Now on pp. 14 and 15.
Appendix IV: Comments from the Department of Defense
Now on p. 15.
Related GAO Products
Best Practices: Setting Requirements Differently Could Reduce
Weapon Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.:
February 11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge
Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:
July 15, 2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead
to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to
Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:
July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better
Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:
April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon
System Program Implement Best Practices. GAO/NSIAD-99-206.
Washington, D.C.: August 16, 1999.
Best Practices: Better Management of Technology Development
Can Improve Weapon System Outcomes. GAO/NSIAD-99-162.
Washington, D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.
GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon
System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Related GAO Products
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: February
24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer Improvements
for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.
(120214)
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and full-text files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Contact:
To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***