Defense Acquisition: DOD Faces Challenges in Implementing Best
Practices (27-FEB-02, GAO-02-469T).
The Department of Defense (DOD) spends close to $100 billion
annually to develop and acquire weapon systems. DOD spends tens
of billions more for services and information technology. From
1995 to 2007, investments in weapon systems are expected to rise
74 percent--from $90 billion to $157 billion. During the next
five years, DOD's request for weapons development and acquisition
funds is estimated at about $700 billion. DOD's spending on
services is also expected to continue to grow, largely because of
increased purchases of information technology services and
professional, administrative, and management support services.
Although continuing to keep legacy systems, DOD plans to fund
newer programs such as Global Hawk and Predator, as well as
future capabilities such as unmanned airplanes, satellite
networks, and information and communication systems. Despite
heavy investments, DOD is not carrying out acquisitions
cost-effectively and the acquisitions themselves are not always
achieving their objectives. Although the military has many
acquisition reform initiatives under way, pervasive problems
persist regarding the use of high risk acquisition strategies;
questionable requirements and solutions that are not the most
cost-effective available; and unrealistic cost, schedule, and
performance estimates. DOD is committed to adopting many best
practices and has already taken steps to change its policies and
procedures. The changes DOD makes must extend well beyond
policies and procedures. Incentives driving traditional ways of
doing business, for example, must be changed, and cultural
resistance to new approaches must be overcome. DOD will need
strong and sustained commitment from its leadership to tackle
these more elusive challenges--not just to initiate changes, but
to continually support them.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-469T
ACCNO: A02808
TITLE: Defense Acquisition: DOD Faces Challenges in Implementing
Best Practices
DATE: 02/27/2002
SUBJECT: Best practices
Defense economic analysis
Defense procurement
Future budget projections
Information resources management
Information systems
Information technology
Internal controls
Performance measures
Risk management
Strategic planning
Weapons systems
Army Crusader System
Joint Tactical Unmanned Aerial Vehicle
DLA Fuels Automated System
DLA Business Systems Modernization
SEI Software Acquisition Capability
Maturity Model
Patriot PAC-3
DOD Composite Health Care System
AIM-9X Missile
Defense Logistics Standard Systems
Program
DOD Standard Procurement System
Global Hawk Unmanned Aerial Vehicle
Joint Strike Fighter
Predator Unmanned Aerial Vehicle
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-469T
United States General Accounting Office
GAO Testimony
Before the Subcommittee on Readiness and Management Support, Committee on
Armed Services, U.S. Senate
For Release on Delivery
Expected at 10:00 a.m., EDT,
Wednesday, February 27, 2002 DEFENSE ACQUISITIONS
DOD Faces Challenges in Implementing Best Practices
Statement for the Record by Jack L. Brock, Jr., Managing Director,
Acquisition and Sourcing Management, and Randolph C. Hite, Director,
Information Technology Architecture and Systems
GAO-02-469T
Mr. Chairman and Members of the Subcommittee:
Thank you for the opportunity to submit this statement for the record. At
your request, we are discussing our work on best practices related to the
acquisition of services, information technology, and weapon systems.
Additionally, we are providing you with information on our other work for
the subcommittee related to acquisitions.
The Department of Defense (DOD) relies heavily on acquisitions. It spends
close to $100 billion annually to research, develop, and acquire weapon
systems and tens of billions more for services and information technology.
Moreover, this investment is expected to grow substantially. From 1995 to
2007, investments in weapon systems are planned to escalate from $90 billion
to $157 billion-about a 74-percent increase. And over the next 5 years,
starting in fiscal year 2003, DOD's request for weapon system development
and acquisition funds is estimated to be about $700 billion. Similarly,
DOD's spending on services is expected to continue to grow, largely
attributable to increased purchases of information technology services and
professional administrative and management support services.
The goals for this investment are ambitious. While continuing to keep legacy
systems, DOD plans to fund newer programs such as Global Hawk and Predator,
as well as future capabilities such as unmanned airplanes, satellite
networks, and information and communication systems. Additionally,
information technology is expected to play a critical role in DOD's business
transformation.
Despite these heavy investments, our work continues to show that DOD is not
carrying out acquisitions cost-effectively and that the acquisitions
themselves are not always achieving DOD's objectives. Although the
department has many acquisition reform initiatives in process, pervasive
problems persist regarding the use of high risk acquisition strategies;
questionable requirements and solutions that are not the most cost-effective
available; and unrealistic cost, schedule, and performance estimates. For
these reasons, we have reported DOD contract management, information
technology management, and weapon systems acquisition as high risk areas for
more than a decade.1
1 U.S. General Accounting Office, High Risk Series: An Update, GAO-01-263
(Washington, D.C.: Jan. 2001).
To help DOD meet these challenges, you have asked us over the past several
years to identify how leading organizations are addressing similar problems.
Our work has identified numerous practices and principles that have
consistently resulted in better outcomes-including dramatic cost savings,
improved services and products, and ultimately, a better return on
investment. The changes leading organizations make often reflected common
sense approaches, but they are nevertheless dramatically different from the
traditional ways of doing business and they each enhance performance. For
example:
* In analyzing just how much was being spent on acquiring services and where
the dollars were going, leading organizations were able to substantially
reduce the number of suppliers and negotiate lower rates. Sometimes,
thousands of suppliers were reduced to just a few.
* Leading organizations were able to make sure that their business systems
could interoperate and truly help to achieve corporate-rather than business
unit-objectives by using enterprise architectures to guide and constrain
their investments. These architectures are essentially blueprints that
define where the organization is going in terms of mission, business
operations, and technology.
* In developing and manufacturing complex products, leading organizations
have learned to treat technology development and product development
differently and manage them separately. Doing so helped them to reduce
design and production difficulties and to deliver more sophisticated
products quicker and cheaper.
DOD is committed to adopting many best practices and has already taken steps
to change its policies and procedures. Implementing these practices,
however, will be extremely challenging. For instance, the sheer size of the
department, the number of acquisitions, and the hundreds of organizations
involved will make it difficult to gain much-needed visibility over spending
on services as well as to implement enterprisewide management and oversight
mechanisms. Moreover, the changes DOD makes must extend well beyond policies
and procedures. Incentives driving traditional ways of doing business, for
example, must be changed, and cultural resistance to new approaches must be
overcome. Undoubtedly, DOD will need strong and sustained commitment from
its leadership to tackle these more elusive challenges-not just to initiate
changes but to continually support them.
Best Practices in the Acquisition of Services
DOD is, by far, the government's largest purchaser of services. In fiscal
year 2000 alone, it bought more than $53 billion in services ranging from
clerical support and consulting, to information technology services, to the
management and operation of facilities. However, this spending is not being
managed efficiently. Responsibility for acquiring services is spread among
individual military commands, weapon system program offices, or functional
units on military bases, with little visibility or control at the DOD-or
military department level. And when it comes to making procurements, our
work, as well as that of other oversight agencies, continues to show that
requirements are not always clearly defined, alternatives are not fully
considered, and contractors are not adequately overseen. DOD leadership has
recognized the need to change current practices for acquiring services and
is seeking to adapt the same revolutionary business and management practices
that helped the commercial sector gain a competitive edge.
GAO Findings
In view of private sector successes with service acquisitions, this
subcommittee asked us to examine how leading companies reengineered their
approach and the extent to which DOD is pursuing a similar approach. Our
first report, which describes the general framework adopted by leading
companies, was issued in January 2002.2
The leading companies we studied made a number of dramatic changes to the
way they bought services and found that these changes, in turn, resulted in
significant cost savings and service improvements. These changes generally
began with a corporate decision to pursue a more strategic approach to
acquiring services. Taking a strategic approach involves a range of
activities-from developing a better picture of what the company is spending
on services, to taking an enterprisewide approach to procuring services, to
developing new ways of doing business. For example:
* The companies we visited analyzed their spending on services to answer the
basic questions of how much was being spent and where the dollars were
going. In doing so, they realized that they were buying similar services
from numerous providers, often at greatly varying prices. The
2U.S. General Accounting Office, Best Practices: Taking A Strategic Approach
Could Improve DOD's Acquisition of Services, GAO-02-230 (Washington, D.C.:
Jan. 18, 2002).
companies used this data to rationalize their supplier base, or in other
words, to determine the right number of suppliers that met their needs.
* The companies we studied changed how they acquired services in significant
ways. They elevated or expanded the role of the company's procurement
organization; designated commodity managers to oversee key services; and
made extensive use of cross-functional teams to help identify their service
needs, conduct market research, evaluate and select providers, and manage
performance.
Bringing about new ways of doing business was challenging. For example, some
companies spent months piecing together data from various financial
management information systems and examining individual purchase orders just
to get a rough idea of what they were spending on services. Other companies
found that in establishing new procurement processes, they needed to
overcome resistance from individual business units reluctant to share
decision-making responsibility and to involve staff that traditionally did
not communicate with each other. To do so, the companies found they needed
to have sustained commitment from their senior leadership; to clearly
communicate the rationale, goals, and expected results from the
reengineering efforts; and to measure whether the changes were having their
intended effects. The figure below highlights specific principles and
practices the companies we studied followed.
Figure 1: Principles and Practices of Leading Companies
Source: GAO analysis.
Taking a strategic approach clearly paid off, as companies found that they
could save millions of dollars and improve the quality of services received
by instituting these changes. In some cases, thousands of suppliers were
reduced to a few, enabling the companies to negotiate lower rates. In other
cases, new information systems enabled companies to better match their
business managers' needs with potential providers.
The strategic approach taken by the leading firms we visited could serve as
a general framework to guide DOD's service contracting initiatives. DOD has
certain elements critical to taking a strategic approach already in place,
such as the commitment by senior leadership to improve its practices for
acquiring services and to adopting best commercial practices. However, DOD
has not conducted a comprehensive analysis of its spending on services or
thoroughly assessed its current structure, processes, and roles-two elements
that companies found to be crucial to reengineering their approaches to
purchasing services. It also lacks a strategic plan that integrates or
coordinates the various initiatives
underway within the department or that provides a road map for identifying
or prioritizing future efforts.
To achieve the significant improvements possible by the use of best
practices in the acquisition of services, we recommended that the secretary
of defense evaluate how a strategic reengineering approach, such as that
employed by the leading companies we visited, could be used as a framework
to guide DOD's reengineering efforts. Specifically, we recommended that DOD
assess (1) whether current or planned financial or management information
systems can provide the type of spending data that DOD needs to identify
opportunities to leverage its buying power and improve oversight and (2)
whether its current organizational structure, processes, and roles are
adequate to support a more strategic approach to acquiring services.
DOD concurred with our recommendations, and it is implementing improvements
on several fronts. For example, it will be upgrading the Federal Procurement
Data System to provide more detail on service acquisitions.
DOD is also required by the National Defense Authorization Act for Fiscal
Year 2002 to establish and implement a management structure for the
procurement of services comparable to the management structure that applies
to the procurement of products by DOD, and to establish a data collection
system to provide management information on each purchase of services in
excess of the simplified acquisition threshold.
Challenges Still Ahead
Undoubtedly, DOD will find it challenging adopting best practices for buying
services. First, DOD's size and the range and complexity of the services it
acquires may mean that it cannot adopt a "one-size-fits-all" approach to
services acquisitions. According to DOD officials, there are individual
commands that are comparable to a Fortune 500 company, each spending
billions of dollars annually on services. Further, while some services can
be acquired departmentwide, others (such as ship support and maintenance)
may be unique to specific commands, units, or geographic locations. Other
challenges that could affect DOD's service contracting initiatives include
existing problems in its information technology and financial management
systems and the unique requirements of the federal environment.
As noted earlier, our January report provided an overall framework of
practices. We plan to assess each practice area in more depth, looking
Best Practices in Information Technology Acquisitions
further at such questions as what are the best practices for conducting a
spending analysis and how to ensure an organization is getting the right
information.
DOD is the federal government's largest consumer of information technology
(IT) resources, spending almost $22 billion on IT in fiscal year 2001. For
this reason, it is critical that DOD adopt effective IT acquisition
practices. Our past reviews have shown that this is not always done.
Particularly, because of inefficient and ineffective processes, DOD is at
risk of pursing systems and services that do not deliver value commensurate
with costs, and that are duplicative, are not well integrated, and do not
help to optimize mission performance.
GAO Findings
Our work in recent years has looked at best practices for acquiring IT
systems and acquiring IT services (e.g., network support or help desk
support). Like the framework we described for services acquisitions, these
practices do not represent cookie cutter approaches; rather, they need to be
tailored to the type of system and service being acquired, how it will be
used, and its importance to an organization.
IT System Acquisitions: The goals of any IT system acquisition, whether
commercial-off-the-shelf (COTS)-based or customized, can be viewed as
threefold: (1) to deliver needed functional and performance capabilities by
a certain time for a certain cost, (2) to reasonably ensure that, over the
system's useful life, these capabilities will provide mission or business
value in excess of costs, and (3) to ensure that the system is defined,
designed, and implemented in a manner that properly fits within the context
of the organizationwide systems environment. In pursuit of these goals, we
have categorized IT system acquisition best practices into three
corresponding groups, and we apply these practices, as appropriate, in our
evaluations of system acquisition across the federal government, including
recent and ongoing work at DOD for the Senate Committee on Armed Services. A
brief description of the three categories follows:
* Is the system being acquired in accordance with mature software
acquisition processes? The Software Engineering Institute (SEI),3
3 SEI is a nationally recognized, federally funded research and development
center established at Carnegie Mellon University to advance the state of
software engineering, development, and acquisition practices.
recognized for its expertise in software processes, publishes best practice
models and methods governing software engineering, acquisition, and
development. Collectively, these best practice tools provide logical
frameworks for understanding the strengths and weaknesses of an
organization's existing software practices, including acquisition practices,
and a structured approach for incrementally implementing them. For example,
SEI has defined a five stage software acquisition capability maturity model
with specific best practices associated with each stage of maturity,
including practices governing acquisition planning, solicitation,
requirements development and management, project management, contract
tracking and oversight, evaluation, and risk management. 4 Other examples of
SEI best practice models include its emerging Integrating Technology by a
Structured Evolutionary Process model, which addresses the unique challenges
associated with COTS-based systems, and its IDEALSM,5 model, which provides
a systematic, five phase best practices-based approach to continuously
improving software practices.
* Is the system being acquired in a series of economically justified
incremental builds? Both federal law and guidance6 advocate the use of
incremental investment management when acquiring or developing large
systems. Incremental investment management can be broken into three major
practices: (1) acquiring/developing the system in a series of smaller system
increments, (2) individually justifying investment in each separate
increment on the basis of costs, benefits, and risks, and (3) monitoring
actual benefits achieved and costs incurred on ongoing increments and
applying these lessons learned to future increments. Using these system
investment practices helps to prevent discovering too late that a given
acquisition/development effort is not cost beneficial.
4 Software Acquisition Capability Maturity Model? (SA-CMM ?), Version 1.02.
Capability Maturity ModelSM is a service mark of Carnegie Mellon University,
and CMM? is registered in the U.S. Patent and Trademark Office.
5 IDEALSM is a service mark of Carnegie Mellon University and stands for
initiating, diagnosing, establishing, acting, and leveraging.
6 Clinger-Cohen Act of 1996, P.L. 104-106, and Office of Management and
Budget Circular A-130 (Nov. 30, 2000).
* Is the system's proposed architecture compliant with the organization's
relevant enterprise architecture(s)? Enterprise architectures7 are essential
tools for effectively and efficiently reengineering business processes and
for acquiring and evolving supporting systems. As such, using them is a
recognized best practice that is embodied in federal guidance.8 An
enterprise architecture can be viewed as a master blueprint that defines
operational and technological change across a given entity, which can be an
organization (e.g., a military service or Defense agency) or a functional or
mission area spanning more than one organization (e.g., financial management
or combat system identification). In some cases, both organizational and
functional/mission area architectures are appropriate because organizations
interrelate closely, sharing functional and mission area responsibilities.
This is the case for DOD and its component organizations.
IT Service Acquisitions: At the request of this committee, we studied IT
services acquisition best practices and captured these practices in a
framework, which includes seven phases as described below. 9 Embedded within
each of the phases are specific practices. For example, during the first
phase, the business and technical reasons for undertaking an outsourcing
effort are explicitly described. This practice ensures, among other things,
that the organization has evaluated the appropriateness of outsourcing in
its environment. During the second phase, the boundary of responsibilities
between provider and acquirer is defined. This takes place prior to even
developing the proposal so that the acquirer understands what resources will
be required of it, and the prospective provider understands its
responsibilities prior to bidding. The remaining phases of the framework
focus on managing and monitoring provider performance.
7 These architectures systematically capture-in useful models, diagrams, and
narrative- the relevant breadth and depth of the mission-based mode of
operation for a given enterprise. Moreover, they describe these operations
in both (1) logical terms, such as interrelated processes, information needs
and flows, work locations, and system applications, and (2) technical terms,
such as hardware, software, data, communications, and security attributes,
and standards. They also provide these perspectives both for the
enterprise's current or "as is" environment for its target or "to be"
environment, as well as a plan or road map for moving between the two
environments.
8 Chief Information Officer's Council, A Practical Guide to Federal
Enterprise Architecture, Version 1.0 (February 2001).
9U.S. General Accounting Office, Information Technology: Leading Commercial
Practices for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov.
30, 2001).
Table 1: Description of Phases for IT Outsourcing
Phase Definition
1. Determine sourcing strategy
Determine whether internal capability or external expertise can more
effectively meet IT needs.
2. Define operational model
Formalize executive leadership, team composition, client responsibilities,
and operating relationships between client and provider organizations.
3. Develop the contract Establish the legal terms for the IT outsourcing
relationship.
4. Select provider(s) Find one or more providers who can help reach IT
outsourcing goals.
5. Transition to provider(s) Transfer responsibility of IT functions to one
or more providers.
6. Manage provider(s) Make sure each provider is meeting performance
performance requirements.
7. Ensure services are Periodically benchmark whether end-users needs are
provided` being met to assess whether the organization is still getting good
value.
In addition, the framework recognizes three critical success factors that
transcend the seven phases: executive leadership, partner alignment, and
relationship management. For example, relationship management describes a
process of managing the vendor's performance that goes beyond the specifics
of the contract. In relationship management, the acquirer and the provider
work together to identify issues and concerns before they evolve into
situations requiring official action.
Our recent work has shown that DOD has not consistently applied best
practices when acquiring IT. For example, we recently reported to this
committee that two Defense Logistics Agency (DLA) system acquisitions- the
Business Systems Modernization (BSM) and the Fuels Automated System
(FAS)10-represented a "tale of two cities" with regard to application of the
best practices embodied in SEI's software acquisition
10BSM is intended to modernize DLA's materiel management business functions,
thereby enabling the agency to manage supply chains. BSM is based on
commercially available software products and is expected to cost $658
million from fiscal years 2000 through 2005. FAS is intended to help the
Defense Energy Support Center annually manage about $5 billion in contracts
with petroleum suppliers. FAS also relies on a commercially available
software package and is expected to cost $293 million from fiscal year 1995
through 2002.
DOD Has Inconsistently Applied Best Practices
maturity model.11 Specifically, while DLA was implementing the vast majority
of these best practices on BSM, it was not on FAS because of resource
constraints. By not following these practices, we concluded that FAS was at
risk of not delivering promised system capabilities on time and within
budget. To address the weaknesses we identified, we made a number of
specific recommendations that DLA intends to implement, including launching
a software process improvement program.
We also recently reported to this committee that DOD components varied in
the degree to which they were implementing software process improvement.12
In particular, we reported that the Air Force, the Army, and the Defense
Finance and Accounting Service generally satisfied the best practice tenets
of SEI's IDEALSM model, as did certain Navy units. However, DLA, the Marine
Corps, and other Navy units did not. This particular model defines a
systematic, five-phased approach for software process improvement.
Accordingly, we made recommendations to correct these weaknesses, which DOD
is implementing.
In July 2000, we reported on DOD system acquisitions that were not employing
incremental investment management best practices. 13 As an example, we
reported that the department had divided its multi-year, billion dollar
Standard Procurement System (SPS)14 into a series of incremental system
releases. However, it had not treated each of these system increments as a
separate investment decision. Instead, it had treated investment in all SPS
increments as a single decision that it made when the acquisition was begun.
Moreover, it was not attempting to validate whether expected system benefits
were actually accruing from deployed system releases. This type of approach
to making investment
11U.S. General Accounting Office, Information Technology: Inconsistent
Software Acquisition Processes at the Defense Logistics Agency Increase
Project Risks, GAO-02-9 (Washington, D.C.: Jan. 10, 2002).
12U.S. General Accounting Office, DOD Information Technology: Software and
Systems Process Improvement Programs Vary in Use of Best Practices,
GAO-01-116 (Washington, D.C.: Mar. 30, 2001).
13U.S. General Accounting Office, DOD Systems Modernization: Continued
Investment in the Standard Procurement System Has Not Been Justified,
GAO-01-682 (Washington, D.C.: July 31, 2001) and U.S. General Accounting
Office, Information Technology: DLA Should Strengthen Business Systems
Modernization Architecture and Investment Activities, GAO-01-631
(Washington, D.C.: June 29, 2001).
14SPS is intended to be DOD's single, standard procurement system and is
expected to cost $3.7 billion over a 10-year period.
decisions has historically resulted in agencies' investing huge sums of
money in systems that do not provide commensurate benefits, and thus has
been abandoned by successful organizations. Accordingly, we made a series of
recommendations to correct the situation. DOD is in the process of
addressing our recommendations.
Lastly, in June 2001, we reported that DOD was in the process of investing
billions of dollars in acquiring various financial and logistics management
systems without having enterprise architectures for either functional area
to guide and constrain these investments.15 As part of these reports, we
made a series of recommendations to systematically correct this IT
acquisition weakness. DOD has initiated steps to implement the
recommendations.
Challenges Ahead
Best Practices for the Acquisition of Weapon Systems
While DOD has taken steps to implement our recommendations, the challenges
ahead are still substantial. To make the most out of its investment in IT,
DOD needs to fully incorporate best practices into its policies and
procedures and implement our recommendations. Until this is done, DOD risks
not meeting its objectives-leading to costly scheduling delays and rework.
Our work for the committee will continue to look at systems acquisitions
important to DOD operations and determine whether best practices are being
effectively applied to them. Currently, we are evaluating how effectively
best practices have been applied to DOD's Composite Health Care System. We
are also working with this committee to identify DOD IT acquisitions to
evaluate against our IT outsourcing framework.
As noted earlier, DOD spends close to $100 billion annually to research,
develop, and acquire weapons systems. Moreover, it is seeking to
considerably ramp up spending to replace a force it believes is becoming
outdated and too costly to operate. (See fig. 2 for DOD's planned
investments in weapon systems.) Our reviews over the past 20 years have
consistently found the same problems with these investments-cost
15U.S. General Accounting Office, Information Technology: Architecture
Needed to Guide Modernization of DOD's Financial Operations,GAO-01-525
(Washington, D.C.: May 17, 2001) and U.S. General Accounting Office,
Information Technology: DLA Should Strengthen Business Systems Modernization
Architecture and Investment Activities, GAO-01-631 (Washington, D.C.:, June
29, 2001).
increases, schedule delays, and performance shortfalls. Clearly, it is
critical to find better ways of doing business-to make sure that weapon
systems are delivered on time, at cost, and effectively. Failure to do so
can jeopardize other programs in the department and limit DOD's ability to
effectively execute warfighting operations.
Figure 2: Research, Development, Test and Evaluation and Procurement Funding
for Fiscal Years 1995 to 2007
Source: DOD.
GAO Findings At the request of the committee, we have undertaken an
extensive body of work that examines weapon acquisition issues from a
different, more cross-cutting perspective-one that draws lessons learned
from the best commercial product development efforts to see if they apply to
weapon system improvement.
This work has consistently shown that leading commercial firms expect that
their program managers will deliver high quality products on time and within
budget. Doing otherwise could result in the customer walking away. Thus, the
firms have created an environment and adopted practices that put their
program managers in a good position to succeed in meeting
these expectations. Collectively, these practices ensure that a high level
of knowledge exists about critical facets of the product at key junctures
during development. Such a knowledge-based process enables decision makers
to be reasonably certain about critical facets of the product under
development when they need it.
The process followed by leading firms can be broken down into three
knowledge points:
* At program launch, when a match must be made between the customer's needs
and the available resources-technology, design, time, and funding;
* Midway through development, when the product's design must demonstrate its
ability to meet performance requirements; and
* At production start, when it must be shown that the product can be
manufactured within cost, schedule, and quality targets. Figure 3 further
illustrates how this process works, while figure 4 highlights some specific
best practices within this process as well as criteria used to move forward.
Figure 3: Knowledge-based Process for Applying Best Practices to the
Development of New Products
Source: GAO analysis.
Figure 4: Highlights of Specific Best Practices
Source: GAO analysis.
Knowledge-based acquisitions embraced in DOD policy
We have found that when DOD programs employed similar practices, they have
experienced outcomes similar to leading firms. The AIM-9X air-to-air missile
program is a good example. By adopting practices that mature technology
before going into product development and stabilized the design by releasing
over 90 percent of the drawings, the program has experienced very minimal
cost increases and scheduling delays.
Conversely, problems occur in programs when best practices are not adopted.
For example, the PAC-3 missile program began nearly 5 years before most of
the technical discovery was complete, and only 20 percent of design drawings
were released at the point when knowledge point 2 should have been achieved.
As a result, the early part of the program was plagued with technical
difficulties that impaired efforts to stabilize design, and manufacturing
the missile has been difficult. The result was costs that doubled and over a
3-year schedule delay.
In 2000 and 2001, DOD made constructive changes to its acquisition policy
that embrace best practices. These focused primarily on (1) making sure
technologies are demonstrated to a high level of maturity before beginning a
weapon system program and (2) taking an evolutionary, or phased, approach to
developing the system.
DOD's policy changes are a positive step. First, they would separate
technology development from a weapon system development program. This would
help to curb incentives to overpromise the capabilities of a new weapon
system and to rely on immature technologies. By having a baseline
requirement, decisionmakers would also have a means for deciding not to
launch a program if a match between requirements and resources was not made.
Second, the changes recommend an evolutionary approach to developing
requirements and making improvements to a system's capabilities. This is
substantially different than the historical approach, which sought to
deliver all desired capabilities in one "big bang." For example, the F-22
fighter program was justified on the basis of achieving stealth, supercruise
propulsion and fuzed avionics in one-leap with the first product off the
production line. But the technologies, and even the funds, were not
available to make good on such a promise.
While DOD's policy changes are a good step, implementation has been mixed.
There have been some successes with evolutionary acquisitions, but they are
exceptional cases in that they required significant and unusual intervention
from top leadership in the services and DOD. For example, the Global Hawk
and Tactical Unmanned Aerial Vehicle programs-both born from Advanced
Concept Technology Demonstrations-have so far
been successful in reducing the time it takes to develop and field a new
weapon. In the Tactical Unmanned Aerial Vehicle program, the top military
acquisition executive met with the head of the user representative's
organization, struck an agreement that the product was to be fielded in
stages, with the first stage being a very basic system, and then enforced
the agreement. The personal involvement of the under secretary of defense
for acquisition, technology, and logistics helped set the stage for Global
Hawk's evolutionary approach to meeting requirements. In both cases, this
top-level intervention allowed requirements to be flexible and gave the
product developers parity with the requirements setters in influencing
requirements. Equally important, we believe the intervention signaled
support for the programs, which eased some of the pressures that normally
accompany efforts to get programs approved.
In some of DOD's larger, more complex programs, best practices recommended
by the new policy have not been effectively implemented. For example, we
recently reported that although the Joint Strike Fighter program has made
good progress in some technology areas, the program is at risk of not
meeting its affordability objective because critical technologies are not
projected to be matured to levels that we believe would indicate a low risk
program at the planned start of product development.16 Earlier this week, we
also reported that while the Crusader program has made considerable progress
in developing key technologies and reducing its size and weight, it was also
likely to enter product development with the majority of its critical
technologies less mature than best practices recommend.17 As stressed in
both reports, failure to make sure technologies are sufficiently mature
before product development could result in increases in both product and
long-term ownership costs, schedule delays, and compromised performance.
Challenges Still Ahead New policies will not produce better outcomes unless
they influence decisions made on weapon systems. A major challenge ahead for
DOD is taking steps necessary to make this happen. Specifically:
16U.S. General Accounting Office, Joint Strike Fighter Acquisition: Mature
Critical Technologies Needed to Reduce Risks, GAO-02-39 (Washington, D.C.:
Oct. 19, 2001).
17U.S. General Accounting Office, Defense Acquisitions: Steps to Improve the
Crusader Program's Investment Decisions, GAO-02-201 (Washington D.C.: Feb.
25, 2002).
* Programs must be structured so that requirements will not outstrip
resources. This means getting requirements-setting organizations to be open
to redefining their needs to better match resources available.
* DOD's funding process must provide assurance to evolutionary programs that
the end-state capability will eventually be achieved. This means getting
decisionmakers to commit to providing funding for later, more improved
versions of a system.
* The role of the science and technology community must change to accept
more responsibility for maturing relevant technologies-without harming DOD's
long-term basic research needs. This may require DOD to increase funding and
support for science and technology.
* Measures for success need to be defined for each stage of the development
process so that decisionmakers can be assured that sufficient knowledge
exists about critical facets of the product before investing more time and
money.
* Responsibility for making decisions must be squarely positioned in those
with authority to adhere to best practices and to make informed tradeoff
decisions.
Our work in this area continues to take aspects of the best practice
framework and look deeper into how specific practices can enhance how weapon
systems are developed and managed. We are currently looking at the
management of product design and manufacturing and the question of how
leading firms reduce total ownership costs of capital equipment.
We were also asked to provide updates on our ongoing work related to (1)
competition under task- or delivery-order contracts, (2) spare parts price
increases, and (3) DOD's use of waivers for certified cost data in
negotiating contracts. The following sections highlight our findings and
describe ongoing work.
Additional Ongoing
Work of Interest to
the Subcommittee
Competition Under Task-The government acquires billions of dollars worth of
products and or Delivery-Order services each year using task- or
delivery-order contracts-also known as Contracts multiple award
contracts-that are available for use by all federal
agencies. A task- or delivery-order contract provides for an indefinite
quantity of supplies or services (within specific limits) to be furnished
during a fixed period, with deliveries scheduled through orders with the
contractor. There have been persistent concerns that agencies avoid
competition when ordering under such contracts. To prevent this from
occurring, the Congress, through the Federal Acquisition Streamlining Act,18
imposed statutory requirements on the use of these contracts. Agencies must
now consider awarding multiple contracts rather than a single contract when
planning a task- or delivery-order contract. Even with this change, concerns
about a lack of competition when ordering under task-and delivery-order
contracts have persisted.
In 1998, we examined how multiple award contracts were being administered by
six organizations, including several within DOD such as the Defense
Information Systems Agency, the Standard Systems Group, and the Electronic
Systems Center's Hanscom Air Force Base operations. We found that efforts to
promote competition for orders placed under multiple award contracts varied
at the six organizations. Two organizations achieved consistent competition
for orders under their contracts while four others experienced more
difficulty obtaining competition. One organization, for example, issued 64
percent of orders (accounting for 20 percent of dollars awarded) on a
sole-source basis through the end of fiscal year 1997. Another organization
named preferred vendors in announcements of opportunities, resulting in only
one proposal being received on most orders.19
In 2000, at the request of this subcommittee, we expanded our review to
examine DOD's use of large orders placed under multiple award contracts to
acquire IT products and services. We reported that most of the 22 large
orders we reviewed had been awarded without competing proposals having been
received. Agencies frequently issued orders on a sole-source basis using one
of the statutory exceptions to the fair opportunity requirement, and
contractors frequently did not submit proposals when provided an opportunity
to do so. In most cases, the proposals received involved incumbent
contractors.20
The DOD Office of the Inspector General (IG) reported in September 2001 that
competition was limited for orders under multiple award contracts
18 P.L. 103-355 (Oct. 13, 1994).
19U.S. General Accounting Office, Acquisition Reform: Multiple-award
Contracting at Six Federal Organizations, GAO/NSIAD-98-215 (Washington,
D.C.: Sept. 30, 1998).
20U.S. General Accounting Office, Contract Management: Few Competing
Proposals for Large DOD Information Technology Orders, GAO/NSIAD-00-56
(Washington, D.C.: Mar. 20, 2000).
DOD organizations administered.21 According to the IG, contracting offices
continued to direct awards to selected sources without providing contractors
a fair opportunity to be considered-304 of the 423 orders reviewed had been
awarded on a sole-source or directed source basis. As a result, the IG
concluded, DOD was not obtaining the benefits of sustained competition and
the reduced costs that the Congress envisioned multiple award contracts
providing.
Also, since our reviews began, steps have been taken by the executive branch
and the Congress to promote broader competition. First, the executive branch
revised procurement regulations in June 1999 to prohibit agencies from
designating preferred vendors for orders-the practice we had reported on the
previous year. Second, the Congress directed, through the National Defense
Authorization Act for Fiscal Year 2000,22 that the procurement regulations
be revised to provide guidance on steps agencies should take to ensure
contractors are provided a fair opportunity to be considered. An initial
revision to the regulations was issued in April 2000 that directed
contracting officers to avoid situations where contractors will specialize
in one or a few areas within the contract's scope, creating the likelihood
that orders in those areas will be awarded on a sole-source basis. The
revision directs contracting officers to consider such factors as the scope
and complexity of the requirement, the expected duration and frequency of
task orders, and the mix of resources a contractor must have to perform
expected task-or delivery-order requirements. The revision also changed the
requirements for placing individual orders under these contracts. An
additional revision to the regulations has been proposed that identifies
issues contracting officers should consider when developing ordering
procedures for multiple award contracts.
The National Defense Authorization Act for Fiscal Year 2000 directs us to
evaluate conformance of the guidance it mandates with existing law. Although
the act indicates this guidance should be in place by April 2, 2000, the
regulations have not yet been issued in final form. We will initiate the
evaluations the act requires once the regulations are finalized, and we look
forward to continuing to work with you and your staff on issues related to
multiple-award contracting.
21Office of the Inspector General, Department of Defense, Multiple Award
Contracts for Services (Sept. 30, 2001).
22 P.L. 106-65 (Oct. 5, 1999).
Increases in Spare Part Prices
In recent years, the military services have expressed concern to the
Congress that spare part prices have been increasing at a higher rate than
inflation and have taken an unanticipated bite out of the limited funds
available to meet readiness requirements. Because the planned price changes
for spare parts drive the ordering units' budget requests, unexpected price
increases could affect their ability to purchase all the parts they need. In
response to these concerns, this subcommittee asked us to determine whether
spare part prices had been increasing and to identify some of the factors
driving the escalation. In 2000, we issued reports on the prices DOD
activities paid for Navy-managed aviation reparable parts, consumable spare
parts23 purchased from DLA, and Marine Corps ground system reparable
parts.24 At the request of the subcommittee, we are following-up on our
prior work at the Navy and are examining the status of DLA's efforts to
address spare part price increases.
Overall, we found that price increases at the Defense organizations we
reviewed were high for certain categories of parts. At the Navy, for
example, the prices increased an average of 12 percent annually, but parts
with high sales volume increased substantially more than parts overall.
Results from our ongoing review indicate that these price increases are
continuing. At DLA, the annual price change was less than 5 percent for most
parts; but for about 14 percent of the parts, price changes were
considerably higher, with a very small percentage experiencing price changes
of 1,000 percent or more.
Our specific findings at the Navy, DLA, and Marine Corps are described
below.
Navy: We found that prices for all Navy-managed aviation parts increased at
an average annual rate of 12 percent from 1994 to 1999. However, prices for
parts with high sales volume increased substantially more, at an average
annual rate of 27 percent. From year to year, there were strong fluctuations
in prices, making it difficult for the Navy to project price
23 Consumable items are those that are consumed in use or discarded when
worn out or broken because they cannot be cost-effectively repaired.
24 U.S. General Accounting Office, Defense Acquisitions: Prices of Marine
Corps Spare Parts Have Increased, GAO/NSIAD-00-123 (Washington, D.C.: July
31, 2000); U.S. General Accounting Office, Defense Acquisitions: Price
Trends for Defense Logistics Agency's Weapon System Parts, GAO-01-22
(Washington, D.C.: Nov. 3, 2000); U.S. General Accounting Office, Defense
Acquisitions: Prices of Navy Aviation Spare Parts Have Increased, GAO-01-23
(Washington, D.C.: Nov. 6, 2000).
changes. These fluctuations were largely driven by dramatic swings in the
surcharge rate. The lack of stability in prices affects ordering units,
which may not have sufficient funds budgeted if unexpected price increases
occur. In addition, we reported that the Navy had sought to alleviate
concerns about high surcharge rates by moving certain overhead costs from
the surcharge rate to the repair cost. This approach merely re-allocated the
overhead costs rather than reducing them.
Our follow-on work indicates that prices continue to increase. We will
report on reasons for the increases later this year. Our work thus far
suggests that the primary reason for the increase is higher material cost
associated with repair of the items. Factors we are looking at as drivers
for the higher material costs include (1) increased cost of parts ordered
from DLA, (2) new, more expensive material being used in the repair process,
(3) increased material usage in the repair process, and (4) a change in the
mix of parts used in repair.
DLA: We found that from 1989 through 1998, an average of 70 percent of the
consumable parts requisitioned by DLA's customers experienced an annual
price change of less than 5 percent. However, a relatively small number of
parts experienced significant annual price increases; that is, increases of
50 percent or more. The proportion of parts with increases of 50 percent or
more had been increasing since 1994, reaching nearly 14 percent in 1998. In
addition, a very small percentage of the parts experienced extreme increases
in price-1,000 percent or more from one year to the next. These extreme
price increases are due to outdated or estimated prices in the catalog that
DOD units consult when ordering parts. When the catalog price reflects an
outdated or estimated price, ordering agencies experience "sticker shock"
when confronted with the actual price of the item, which in some cases is
significantly higher than the listed-and anticipated-price.
Since our review, DLA has undertaken a range of efforts to respond to
concerns about significant spare part price increases. For example, the
agency has recently completed two procurement management reviews concerning
price reasonableness determinations and is developing two computer software
programs to assist buyers in evaluating contractor-offered prices. These
efforts, however, are in various stages of completion and it is too early to
assess the results. In March 2001, DLA reported to the secretary of defense
on price increases for commercially available spare parts. DLA reported
that, from fiscal year 1993 to 2000, materiel costs grew 10.8 percent for
competitively purchased commercial items, but increased more than twice as
much for noncompetitive purchases. DLA is examining
the causes of the price increases and plans to provide the Secretary of
Defense with more detailed explanations of cost growth disparities and any
remedies. This analysis will be part of DOD's third report to Congress as
required in the Strom Thurmond National Defense Authorization Act for Fiscal
Year 1999.25
Marine Corps: We focused our work on the ground system spare parts that
end-users actually procured during fiscal years 1997-99. We found that the
prices for these parts had increased at an average annual rate of about 14
percent from 1995-99. Increases in the surcharge rates charged by the Marine
Corps were a major cause of the price escalation. We also found that the
Marine Corps did not follow DOD pricing regulations in setting prices and
that mathematical and computer errors had occurred in price-setting. As a
result, the prices of most parts sold to Marine Corps customers were not
correct. The Marine Corps has since corrected its prices and implemented a
number of corrective actions pertaining to its pricing methodology.
Waiving the Requirement
for Certified Cost Data in
Negotiations of Contracts
To maximize the value of taxpayer dollars, the federal government generally
seeks to compete its contracts. However, DOD buys many unique products and
services for which it cannot always rely on competitive forces of the
marketplace to get fair prices and values. Instead, it must turn to just a
few sources or even a sole source for its procurements. Each year, DOD
purchases billions of dollars in weapons systems without competition.
In these cases, contractors and subcontractors normally provide the
government with cost or pricing data supporting their proposed prices and
they certify that the data submitted are accurate, complete, and current.
This requirement, established by the Truth-in-Negotiation Act, is meant to
put the government on an information parity with sole-source contractors and
protect against inflated prices. The act, as amended, specifically provided
that the requirements did not apply in "exceptional cases," but it did not
include an explanation of what constituted an exceptional case and it has
never been amended to define that term. In September 1995, the Federal
Acquisition Regulation was amended to allow the head of contracting
activities to authorize exceptional case waivers if contracting
25 P.L. 105-261.
officers have sufficient information available to negotiate fair and
reasonable contract prices without requiring certified data.
Concerned about the waiver process, this subcommittee requested that we look
at cases where waivers have been made to identify the circumstances in which
the waiver was used and techniques used to negotiate prices in place of
requiring certified data. We identified 20 cases in which the requirement
for certified data was waived, covering fiscal year 2000 contracting actions
of $5 million or more at six buying activities. The total contract value for
which certified data was waived was $4.4 billion. Contracting officers cited
the authority contained in the Federal Acquisition Regulation as the basis
for all 20 waivers we reviewed. We are evaluating the agencies' bases for
the decisions to waive cost and pricing data. We will be reporting on our
findings later this year.
This concludes our statement. We appreciate the opportunity to have it
placed in the record. If you have questions about our work on service
acquisitions and other contracting issues, please call David Cooper at (202)
512-4125, on information technology issues, please call Randolph Hite at
(202) 512-3439, and on weapon system issues, please call Katherine Schinasi
at (202) 512-4841.
Related GAO Products
Contract Management
Acquisition Reform: Multiple-award Contracting at Six Federal Organizations.
GAO/NSIAD-98-215. Washington, D.C.: September 30, 1998.
Acquisition Reform: Review of Selected Best-Value Contracts.
GAO/NSIAD-99-93R. Washington, D.C.: April 14, 1999.
Best Practices: Taking a Strategic Approach Could Improve DOD's Acquisition
of Services. GAO-02-230. January 18, 2002.
Contract Management: Few Competing Proposals for Large DOD Information
Technology Orders. GAO/NSIAD-00-56. March 20, 2000.
Contract Management: Not Following Procedures Undermines Best Pricing Under
GSA's Schedule. GAO-01-125. November 28, 2000.
Contract Management: Service Contracting Trends and Challenges.
GAO-01-1074R. August 22, 2001.
Contract Management: Trends and Challenges in Acquiring Services.
GAO-01-753T. May 22, 2001.
Information
Technology
DOD Information Technology: Software and Systems Process Improvement
Programs Vary in Use of Best Practices. GAO-01-116. March 30, 2001.
DOD Systems Modernization: Continued Investment in the Standard Procurement
System Has Not Been Justified. GAO-01-682. July 31, 2001.
DOD's Standard Procurement System: Continued Investment Has Yet to Be
Justified. GAO-02-392T. February 7, 2002.
Information Technology: Architecture Needed to Guide Modernization of DOD's
Financial Operations. GAO-01-525. May 17, 2001.
Information Technology: DLA Should Strengthen Business Systems Modernization
Architecture and Investment Activities. GAO-01-631. June 29, 2001.
Information Technology: Inconsistent Software Acquisition Processes at the
Defense Logistics Agency Increase Project Risks. GAO-02-9. January 10, 2002.
Information Technology: Leading Commercial Practices for Outsourcing of
Services. GAO-02-214. November 30, 2001.
Defense Acquisitions: Collection and Reporting of Information Technology
Purchases. GAO-02-331. January 28, 2002.
Weapon System Acquisition
Best Practices: A More Constructive Test Approach Is Key to Better Weapon
System Outcomes. GAO/NSIAD-00-199. July 31, 2000.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. July 30, 1999.
Best Practices: Better Matching of Needs and Resources Will Lead to Better
Weapon System Outcomes. GAO-01-288. March 8, 2001.
Best Practices: Commercial Quality Assurance Practices Offer Improvements
for DOD. GAO/NSIAD-96-162. August 26, 1996.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. March 17, 1998.
Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/NSIAD-99-206. August 16, 1999.
Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. February 24, 1998.
Defense Acquisition: Best Commercial Practices Can Improve Program Outcomes.
GAO/T-NSIAD-99-116. March 17, 1999.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon System
Decisions. GAO/T-NSIAD-00-137. April 26, 2000.
Defense Acquisition: Improved Program Outcomes Are Possible T-NSIAD-98-123.
March 18, 1998.
Defense Acquisitions: Steps to Improve the Crusader Program's Investment
Decisions. GAO-02-201. February 25, 2002.
Joint Strike Fighter Acquisition: Mature Critical Technologies Needed to
Reduce Risks. GAO-02-39. October 19, 2001.
Pricing Issues Defense Acquisitions: Prices of Navy Aviation Spare Parts
Have Increased. GAO-01-23. November 6, 2000.
Defense Acquisitions: Price Trends for Defense Logistics Agency's Weapon
System Parts. GAO-01-22. November 3, 2000.
Defense Acquisitions: Prices of Marine Corps Spare Parts Have Increased.
GAO/NSIAD-00-123. July 31, 2000.
Additional GAO High Risk Series: An Update. GAO-01-263. January 2001.
Reports Major Management Challenges and Program Risks: Department of
Defense. GAO-01-244. January 2001.
*** End of document. ***