NASA Procurement: Challenges Remain in Implementing Improvement Reforms (Chapter Report, 08/18/94, GAO/NSIAD-94-179). In recent years, the National Aeronautics and Space Administration (NASA) has experienced a variety of development and performance problems with expensive space hardware. These problems have been a primary impetus behind the mounting Congressional pressures for NASA to improve its procurement practices. NASA has partly or fully implemented all but one of the eight key procurement management improvement initiatives it was working on in June 1993. Some were developed slowly and implemented quickly or achieved goals ahead of schedule. GAO questions whether some of the initiatives will be fully effective as a result of planning and implementation problems. Specifically, (1) the proposed contractors' liability policy will be hard to administer, may produce higher contract prices, and could harm subcontractors who are unwilling or unable to risk increased liability; (2) the potential effect of the proposed contractor liability policy on small and disadvantaged businesses has not been evaluated; and (3) the policy guidance for the training of contracting officers' technical representatives is incomplete. --------------------------- Indexing Terms ----------------------------- REPORTNUM: NSIAD-94-179 TITLE: NASA Procurement: Challenges Remain in Implementing Improvement Reforms DATE: 08/18/94 SUBJECT: Federal procurement Contract administration Aerospace contracts Contract modifications Contract costs Contractor performance Work measurement standards Small business contractors Human resources training Contractor responsibility ************************************************************************** * This file contains an ASCII representation of the text of a GAO * * report. Delineations within the text indicating chapter titles, * * headings, and bullets are preserved. Major divisions and subdivisions * * of the text, such as Chapters, Sections, and Appendixes, are * * identified by double and single lines. The numbers on the right end * * of these lines indicate the position of each of the subsections in the * * document outline. These numbers do NOT correspond with the page * * numbers of the printed product. * * * * No attempt has been made to display graphic images, although figure * * captions are reproduced. Tables are included, but may not resemble * * those in the printed version. * * * * A printed copy of this report may be obtained from the GAO Document * * Distribution Facility by calling (202) 512-6000, by faxing your * * request to (301) 258-4066, or by writing to P.O. Box 6015, * * Gaithersburg, MD 20884-6015. We are unable to accept electronic orders * * for printed documents at this time. * ************************************************************************** Cover ================================================================ COVER Report to the Chairman, Committee on Government Operations, House of Representatives August 1994 NASA PROCUREMENT - CHALLENGES REMAIN IN IMPLEMENTING IMPROVEMENT REFORMS GAO/NSIAD-94-179 NASA Procurement Abbreviations =============================================================== ABBREV COTR - contracting officer's technical representative CPAF - cost-plus-award-fee FAR - Federal Acquisition Regulation GAO - General Accounting Office NASA - National Aeronautics and Space Administration SBIR - Small Business Innovative Research SDB - small and disadvantaged business Letter =============================================================== LETTER B-257226 August 18, 1994 The Honorable John Conyers, Jr. Chairman, Committee on Government Operations House of Representatives Dear Mr. Chairman: This report responds to your request that we review the status of the National Aeronautics and Space Administration's (NASA) procurement management improvement initiatives and assess their planning and implementation. Our recommendations to the NASA Administrator focus on improving the agency's procurement initiatives, including providing additional guidance and evaluating the potential for unintended consequences. As requested, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the Administrator, NASA; the Director, Office of Management and Budget; and other interested parties. If you or your staff have any questions about this report, please contact me at (202) 512-8412. Major contributors to this report are listed in appendix V. Sincerely yours, Donna M. Heivilin Director, Defense Management and NASA Issues EXECUTIVE SUMMARY ============================================================ Chapter 0 PURPOSE ---------------------------------------------------------- Chapter 0:1 In recent years, the National Aeronautics and Space Administration (NASA) has experienced a variety of development and performance problems with expensive space hardware. Such problems have contributed to an increasing awareness of NASA's difficulties in managing its contracts and in buying hardware that works properly, and they have been a primary impetus behind the mounting pressure from the Congress for NASA to improve its procurement practices. The Chairman of the House Committee on Government Operations asked GAO to (1) determine the status of NASA's initiatives to improve and streamline its procurement activities and (2) assess NASA's planning and implementation of the initiatives. BACKGROUND ---------------------------------------------------------- Chapter 0:2 NASA spends about 90 percent of its funds--$13.2 billion in fiscal year 1993--on contracts. It is the second-largest civilian contracting agency in the federal government. Since 1987, NASA has acknowledged that its procurement and contract management are vulnerable to waste and mismanagement, based on its own internal management reviews and audits by the NASA Inspector General. Well-publicized performance problems and numerous assessments by NASA management, the NASA Inspector General, and GAO have demonstrated the need for NASA to improve its procurement practices. RESULTS IN BRIEF ---------------------------------------------------------- Chapter 0:3 NASA has partly or fully implemented all but one of the eight key procurement management improvement initiatives it was working on in June 1993. Some were developed slowly and raised concerns inside and outside the agency. Others were developed and implemented quickly or achieved goals ahead of schedule. GAO questions whether some of the initiatives will be fully effective due to planning and implementation problems. Specifically, (1) the proposed contractor liability policy will be difficult to administer, may result in higher contract prices, and could adversely affect subcontractors who are unwilling or unable to risk increased liability; (2) the potential effect of the proposed contractor liability policy on small and disadvantaged businesses has not been evaluated; and (3) the policy guidance for the training of contracting officers' technical representatives is incomplete. PRINCIPAL FINDINGS ---------------------------------------------------------- Chapter 0:4 STATUS OF INITIATIVES FOR IMPROVING PROCUREMENT MANAGEMENT -------------------------------------------------------- Chapter 0:4.1 Over the years, there have been a variety of serious weaknesses in NASA's award fee contracting. For more than 2 years, NASA planned and developed a new award fee policy. The new policy was issued in October 1993. It increases emphasis on selection of contract type, cost control, end product, and overall contractor performance. It also eliminates the use of rollovers (allowing fee that was not earned in a prior period to be available in a subsequent period); restricts base fee (fee paid regardless of performance); provides for a uniform and simplified scoring system throughout the agency; and requires that some award fee contracts include incentives to reward or penalize contractors based on the performance of delivered hardware. In November 1992, the Congress directed NASA to review options and develop a policy for sharing risk with research and development contractors. This direction stemmed from concerns about the agency's payments to contractors for the costs of repairing or replacing articles when the defects were caused by contractors' negligence. In June 1993, NASA reported the results of its assessment, and, in February 1994, it published a proposed policy for public comment. The proposed policy, if implemented as published in draft, would increase contractors' liability for correcting defects in material or workmanship, or other failures to conform to requirements of high-dollar value research and development, cost-type contracts. The new policy would increase a contractor's liability if NASA determined that the contractor (1) had not applied its best efforts toward the research and development objectives of the contract or (2) had not followed proper procedures conforming to generally accepted practices in performing routine operations, such as moving equipment or conducting standardized tests. A NASA procurement official estimated that a policy increasing contractor liability probably will become effective in summer 1994. In January 1993, 6 months after the NASA Administrator directed that it be developed, the agency implemented a system of contractor metrics, a set of standardized data for reporting specific measures of contractor performance. The system provides agency and corporate senior managers with an overview of contractor performance on major NASA contracts. NASA designed this agencywide system to meet two recognized needs: (1) establish an ongoing dialogue between upper-level NASA and corporate management and (2) simplify and standardize the information given to high-level NASA officials on active contracts. A fourth initiative focused on reducing the number and value of the agency's unpriced contract changes. When contract changes are unpriced, the government's cost-risk increases. The longer changes remain unpriced, the greater the risk. Past efforts to manage unpriced contract changes more effectively have either failed to meet goals or have not resulted in lasting reductions in the value of such changes. In June 1993, the agency designated its ongoing efforts as a formal procurement initiative. In May 1994, NASA implemented an agencywide policy on pricing such changes that requires top management approval of high-value changes--a control element absent from prior attempts to significantly and permanently lower the value of outstanding unpriced changes. The four remaining initiatives focused on (1) establishing minimum training requirements for contracting officers' technical representatives, (2) achieving small and disadvantaged business goals, (3) communicating with industry representatives about procurement issues and concerns, and (4) testing methods to reduce the time and effort applied to the majority of NASA's contracting actions without adversely affecting their quality. PROBLEMS IN PLANNING AND IMPLEMENTING PROCUREMENT INITIATIVES -------------------------------------------------------- Chapter 0:4.2 NASA's proposed contractor liability policy will be difficult to administer. The policy increases contractors' liability for the cost of replacement or correction of defects. A NASA official said he assumes that NASA would apply this policy even in cases where the agency does not intend to replace or repair a faulty item. This option is not addressed, however, either in the existing contract clause or in the proposed policy. Consequently, NASA's decision to assess liability in cases when the agency does not intend to replace or repair an item could be disputed by the contractor. NASA's determinations of contractor liability for the cost of replacement or correction of defects requires applying general concepts that are yet to be precisely defined, such as "best efforts," "routine operations," and "generally accepted industrial or engineering practices." NASA's decision to assess liability can be disputed by the contractor and could result in protracted negotiations and potentially costly litigation. Also, the proposed policy could (1) lead to higher contract prices and (2) have unintended adverse consequences on subcontractors with limited capital. NASA has not evaluated the impact of its proposed contractor liability policy on its goals for contracting with small and disadvantaged businesses. The policy, if implemented, could affect the ability or willingness of subcontractors that are small and disadvantaged businesses or universities to work on NASA contracts. Specifically, the policy could result in a potentially significant risk of liability, which could, in some cases, undermine the economic viability of such subcontractors. Also, liabilities for hardware that fails to perform properly or for other failures to comply with contractual requirements could affect large contractors' willingness to subcontract with such businesses, which could inadvertently affect NASA's ability to achieve future small and disadvantaged business contracting goals, especially in high-technology areas. NASA's guidance on new requirements for the training of contracting officers' technical representatives is incomplete. It does not discuss available options for fulfilling mandatory training requirements and for establishing and applying criteria to determine when no additional training is needed. RECOMMENDATIONS ---------------------------------------------------------- Chapter 0:5 GAO recommends that the NASA Administrator evaluate the potential impact of the proposed contractor liability policy on small and disadvantaged businesses and consider options to mitigate any adverse impacts identified; define "high-technology" work and develop and implement a methodology for consistently measuring progress toward NASA's goal of increasing small and disadvantaged businesses' participation in such work, either as prime contractors or subcontractors; direct that special reviews be conducted at centers to ensure that training courses for contracting officers' technical representatives sufficiently address new minimum training requirements; and provide guidance to the centers on establishing and applying criteria to determine when no additional training is needed to meet new minimum training requirements for contracting officers' technical representatives. AGENCY COMMENTS ---------------------------------------------------------- Chapter 0:6 NASA agreed with GAO's recommendations and generally agreed with the report's contents. NASA provided comments on GAO's analyses of some of the initiatives, and these comments were considered in preparing this report. NASA's comments are reprinted in appendix II. INTRODUCTION ============================================================ Chapter 1 The National Aeronautics and Space Administration's (NASA) procurement budget is one of the largest of all civilian agencies in the federal government. Each year, the agency spends about 90 percent of its funds on contracts. Over the last decade, the reported value of NASA's annual procurements in 1993 dollars has increased by almost 30 percent--from $10.4 billion to $13.2 billion. NASA employs more than 1,000 people in procurement activities throughout the agency. CONCERN ABOUT NASA'S PROCUREMENT MANAGEMENT ---------------------------------------------------------- Chapter 1:1 Based on its own internal management reviews and audits by the NASA Inspector General, the agency initially acknowledged in 1987 that its procurement and contract management were vulnerable to waste and mismanagement. Since then, there have been a number of well-publicized problems with expensive space hardware, including the Hubble Space Telescope and an advanced series of weather satellites (GOES-Next). These problems have contributed to an increasing awareness outside of NASA of the agency's difficulties in managing its contracts and in buying hardware that works properly. This awareness has been a primary factor in mounting pressure from the Congress that NASA improve its procurement practices. Among other areas, the effectiveness of NASA's cost-plus-award-fee (CPAF) contracts in motivating contractors to control costs and to deliver quality products has been questioned. The Congress has also raised concerns about NASA contracts that require contractors to be paid for repairing defects caused by their own negligence. We have noted that NASA's procurement management problems are linked to the agency's overall program management process.\1 NASA recognizes this interdependence and, in recent years, has conducted many internal reviews to identify problems and develop methods for improving agency operations and programs. In early 1993, NASA announced a set of initiatives to improve both program and procurement management. Well before that, however, NASA had been working to improve its procurement activities. Over the past several years, NASA has had a total of 19 procurement improvement initiatives. Some of these have been completed; others have been added or dropped as management priorities changed based on updated information. Earlier initiatives included establishing a new Contract Management Division within the Office of Procurement at NASA headquarters, improving the training of procurement personnel, and streamlining grant processing. As of June 1993, NASA was working on eight procurement management improvement initiatives. Since then, two more have been added. Although these varied efforts are intended to improve and streamline NASA's procurement process, agency officials acknowledged that associated potential savings are not easily quantifiable or likely to be realized or significant in the near term. -------------------- \1 NASA Contract Management (GAO/HR-93-11, Dec. 1992). OBJECTIVES, SCOPE, AND METHODOLOGY ---------------------------------------------------------- Chapter 1:2 The Chairman, House Committee on Government Operations, requested that we review NASA's efforts to improve and streamline its procurement activities. Specifically, we were asked to (1) determine the status of NASA's procurement management improvement initiatives and (2) assess their planning and implementation. We reviewed the eight procurement initiatives NASA identified in June 1993. In response to another congressional request, we were already evaluating one of the initiatives.\2 Consequently, we focused most of our efforts on the seven remaining. To satisfy our objectives, we interviewed cognizant officials in the Offices of Procurement and Small and Disadvantaged Business Utilization at NASA headquarters, and we reviewed documents related to the initiatives, including current, proposed, and final policies; public comments NASA received on those initiatives published in the Federal Register; and center responses to some proposed policy changes. We also reviewed sections of the Federal Acquisition Regulation (FAR) and the NASA FAR Supplement, NASA handbooks and other guidance, NASA's annual procurement reports, reports required by the Federal Managers' Financial Integrity Act, NASA Inspector General and internal management review reports, and congressional hearings. During the course of our review, NASA announced two new initiatives. We did not include these in the scope of our review, but we did obtain preliminary information on them. A description of these initiatives is included in appendix I. In commenting on our draft report, NASA provided information on a new initiative on past contractor performance. This information is included in appendix II, where NASA's comments are reprinted in their entirety. We performed our work at NASA headquarters in Washington, D.C., from August 1993 through June 1994 in accordance with generally accepted government auditing standards. -------------------- \2 We subsequently reported on this initiative in NASA Procurement: Planning for Pilot Test of New Procurement Procedures Is Adequate (GAO/NSIAD-94-67, Nov. 4, 1993). NASA'S EFFORTS TO DEVELOP AND IMPLEMENT PROCUREMENT INITIATIVES ============================================================ Chapter 2 As of June 1993, NASA was working on eight key procurement management improvement initiatives; all but one have been partly or fully implemented. Some of the improvement efforts developed slowly, raised the concerns of industry and agency personnel, or failed to meet goals. On the other hand, others were developed and implemented quickly or achieved goals ahead of schedule. Table 2.1 provides a summary of information and comments on the eight initiatives discussed in chapters 2 and 3. Table 2.1 Overview of NASA's Eight Procurement Initiatives Problems/ Date Focus of concerns initiative Actions Actions Observations/ initiative addressed announced\a taken remaining comments ----------- ----------- ----------- ----------- ----------- --------------- Improve Excessive 3/92 Policy Complete Complete award fee fees implemented development restructuring contracting awarded, in October of of award fee little 1993. agencywide contracting. emphasis on course. Increases end Compliance emphasis on end results, to be results, product monitored product performance by performance, , and cost headquarter and cost control. s' reviews control. Inconsisten and t field procurement center management practices. surveys. Increase Contractors 6/93 Published Estimated Increases contractor reimbursed proposed implementat contractor liability for repairs policy in ion summer liability. when February 1994. Difficult to defects 1994. administer. were caused Could lead to by higher contract contractors prices and ' adversely negligence. affect small and disadvantaged businesses (SDB). Report on Limited 4/93 Designed Continue New system contractor high-level and regular providing performance communicati implemented reporting increased on. Time- agencywide cycles. communication consuming reporting with industry review of system in and simplified contract January review within status in 1993. NASA. Could nonstandard report on SDB format. high- technology work. Reduce Billions of 6/93 Implemented Continued Part of value of dollars in agencywide monitoring continuing unpriced unpriced policy in of value of effort. Recent contract change May 1994. unpriced effort resulted changes orders. changes at in significant Contractor centers. reduction in has limited value of incentive unpriced to control contract costs. changes. Strengthen Problems 6/93 Issued All COTRs New requirement training with COTRs' agencywide to be to ensure COTRs for understandi guidance in trained by adequately contracting ng of January April 1, trained. Does officers' roles, 1994. 1995. not ensure technical responsibil sufficiency of representat ities, and COTR training ives (COTR) scope of courses. authority. Guidance is Amount and incomplete. quality of COTR training inconsisten t. Achieve SDB Agency 3/92 Developed Increased Designed to goals directed to plan in emphasis on meet mandated award 8 1992 to high- contracting percent of achieve 8- technology goal. Achieved contract percent contracts. goal ahead of dollars to goal. schedule. SDBs. Developed strategies for achieving future goals. Communicate Issues 3/92 Established Continue Enables NASA to with hindering Industry dialogue consider and industry the Process with address effectivene Action Team industry on industry ss of in October procurement concerns on NASA's 1991. issues. procurement acquisition issues. process. Streamline Disproporti 3/92 Began pilot Expand Designed to midrange onate test of new pilot test reduce procurement amount of procedures to all NASA procurement procedures time and at Marshall centers in time and effort for Space near effort. procurement Flight future. Preliminary s in Center in Must obtain test results $25,000 to July 1993. waivers for show decrease $500,000 exclusive in average lead annual use of time for range. electronic awarding bulletin midrange board. contracts. -------------------------------------------------------------------------------- \a Indicates when NASA formally announced the initiative, not when the problem was first identified or work actually began. This chapter discusses the problems and issues addressed by the first seven initiatives and the status of efforts to develop and implement them. We discussed the planning for the last initiative in an earlier report; this chapter provides updated information on that effort. Chapter 3 discusses factors that may impede the initiatives' effectiveness, including problems with administration, potential adverse effects, and policy guidance. CHANGING AWARD FEE CONTRACTING ---------------------------------------------------------- Chapter 2:1 Over the years, reviews of CPAF contracting practices at NASA have identified a variety of serious weaknesses, including the award of excessive fees and limited emphasis on end results, product performance, and cost control. For example, in 1993, the NASA Inspector General reported that contract cost overruns were not adequately considered as part of the cost incentive evaluation criteria in determining fees, and contractors were paid millions of dollars in fees on contracts with hundreds of millions of dollars in cost overruns. Other issues addressed in recent internal reviews of CPAF contracting practices include (1) inconsistencies among NASA field centers in how they evaluate contractor performance and (2) the potential for requiring contractors to return fees previously awarded. These issues were similar to those raised a decade ago in earlier reviews of CPAF contracting at NASA. However, those earlier reviews did not lead to effective corrective actions. NASA began its award fee initiative in August 1991 but did not issue a new award fee contracting policy until more than 2 years later. One factor contributing to the slow development was the need to address concerns expressed by industry and by the NASA centers' program, technical, and procurement personnel. Due to industry's concerns, NASA rescinded an earlier proposed change to the policy only 5 days after it was published in the Federal Register. Industry concerns centered mainly on the proposed use of negative performance incentives and a prohibition on the use of base fees.\1 The use of negative performance incentives did not change in the final policy; the base fee prohibition was modified as discussed below. Many of the NASA centers' concerns were similar to those of contractors, but they were also concerned about the prohibition on rollovers (the practice of making a fee that was not earned in a prior period available to be earned in a subsequent period). That prohibition remained in the final version of the policy. The new award fee policy applies to all solicitations issued for CPAF contracts after October 8, 1993. The agency developed a handbook on the policy and is working on an agencywide training course. To assess compliance with the new policy, NASA headquarters officials and procurement management survey teams will review pre-award activities for selected cost-plus-award-fee contracts. Procurement survey teams will also assess center compliance with post-award provisions. However, such surveys are conducted infrequently and address a broad range of subjects with limited time and resources. Consequently, procurement management surveys may be limited in their ability to identify some compliance problems. The potential for noncompliance is an area of special concern. For example, the NASA Inspector General recently reported that a major procurement center, which expended $1.2 billion on award fee contracts in fiscal year 1993, was not consistently adhering to award fee policies, procedures, and guidelines. Our past reviews have also shown that a primary cause of compliance problems at centers was ineffective oversight by NASA headquarters. The new award fee policy emphasizes the importance of selecting the appropriate type of contract to reasonably allocate performance risk between the contractor and the government. In particular, the policy notes that CPAF contracts should be used only where there is substantial technical, cost, or schedule uncertainty and the evaluation of contractor performance must be subjective. Where uncertainties are fewer and risk is lower, the new policy suggests the use of other contract types, including cost-plus-incentive-fee, fixed-price-incentive, or firm-fixed-price (where the contractor assumes maximum risk). -------------------- \1 Base fees on cost-type contracts were paid regardless of contractor performance. AWARD FEE POLICY -------------------------------------------------------- Chapter 2:1.1 The award fee policy states that cost control will be emphasized in all CPAF contracts. When weighted evaluation factors are used, the assigned weight for cost control will be no less than 25 percent of the total award fee, excluding any base fee. The contractor must receive a rating of "very good" or "excellent" on technical factors before being rewarded for cost savings. The policy also prescribes a uniform and simplified award fee scoring system for the entire agency; establishes 6 months as a standard evaluation period; eliminates the use of rollovers; and emphasizes the timely payment of provisional, interim award fees. Under the previous award fee policy, base fee was allowed on all CPAF contracts. NASA's new policy eliminates base fee on service contracts, but allows its use on other contracts, such as those for study, design, or hardware, provided contractor performance is at least "satisfactory" for the entire contract. Another key element of the new award fee policy is increased emphasis on overall contractor performance and the end product, rather than on interim progress. For example, on contracts for study, design, or hardware, where the quality of contractor performance cannot be measured until the end of the contract, the award fee earned is linked to a final, comprehensive rating. The policy also requires the use of both positive and negative performance incentives on all CPAF contracts when the total estimated cost and fee is greater than $25 million and the primary deliverable is hardware. Performance incentives are based on measurements of hardware performance against objective criteria. They are separate and distinct from award fee, which is subjective and measures contractor performance up to delivery and acceptance.\2 The new policy stipulates that at least one-third of the total potential contract fees must be placed in the performance incentive pool. The contractor earns no performance incentive when the hardware performs at the minimum acceptable level established in the contract, since this performance level is covered by award fee payments. Performance incentives reward contractors when delivered hardware performs above minimum contract requirements. Conversely, the contractor is penalized if the product does not meet such requirements. In the event of a total, immediate, and permanent hardware failure, the contractor is to return an amount equal to the entire earned award fee, including any base fee. In certain circumstances, the payback amount would be based on the maximum potential award fee the contractor could earn, not the amount actually earned. Appendix III describes how a performance incentive might be structured. -------------------- \2 Unlike award fee decisions, both negative and positive performance incentive decisions can be legally disputed. PROJECTED EFFECT OF AWARD FEE CHANGES -------------------------------------------------------- Chapter 2:1.2 NASA procurement officials believe that the new award fee policy will encourage contractors to deliver quality products and services at a reasonable cost and in a timely manner. However, they noted, it will be several years before they will be able to fully assess the effectiveness of award fee contracting changes. The new policy is based on overall contract performance, rather than on interim progress. Consequently, officials said, they will be able to gauge definitive, long-term results only after contract completion. Analysis of contract costs will be critical at this juncture because, according to a NASA procurement official, the award fee policy may not be entirely effective in ensuring that contractors bid realistically on contracts. The impact of the new performance incentive requirement will be limited initially because, in fiscal year 1993, NASA awarded only two contracts that would have been required to incorporate performance incentives had the policy been in effect. A NASA procurement official believes that, although there is no requirement to do so, performance incentives will also be used on some hardware contracts with estimated cost and fee under $25 million. Procurement officers must approve the use of performance incentives on such contracts after determining that expected benefits will outweigh the associated administrative burden. NASA was not able to estimate the likely frequency of the use of incentives on CPAF contracts under the $25-million threshold. While the effect of the performance incentive feature may be limited, the award fee contracting policy should have a substantial impact on agency operations because the percentage of contract dollars that NASA awards to business firms through CPAF contracts has been increasing for some years, as illustrated in figure 2.1. In fiscal year 1986, CPAF awards represented only about 43 percent of the total awards to business firms. By fiscal year 1993, contracts and modifications to contracts having CPAF provisions accounted for 76 percent, or $7.8 billion, of the total value of awards to business firms, as shown in figure 2.2. This increase was primarily because of the award of a few very large CPAF contracts, including those for the space station and solid rocket motors for the space shuttle. Figure 2.1: Eight-Year Trend in the Percentage of Dollars Awarded Annually to Business Firms for Cost-Plus-Award-Fee Contracts (See figure in printed edition.) Note: Excludes smaller procurements, generally those of $25,000 or less. Figure 2.2: Percentage of Dollars Directly Awarded to Business Firms by Contract Type for Fiscal Year 1993 (See figure in printed edition.) Note: Excludes smaller procurements, generally those of $25,000 or less and orders under GSA Federal Supply Schedule contracts. NASA uses CPAF contracts more than any other government organization does. For example, in fiscal year 1993, the Department of Defense used contracts and modifications to contracts having CPAF provisions for less than 8 percent of the total value of its procurements, while NASA used CPAF provisions for more than three-quarters of the total value of such awards to businesses. NASA uses CPAF contracts for operations and maintenance activities at its field centers, as well as for procuring design, development, and initial fabrication of state-of-the-art hardware, such as a new type of planetary probe. According to NASA officials, NASA managers prefer CPAF contracts because they give them more control and flexibility. Under CPAF contracts, contractors' profits are linked to NASA's evaluations of contractor performance, and NASA can adjust the evaluation criteria periodically to reflect changes in management emphasis or concern. INCREASING CONTRACTOR LIABILITY ---------------------------------------------------------- Chapter 2:2 Like the award fee contracting initiative, the effort to increase contractor liability developed slowly, partly because a number of individuals and organizations raised concerns about the use and impact of the proposed policy. NASA developed this policy primarily to address congressional concerns about NASA's payments to contractors for the costs of repairing or replacing articles when the defects were caused by contractors' negligence. Currently, contractors' liability for the cost to correct failures to comply with requirements under cost-type, research and development contracts is limited. Federal agencies, including NASA, currently assume total responsibility for the cost of correcting defects under such contracts, except in cases of fraud, willful misconduct, and lack of good faith by contractors' senior management. In November 1992, the Congress directed NASA to review options and develop a policy for sharing risk with research and development contractors.\3 In June 1993, NASA reported the results of its assessment and, in February 1994, published for public comment a proposed policy that may be modified before it is implemented. The proposed policy would increase contractor liability for correcting defects in material or workmanship or other failures to conform to requirements of high-dollar value, research and development, cost-type contracts. If implemented after a public comment period that ended on April 18, 1994, the policy probably will become effective in summer 1994, according to a NASA procurement official. The proposed policy adds two situations under which NASA could impose on contractors additional responsibility to remedy failures to comply with contract requirements. Specifically, a contractor would be financially liable for correction or replacement if NASA determined that the contractor (1) had not applied its best efforts toward the research and development objectives of the contract or (2) had not followed proper procedures conforming to generally accepted practices in performing routine operations, such as moving equipment or conducting standardized tests. In these instances, NASA could hold the contractor liable for 50 percent of the cost for remedying each failure or 10 percent of the contract value at the time the failure occurred, whichever is less.\4 NASA developed its contractor liability policy slowly for two main reasons. First, NASA had focused its efforts primarily on developing a new approach to award fee contracting because the agency anticipated that award fee changes would be sufficient to deal with hardware performance concerns and a separate contractor liability policy would not be needed. Second, industry, the Department of Defense, the General Service Administration, and others expressed concerns about the initiative. In March 1993, NASA published preliminary information about proposed changes in the Federal Register. Comments on the proposed contractor liability policy, as well as those from NASA personnel during the policy development process, focused on a variety of issues, including concerns that contractors would focus on reducing risk by avoiding technically risky, cutting-edge projects and solutions; NASA's contracts applying this policy would be less attractive to potential offerors, thus reducing competition; increased allocation of risk to the contractor is inconsistent with the use of cost-type contracts and would, in effect, convert cost-type contracts into fixed-price contracts; and use of the negative incentive fee provision in the new award fee policy concurrently with the proposed contractor liability policy would be tantamount to penalizing the contractor twice should a product fail to perform. A NASA procurement official said he does not believe that contractors will avoid high-risk projects or decline to bid for contracts if the proposed policy is implemented. He noted that the proposed policy explicitly exempts failures in high-risk projects that occur despite the contractor's best efforts. The official also noted that contractors will probably be willing to compete and assume more risk because there are relatively few large dollar contracting opportunities at NASA. In any event, NASA's judgment to cite a contractor for not applying its best effort or for not following proper procedures can be challenged by the contractor. The new contractor liability rule would apply only to hardware contracts with an estimated award value of $50 million or more. In fiscal year 1993, NASA awarded only one such contract. Only one additional contract awarded in fiscal year 1993 would have been covered by the proposed policy if the threshold were lowered to $25 million or more. -------------------- \3 Section 401 of the National Aeronautics and Space Administration Authorization Act for Fiscal Year 1993 (P.L. 102-588) directed the agency to review its contracting procedures. \4 Ten percent is the maximum liability for the entire contract, not for each incident. These limits were not intended to bear a true relationship to the damages suffered by NASA, according to NASA officials. REPORTING ON CONTRACTOR PERFORMANCE ---------------------------------------------------------- Chapter 2:3 In January 1993, 6 months after the NASA Administrator directed that it be developed, the agency implemented a system of contractor metrics, a set of standardized data for reporting specific measures of contractor performance. NASA designed this agencywide system to meet two recognized needs. Specifically, its purpose is to (1) establish an ongoing dialogue between upper-level NASA and corporate management and (2) simplify and standardize the information given to high-level NASA officials on active contracts. Previously, NASA had communicated with its contractors mainly at lower levels, and high-level NASA officials, in a time-consuming process, reviewed large amounts of data presented in a variety of formats. NASA officials also believe that the metrics system may help management identify troubled programs that need additional review or should be considered for cancellation because of cost, schedule, or technical problems. Implementation began in January 1993, and the present system may be expanded or modified. Currently, the metrics system reports on approximately 30 NASA contracts and uses data already routinely collected on all NASA contracts with an estimated value greater than $25 million. Contracts for metrics reporting are selected annually and are usually those with substantial near-term funding requirements. In fiscal year 1993, the metrics system was used to report on the performance of 14 contractors with 30 contracts valued at $5.9 billion, or about 60 percent of NASA's fiscal year 1993 funding to business firms. The contractor metrics system addresses seven key categories. Trend information is reported in the following four areas: cost, schedule, subcontracting plan, and award fee. Information includes the extent to which the contractor has met cost, schedule, and subcontracting plans, and the percentage of available award fee paid. A sample cost metric for a service contract is provided in appendix IV. There is no single measure, or set of measures, used on contracts for two other performance standards--technical and safety and mission-assured performance. These metrics are uniquely tailored to each contract and may change as the program evolves. The technical metric could focus on a number of different areas, including compliance with weight specifications or payload status. The safety and mission-assured performance metric could report on such items as defects per unit. These six metrics are reported on a contract-by-contract basis. The final metric--continual improvement--is a voluntary metric reported by contractors that addresses improvements in their overall business practices. Nine of 14 contractors described their continual improvement activities in the most recent reporting cycle. The seven metrics, together with contract status summary information and an overall assessment by the NASA project manager, is reported semiannually to each contractor's chief executive officer. For comparison purposes, these reports to contractors also include unlabeled information on other contracts' metrics, including data on estimated cost growth at contract completion. In March 1994, these estimates ranged from about minus 20 percent to about 270 percent and averaged 50 percent. NASA has issued reports to contractors twice--first in July 1993 and again in March 1994. The status summaries and project manager assessments and two metric categories--cost and schedule--are updated and reviewed quarterly within NASA. REDUCING THE VALUE OF UNPRICED CONTRACT CHANGES ---------------------------------------------------------- Chapter 2:4 In an effort to manage contract changes more effectively and to help control cost growth, NASA has taken several steps over a 3-year period to reduce the number and value of the agency's unpriced contract changes and to price them in a more timely manner. Although earlier efforts have not generally brought about lasting improvements, recent actions have resulted in a significant reduction in the value of unpriced contract changes. Government contracts contain a clause that allows (1) contracting officers to make certain changes within the general scope of the contract and (2) contractors to start work and incur costs before agreeing with the contracting agency on terms and conditions of the change, including price. However, issuing unpriced contract changes should not be standard operating procedure, and, when they are used, they should be priced as soon as practicable. Until firm prices are negotiated for contract changes, the contractor has limited incentive to control costs. Also, if work is completed before pricing the change, the government will not have had the opportunity to review proposed costs and consider more efficient production methods or management controls. NASA designated the effort to reduce the value of unpriced contract changes as a formal procurement initiative in June 1993, and in May 1994, NASA implemented an agencywide policy on pricing such changes. To monitor the effectiveness of this policy, the agency will continue to gather data on the number, value, and age of unpriced change orders at field centers. NASA's new policy requires that unpriced change orders be issued on a strict exception basis; center directors approve unpriced change orders with an estimated value of more than $1 million; and the cognizant Deputy Associate Administrator approve unpriced change orders for the space station with an estimated value of more than $10 million.\5 However, approval requirements may be waived in certain instances, including changes that involve safety issues or decreases in contract value. NASA's current initiative also focuses on improving the change order management process, including the need to develop a clear and complete technical definition of changes, as well as realistic cost estimates. Before agencywide implementation, the new policy had been in effect since October 1993 at the four NASA centers that accounted for about 90 percent of the value of unpriced change orders. During a 9-month period, from October 1, 1993, when the policy became effective, through June 30, 1994, the value of unpriced change orders at these four centers dropped from almost $5 billion to about $1.8 billion,\6 a 63-percent reduction. Also during this 9-month period, the value of unpriced change orders over 180 days old dropped by 87 percent at the four centers. In 1992 and 1993, the two centers with the largest backlogs of older, unpriced change orders reviewed and revised center policies to improve pricing timeliness. These actions, however, did not lead to a lasting reduction in the value of change orders unpriced for more than 180 days.\7 As of September 30, 1993, there were about $3.4 billion in change orders unpriced for more than 180 days at these two centers, an increase of about $500,000 from the previous year. The value of such unpriced change orders has decreased since October 1, 1993, when NASA's new policy took effect at four centers, as discussed above. Earlier agencywide efforts to reduce unpriced change orders began in late 1990, when NASA headquarters increased oversight and reporting requirements for unpriced contract changes. This effort significantly reduced the number of unpriced contract changes outstanding for extremely long periods of time. For example, in 1991, the value of unpriced contract changes over 2 years old dropped from more than $3 billion to just over $131 million in a 6-month period. NASA, however, did not meet other goals established in October 1991, more than a year and a half before the agency formally announced its initiative to reduce the value of unpriced contract changes. For example, NASA had wanted to eliminate all unpriced changes over 540 days old by the end of March 1992. But, by that date, 17 changes, with an estimated total value of more than $114 million, were over 540 days old. The average age of these unpriced changes was 750 days, and one change was more than 1,300 days old. NASA also wanted to eliminate all unpriced changes over 360 days old by the end of September 1992. As of September 30, 1993, 1 year past the target date, 69 changes, with an estimated total value of $1.9 billion, were over 360 days old. A NASA official noted that increasing the awareness of risk is one major requirement for success in efforts to achieve a long-term, permanent reduction in the value of unpriced change orders. Specifically, he said, NASA must "sensitize people" to the pitfalls of issuing such changes under cost-type contracts. This official believes that NASA has begun to sensitize both procurement and program office personnel to the inherent risk of unpriced contract changes through various means, including offering formal training classes and using employee teams to develop methods to streamline the pricing process. -------------------- \5 A NASA official said the space station program approval level and authority is different because managers monitor these changes very closely, and the value of such changes is likely to be large because of the relative size of the program. \6 About 75 percent of the total $1.8 billion in unpriced change orders was due to changes that reduced the scope and value of two contracts. \7 NASA officials generally use the 180-day period as a guideline for completing negotiations on unpriced contract changes. STRENGTHENING TRAINING FOR CONTRACTING OFFICERS' TECHNICAL REPRESENTATIVES ---------------------------------------------------------- Chapter 2:5 Contracting officers appoint contracting officers' technical representatives (COTRs) to assist them in managing the technical aspects of contracts. NASA's efforts to strengthen training for COTRs stemmed directly from evaluations demonstrating the need for, or recommending changes in, such training. These evaluations identified problems with COTRs' understanding of their roles, responsibilities, and scope of authority; noted instances where COTRs' actions eroded the authority of contracting officers; and cited cases where unauthorized technical representatives directed contractors to work beyond the contracts' original requirements and inappropriately issued task orders. The amount and quality of training for COTRs have been inconsistent throughout NASA. NASA officials acknowledge that this may have been a contributing factor to significant deficiencies in the management of prime and subcontracts. In 1992, we recommended that the NASA Administrator establish and enforce minimum training requirements for technical representatives--requirements that emphasize COTRs' roles and responsibilities, scope of authority, and relationship to other members of the procurement management team.\8 NASA identified COTR training as an agencywide initiative in June 1993, and, in January 1994, it issued agencywide guidance that specifies mandatory COTR training areas, including contracting authority, contract modifications, surveillance plans, and government property. The guidance makes procurement officers at each center responsible for ensuring that the COTR training provided addresses these areas in sufficient detail. All COTRs must be fully trained in these required areas by April 1, 1995. After that date, contracting officers will be required to verify that technical personnel have been trained in the mandatory topics before appointing them as COTRs. The new policy does not require formal training for those COTRs who contracting officers believe have met the requirement through previous training and experience. -------------------- \8 NASA Procurement: Agencywide Actions Needed to Improve Management of Contract Modifications (GAO/NSIAD-92-87, Mar. 2, 1992). ACHIEVING SMALL AND DISADVANTAGED BUSINESS GOALS ---------------------------------------------------------- Chapter 2:6 NASA's efforts to increase small and disadvantaged business (SDB) contracting have helped the agency exceed established goals 1 year ahead of schedule. In 1989, the Congress directed NASA to establish and implement a plan to award at least 8 percent of its contract dollars to SDBs. At that time, NASA was doing about 5.5 percent of its business with SDBs. As currently defined, SDBs include small business concerns or other organizations owned or controlled by minorities, women, historically black colleges and universities, and educational institutions serving other minorities.\9 In 1990, NASA set the end of fiscal year 1994 as the target date for meeting its congressionally mandated 8-percent goal. However, by the end of fiscal year 1993, 1 year earlier than planned, NASA exceeded its goal by awarding almost $1 billion, or 8.5 percent of its total awards to business, to SDBs. This amount included about $443 million awarded to prime contractors and about $557 million awarded to subcontractors. The NASA Associate Administrator, Office of Small and Disadvantaged Business Utilization, projects that the agency will meet its SDB contracting goal again in fiscal year 1994. Pivotal to NASA's efforts to achieve its SDB contracting goal was a 1992 plan that included provisions for rating upper-level agency managers on their performance in meeting socioeconomic goals; directing high-level managers to report on actions to increase SDB subcontracting on NASA's top 100 contracts; requiring headquarters to review and approve of contract consolidations--combining several small contracts into one large contract for administrative efficiency--that might reduce prime awards to SDBs; and setting aside 26 contracts with an estimated value of more than $300 million for SDBs. (As of June 24, 1994, NASA had awarded 16 of these contracts--an estimated total value of $97.5 million.) NASA's future plans call for increasing the value of SDB contracts in high-technology areas, an area of interest to both the Congress and the NASA Administrator. NASA has taken several steps to achieve this goal, including establishing a committee to advise the agency on increasing awards to SDBs, with emphasis on high-technology areas. The agency has developed a SDB/aeronautics forum to provide SDBs with high-technology capabilities an opportunity to brief center technical personnel about their qualifications and experience. In addition, by October 1994, NASA plans to implement a mentor/protege program to encourage NASA contractors to assist SDBs in enhancing their business and technical capabilities and to increase their participation on NASA contracts. -------------------- \9 These minorities include Hispanic and Asian Americans, and native Americans, Alaskans, and Hawaiians. COMMUNICATING WITH INDUSTRY ---------------------------------------------------------- Chapter 2:7 In October 1991, NASA established the NASA/Industry Process Action Team to identify and discuss procurement-related issues that hinder the effectiveness and efficiency of NASA's acquisition process. The team currently has 31 members, including NASA contractors, a law firm, and NASA representatives. Membership is rotated each year among interested industry organizations through a random selection process. The team has representatives from all segments of NASA's contractor community, including both large and small NASA contractors and SDBs. Meetings are held about 8 to 10 times per year, or as often as necessary to discuss major procurement issues and initiatives. Team members present their individual or organizational ideas and comments. The team is not permitted to vote on the issues presented or to form a consensus on proposed changes. NASA briefs the team, which, in turn, provides input on many of its procurement improvement initiatives, including contractor metrics, contractor liability, award fee policy, and cooperative agreements. The team provides NASA policy representatives with an opportunity to learn about various industry perspectives before developing draft policy that will be available for public comment through the Federal Register. NASA is not obligated to accept any of the viewpoints offered. The initial team assisted in the development and presentation of a NASA/industry conference in Houston, Texas, in March 1992. During this conference, the team cataloged 61 areas of possible improvement, and NASA has completed action on all but 10 of these. In some cases, after evaluation, NASA decided not to implement the team's suggestions. However, NASA acted on many, including those related to small business and SDB goals, early industry involvement in NASA procurements, more timely proposal evaluation and source selection, subcontract management, and change orders. One issue that was originally closed--termination liability--was subsequently reopened when industry members wanted to further discuss the financial risk to industry. Following the team discussions, NASA drafted an amendment to the NASA FAR Supplement, establishing a mechanism for funding termination liability on cost-type contracts. This change addressed the major concerns of team members, while providing NASA headquarters personnel with a better estimate of potential liability. PROGRESS OF PILOT TEST ON STREAMLINING MIDRANGE PROCUREMENT PROCEDURES ---------------------------------------------------------- Chapter 2:8 NASA's new midrange procurement procedures were designed to reduce procurement time and effort without adversely affecting quality. As currently designed, each midrange acquisition of supplies and services is reserved exclusively for small business concerns. In fiscal year 1993, NASA awarded 1,863 new contracts, of which 1,579, or approximately 85 percent, were midrange procurements with annual values between $25,000 and $500,000 and total 5-year values up to $2.5 million. In July 1993, NASA began a pilot test of midrange procurement procedures at Marshall Space Flight Center. By the end of March 1994, Marshall had awarded 72 contracts, all for Small Business Innovative Research (SBIR), under the pilot test. Preliminary results show a reduction in average lead time for awarding such contracts. A NASA official noted that this reduction is especially significant. Since Marshall already had the lowest SBIR lead time in the agency, lead time reductions are likely to be achievable at other centers. A NASA official noted that the introduction of an electronic commerce system (a computer bulletin board) as the sole means of providing information about procurements is also likely to reduce average lead time. The exclusive use of an electronic commerce system for expediting communication with prospective offerors in announcing and awarding midrange procurements has not yet been approved. Currently, all proposed procurement actions over $25,000 must be published in the Commerce Business Daily.\10 This process, including required response waiting times, is slow. The agency cannot yet gauge the ultimate impact of midrange procurement procedures on the business community because only limited data are available. However, NASA has not received any unfavorable comments from offerors, and no formal bid protests have been filed. Also, a NASA official noted Marshall personnel are enthusiastic about the midrange procurement procedures, and other centers have asked headquarters to expand the pilot. NASA expects to begin implementation at Johnson Space Center and at its headquarters' Acquisition Division in the summer of 1994. -------------------- \10 NASA must obtain waivers before it can use an electronic bulletin board as the sole means of communicating requirements and providing copies of solicitations. PROBLEMS IN PLANNING AND IMPLEMENTING INITIATIVES ============================================================ Chapter 3 The effectiveness of some of NASA's procurement improvement initiatives may be limited. Specifically, (1) the proposed contractor liability policy will be difficult to administer, may result in higher contract prices, and could adversely affect subcontractors who are unwilling or unable to risk increased liability; (2) the potential effect of the proposed contractor liability policy on SDBs has not been evaluated; and (3) the policy guidance for the training of COTRs is incomplete. POTENTIAL PROBLEMS WITH PROPOSED CONTRACTOR LIABILITY POLICY ---------------------------------------------------------- Chapter 3:1 The concept behind NASA's proposed contractor liability policy appears to be a simple one--contractors working under cost-type, research and development contracts should bear additional financial responsibility for failures to comply with contract requirements. The proposed policy, if properly implemented, may improve NASA procurements under some high-dollar value contracts. However, this policy will be difficult to administer, may lead to higher contract prices, and could affect subcontractors with limited capability to risk increased liability. The proposed policy increases contractors' liability for the cost of replacement or correction of defects. A NASA official said he assumes that NASA would apply this policy even in cases where the agency does not intend to replace or repair a faulty item. This option is not addressed, however, either in the existing contract clause or in the proposed policy. Consequently, NASA's decision to assess liability in cases when the agency does not intend to replace or repair an item could be disputed by the contractor. The proposed clause also requires that NASA determine liability for the cost of replacement or correction of defects. NASA's determinations of contractor liability for the cost of replacement or correction of defects requires applying general concepts that are yet to be precisely defined, such as "best efforts," "routine operations," and "generally accepted industrial or engineering practices." Even determining "failures to comply with the requirements of the contract" could be difficult, especially in a cost-type, research and development environment, where initial requirements are frequently broadly stated. NASA's decision to assess liability can be disputed by the contractor and could result in protracted negotiations and potentially costly litigation. A NASA procurement official acknowledged that the proposed clause will be difficult to administer and will probably lead to an increasingly adversarial relationship between NASA and its contractors. He noted, however, that NASA has tried to address some concerns. For example, the policy reduces the potential for disputes and litigation by requiring the use of advance agreements to identify routine operations and research and development activities and by placing the burden of proof on the agency for determining liability.\1 Although contractors could not claim additional costs under the contract to cover their increased risk of liability, contract prices could increase if contractors request higher award fees to compensate for this increased risk. Moreover, their costs could be higher if they choose to increase their supervision and internal review activities and/or develop and implement stricter standards and procedures to reduce the chances of a partial or complete failure. A NASA procurement official acknowledged that individual contracts may cost more as a result of the contractor liability policy but noted that NASA hopes ultimately to save money by avoiding catastrophic losses. As previously noted, in fiscal year 1993, NASA awarded only one contract that would have been subject to the proposed contractor liability provision had it been in effect. Furthermore, in view of NASA's constrained budget situation, it is unlikely that many such contracts--cost-type, research and development contracts for hardware with an estimated value of $50 million or more--will be awarded in the foreseeable future. Therefore, major aerospace companies and smaller subcontractors probably will not be immediately affected by the policy. In the future, however, even with a limited number of prime contractors, larger numbers of subcontractors, including SDBs and universities, could be affected if prime contractors pass on a share of the potential liability through "flow-down" provisions in their contracts. Subcontractor liability under such provisions would be based on the value of the subcontract and apply to the activities performed by the subcontractor. Comments in response to the proposed policy published in the Federal Register noted that it could affect the ability or willingness of subcontractors who are SDBs or universities to work on NASA contracts because of the potentially significant risk of liability that could, in some cases, undermine their economic viability. -------------------- \1 An earlier draft of the proposed policy required that the contractor prove that it was not liable. EFFECT ON SDBS NOT THOROUGHLY EVALUATED ---------------------------------------------------------- Chapter 3:2 NASA developed and implemented its contractor liability and SDB initiatives independently. Consequently, the agency has not yet evaluated the potential effects the proposed contractor liability policy may have on SDBs or on NASA's ability to achieve future SDB goals. As previously noted, increasing financial liability for contractors could affect the ability of SDBs with limited resources to work on NASA subcontracts. In addition, liability for hardware that fails to perform properly or for other failures to comply with contractual requirements could affect large contractors' willingness to subcontract work to SDBs, particularly in high-technology areas. An official in the Office of Small and Disadvantaged Business Utilization at NASA headquarters acknowledged that the agency has not fully considered the potential impact of the proposed contractor liability policy on subcontractors, including SDBs. He said the proposed contractor liability policy may make it more difficult for some SDBs to subcontract with NASA and noted that some prime contractors are already hesitant to do business with SDBs, especially those working in high-technology areas. In June 1993, a report by NASA's Minority Business Resources Advisory Committee\2 also raised concerns about the willingness of prime contractors to subcontract high-technology work to SDBs and noted that some contractors classify such subcontracting as "too risky" to consider. Even though NASA has coordinated initiatives in some instances, there are further opportunities for enhancing coordination. For example, NASA revised its award fee metrics to reflect changes in the new award fee process. The contractor metrics system also supports the SDB initiative by presenting information on contractors' progress in meeting SDB goals. However, contractor metrics could provide additional support for the SDB goal of increasing subcontracting opportunities for SDBs in high-technology areas. The contractor metrics system does not currently report information on the percentage of SDB subcontracting work that could be classified as "high-technology." -------------------- \2 The committee was established by the NASA Administrator. It includes representatives from SDBs and academia. GUIDANCE FOR COTR TRAINING INCOMPLETE ---------------------------------------------------------- Chapter 3:3 NASA has issued incomplete guidance on a new policy that is intended to strengthen COTR training requirements. This guidance identifies mandatory training areas and specifies that COTRs must be fully trained in these areas by April 1, 1995. A NASA procurement official told us that new training would not be required for those technical personnel who are judged to already have sufficient knowledge of the mandatory areas as a result of previous training and experience. Contracting officers are responsible for determining if technical personnel need additional training before appointing them as COTRs, he noted. The training policy guidance, however, does not provide any information on available options for fulfilling mandatory training requirements and on establishing and applying criteria to determine when no additional training is needed. The lack of guidance in these areas is likely to lead to inconsistent agencywide application of the new policy. In commenting on the proposed new policy, several centers expressed concerns that limited funding could make it difficult to meet the new COTR training requirements. Constrained training budgets and heavy competing demands do not bode well for the strict, uniform application of the new policy from center to center. Additional guidance would help ensure more consistency in interpreting and applying the new COTR training requirements. CONCLUSIONS, RECOMMENDATIONS, AND AGENCY COMMENTS ============================================================ Chapter 4 CONCLUSIONS ---------------------------------------------------------- Chapter 4:1 For 7 successive years, NASA has acknowledged that its procurement system and contract management had serious internal control deficiencies; over the years, the agency has designed, developed, and, in many cases, implemented a variety of improvement initiatives to address these and other problems. The initiatives we reviewed address a variety of problems, some of which are long standing, persistent, and complex. Correcting them will be a demanding process, and much hard work remains in order to achieve effective, long-term results. NASA has made uneven progress in planning and implementing its procurement improvement efforts. The agency's cycle of successes and failures in its efforts to reduce the value of its unpriced change orders provides a cogent example of why prolonged, intensive effort is frequently needed to correct persistent problems. Progress on some initiatives has been slow for several reasons, including centers' and contractors' objections to changing the traditional approaches to motivating and rewarding contractors and allocating risk. NASA has dealt with such resistance by soliciting center and industry comments and making changes to address some of their concerns as it developed new policies. Some of NASA's procurement improvement initiatives have the inherent potential to create a more adversarial relationship between NASA and its contractors or have other unintended, potentially adverse side effects. Key among these are the potential effects of increasing contractors' liability on SDBs, especially those qualified to do high-technology work--a focus of NASA's SDB initiative. Where such potential effects are identified, NASA may need to consider options for mitigating them. Such options should not involve lowering applicable standards or exempting SDBs from potential liability. Instead, options could include offering further incentives through the award fees to prime contractors that subcontract with SDBs, especially in high-technology areas; providing advice and assistance to SDBs, if needed, through training and mentoring activities; and redoubling ongoing efforts to acquaint center personnel with SDBs with high-technology capabilities. To further support its efforts, NASA could adjust its SDB metric category to present information on the percentage of high-technology work subcontracted to SDBs. To do this, NASA needs to establish a standard definition of high-technology work and to develop official statistics on the value of such work awarded to SDBs, either as prime contractors or subcontractors. Using its metrics reporting system would enable NASA to effectively identify and track its progress toward involving SDBs in high-technology work for critical contracts that account for more than half of the agency's funding to business firms. NASA's procurement reform initiatives are a vital part of its overall efforts to improve operations. The problems or difficulties the agency has had in planning and implementing its initiatives must not be confused with failure. They do, however, demonstrate the need for sustained management attention, routine monitoring, and effective followup. Particularly critical are NASA's plans for monitoring centers' compliance with the new award fee policy because of the nature and scope of revisions, their impact on agency operations, the difficulty of implementing changes affecting the agency's traditional ways of conducting business, and a history of compliance problems at field centers. Close monitoring is needed to ensure that center personnel have properly interpreted and are adequately complying with new rules. Continuing commitment to improving procurement activities is essential because correcting problems is critical to safeguarding increasingly scarce resources and ensuring their efficient and effective use on behalf of the government. Without effective procurement and contract management, NASA cannot reasonably ensure that billions of dollars in contract funding will be spent effectively and that the products it purchases will function properly. RECOMMENDATIONS ---------------------------------------------------------- Chapter 4:2 We recommend that the NASA Administrator evaluate the potential impact of the proposed contractor liability policy on SDBs and consider options to mitigate any adverse impacts identified; define "high-technology" work and develop and implement a methodology for consistently measuring progress toward NASA's goal of increasing SDB participation in such work, either as prime contractors or subcontractors; direct that special reviews be conducted at centers to ensure that training courses for COTRs sufficiently address new minimum training requirements; and provide guidance to the centers on establishing and applying criteria to determine when no additional training is needed to meet new minimum training requirements for COTRs. AGENCY COMMENTS ---------------------------------------------------------- Chapter 4:3 NASA agreed with our recommendations and generally agreed with the report's contents. NASA provided comments on our analyses of some of the initiatives. We considered these comments in preparing this report. NASA's comments are reprinted in their entirety in appendix II. NEW INITIATIVES: PROCUREMENT REINVENTION LABORATORY AND COOPERATIVE AGREEMENTS =========================================================== Appendix I Since our review began, NASA has started two new procurement improvement initiatives. In October 1993, NASA established a procurement reinvention laboratory to identify and implement innovative methods for acquiring goods and services. The initiative resulted from, and operates under the auspices of, the National Performance Review. The headquarters' Acquisition Division, which accounts for about 6 percent of the total value of the agency's acquisitions, was designated as the reinvention laboratory. The Associate Administrator for Procurement has waived the requirement for the laboratory to comply with the NASA Supplement to the Federal Acquisition Regulation. The laboratory also has requested or has received waivers from other agencies' internal procedures that affect NASA's procurement process. NASA's laboratory is currently involved in acquiring technology for small spacecraft and is using this procurement to develop and test strategies for focusing on reducing procurement time, streamlining internal reviews, and motivating contractor performance. Although the laboratory was originally expected to operate for 1 year, NASA officials are considering extending the duration to 2 years. At that time, NASA will evaluate the results and decide if the initiative will be continued or expanded. In February 1994, NASA identified cooperative agreements with profit-making organizations as a procurement improvement initiative. NASA expects to use these agreements to increase technology transfer to industry. Although NASA has used cooperative agreements with universities and nonprofit organizations in the past, cooperative agreements between NASA and members of the private sector is a new concept. Cooperative agreement notices will be published in the Commerce Business Daily to solicit industry interest in a particular effort. NASA will also consider unsolicited proposals from industry. Although NASA prefers working with teams of representatives from different companies, an agreement with a single firm will be considered. NASA has developed procedures for structuring cooperative agreements and for monitoring progress--both technical and financial. While cooperative agreements will not replace contracts, NASA believes that such agreements will foster a closer working relationship between government and private industry and will assist for-profit firms in advancing and commercializing technologies in which the government has unique capabilities. (See figure in printed edition.)Appendix II COMMENTS FROM THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION =========================================================== Appendix I (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) The following are GAO's comments on the NASA's letter dated July 21, 1994. GAO COMMENTS --------------------------------------------------------- Appendix I:1 1. We discuss a variety of factors that demonstrated the need for change and led NASA to develop and implement procurement improvement efforts and initiatives. Other than NASA Inspector General and GAO reports, factors mentioned include performance problems, assessments by NASA management, and direction from the Congress and the NASA Administrator. 2. The results of the change order initiative are not diminished by the presentation. Rather, the presentation provides a historical perspective on the difficulty of correcting this long-standing problem. Moreover, we note in the first paragraph of the text on the change order initiative that "recent actions have resulted in a significant reduction in the value of unpriced contract changes." We also provide details of NASA's current initiative and note the 63-percent reduction over a 9-month period at four centers. PERFORMANCE INCENTIVES IN NASA'S NEW AWARD FEE POLICY ========================================================= Appendix III NASA provided the following example to illustrate how a performance incentive might be structured under the agency's new award fee policy guidance: In a contract requiring the delivery of a spacecraft, the performance incentive unit of measurement is useful months in orbit. If 12 months is the expected performance level, the 12th month could be identified as the standard performance for which no incentive is earned. If 24 months is the maximum useful life of the spacecraft, the 24th month could be identified as the maximum performance level, at which the contractor would earn the maximum performance incentive. Interim measures of spacecraft life from 12 to 24 months would then be identified with fees ranging from $0 to the maximum positive incentive. The amounts associated with these interim measures should correspond to the relative value to the government of each additional month in orbit. NASA will not reward the contractor for above standard performance that does not benefit the government. A similar scale would be established for the negative incentive, ranging from the 12th month for standard performance, for which no penalty would be due, to a total, immediate, and permanent system failure, for which the maximum negative performance incentive would be due from the contractor. The amount of the maximum negative incentive would depend on the type of hardware provided under the contract. For research and development efforts (that is, the first and second units produced), the maximum negative performance incentive would be equal to the total amount of award fee actually paid. For production efforts (that is, the third and all subsequent units of any hardware item), the maximum negative performance incentive would be equal to the total amount of award fee the contractor could have earned. In other words, if there were a total, complete, and permanent failure of a production item, the contractor could experience a net loss on the contract. SAMPLE METRIC: CONTRACT BASELINE COST TREND ANALYSIS FOR A SERVICE CONTRACT ========================================================== Appendix IV (See figure in printed edition.) Note: Trend analysis chart presentation may include narrative and supplementary data, including information on whether cost variations are due to contractor or NASA changes. Source: NASA. MAJOR CONTRIBUTORS TO THIS REPORT =========================================================== Appendix V NATIONAL SECURITY AND INTERNATIONAL AFFAIRS DIVISION, WASHINGTON, D.C. David R. Warren Frank Degnan Sandra D. Gove Roberta H. Gaston