Weather Forecasting: Improvements Needed in Laboratory Software
Development Processes (Letter Report, 12/14/94, GAO/AIMD-95-24).
This report reviews the software development capabilities of the
organizations responsible for producing a critical upgrade of the
National Weather Service's (NWS) Advanced Weather Interactive Processing
System. By automating functions now performed by forecasters, this
system enhancement would streamline the forecast process and improve
forecaster productivity. The National Oceanic and Atmospheric
Administration (NOAA) is developing the system enhancement jointly with
NWS. The plan is to define software specifications through a series of
prototype systems and then develop production-quality software for use
in the Advanced Weather Interactive Processing System. This report
discusses whether the software development processes of NOAA and NWS are
adequate to support (1) software prototyping and (2) production-quality
software development.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-95-24
TITLE: Weather Forecasting: Improvements Needed in Laboratory
Software Development Processes
DATE: 12/14/94
SUBJECT: Earth resources satellites
Meteorological research
Systems design
Weather forecasting
Computer software verification and validation
Environmental monitoring
Information systems
IDENTIFIER: NWS Advanced Weather Interactive Processing System
NWS AWIPS Forecast Preparation System
NOAA Geostationary Operational Environmental Satellite
NWS Next Generation Weather Radar
NOAA/NASA GOES-Next Satellite Program
*******************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
*******************************************************************************
Cover
========================================================================== COVER
Report to Congressional Requesters
December 1994
WEATHER FORECASTING - IMPROVEMENTS
NEEDED IN LABORATORY SOFTWARE
DEVELOPMENT PROCESSES
GAO/AIMD-95-24
Weather Service Software Development
Abbreviations
========================================================================= ABBREV
AFPS - AWIPS Forecast Preparation System
AWIPS - Advanced Weather Interactive Processing System
FSL - Forecast Systems Laboratory
NOAA - National Oceanic and Atmospheric Administration
NWS - National Weather Service
PRC - Planning and Research Corporation
SCM - software configuration management
SEI - Software Engineering Institute
SQA - software quality assurance
TDL - Techniques Development Laboratory
Letter
========================================================================= LETTER
B-259114
December 14, 1994
The Honorable George E. Brown, Jr.
Chairman
The Honorable Robert S. Walker
Ranking Minority Member
Committee on Science, Space,
and Technology
House of Representatives
The Honorable Ernest F. Hollings
Chairman, Committee on Commerce,
Science, and Transportation
United States Senate
This report responds to your request that we review the software development
capabilities of the organizations responsible for developing a critical
enhancement to the National Weather Service's (NWS) Advanced Weather
Interactive Processing System (AWIPS), known as the AWIPS Forecast Preparation
System (AFPS). AWIPS is an information processing system to support
forecasters in acquiring, analyzing, and disseminating weather data from
various sources. The AFPS enhancement is to automate additional functions
currently performed by the forecasters, thereby streamlining the forecast
process and improving forecaster productivity. The National Oceanic and
Atmospheric Administration (NOAA), of which NWS is a part, is jointly
developing AFPS with NWS through the respective laboratories. Their plan is to
first define an AFPS software specification through a series of prototype
systems and then develop AFPS production-quality software\1 for direct
integration into AWIPS.
Our objective was to determine whether the software development processes of
NOAA's Forecast Systems Laboratory (FSL) and NWS' Techniques Development
Laboratory (TDL) are adequate to support (1) AFPS software prototyping and (2)
AFPS production-quality software development. As a rule, the software
development processes needed to write production-quality software are much more
rigorous, disciplined, and formal than those used for software prototyping.
The laboratories' software development processes in question are software
requirements management, project planning, quality assurance, configuration
management, and tracking and oversight. A detailed explanation of these
processes is presented in appendix I.
--------------------
\1 Production-quality software is software that (1) satisfies its specified
functional, performance, and operational requirements and thus is ready for
day-to-day use and (2) has effective documentation (for example, programmers'
manuals, users' manuals, comments on code, testing procedures and results,
etc.) to support system operations and maintenance.
RESULTS IN BRIEF
---------------------------------------------------------------------- Letter :1
The software development processes that FSL and TDL have in place and are
following in developing AFPS are adequate to support NWS' near-term AFPS
development activities--prototyping for the purpose of defining AFPS
requirements and preparing a software specification. However, the
laboratories' processes are not adequate to achieve NWS' ultimate
objective--developing production-quality AFPS code that NWS can give to the
AWIPS contractor for direct integration into AWIPS. Unless the laboratories
introduce formality, rigor, and discipline into their software development
processes before they begin writing production-quality code, AFPS and AWIPS
will likely perform poorly, be delivered late, and cost considerably more than
planned. Officials for both laboratories acknowledged their respective process
limitations relative to developing production-quality code, and stated that
needed improvements are planned.
BACKGROUND
---------------------------------------------------------------------- Letter :2
Since the early 1980s, NWS has been modernizing its observing, information
processing, and communications systems to improve the accuracy, timeliness, and
efficiency of weather forecasts and warnings. The modernization includes four
new major systems--AWIPS, the Next Generation Geostationary Operational
Environmental Satellites, Next Generation Weather Radars, and Automated Surface
Observing Systems. The Department of Commerce's Deputy Under Secretary for
Oceans and Atmosphere is responsible for the modernization.
AWIPS DESCRIPTION AND STATUS
-------------------------------------------------------------------- Letter :2.1
As discussed in our March 1994 report,\2 the heart of the modernization is
AWIPS. AWIPS will allow forecasters to integrate, manipulate, and analyze the
vast amounts of data that are expected to be available from the new or improved
observing and information processing systems.
NOAA awarded the AWIPS contract to the Planning and Research Corporation (PRC)
in December 1992 with an atypical arrangement in which NWS would supply PRC
with certain specifications in the form of existing and to-be-developed code.
Under this arrangement, PRC would then decide, with NOAA's and NWS' approval,
whether the code was of sufficient quality to be directly incorporated into
AWIPS or whether it required rewriting.
NWS is now in the process of restructuring the AWIPS program and renegotiating
the AWIPS contract with PRC. These changes are in response to AWIPS design
problems and program delays. Although all the details surrounding program and
contract changes have not been defined, NWS officials told us that the
restructured program will, among other things, assign responsibility for
developing AWIPS application software to the government. This
government-developed application software will then be provided to PRC, which
will be responsible for providing the AWIPS platform (that is, hardware and
systems software environment), integrating the applications with the platforms,
and ensuring overall system quality. According to these officials, PRC will be
contractually required to use the government-furnished code, including the AFPS
code, as delivered, and integrate it into AWIPS (that is, treating it as
government-furnished equipment). However, the specific contract terms that
define such things as system quality and responsibility for poor system
performance have not been specified.
--------------------
\2 Weather Forecasting: Systems Architecture Needed for National Weather
Service Modernization (GAO/AIMD-94-28, March 11, 1994).
AFPS DESCRIPTION AND STATUS
-------------------------------------------------------------------- Letter :2.2
AFPS is part of the NWS to-be-provided application software. Estimates of its
size range from 100,000 to 250,000 lines of code. The system is expected to
streamline the forecast process and improve forecaster productivity through
automated:
Forecast visualization - The process of forming an on-screen graphical forecast
image based on observations, direct model input, numerical and statistical
forecasts, climatology, and other data, as well as experience and training.
Graphical forecast editing - The process of graphically entering and revising
watch, warning, and advisory information used in the original visualized
forecast.
Text generation - The process of automatically producing text based on the
graphical representation of the forecast, thus relieving the forecasters of
repetitive and time-consuming typing responsibilities.
The laboratories are currently defining and validating AFPS requirements
through the use of prototyping techniques. Prototyping is the iterative
process of quickly coding, evaluating, and refining less than complete versions
of a system. As such, a system prototype is not intended to be an end product
or final system; instead, the prototype is a learning tool or a series of
learning tools. Prototyping may be undertaken to conduct research (for
example, to prove a concept, determine a project's feasibility, or define
system requirements), and thus, once the goals are achieved, the prototype can
be discarded. However, prototypes can also be retained and used as a
foundation for enhancement or be repackaged as a final product.
To date, the laboratories have developed four AFPS iterations. According to
laboratory officials, their initial AFPS prototype was a "throw away." The next
prototype, however, formed the foundation on which additional system functions
and features have and will continue to be added in building increasingly more
capable iterations. The laboratories' plan is to continue prototyping until
AFPS is refined to the point that it can be placed in an NWS field office for
validation by weather forecasters in an operational setting. On the basis of
what is learned in the field office, AFPS is to be further refined before
providing it to NWS for additional testing and refinement. NWS is then to
deliver the code to PRC for integration into AWIPS. According to laboratory,
NOAA, and NWS officials, the current plan is to deliver code that requires no
rewrite before it is integrated into AWIPS. The officials emphasized that
their goal is to develop production-quality code.
IMPORTANCE OF SOFTWARE
DEVELOPMENT DISCIPLINE
-------------------------------------------------------------------- Letter :2.3
Software development has proven to be the "Achilles heel" of many system
development projects. Frequently these projects are delivered late, exceed
budgets, and perform poorly because the software development was not guided by
disciplined engineering processes.
The rigor and formality of the processes needed to successfully develop
software are determined by the desired outcome. If the goal is to develop one
or more prototypes that are to be used as learning tools and then discarded,
formal software processes and extensive documentation are unnecessary.
However, if the desired outcome is production-quality code for the final
system, more rigorous and stringent software engineering processes should guide
the development effort.\3
--------------------
\3 Rigorous software engineering processes are those that are structured, well
documented, and systematically implemented and monitored. See appendix II for
specific examples of such processes.
SCOPE AND METHODOLOGY
---------------------------------------------------------------------- Letter :3
To determine the laboratories' development capabilities, we reviewed key
software development documents, including the AFPS development plan, software
development and documentation guidelines, summaries of prototyping cycles, and
quarterly reports. In addition, we interviewed NOAA and NWS officials at TDL
and FSL about their processes. To identify the software development processes
that can reasonably be expected to be in place when prototyping a system versus
developing production-quality software, we researched relevant literature and
government and industry standards, reviewed the Capability Maturity Model
developed by Carnegie Mellon University's Software Engineering Institute (SEI),
and interviewed SEI officials. SEI's Capability Maturity Model is a tool for
assessing organizations' ability to develop software in accordance with modern
software engineering methods. This tool focuses on the maturity of certain
software development processes. The processes applicable to the laboratories
are (1) software requirements management, (2) project planning, (3) quality
assurance, (4) configuration management, and (5) tracking and oversight.
We performed our work at NOAA and NWS headquarters and at NWS' TDL in Silver
Spring, Maryland; NOAA's FSL in Boulder, Colorado; and SEI in Pittsburgh,
Pennsylvania. Our work was performed from May 1994 through October 1994, in
accordance with generally accepted government auditing standards.
LABORATORIES' PROCESSES ARE ADEQUATE
FOR AFPS SOFTWARE PROTOTYPING
---------------------------------------------------------------------- Letter :4
The processes that the laboratories have in place for software requirements
management, project planning, quality assurance, configuration management, and
tracking and oversight are adequate for AFPS prototyping. Laboratory
activities in each of the five process areas that we reviewed satisfied the
level of structure and documentation advocated by SEI for determining
requirements and defining software specifications through system prototyping.
In particular, SEI officials stated that prototyping is an investigative
process that typically requires flexible processes so as not to overwhelm the
developers' creativity. Accordingly, FSL has adopted a process known as the
spiral model of software development to define requirements. This process
involves iterative builds, each enhancing the previous one. TDL and FSL have
also augmented the use of this model with some software development formality,
such as the use of documented software coding guidelines and a documented
software development plan.
LABORATORIES' PROCESSES ARE NOT
ADEQUATE FOR DEVELOPING AFPS
PRODUCTION-QUALITY SOFTWARE
---------------------------------------------------------------------- Letter :5
The laboratories do not, however, have in place the software development
processes that are needed to develop high-quality, production code. Instead,
their processes are largely informal, relying more on the capabilities of
laboratory staff rather than clearly defined and documented processes. While
certain laboratory process areas (for example, requirements management) are
stronger than others (for example, software quality assurance), none possess
the full complement of activities that the SEI Capability Maturity Model
advocates. Specific examples of where the laboratories are deficient in each
process area are described below. Additional examples are provided in appendix
II.
Requirements management: Organizations developing production-quality software
should have a software engineering group that reviews and agrees to
requirements before code is written to incorporate these requirements into the
software. Neither TDL nor FSL have software engineering groups.
Project planning: The software development plan is the culmination of software
project planning. To the laboratories' credit, they have an AFPS software
development plan that defines the project's purpose, goals, organizational
development responsibilities, software development methodology, and software
development schedules. However, their plan lacks other important elements,
including software verification and validation provisions and software metrics.
Quality assurance: The software quality assurance plan is the centerpiece of
an effective quality assurance program. This plan defines the activities
necessary to ensure that software development processes and products conform to
applicable requirements and standards. In addition, an independent software
quality assurance group should review and audit the software engineering
activities to ensure compliance. The laboratories do not have a software
quality assurance plan or group.
Configuration management: A software configuration management plan should
exist that clearly defines the procedures for identifying, accounting for, and
reporting on changes to software items that are under configuration control.
Neither laboratory has a software configuration management plan.
Tracking and oversight: To ensure that a software development project is
proceeding as planned, formal reviews to communicate accomplishments and
results of the project software engineering should be conducted at selected
project milestones. The laboratories do not conduct such reviews.
FSL and TDL officials agreed that their software development processes are not
adequate for production-quality development activities. They stated that
actions are planned to improve the maturity of these processes. For example,
FSL plans to hire an individual to establish and manage an independent quality
assurance program, including development and implementation of a quality
assurance plan. Also, FSL plans to improve its requirements management process
by (1) documenting the process, (2) developing a technique for mapping system
requirements to the system and software designs, and (3) documenting changes to
the requirements baseline.
CONCLUSIONS
---------------------------------------------------------------------- Letter :6
Although FSL and TDL have adequate software development processes for defining
AFPS requirements via prototyping, neither has adequate processes for
developing production-quality code, which is NWS' ultimate goal on AFPS.
Without these processes, NWS is exposing AFPS, as well as AWIPS, to unnecessary
cost, schedule, and performance risks.
RECOMMENDATIONS
---------------------------------------------------------------------- Letter :7
In light of NWS' plan to provide the AWIPS contractor with production-quality
software for direct integration into AWIPS, we recommend that the Secretary of
Commerce direct the Deputy Under Secretary for Oceans and Atmosphere to have
FSL and TDL strengthen their software development processes for requirements
management, project planning, quality assurance, configuration management, and
tracking and oversight before beginning development of any production-quality
code. The crucial processes are outlined in appendix II of this report.
-------------------------------------------------------------------- Letter :7.1
We discussed the contents of this report with TDL and FSL officials, the AWIPS
Technical Director, and the NWS Modernization Systems Manager. These officials
generally agreed with the information presented. We have incorporated their
comments where appropriate.
We are sending copies of this report to the Secretary of Commerce, the Director
of the Office of Management and Budget, and interested congressional
committees. Copies will also be
Please call me at (202) 512-6253 if you or your staff have any questions
concerning this report. Other major contributors are listed in appendix III.
Joel C. Willemssen
Director, Information Resources
Management/Resources, Community,
and Economic Development
SOFTWARE DEVELOPMENT PROCESS AREAS
===================================================================== Appendix I
Table I.1 provides the software development process areas we evaluated and
their definitions.
Table I.1
Software Development Process Areas
------------------ ----------------------------------------
Software This process involves establishing and
requirements maintaining an agreement with the
management customer for both the technical and
nontechnical requirements for the
software project. This agreement forms
the basis for estimating, planning,
performing, and tracking the software
project's activities throughout the
software life cycle.
Software project This process is a subset of the overall
planning project planning. It involves defining
each major software project task,
estimating the time and resources
required to accomplish it, and tracking
and controlling actual software
production against these and other
production goals. The centerpiece of
software project planning is the
software development plan.
Software quality This process is the planned and
assurance systematic set of actions necessary to
provide sufficient confidence that the
software product conforms to established
requirements. Effective quality
assurance should be (1) managed by an
organization independent of the
organizations managing the project and
developing the software products,
(2) managed by an organization that has
the authority to establish and enforce
software quality standards and
procedures, (3) based on predetermined
software metrics, and (4) managed by
documented processes that are shared
among parties participating in the
project.
Software This process is the means by which
configuration changes to software products are
management controlled. It includes identification
of software products to be controlled,
accounting for changes to these
controlled products, and reporting on
the status of these products.
Software tracking This process provides insight into
and oversight project progress and provides a basis
for reporting on project status. It is
accomplished by measuring ongoing
software development activities and
comparing the measurements against
documented estimates, commitments,
plans, and industry norms.
------------------------------------------------------------
COMPARISON OF CRUCIAL PRODUCTION
SOFTWARE DEVELOPMENT ACTIVITIES WITH
THOSE AT TDL AND FSL
==================================================================== Appendix II
This appendix identifies the activities\1 that are expected of leading software
development organizations and contrasts these with the activities currently
performed at TDL and FSL. While this is not a comprehensive list of
activities, it highlights the most crucial activities desired in each of the
software development process areas.
Table II.1
Software Requirements Management Process
Crucial activities TDL and FSL activities
----------------------------- -----------------------------
The requirements are The requirements are
documented in a consistent documented with each
format and are clearly prototype iteration. Each
stated, verifiable, and iteration more clearly
testable. defines the baseline
requirements.
The software engineering A formal software engineering
group reviews and agrees to group does not exist.
the requirements before they
are incorporated into the
software efforts. The AFPS requirements are
still being developed, and
The requirements form the thus such documents and
basis for the software plans, activities do not yet exist.
products, and activities.
Requirements changes are
reviewed by a technical
Changes to the requirements coordinator and are
are appropriately reviewed incorporated into the next
and incorporated into the prototype iteration.
software efforts.
------------------------------------------------------------
Table II.2 Software Project Planning
Process
Crucial activities TDL and FSL activities
----------------------------- -----------------------------
The software engineering There is no formal software
group is an active engineering group, but both
participant in proposing and TDL and FSL programmers
planning the project participate in the project's
throughout its life. planning activities.
The AFPS software development
Software planning is plan was developed in
initiated in the early stages conjunction with the AFPS
of, and in parallel with, concept document.
overall project planning.
This activity does not
Software planning data are exist.
recorded for use by the
project.
No arrangements with external
Senior management reviews and organizations exist.
approves all commitments made
to individuals and groups
external to the A software life-cycle model
organization. does not exist.
A software life-cycle model
with predetermined stages of
manageable size is identified The AFPS software development
or defined. plan is documented in the
AFPS concept document and
The project's software addresses most software
development plan is developed activities.
according to a documented
procedure and addresses all Software product and process
software activities. specifications (for example,
design documents, programming
Software products and guidelines) exist but are not
software process identified as controlled
specifications that are project baseline items.
needed to establish and
maintain stability of the A documented procedure to
software activities are derive these items does not
explicitly identified as exist.
controlled project baseline
items.
Estimates for the size of the
software products, the
software development Such risks are identified and
resources and costs, the assessed informally; however,
critical target computer they are not documented.
resources, and the software
schedule are derived
according to a documented
procedure.
The software technical, cost,
resource, and schedule risks
are assessed and documented.
------------------------------------------------------------
Table II.3
Software Quality Assurance Process
Crucial activities TDL and FSL activities
----------------------------- -----------------------------
A software quality assurance An SQA plan does not exist.
(SQA) plan is prepared for
each software project
according to a documented
procedure, and SQA activities
are performed in accordance
with the SQA plan. An SQA group does not exist.
The SQA group participates in
the preparation, review, and
approval of the project's
software development plan,
process specifications, The FSL Technical Coordinator
standards, and procedures. reviews code and conducts
code walk-throughs. TDL staff
The SQA group reviews and have peer code reviews and
audits the software walk-throughs. However, a
engineering activities to separate SQA group does not
ensure process compliance. exist.
Such activities do not
occur.
The SQA group reviews
representative samples of
deliverable and designated
nondeliverable software
products to ensure compliance
with the designated process Such activities do not
requirements. occur.
The SQA group regularly
reports its reviews and
audits to the software Deviations identified as a
engineering staff and result of the FSL Technical
managers. Coordinator and TDL peer
reviews are addressed but not
Deviations identified in the documented nor handled
software engineering according to documented
activities are documented and procedures.
handled according to a
documented procedure.
------------------------------------------------------------
Table II.4
Software Configuration Management
Process
Crucial activities TDL and FSL activities
----------------------------- -----------------------------
A documented software An SCM plan does not exist.
configuration management
(SCM) plan exists and is used
as the basis for performing
SCM activities. A configuration management
library system for managing
A configuration management the software baseline does
library system is established not exist. However, the
as a repository for the laboratories use documented,
software baselines. automated tools for
configuration control of the
software products.
Such products and processes
The software engineering have not been identified for
products and process placement under configuration
specifications to be placed control.
under configuration
management are identified. A documented procedure does
not exist for tracking and
A documented procedure is controlling change requests
followed for initiating, and trouble reports.
recording, reviewing,
approving, and tracking
change requests and trouble A documented procedure does
reports for all configuration not exist to control software
items. baselines.
A documented procedure is
followed to create and Such documented procedures do
control the release of not exist.
software baseline products.
A documented procedure is
followed to record the status
of configuration items and
change requests, and to
control changes to
configuration items.
------------------------------------------------------------
Table II.5
Software Tracking and Oversight Process
Crucial activities TDL and FSL activities
----------------------------- -----------------------------
A documented software A software development plan
development plan is used for is not used for tracking the
tracking the software software activities; however,
activities and communicating such activities are tracked
status. and communicated through
quarterly reports.
Senior management reviews and No arrangements with external
approves all commitments to organizations exist.
individuals and groups
external to the
organization. Software changes are
communicated informally
Approved changes to software between laboratory officials.
commitments or commitments
affecting the software
activities are explicitly
communicated to the staff and
managers of the software These activities are not
engineering group. tracked by any formal means.
The project's size, costs,
critical target computer
resources, software schedule, These risks are not tracked
and software engineering by any formal means.
activities are tracked and
corrective actions are
taken. Such activities do not
occur.
The software technical, cost,
resource, and schedule risks
are tracked throughout the
life of the project.
Such regular reviews to track
Actual measured data and such items against software
replanning data for the development plans do not
software project tracking occur.
activities are recorded for
use by the software
engineering staff and Formal reviews are not
managers. conducted.
The software engineering
staff and managers conduct
regular reviews to track
technical progress, plans,
performance, and issues
against software development
plans.
Formal reviews to address the
accomplishments and results
of project software
engineering are conducted at
selected project milestones.
------------------------------------------------------------
--------------------
\1 The activities were derived from SEI's Capability Maturity Model.
MAJOR CONTRIBUTORS TO THIS REPORT
=================================================================== Appendix III
ACCOUNTING AND INFORMATION MANAGEMENT
DIVISION, WASHINGTON, D.C.
Dr. Rona B. Stillman, Chief Scientist for Computers and Communications
Randolph C. Hite, Assistant Director
Keith A. Rhodes, Technical Assistant Director
David A. Powner, Evaluator-in-Charge
Colleen M. Phillips, Senior Evaluator
*** End of document. ***