Federal Student Aid: Progress in Integrating Pell Grant and
Direct Loan Systems and Processes, but Critical Work Remains
(31-DEC-02, GAO-03-241).
To address system problems and other long-standing management
weaknesses, in 1998, the Congress created a discrete unit within
the Department of Education, the Office of Federal Student Aid
(FSA). This office subsequently adopted a new approach to systems
integration using middleware (a type of software that can allow
an application to access data residing in different databases)
and Extensible Markup Language (XML)--a flexible, nonproprietary
set of standards that is intended to make it easier to identify,
integrate, and process information widely dispersed among systems
and organizations. FSA's first use of this approach is the Common
Origination and Disbursement (COD) process for the Direct Loan,
Pell Grant, and campus-based programs. GAO initiated a follow-up
review to assess FSA's progress in implementing this process.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-241
ACCNO: A05785
TITLE: Federal Student Aid: Progress in Integrating Pell Grant
and Direct Loan Systems and Processes, but Critical Work Remains
DATE: 12/31/2002
SUBJECT: Best practices
Direct loans
Federal aid programs
Government information dissemination
Grants
Information resources management
Information technology
Student financial aid
Systems compatibility
Department of Education Common
Origination and Disbursement System
Extensible Markup Language
Federal Pell Grant Program
William D. Ford Federal Direct Loan
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-241
A
Report to the Secretary of Education
December 2002 FEDERAL STUDENT AID Progress in Integrating Pell Grant and
Direct Loan Systems and Processes, but Critical Work Remains
GAO- 03- 241
a
GAO United States General Accounting Office
FSA has made progress in implementing the COD process. Specifically, it
has implemented a new information technology infrastructure that uses
middleware to
enable data exchange among disparate systems;
the initial version of the basic COD system, which replaces two existing
systems and is being used by schools participating in the Pell Grant and
Direct Loan programs;
middleware into existing systems to support the COD process; and
a common record based on XML that schools can use to submit student
financial data for the Pell Grant and Direct Loan programs.
However, the implementation of the COD process is behind schedule, and its
ultimate success hinges on FSA*s completing critical work, including
addressing serious postimplementation operational problems, and having
thousands of postsecondary schools implement the common record. Further,
there are important elements to managing any information technology
investment that FSA has not yet completed:
Determining whether expected benefits are being achieved. As illustrated
below, FSA has only some of the metrics, baseline data, and tracking
processes necessary to determine whether it is achieving all expected
benefits.
Tracking lessons learned. FSA has relied on an ad hoc approach for
gathering and disseminating lessons learned related to schools*
implementation of the common record. To address this issue, FSA plans to
include lessons learned as part of an update to its school testing guide.
However, this does not replace the need for an ongoing mechanism to
capture and disseminate lessons learned, without which schools may
encounter problems that could have been avoided or mitigated. Status of
the Office of Federal Student Aid Tracking of Actual Benefits of the COD
System Expected benefit
Metrics defined? Baseline data
available? Benefits tracked?
Reduced cost Yes Yes Yes a Increased customer satisfaction Partially b
Partially No Increased employee satisfaction N/ A c N/ A N/ A Increased
financial integrity Partially d Partially Partially Integration and
modernization of legacy systems Yes Yes Yes
a In providing comments on a draft of this report, FSA stated that it has
begun tracking this expected benefit and provided supporting
documentation, which we did not validate. b FSA stated that at this time,
it did not have approval from the Department of Education to perform
the surveys that it had planned to use to validate this benefit. However,
FSA stated that it is trying to identify alternative metrics to measure
customer satisfaction improvements due to COD. C Not applicable; although
this was identified in the COD business case as an expected benefit, in
commenting on a draft of this report, FSA stated that it is no longer
using this as a COD benefit. d Not all metrics for this expected benefit
have been defined.
Source: GAO analysis on the basis of FSA documentation. FEDERAL STUDENT
AID
Progress in Integrating Pell Grant and Direct Loan Systems and Processes,
but Critical Work Remains
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 241. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact David A. Powner at (202) 512- 9286 or pownerd@ gao.
gov. Highlights of GAO- 03- 241, a report to the
Secretary of Education
December 2002
To address system problems and other long- standing management weaknesses,
in 1998, the Congress created a discrete unit within the
Department of Education, the Office of Federal Student Aid (FSA). This
office subsequently adopted a new approach to systems integration using
middleware (a type of software that can allow an application to access
data residing in different databases) and Extensible Markup Language
(XML)* a flexible, nonproprietary set of standards that is intended to
make it easier to
identify, integrate, and process information widely dispersed among
systems and organizations.
FSA*s first use of this approach is the Common Origination and
Disbursement (COD) process for the Direct Loan, Pell Grant, and
campus- based programs. GAO initiated a follow- up review to assess FSA*s
progress in implementing this process. Among GAO*s recommendations is
that the Secretary of Education direct the Chief Operating Officer, FSA,
to establish a process to
capture and disseminate lessons learned to schools.
In commenting on a draft of this report, FSA provided updated information
and technical comments. GAO modified the report to address this new
information, as appropriate.
Page i GAO- 03- 241 Federal Student Aid Systems Integration Letter 1
Results in Brief 3 Background 4
Progress in Implementing the COD Process, but Critical Work Remains, and
Benefits and Lessons Learned Are Not Being Tracked 7 FSA Is Not Completely
Tracking Actual COD Benefits or Lessons
Learned Related to Schools* Implementation of the Common Record 10
Conclusions 13 Recommendations 13 Agency Comments and Our Evaluation 14
Appendix I Systems and Technologies Supporting the Common Origination and
Disbursement Process, as of November 2002 16
Appendix II Comments from the Department of Education*s Office of Federal
Student Aid 18
GAO Comments 22
Tables
Table 1: Status of FSA Tracking of Actual Benefits of the COD System 11
Table 2: Examples of Lessons Learned Related to School Migration
to the Common Record 13
Abbreviations
COD Common Origination and Disbursement EAI Enterprise Application
Integration FSA Office of Federal Student Aid IT information technology
XML Extensible Markup Language Contents
Page 1 GAO- 03- 241 Federal Student Aid Systems Integration December 31,
2002 The Honorable Roderick R. Paige
The Secretary of Education Dear Mr. Secretary: Over the past decade, the
Department of Education has spent millions of dollars to modernize and
integrate 1 its disparate financial aid systems in an effort to improve
the administration of its programs and provide more information and
greater service to its customers* students, parents, schools, and lenders.
However, as we reported last year, the department*s efforts had achieved
limited success. 2 In January 1993, for example, the department awarded a
5- year, $39 million contract for the development and maintenance of the
National Student Loan Data System, which was to provide information on
systems across programmatic boundaries, yet it often lacks the most recent
information from other financial aid systems.
To address such systems problems and other long- standing management
weaknesses, in 1998, as you know, the Congress created the federal
government*s first performance- based organization, 3 currently called the
Office of Federal Student Aid (FSA). 4 Under the performance- based
organization concept, FSA is a discrete organizational unit within the
Department of Education, led by its own Chief Operating Officer. FSA
focuses on managing the operation of the student financial assistance
1 Information integration is defined by the National Institute of
Standards and Technology as the establishment of the appropriate computer
hardware/ software, methodology, and organizational environment to provide
a unified and shared information management
capability for a complex business enterprise. 2 U. S. General Accounting
Office, Student Financial Aid: Use of Middleware for Systems Integration
Holds Promise, GAO- 02- 7 (Washington, D. C.: Nov. 30, 2001). 3 The Higher
Education Amendments of 1998, which amended the Higher Education Act of
1965, states that the responsibilities of the performance- based
organization include integrating the information systems supporting
federal student financial assistance programs; implementing an open,
common, and integrated system for delivery of student financial assistance
under title IV; and developing and maintaining a student financial
assistance system that contains complete, accurate, and timely data to
ensure program integrity.
4 Financial aid programs are administered by an office previously known as
the Office of Student Financial Assistance, which was changed to the
Office of Federal Student Aid on March 6, 2002. United States General
Accounting Office Washington, DC 20548
Page 2 GAO- 03- 241 Federal Student Aid Systems Integration programs,
while the Department of Education focuses on policy- making functions.
FSA subsequently adopted a systems integration approach that uses
middleware 5 and Extensible Markup Language (XML) 6 technologies. In
November 2001, we reported that in selecting middleware, FSA adopted a
viable, industry- accepted mechanism for addressing its long- standing
systems integration problems. 7 We also reported that FSA*s first use of
middleware and XML as part of a process for delivering Direct Loan and
Pell Grant aid to students was expected in early 2002. This process,
called the Common Origination and Disbursement (COD) process, was expected
to use a new system, also called COD, and be supported by a new
information technology (IT) infrastructure and various existing systems.
In the summer of 2002, we initiated a follow- up review to our November
2001 report. Our objective was to assess FSA*s progress in implementing
the COD process. In doing this work, we reviewed applicable FSA and
Accenture (the prime contractor) documentation, including FSA*s
modernization blueprint, the COD business case, the COD system*s
requirements, the COD and Enterprise Application Integration
implementation timelines, Accenture project status briefings, and test
guides and results. In addition, we reviewed reports by an independent
verification and validation contractor. We also interviewed applicable
officials from FSA IT and program offices and Accenture.
We performed our work at FSA headquarters in Washington, D. C., between
August and November 2002, in accordance with generally accepted government
auditing standards.
5 Middleware is a type of software that can allow an application to access
data residing in different databases. In addition, middleware can enable
dissimilar systems to communicate and work together as if they all resided
on a single platform. 6 XML is a flexible, nonproprietary set of standards
that offers the promise of making it
significantly easier to identify, integrate, and process information that
is widely dispersed among systems and organizations.
7 GAO- 02- 7.
Page 3 GAO- 03- 241 Federal Student Aid Systems Integration FSA has made
progress in implementing the COD process. 8 Specifically, it has
implemented (1) a new IT infrastructure that uses middleware to
enable data exchange between disparate systems, (2) the initial version of
the basic COD system, (3) middleware into existing systems to support the
COD process, and (4) a common record based on XML that schools can use to
submit student financial data for both the Pell Grant and Direct Loan
programs. Nevertheless, the implementation of the COD process is behind
schedule, and its ultimate success hinges on FSA*s completing critical
work, including the implementation of the basic COD system requirements,
addressing serious postimplementation operational problems, and having
thousands of postsecondary schools implement the common record.
Important elements of managing an IT investment are determining whether
expected benefits are being achieved and tracking lessons learned.
However, at this time, FSA is not fully tracking whether it is achieving
certain expected benefits, such as increased customer satisfaction. In
this instance, FSA COD officials stated that they did not have approval
from the Department of Education to perform the survey that they had
planned to use to validate this expected benefit, but they are trying to
identify alternative metrics. Without such data, FSA lacks vital
information on whether it is achieving all of its investment goals.
Regarding lessons learned, FSA relies on an ad hoc approach to gathering
and disseminating such information, which it believes is an adequate
approach. However, such an ad hoc process may not ensure that all schools
obtain these critical data. As a result, schools may encounter problems
that could have been avoided or mitigated had they known of other schools*
experiences.
We are making recommendations to improve the management of FSA*s COD
program, including recommending that you direct FSA*s Chief Operating
Officer to develop (1) metrics and baseline data that are related to
increased customer satisfaction and increased financial integrity and (2)
a tracking process to assess the extent to which the expected results
are being achieved. 8 Accenture was responsible for providing the project
management, software, testing, and operation of the technology elements of
the COD process. However, as used here, we ascribe the action to FSA
because it made the decision to accept and deploy the contractor*s work.
Results in Brief
Page 4 GAO- 03- 241 Federal Student Aid Systems Integration In commenting
on a draft of this report, FSA*s Chief Operating Officer provided updated
information and technical comments, but did not
comment on our recommendations. Among the comments provided, the Chief
Operating Officer (1) stated that the draft report did not adequately
portray the level of COD progress that had been made, (2) agreed that FSA
was not tracking all expected benefits at this time and provided new
information and supporting documentation on its recent efforts in this
area, and (3) stated that FSA*s *informal* process for communicating
lessons had worked well but planned to provide written lessons learned as
part of a planned update of its school testing guide. We updated our
report to reflect new FSA processes and information, as appropriate.
However, we believe that we have accurately portrayed FSA*s progress,
particularly in light of its need to address operational problems and
facilitate the
implementation of the common record at thousands of postsecondary schools.
In addition, we continue to believe that FSA*s ad hoc processes for
capturing and disseminating lessons learned do not provide assurance that
it has captured and disseminated lessons learned related to schools*
implementation of the common record and could overlook important
improvements that could be made.
The Department of Education*s FSA manages and administers student
financial assistance programs authorized under title IV of the Higher
Education Act of 1965, as amended. These postsecondary programs include
the William D. Ford Federal Direct Loan Program (often called the Direct
Loan program), the Federal Family Education Loan Program (often called the
Guaranteed Loan program), the Federal Pell Grant Program, and campus-
based programs. 9 Annually, these programs together provide about $50
billion in student aid to approximately 8 million students and their
families.
During the past three decades, the Department of Education has created
many disparate information systems to support these various student
financial aid programs. In many cases, these systems* run on multiple
operating platforms using different network protocols 10 and maintained
9 Campus- based programs, which include the Federal Work- Study Program,
the Federal Perkins Loan Program, and the Federal Supplemental Educational
Opportunity Grant Program, are administered jointly by FSA and
postsecondary educational institutions.
10 For example, the FSA systems environment includes operating platforms,
such as IBM OS/ 390 mainframe, Sun Solaris on Sparc, and Windows NT, and
network protocols, such as the Transmission Control Protocol/ Internet
Protocol and Systems Network Architecture. Background
Page 5 GAO- 03- 241 Federal Student Aid Systems Integration and operated
by a host of different contractors* were unable to easily exchange the
timely, accurate, and useful data needed to ensure the
proper management and oversight of the various student financial aid
programs. For example, as we reported in 1997, neither the National
Student Loan Data System nor other systems were designed for efficient
access to reliable student financial aid information, since many systems
were incompatible and lacked data standards and common identifiers. 11 In
addition, because FSA used three separate systems to originate and/ or
disburse title IV funds, access to student and school data was fragmented
and unreliable. As a result, FSA found it increasingly difficult to
quickly access data to support day- to- day operational and management
decisions, and schools could not easily access data to obtain a clear
picture of the
title IV student aid that had been disbursed. In September 1999, FSA
issued its initial modernization blueprint, 12 which was subsequently
updated in July 2000, to transform the title IV student financial aid
systems using technology. COD is one of four school service business
processes 13 in FSA*s blueprint and is intended to implement a simplified
process for the operation of the Direct Loan and Pell Grant programs.
According to FSA*s modernization blueprint, the common origination and
disbursement process is composed of seven steps involving students, the
Department of Education, and schools: (1) obtain applicant data, (2)
determine eligibility, (3) determine award, (4) notify the Department of
Education of the intent to disburse, (5) obtain funds from Education, (6)
disburse funds to student, and (7) close out. A common process to support
origination and disbursement is considered critical to FSA*s goal of
achieving an enterprisewide solution that provides real- time data to
students, schools, and financial partners via Web portals.
To implement COD, FSA is using middleware and XML technologies.
Specifically, middleware is being used to integrate FSA systems that
11 U. S. General Accounting Office, Student Financial Aid Information:
Systems Architecture Needed to Improve Programs* Efficiency, GAO/ AIMD-
97- 122 (Washington, D. C.: July 29, 1997).
12 FSA*s modernization blueprint describes the agency*s business
requirements, business and technical architecture, and sequencing plan. 13
The other three processes are program eligibility, program support, and
financial transactions.
Page 6 GAO- 03- 241 Federal Student Aid Systems Integration support the
COD process. 14 Traditionally, systems integration would require building
separate point- to- point interfaces between every two
applications. Although this approach can be effective, it creates several
problems, such as (1) every connection between two applications requires
custom programming; (2) a lot of connections have to be developed when
there are multiple data sources; and (3) whenever the logic or data in one
application changes, the accompanying interface often also needs to be
altered. Middleware represents an alternative means to the traditional
approach, and it can provide a quicker and more robust solution to systems
integration. In essence, middleware separates the business application
from the technical details of interapplication communications. Thus,
middleware can simplify and reduce the number of interfaces for multiple
systems because it can handle differences in data formats and record
layouts.
As part of the COD process, XML is being used to consolidate multiple
legacy record formats previously used by schools to submit data on the
Pell Grant and Direct Loan programs. 15 By using an XML- based common
record, schools can transmit one file with all of the student*s data
instead of submitting separate legacy records with redundant student and
school information.
Appendix I provides a high- level depiction of the systems and
technologies supporting the COD process as of November 2002. As depicted,
the COD system can translate or convert legacy records by using
middleware. In addition, middleware has been built into several existing
systems so that they can establish connectivity and exchange data with the
COD system through a common IT infrastructure. This IT infrastructure,
called the Enterprise Application Integration (EAI) bus, is also
implemented using middleware to route data between systems in a correct
format. In addition, as part of the COD process, some schools have begun
submitting Pell Grant and Direct Loan data using the XML- based common
record.
14 For additional information on middleware and how it works, see GAO- 02-
7 and North Carolina Information Resource Management Commission, North
Carolina Statewide Technical Architecture, which can be found at http://
irm. state. nc. us/ techarch/ chaps/ pdffiles/ pdflist. htm. 15 For
additional information on XML and how it works, see U. S. General
Accounting Office, Electronic Government: Challenges to Effective Adoption
of the Extensible Markup Language, GAO- 02- 327 (Washington, D. C.: Apr.
5, 2002).
Page 7 GAO- 03- 241 Federal Student Aid Systems Integration FSA hired
Accenture as its *modernization partner* to help carry out its
modernization blueprint, including the implementation of the COD
process. Accenture is the prime contractor 16 providing leadership of
critical planning activities that are essential to the success of FSA*s
modernization. Regarding the COD system part of FSA*s modernization, FSA
also hired an independent verification and validation contractor to review
the initial release of this system, which was completed earlier this year.
FSA has made progress in implementing the new COD process. In particular,
it has begun implementing (1) its middleware solution in its IT
infrastructure and various existing systems, (2) the COD system, and (3)
an XML- based common record. However, FSA*s implementation of COD is
behind schedule, and critical work remains to be completed. For example,
the basic COD system was to be completed by mid- October 2002; however,
only about three- quarters of the COD basic system requirements had been
implemented as of October 23, 2002. In addition, FSA is not tracking
whether it is achieving certain benefits because it is still in the
process of defining applicable metrics to measure progress. Without such
tracking processes, FSA lacks critical information about whether it is
achieving expected benefits. Finally, FSA lacks assurance that it has
captured and disseminated important lessons learned related to schools*
implementation of the common record because it believes that its current
ad hoc process is adequate. Accordingly, the thousands of schools that
have not yet implemented the common record may not benefit from the
experience of those that have.
FSA has made progress in implementing COD. The following are significant
elements of the COD process that have been implemented:
Deployment of the EAI bus. As a prerequisite to implementing COD, in
late October 2001, FSA deployed its middleware solution in an EAI *bus**
an IT infrastructure that uses middleware to access data from disparate
systems, transform the data formats as necessary, and route the data to
the appropriate requesting systems, thus enabling data exchange among
16 Accenture also uses subcontractors to perform some of the requirements
related to this contract. Progress in
Implementing the COD Process, but Critical Work Remains, and Benefits and
Lessons Learned Are Not Being Tracked
FSA Has Begun Implementing COD Capabilities, but Critical Work Remains
Page 8 GAO- 03- 241 Federal Student Aid Systems Integration disparate
systems. The EAI bus provides the set of technical capabilities necessary
to integrate FSA*s disparate systems.
Initial implementation of basic COD system (release 1. x). 17 On April
29, 2002, FSA went live with version 1.0 of the basic COD system. As of
midNovember 2002, FSA had released an additional five sub- versions of the
COD system (e. g., version 1.1). The COD system replaces the Direct Loan
Origination System and the Recipient Financial Management System, and it
currently processes files for all schools participating in the Pell Grant
and Direct Loan programs. 18 According to FSA, in the first 6 months of
its operation, the COD system processed just under 16 million
transactions, representing Pell Grant and Direct Loan awards totaling
almost $10 billion to over 5 million recipients.
Implementation of middleware in selected systems. As of mid- November
2002, FSA had built middleware into seven systems so that these systems
can interact with the COD system through the EAI bus. These systems
include (1) the Central Processing System, which determines students*
eligibility and award levels, and (2) the National Student Loan Data
System, which contains loan- and grant- level information and is used by
schools to screen student aid applicants to identify ineligible borrowers.
Development and implementation of the common record. Using XML,
FSA developed and began implementing a common record that schools can use
to submit student financial aid data to the COD system. The common record,
designed with assistance from members of the National Council of Higher
Education Loan Programs and the Postsecondary Electronic Standards
Council, consolidates multiple legacy file formats previously used by the
Pell Grant and Direct Loan programs. 19 17 *X* represents the sub- version
number of the COD system. As of mid- November 2002,
FSA planned to deploy a total of seven sub- versions of release 1. x. 18
FSA plans to keep these two systems functioning until it (1) completes the
reconciliation of prior years* processing for direct loans, which is
expected to occur by the end of this fiscal year, and (2) migrates the
Recipient Financial Management System data to COD, which is expected by
the summer of 2003.
19 The common record also includes data blocks for campus- based programs
to simplify school reporting. At this time, schools cannot report these
data to FSA because the campus- based program part of the COD system is
not yet available. According to a COD official, the campus- based data
blocks in the common record are expected to be enabled in the summer of
2003.
Page 9 GAO- 03- 241 Federal Student Aid Systems Integration Although FSA
has made progress in implementing the COD process, critical work remains
to be completed. First, FSA is behind schedule in
implementing the basic COD system. Although FSA had planned to complete
the basic COD system by mid- October 2002, only about threequarters 20 of
the COD basic system requirements had been implemented as
of October 23, 2002. 21 For example, as of early November 2002, one of the
basic business functions that remains to be implemented is to enable FSA
to make automated adjustments in batches to school current funding levels.
FSA now estimates that most of the remaining functionality will be
completed by the end of September 2003. According to FSA IT and
program officials, the implementation of the basic COD functionality was
delayed to allow adequate time for testing to ensure the quality of the
system. 22 Second, as of November 19, 2002, Accenture reported several
operational problems that needed to be addressed. For example, in some
cases, the
COD system was incorrectly processing school batch data that contained
multiple change records for an individual student. According to COD and
contractor officials, the causes of operational problems included unclear
requirements and software design defects. An independent verification and
validation contractor also found problems with the requirements and design
aspects of release 1.0. The COD Contracting Officer*s Representative
characterized these operational problems as very serious and stated that
they could impede operations and the delivery of future COD releases. This
same official noted that FSA and Accenture are currently undertaking
efforts to address these problems. For example, FSA has established
production teams composed of agency and contractor staff to address
problems in specific areas. In addition, FSA has established a continuous
improvement process to more rigorously manage its relationship with
Accenture.
Third, fewer postsecondary schools than planned have implemented the
common record. FSA had estimated that 50 schools (out of about 5,500)
would implement the common record in fiscal year 2002. However, as of
20 This percentage does not include functions initially planned to be
included in the COD system but subsequently cancelled. 21 In addition, on
September 4, 2002, the COD Development Manager estimated that Accenture
had met about 88 percent of the COD production acceptance criteria.
22 Once release 1. x is completed, FSA plans to implement other
functionality, including campus- based program reporting.
Page 10 GAO- 03- 241 Federal Student Aid Systems Integration November 26,
2002, only 22 schools 23 had implemented and tested the common record with
FSA. FSA COD officials attributed the fewer- thanexpected
number of schools using the common record to schools and vendors not being
ready to implement it. FSA expects that the number of schools using the
common record will be considerably higher during the next award year
(2003* 2004) because, by April 2003, it plans to implement and test the
common record with EDExpress, a software application FSA distributes free
of charge to about 3,000 schools for use in submitting data. In addition,
FSA expects that all schools will be using the common record format by
March 2004, in time for the 2004* 2005 award year.
In its COD business case, FSA outlined five expected benefits: (1) reduced
cost, (2) increased customer satisfaction, (3) increased employee
satisfaction, (4) increased financial integrity, and (5) the integration
and modernization of legacy systems. An important aspect of implementing
an IT investment cited by the Office of Management and Budget 24 and our
IT
investment management guide 25 is evaluating the results of the investment
by determining whether such expected benefits are being achieved. However,
as illustrated in table 1, at this time FSA has only some of the data
necessary to determine whether it is achieving all expected benefits. In
particular, for the increased customer satisfaction and financial
integrity benefits, FSA (1) has not fully defined the performance metrics
to be used, (2) does not have all baseline data, and/ or (3) is not fully
tracking whether the benefits are being achieved. In these cases, FSA COD
officials stated that they were in the process of developing relevant
metrics, which would be tracked to measure the project*s performance
against expected benefits. However, until FSA develops these data and
begins tracking actual benefits and comparing them with expected benefits,
it will lack vital data with which to demonstrate actual investment
results.
23 In addition, as of mid- October 2002, five vendors had implemented and
tested the common record with FSA. Although FSA does not know how many
schools are serviced by these vendors because the vendors consider this
information proprietary, some of the 22 schools that have implemented the
common record use these vendors. 24 Office of Management and Budget
Circular A- 130, Management of Federal Information Resources (Nov. 30,
2000). 25 U. S. General Accounting Office, Information Technology
Investment Management: A Framework for Assessing and Improving Process
Maturity, GAO/ AIMD- 10- 1. 23, Exposure Draft (Washington, D. C.: May
2000). FSA Is Not
Completely Tracking Actual COD Benefits or Lessons Learned Related to
Schools* Implementation of the Common Record
Page 11 GAO- 03- 241 Federal Student Aid Systems Integration Table 1:
Status of FSA Tracking of Actual Benefits of the COD System Expected
benefit Metrics defined? Baseline data available? Benefits tracked?
Reduced cost Yes* The COD business case identifies decreased costs for
systems operations, customer service, and the virtual data center.
Yes Yes a Increased customer satisfaction Partially* The COD business case
identifies
the use of the American Customer Satisfaction Index scores to demonstrate
clarity of instructions, ease of submitting data, and accuracy of records.
However, FSA COD officials stated that at this time, they did not have
approval from the Department of Education to perform this survey;
therefore, they are trying to identify alternative metrics that could be
used to measure customer satisfaction improvements due to COD.
Partially* According to FSA, baseline data related to the American
Customer Satisfaction Survey exist, but other planned metric( s) are not
yet defined.
No Increased employee satisfaction Not applicable* The COD business case
identifies a metric for this expected benefit,
but in commenting on our draft report, FSA stated that it has decided to
discontinue the use of employee satisfaction as a key
performance measure and thus will not be tracking this as an expected
benefit of the COD system.
Not applicable Not applicable Increased financial integrity Partially* FSA
COD officials stated that
they have identified increased school compliance with its requirements
that schools substantiate the amount of funds withdrawn within certain
time frames as a metric. They
also stated that they are currently developing other performance metrics
for increased financial integrity but have not set a target date for
completing them.
Partially* Baseline data related to the identified metric exist, b but
other planned metric( s) are not yet defined.
Partially* Tracking the identified metric only.
Integration and modernization of legacy systems Yes* The COD business case
identifies a
variety of quantitative and qualitative measures, including the retirement
of two legacy applications and the simplification of testing and ongoing
development.
Yes Yes a In providing comments on a draft of this report, FSA*s Chief
Operating Officer stated that after the completion of our review, FSA had
begun tracking this expected benefit against the metrics defined in the
business case. FSA also provided supporting documentation that previously
was not available. We did not validate these new data. b In providing
comments on a draft of this report, FSA*s Chief Operating Officer stated
that estimated
baseline data for the identified metric are available, but FSA is working
to develop more formal baseline data for this performance metric.
Source: GAO analysis on the basis of FSA documentation.
FSA IT officials also stated that they plan to have a contractor conduct a
postimplementation review of the COD basic system in fiscal year 2003,
which is expected to look at the achievement of expected benefits. While
Page 12 GAO- 03- 241 Federal Student Aid Systems Integration this is an
important initiative that could provide FSA with valuable information, it
does not take the place of a continuing and systematic
process of tracking actual benefits. According to our IT investment
management guide, another critical activity is establishing a process for
developing and capturing lessons learned in a written product or knowledge
base and disseminating them to decision- makers. 26 Lessons- learned
mechanisms serve to communicate acquired knowledge more effectively and to
ensure that beneficial information is factored into planning, work
processes, and activities. Lessons learned can be based on positive
experiences that save money or on negative experiences that result in
undesirable outcomes.
FSA has recognized the importance of generating lessons learned in certain
areas. For example, it has implemented a process for developing lessons
related to managing the relationship between the agency and its
prime contractor. However, FSA lacks such a process for capturing or
disseminating lessons related to school migration to the common record.
FSA COD officials stated that lessons learned pertaining to school
migration to the common record are addressed through periodic discussions
during biweekly conference calls with schools undergoing testing with FSA
and during portions of various FSA- sponsored conferences. FSA COD
officials stated that they believed this process for
capturing and disseminating lessons learned was adequate. However, by
relying on such an ad hoc process, FSA lacks assurance that it has
captured and disseminated all key lessons learned related to schools*
implementation of the common record and could overlook important
improvements that could be made. In addition, schools that do not attend
the conferences may not receive and benefit from the lessons identified in
the initial phase of implementation. As a result, schools may encounter
problems that could have been avoided or mitigated had they known of other
schools* experiences. This could hamper FSA*s ability to facilitate the
transition of schools to the new common record and thus the agency*s
ability to fully implement the new COD process and achieve the expected
benefits.
In commenting on a draft of this report, FSA stated that it plans to
provide lessons learned as part of a planned update to its school testing
guide.
26 GAO/ AIMD- 10- 1. 23.
Page 13 GAO- 03- 241 Federal Student Aid Systems Integration While this is
a positive step, it does not replace the need for mechanisms to
continuously capture and disseminate acquired knowledge as schools
implement the common record.
Table 2 includes examples of lessons learned provided by FSA at our
request that were drawn from schools* initial implementation of the common
record for the 2002* 2003 award year. Such information would be important
for the thousands of schools that have not yet implemented the
common record so that they can avoid problems during the common record
implementation and testing processes.
Table 2: Examples of Lessons Learned Related to School Migration to the
Common Record Lesson learned Effect
Several schools and vendors did not have the technical staff to support
the switch to XML. FSA had fewer schools participating in testing than
initially expected. Many schools had only a technical contact or a
business contact, but not both. Instructions FSA provided to schools may
not be translated properly
between the business and technical contacts. Schools did not sufficiently
test their applications before testing them with COD. Schools and vendors
required more support from FSA than
anticipated. Service- level expectations were not well- defined before the
start of testing and were not made available to all parties involved.
Schools had unrealistic expectations of the turnaround time required to
process their records.
Schools should use an XML tool to validate school XMLbased applications
before testing with the COD system. Schools that did not participate in
such validations had more errors
and a greater chance of their records being rejected upon submission to
the COD system.
Source: FSA.
FSA has taken important steps toward achieving full implementation of the
new COD process. However, critical actions, such as completing the basic
functionality of the COD system and the implementation of the
common record at thousands of affected schools, must still be undertaken.
In addition, FSA has not yet fully established the metrics and processes
to track actual benefits related to all of its expected benefits or the
lessons that have been generated by the few schools that have implemented
the common record thus far. By not tracking actual benefits, FSA lacks
information that is critical to determining whether it is meeting all of
its goals. Further, not capturing and disseminating information to schools
regarding lessons learned could make achieving these goals more difficult.
To determine the extent to which the new COD process is achieving expected
results related to customer satisfaction and financial integrity, we
recommend that you direct FSA*s Chief Operating Officer to expeditiously
develop metrics and baseline data to measure these benefits Conclusions
Recommendations
Page 14 GAO- 03- 241 Federal Student Aid Systems Integration and develop a
tracking process to assess the extent to which the expected results are
being achieved.
To ensure that the schools that have not yet implemented the common record
benefit from the experiences of those that have, we recommend that you
direct FSA*s Chief Operating Officer to establish a process for capturing
lessons learned in a written product or knowledge base and for
disseminating them to these schools.
In providing written comments on our draft report, FSA*s Chief Operating
Officer provided technical comments and updated information, but did not
comment on our recommendations. Specifically, The Chief Operating
Officer did not believe the report adequately
portrayed the level of COD progress that had been made. In particular, she
took issue with our using the completion of 75 percent of COD requirements
as an indication of progress. Although the Chief Operating Officer did not
disagree with the accuracy of this figure, she stated that FSA*s informal
analysis indicated that between 85 to 90 percent of COD functions had been
implemented, which she believed was a better gauge of progress. We believe
that we have accurately portrayed FSA*s progress in implementing the COD
process. First, since FSA*s analysis was *informal,* and FSA*s supporting
documentation had limited detail that we could not validate, we do not
agree that this should be the primary basis for an analysis of COD*s
progress. Second, we included both the percentage of COD*s requirements
that had been implemented and FSA*s estimate in our report. Nevertheless,
we modified our report to include additional data provided by FSA
regarding the number of transactions processed by the COD system to
further indicate progress.
The Chief Operating Officer agreed that the tracking of all of the
expected benefits is not in place at this time, but stated that work is
under way in this area. FSA also provided updated information and
supporting documentation related to the tracking of some of the expected
benefits. We made changes to the report reflecting this new information,
as appropriate. The Chief Operating Officer agreed that it is important
and beneficial to
communicate lessons learned, but stated that FSA*s informal method for
communicating lessons related to school migration to the common record
worked well in the first year of COD implementation. FSA also noted that
it plans to include lessons learned in a planned update to its school
testing Agency Comments
and Our Evaluation
Page 15 GAO- 03- 241 Federal Student Aid Systems Integration guide. We
modified the report to reflect this initiative, but we do not agree that
FSA*s informal method or its plan to include lessons learned in its
testing guide is adequate because these approaches do not provide a
continuous process for actively capturing and disseminating lessons
learned. As a result, some important lessons may be overlooked, and all
schools may not be aware of potential problems associated with
implementing the common record.
FSA*s written comments, along with our responses, are reproduced in
appendix II. We are sending copies of this report to the appropriate
congressional
committees, the Secretary of Education, the Chief Operating Officer of
Education*s Office of Federal Student Aid, and the Director of the Office
of Management and Budget. This report will also be available at no charge
on the GAO Web site at http:// www. gao. gov.
If you have any questions on matters discussed in this report, please
contact me at (202) 512- 9286 or Linda J. Lambert, Assistant Director, at
(202) 512- 9556. We can also be reached by E- mail at pownerd@ gao. gov
and lambertl@ gao. gov, respectively. Other individuals making key
contributions to this report included Jason B. Bakelar and Anh Q. Le.
Sincerely yours, David A. Powner Director (Acting), Information Technology
Management Issues
Appendix I: Systems and Technologies Supporting the Common Origination and
Disbursement Process, as of November 2002
Page 16 GAO- 03- 241 Federal Student Aid Systems Integration a This is a
temporary interface. DLOS is targeted to be retired in fiscal year 2003. b
MQSeries is IBM*s proprietary message format. Appendix I: Systems and
Technologies Supporting the Common Origination and
Disbursement Process, as of November 2002
COD system DLOS CPS DLSS LOWeb
system FMS NSLDS PEPS FTP files a
XML- based common record
(MQSeries) b XML- based common record
(MQSeries) Middleware Middleware Middleware Middleware Middleware
Middleware Middleware
Middleware Legacy records (MQSeries)
Enterprise Application Integration "Bus" c Student Aid Internet Gateway
Internet Single common record file (XML)
18 legacy files (secured FTP)
About 5,500 postsecondary
schools 22
postsecondary schools
Legacy records (MQSeries)
Legend: COD Common Origination and Disbursement CPS Central Processing
System DLOS Direct Loan Origination System DLSS Direct Loan Servicing
System FMS Financial Management System
FTP File Transfer Protocol LOWeb Loan Origination Web NSLDS National
Student Loan Data System PEPS Postsecondary Education Participants System
XML Extensible Markup Language
Source: GAO analysis of FSA documents.
Appendix I: Systems and Technologies Supporting the Common Origination and
Disbursement Process, as of November 2002
Page 17 GAO- 03- 241 Federal Student Aid Systems Integration C This is an
information technology infrastructure that enables data exchange among
disparate systems.
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 18 GAO- 03- 241 Federal Student Aid Systems Integration Appendix II:
Comments from the Department of Education*s Office of Federal Student Aid
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 3. See comment 2. See comment 1.
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 19 GAO- 03- 241 Federal Student Aid Systems Integration See comment
5. See comment 4.
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 20 GAO- 03- 241 Federal Student Aid Systems Integration See comment
6.
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 21 GAO- 03- 241 Federal Student Aid Systems Integration See comment
8. See comment 7.
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 22 GAO- 03- 241 Federal Student Aid Systems Integration The following
are GAO*s comments on the Department of Education*s Office of Federal
Student Aid*s letter dated December 18, 2002.
1. We revised the draft report title to clarify that our follow- up review
was focused on assessing FSA*s progress in implementing the COD process.
2. Information related to operational problems was contained in the draft
report. We asked the COD Contracting Officer*s Representative to
characterize these problems, and he stated that they were very serious. In
addition, we confirmed the seriousness of these problems at the conclusion
of our review with FSA IT and program officials.
3. We modified this report to include data on the number of transactions
processed. We also modified our report to clarify that all schools
participating in the Pell Grant and/ or Direct Loan programs currently use
the COD system.
4. We do not agree that the report implies that FSA*s use of a phased- in
approach in implementing the common record increases risks. Instead, this
report notes that the implementation is not yet complete. 5. We believe
that we have accurately portrayed FSA*s progress in
implementing the COD process. First, since FSA*s analysis was *informal,*
and FSA*s supporting documentation had limited detail that we could not
validate, we do not agree that the COD Development Manager*s functionality
estimate should be the primary basis for an analysis of COD*s progress.
Second, we included both the percentage of COD*s requirements that had
been implemented and FSA*s estimate in our report.
6. We modified our report to reflect this updated information as
appropriate. 7. We do not agree that FSA*s informal process for capturing
and
disseminating lessons learned was adequate because (1) it may lead to
important lessons being overlooked and (2) all schools may not be aware of
potential problems associated with implementing the common record.
8. We modified this report to reflect that FSA plans to include lessons
learned in a planned update to its school testing guide. While this is a
GAO Comments
Appendix II: Comments from the Department of Education*s Office of Federal
Student Aid
Page 23 GAO- 03- 241 Federal Student Aid Systems Integration positive
step, it does not replace the need for mechanisms to continuously capture
and disseminate acquired knowledge as schools
implement the common record.
(310442)
The General Accounting Office, the investigative arm of Congress, exists
to support Congress in meeting its constitutional responsibilities and to
help improve the performance and accountability of the federal government
for the American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations, and
other assistance to help Congress make informed oversight, policy, and
funding decisions. GAO*s commitment to good government is reflected in its
core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail
this list to you every afternoon, go to www. gao. gov and select
*Subscribe to daily E- mail alert for newly released products* under the
GAO Reports heading.
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to: U. S. General Accounting Office 441 G Street NW, Room LM
Washington, D. C. 20548 To order by Phone: Voice: (202) 512- 6000
TDD: (202) 512- 2537 Fax: (202) 512- 6061
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470 Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512-
4800
U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548 GAO*s Mission Obtaining Copies of
GAO Reports and Testimony
Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs Public Affairs
*** End of document. ***