Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity (01-MAR-04,
GAO-04-394G).
In 2000, GAO published an exposure draft of Information
Technology Investment Management: A Framework for Assessing and
Improving Process Maturity (ITIM). Built around the
select/control/evaluate approach described in the Clinger-Cohen
Act of 1996--which establishes statutory requirements for IT
management--the framework provides a method for evaluating and
assessing how well an agency is selecting and managing its IT
resources. The exposure draft reflected current accepted or best
practices in IT investment management, as well as the reported
experience of federal agencies and other organizations in
creating their own investment management processes. This new
version updates the exposure draft to take into account comments
that GAO has received; GAO's experiences in evaluating several
agencies' implementations of investment management processes and
the lessons learned by these agencies; and the importance of
enterprise architecture (EA) as a critical frame of reference in
making IT investment decisions. Using the framework to analyze an
agency's IT investment management processes provides: (1) a
rigorous, standardized tool for internal and external evaluations
of these processes; (2) a consistent and understandable mechanism
for reporting the results of assessments; and (3) a road map that
agencies can follow in improving their processes.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-394G
ACCNO: A09396
TITLE: Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity
DATE: 03/01/2004
SUBJECT: Best practices
Best practices methodology
Cost effectiveness analysis
Information resources management
Information technology
Procurement evaluation
Procurement planning
Strategic information systems planning
Enterprise architecture
Maturity model
GAO Information Technology Investment
Management Framework
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-394G
United States General Accounting Office
GAO Executive Guide
March 2004
Version 1.1 INFORMATION
TECHNOLOGY INVESTMENT MANAGEMENT
A Framework for Assessing and Improving Process Maturity
a
GAO-04-394G
Highlights of GAO-04-394G, an executive guide.
In 2000, GAO published an exposure draft of Information Technology
Investment Management: A Framework for Assessing and Improving Process
Maturity (ITIM). Built around the select/control/evaluate approach
described in the Clinger-Cohen Act of 1996-which establishes statutory
requirements for IT management-the framework provides a method for
evaluating and assessing how well an agency is selecting and managing its
IT resources. The exposure draft reflected current accepted or best
practices in IT investment management, as well as the reported experience
of federal agencies and other organizations in creating their own
investment management processes. This new version updates the exposure
draft to take into account comments that GAO has received; GAO's
experiences in evaluating several agencies' implementations of investment
management processes and the lessons learned by these agencies; and the
importance of enterprise architecture (EA) as a critical frame of
reference in making IT investment decisions. Using the framework to
analyze an agency's IT investment management processes provides: (1) a
rigorous, standardized tool for internal and external evaluations of these
processes; (2) a consistent and understandable mechanism for reporting the
results of assessments; and (3) a road map that agencies can follow in
improving their processes.
www.gao.gov/cgi-bin/getrpt?GAO-04-394G.
To view the full product, click on the link above. For more information,
contact David Powner, 202-512-4299, [email protected]., or Lester Diamond,
202512-7957, [email protected].
March 2004
INFORMATION TECHNOLOGY INVESTMENT MANAGEMENT
A Framework for Assessing and Improving Process Maturity
The ITIM framework is a maturity model composed of five progressive stages
of maturity that an agency can achieve in its IT investment management
capabilities. These maturity stages are cumulative; that is, in order to
attain a higher stage of maturity, the agency must have institutionalized
all of the requirements for that stage in addition to those for all of the
lower stages. The framework can be used both to assess the maturity of an
agency's investment management processes and as a tool for organizational
improvement. For each maturity stage, the ITIM describes a set of critical
processes that must be in place for the agency to achieve that stage. The
figure below shows the five stages and lists the critical processes for
each stage.
At the Stage 1 level of maturity, an agency is selecting investments in an
unstructured, ad hoc manner. Project outcomes are unpredictable and
successes are not repeatable; the agency is creating awareness of the
investment process. Stage 2 critical processes lay the foundation for
sound IT investment processes by helping the agency to attain successful,
predictable, and repeatable investment control processes at the project
level. Stage 3 represents a major step forward in maturity, in which the
agency moves from project-centric processes to a portfolio approach,
evaluating potential investments by how well they support the agency's
missions, strategies, and goals. At Stage 4, an agency uses evaluation
techniques to improve its IT investment processes and its investment
portfolio. It is able to plan and implement the "de-selection" of
obsolete, high-risk, or low-value IT investments. The most advanced
organizations, operating at Stage 5 maturity, benchmark their IT
investment processes relative to other "best-inclass" organizations and
look for breakthrough information technologies that will enable them to
change and improve their business performance.
The ITIM Stages of Maturity with Critical Processes
Contents
Preface 1
Section 1: Introduction 5
Changes from the Exposure Draft 6
Investment Management Overview 7
Section 2: Overview of 11
The Stages of Maturity 11
ITIM Progressing through the Stages of Maturity 15
Section 3: Components 21
of ITIM ITIM Hierarchy 21
Section 4: Uses of ITIM 23
Principles Guiding the Use and Interpretation of the Framework 23
Tool for Organizational Improvement 24
Tool for Assessing the Maturity of an Organization 26
Limitations and Boundaries 27
Section 5: Critical 29
Stage 1: Creating Investment Awareness 30
Processes for the ITIM Stage 2: Building the Investment Foundation 33
Stages Stage 3: Developing a Complete Investment Portfolio 63
Stage 4: Improving the Investment Process 90
Stage 5: Leveraging Information Technology for Strategic
Outcomes 102
Appendixes
Appendix I: Glossary 113
Appendix II: Conducting an ITIM Assessment 121
Using ITIM to Assess IT Investment Decision-Making
Processes 121
Summary of ITIM Assessment Process 122
Appendix III: Acknowledgments 135
Contents
Figures Figure 1: Figure 2: Figure 3:
Figure 4: Figure 5: Figure 6:
Figure 7:
Figure 8: Figure 9:
Fundamental Phases of the IT Investment Approach 8
The Five Stages of Maturity Within ITIM 11
Critical Maturation Steps Required to Move to the Next
Stage 16
The Components of an ITIM Critical Process 22
The ITIM Stages of Maturity with Critical Processes 29
The ITIM Stages of Maturity with No Stage 1 Critical
Processes 30
The ITIM Stages of Maturity with Stage 2 Critical
Processes 33
Instituting the Investment Board 35
Meeting Business Needs 41
Figure 10: Selecting an Investment 46 Figure 11: Providing Investment
Oversight 51 Figure 12: Capturing Investment Information 58 Figure 13: The
ITIM Stages of Maturity with Stage 3 Critical
Processes 63 Figure 14: Defining the Portfolio Criteria 65 Figure 15:
Creating the Portfolio 71 Figure 16: Evaluating the Portfolio 78 Figure
17: Conducting Postimplementation Reviews 84 Figure 18: The ITIM Stages of
Maturity With Stage 4 Critical
Processes 90 Figure 19: Improving the Portfolio's Performance 92 Figure
20: Managing the Succession of Information Systems 97 Figure 21: The ITIM
Stages of Maturity with Stage 5 Critical
Processes 102 Figure 22: Optimizing the Investment Process 104 Figure 23:
Using IT to Drive Strategic Business Change 109 Figure 24: Phases in an
ITIM Assessment 122
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
Preface
Investments in information technology (IT) can enrich people's lives and
improve organizational performance. For example, during the last decade
the Internet has matured from being a means for academics and scientists
to communicate with each other to a national resource where citizens can
interact with their government in many ways, for example, by receiving
services, supplying and obtaining information, asking questions, and
providing comments on proposed rules. Although they have the potential to
improve lives and organizations, IT projects can also become risky,
costly, unproductive mistakes. As we have described in numerous reports
and testimonies, federal IT projects too frequently incur cost overruns
and schedule slippages while contributing little to mission-related
outcomes.
The Paperwork Reduction Act (PRA)1 requires federal agencies to be
accountable for their IT investments and responsible for maximizing the
value and managing the risks of their major information systems
initiatives. The Clinger-Cohen Act of 19962 establishes a more definitive
framework for implementing the PRA's requirements for IT investment
management. It requires federal agencies to focus more on the results they
have achieved through IT investments, while concurrently improving their
IT acquisition processes. The Clinger-Cohen Act3 also introduces more
rigor and structure into how agencies are to select and manage IT
projects. Among other things, it lays out specific aspects of the process
that agency heads are to implement in order to maximize the value of the
agency's IT investments and assess, manage, and evaluate the risks of its
IT acquisitions.4 The E-Government Act of 20025 provides additional
guidance on IT management practices across federal agencies.
Through our research into IT management best practices and our evaluation
of agency IT management performance, we have identified a set of essential
and complementary management disciplines. These include
144 U.S.C. S: 3506(h).
2The fiscal year 1997 Omnibus Consolidated Appropriations Act, Pub. L.
104-208, renamed both Division D (the Federal Acquisition Reform Act) and
E (the Information Technology Management Reform Act) of the 1996 DOD
Authorization Act, Pub. L. 104-106, as the Clinger-Cohen Act of 1996.
340 U.S.C. S:S: 11312-11313.
444 U.S.C. S: 3506(h)(5); 40 U.S.C. S:S: 11312-11313.
5E-Government Act of 2002, Public Law 107-347 (Dec. 17, 2002).
Preface
o investment management,
o strategic planning,
o software/system development and acquisition management,
o IT services acquisition management,
o human capital management,
o information security management, and
o enterprise architecture management.
Using the results of this research and evaluation, we have developed
various management frameworks and guides. In 1997 we developed guidance,6
based primarily on the Clinger-Cohen Act, that provides a method for
evaluating and assessing how well a federal agency is selecting and
managing its IT resources. This guidance also identifies specific areas
where improvements can be made. The Information Technology Investment
Management (ITIM) Framework enhances this guidance by identifying critical
processes for successful investment and organizing these processes into a
framework of increasingly mature stages.
Maturity models have been proven to be a highly effective evaluative
technique for the Software Engineering Institute, which is well regarded
for its collection of Capability Maturity Models SM (e.g., Capability
Maturity Model for Software).7,8 Other researchers have proposed similar
approaches based on maturity models. 9
6U.S. General Accounting Office, Assessing Risk and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision-making,
GAO-AIMD-10.1.13 (Washington, D.C.: February 1997).
7Capability Maturity Model is a service mark of Carnegie Mellon
University.
8M. Paulk et al., Capability Maturity Model for Software (Version 1.1),
SEI-93-TR-024.
9Giga Information Group, Inc., Total Economic Impact, Part 2: Defining and
Measuring IT Value (P-1297-009).
Preface
The maturity framework approach generally
o offers a comprehensive model for assessing process capability within an
organization;
o can be applied to multiple types of disciplines, such as IT asset
acquisition, human capital, and systems engineering; and
o can serve as a valuable tool for organizations to use to improve their
technical development and management processes.
The initial ITIM exposure draft that we issued in May 200010 reflected
both a maturation of thinking in the area of IT investment management and
input we had received from organizations and federal agencies based on
their experiences in creating their own investment mechanisms and
processes. This updated version has been modified based on comments we
received on the initial exposure draft and on our experiences in
evaluating and learning from agencies that are implementing investment
management processes. Moreover, this version of the ITIM is consistent
with and supports other maturity frameworks, including GAO's Enterprise
Architecture Management Maturity Framework (EAMMF).11 Among other things,
this version of the ITIM addresses the importance of an enterprise
architecture (EA) as a critical frame of reference for organizations when
they are making IT investment decisions.
The ITIM can be used to analyze an organization's investment management
processes and to determine its level of maturity. Since its release in
exposure draft in May, 2000, the ITIM has been GAO's primary tool for
evaluating investment management capabilities. In addition, a number of
agencies have used the framework as they worked to improve their
investment processes.
If you have any questions about the Information Technology Investment
Management Framework or the IT investment management approach,
10U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
Exposure Draft, GAO-AIMD10.1.23 (Washington, D.C.: May 2000).
11U.S. General Accounting Office, Information Technology: A Framework for
Assessing and Improving Enterprise Architecture Management (Version 1.1),
GAO-03-584G (Washington, D.C.: April 2003).
Preface
please contact me at (202) 512-4299 or [email protected]; or Lester
Diamond, Assistant Director at (202) 512-7957 or [email protected]. Other
key contributors to this report were Joanne Fiorino, Sabine R. Paul,
Tomas Ramirez, Thomas Wright, and Neil Doherty.
David A. Powner
Director, Information Technology Management Issues
Section 1: Introduction
The Information Technology Investment Management Framework identifies-and
organizes into a framework of increasingly mature stages- thirteen
processes that are critical for successful investment. The original
exposure draft of ITIM expanded the widely accepted federal management
framework for IT investment decision making that was embodied in OMB and
GAO guidance12 and shifted the content from a guidance-based focus to an
activity- and maturity-based focus. Such a maturity framework can be used
either to analyze an organization's investment management process or to
determine the maturity of its investment process. The framework provides
three key capabilities that are of use to many federal agencies: (1) a
rigorous, standardized tool for internal and external evaluations of an
agency's IT investment management process; (2) a consistent and
comprehensible mechanism for reporting the results of these assessments to
agency executives, the Congress, and other interested parties; and (3) a
road map that agencies can use for improving their investment management
processes. It should be noted, however, that an organization's achievement
of more mature investment management stages depends on its instituting
other good management practices and attributes, such as strategic
planning, project management, enterprise architecture (EA) management,
human capital management, and software and system acquisition management.
In May 2000 we released an exposure draft of the ITIM framework for trial
and comment. Since that time, the framework has been used by a number of
federal agencies in developing and enhancing their investment management
strategies. In addition, we have used it to evaluate several
12Evaluating Information Technology Investments, A Practical Guide,
Executive Office of the President, Office of Management and Budget,
November 1995, and U.S. General Accounting Office, Assessing Risks and
Returns: A Guide for Evaluating Federal Agencies' IT Investment
Decision-making, GAO/AIMD-10.1.13 (Washington D.C.: February 1997).
Section 1: Introduction
agencies.13 This release includes lessons learned from our use of the
framework in these evaluations and from lessons conveyed to us by users of
the framework at a number of agencies. In order to validate the
appropriateness of our changes and to gain the advantage of their
experience, we provided this release for review to several outside experts
who are familiar with the ITIM exposure draft and with investment
management in a broad array of organizations, both public and private.
This version also includes a much fuller description of the relationship
between ITIM and EA. Based on our experience, employing ITIM and EA in
concert can greatly increase the chances that an organization's
operational and IT environments will be pursued in a way that optimizes
mission performance. The EA provides a clear and comprehensive picture of
the structure of an entity, whether it is an organization or a functional
or mission area. It defines an organization's operations in logical (i.e.,
information flows) as well as technical terms (i.e., hardware and
software). The EA also describes these perspectives both for the
organization's current or "as-is" environment and for its target or
"to-be" environment as well as for a transition or sequencing plan for
moving from the "as-is" to the "to-be" environment.
Changes from the Exposure Draft
Stage 2 has been the primary beneficiary of the lessons learned from the
use of the framework, because most agencies that we have evaluated are
still operating at Stage 2. In Stage 2 we have tried to clarify aspects of
critical processes that previously have led to diverse interpretations. In
addition, we have moved what was previously the critical process for
Authority Alignment of IT Investment Boards from Stage 3 in the exposure
draft into Stage 2 in this release; it is now part of the critical process
for Instituting the Investment Board. Through our work, we have found that
13U.S. General Accounting Office, Information Technology: INS Needs to
Strengthen Its Investment Management Capability, GAO-01-146 (Washington,
D.C.: Dec. 29, 2000); Information Technology: DLA Needs to Strengthen Its
Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar. 15,
2002); United States Postal Service: Opportunities to Strengthen IT
Investment Management Capabilities, GAO-03-3 (Washington D.C.: Oct. 15,
2002); U.S. General Accounting Office, Bureau of Land Management: Plan
Needed to Sustain Progress in Establishing IT Investment Management
Capabilities, GAO-03-1025 (Washington, D.C.: Sept. 12, 2003); U.S. General
Accounting Office, Information Technology: Departmental Leadership Crucial
to Success of Investment Reforms at Interior, GAO-03-1028 (Washington,
D.C.: Sept. 12, 2003).
Section 1: Introduction
instituting multiple boards was not unusual for organizations working in
Stage 2 and that these boards occasionally were not well aligned.
Stage 3 has been enhanced to better explain the organization and use of
portfolio management for investments. In this area we gained knowledge
from the experiences of others, both directly from individuals using IT
portfolio management in agencies as well as from literature that has been
released during the last few years. In addition, we moved the critical
process for Postimplementation Review and Feedback from Stage 4 in the
exposure draft to Stage 3 in this release. We did this so we could ensure
that organizations that have completed Stage 3 are meeting the requirement
for having selection, control, and evaluation processes in place, as
required by the Clinger-Cohen Act.
Stages 4 and 5 have been modified only to reflect new names for critical
processes and to relocate to Stage 3 the critical process for
Postimplementation Review and Feedback. We have not gained substantial new
experience in these stages, because few organizations are operating at
these levels of maturity. We anticipate modifying these stages in the
future, when we have learned more from organizations' experiences.
Investment A central tenet of the federal approach to IT investment
management has been the select/control/evaluate model. This model was
initially identified
Management Overview in our Strategic Information Management (SIM)
Executive Guide,14 expanded in the Office of Management and Budget's IT
investment guidance,15 and then refined in our subsequent guidance.16 It
provides a systematic method for agencies to minimize risks while
maximizing the returns of investments. Figure 1 illustrates the central
components of this model.
14U.S. General Accounting Office, Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology,
GAO/AIMD-94-115 (Washington, D.C.: May 1994).
15Evaluating Information Technology Investments, A Practical Guide,
Executive Office of the President, Office of Management and Budget,
November 1995.
16U.S. General Accounting Office, Assessing Risks and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision-making,
GAO/AIMD-10.1.13 (Washington, D.C.: February 1997).
Section 1: Introduction
Figure 1: Fundamental Phases of the IT Investment Approach
Source: GAO.
During the select phase the organization (1) identifies and analyzes each
project's risks and returns before committing significant funds to any
project and (2) selects those IT projects that will best support its
mission needs. This process should be repeated each time funds are
allocated to projects, reselecting even ongoing investments as described
below.
During the control phase the organization ensures that, as projects
develop and investment expenditures continue, the project continues to
meet mission needs at the expected levels of cost and risk. If the project
is not meeting expectations or if problems have arisen, steps are quickly
taken to address the deficiencies. If mission needs have changed, the
organization is able to adjust its objectives for the project and
appropriately modify expected project outcomes.
During the evaluate phase, actual versus expected results are compared
after a project has been fully implemented. This is done to (1) assess the
project's impact on mission performance, (2) identify any changes or
modifications to the project that may be needed, and (3) revise the
investment management process based on lessons learned.
The investment process does not end with the evaluation phase. A project
can be active concurrently in more than one phase of the
select/control/evaluate model. After a project has been designated for
Section 1: Introduction
initial funding in the select phase, it becomes the subject of evaluation
throughout the control phase for the purposes of reselection. Reselection
is an ongoing process that continues for as long as a project is receiving
funding. If a project is not meeting the goals and objectives that were
originally established when it was selected, or if the goals have been
modified to reflect changes in mission objectives-and corrective actions
are not succeeding-a decision must be made on whether to continue to fund
the project. Ultimately, "deselection" can be one of the most difficult
steps to implement, but it is necessary if funds can be better utilized
elsewhere. Once projects are operating and being maintained, they remain
under constant review for reselection.
Section 1: Introduction
Section 2: Overview of ITIM
The Stages of Maturity ITIM is comprised of five stages of maturity. Each
stage builds upon the lower stages and enhances the organization's ability
to manage its IT investments. Figure 2 shows the five ITIM stages and
gives a brief description of each stage.
Figure 2: The Five Stages of Maturity Within ITIM
Source: GAO.
Stage 1: Creating Investment Awareness
Stage 1 is characterized by ad hoc, unstructured, and unpredictable
investment processes. For example, in a Stage 1 organization, there is
generally little relationship between the success or failure of one
project and the success or failure of another project. If an IT project
succeeds and is seen as a good investment, it is largely due to
exceptional actions on the part of the project team, and thus its success
might be difficult to repeat. Investment processes that are important for
success may be known, but only to isolated teams; this process knowledge
is not widely shared or institutionalized.
Most organizations with Stage 1 maturity have some type of project
selection process in place as part of their annual budgeting activity.
Section 2: Overview of ITIM
However, the selection process is frequently rudimentary, poorly
documented, and inconsistently applied.
The unstructured and unpredictable investment processes that characterize
a Stage 1 organization also mean that even if it recognizes that a given
project is in trouble, it may not have adequate processes to consistently
address and resolve the project's problems. Additionally, a focus on
project results in terms of business benefits is often missing in these
organizations.
Stage 2: Building the Investment Foundation
One focus of Stage 2 maturity is to establish basic selection
capabilities. Basic selection capabilities are driven by the development
of project selection criteria, including benefit and risk criteria, and an
awareness of organizational priorities when identifying projects for
funding. No longer are projects being funded solely on an ad hoc basis.
The basic selection processes established in Stage 2 lay the foundation
for more mature selection capabilities in Stage 3. Therefore, the
organization also focuses on defining and developing its IT investment
board(s), identifying the business needs or opportunities to be addressed
by each IT project, and using this knowledge in the selection of new IT
proposals.
An organization working to complete Stage 2 should be starting to develop
an ITIM decision-making process that utilizes its EA-to the extent that an
EA exists. An organization's "as-is" architecture may provide some of the
basic information that is needed by decision makers, such as what systems
currently exist and what potential functional overlap may occur with a new
investment. In addition, an organization's EA tool may serve as a
repository for investment information, although this may require modifying
the manner in which the tool is currently being used. Criteria for
selecting new and ongoing investments should be established, and the
requirement to comply with the target EA may serve as an important guide
in investment decisions. In addition, to gain further confidence that each
investment is providing specific value to the organization, an
organization's policies and procedures should provide for identifying the
business needs and the associated users of each IT project.
An equally important focus is to attain repeatable, successful IT
investment control techniques at the project level. For an organization to
develop a sound IT investment process, it must first be able to control
its investments so that they finish predictably within established
schedule and budget ranges. In addition, it must be able to identify
potential exposures to risk
Section 2: Overview of ITIM
and put in place strategies to mitigate that risk. In the absence of
predictable, repeatable, and reliable investment control processes,
selected investments will be subject to a higher risk of failure despite
rigorous analysis of the estimates used to justify them. Further, the
absence of repeatable control processes will result in ineffective
evaluation processes and contradictory efforts at process improvement.
To ultimately succeed, most IT investments require a relentless focus on
interim results and successful risk management strategies, among other
things. Taking this into account, an organization can begin by (1)
focusing on gaining control of its existing collection of projects and (2)
following a disciplined process for improving project outcomes over time
by regularly tracking and overseeing each project's cost and schedule
milestones and by monitoring expected benefits and risks. Supporting these
activities requires collecting investment information to ensure that the
organization knows fundamental facts about its IT assets, such as their
location, cost, and ownership.
Stage 3: Developing a Complete Investment Portfolio
Stage 3 critical processes depend specifically on the successful
implementation of Stage 2 critical processes. In order to operate
successfully at Stage 3, the organization must have in place the structure
and repeatability of the project-centric management processes described
above. In addition, the project-specific performance data being used for
oversight and reselection in Stage 2 are crucial for the successful
management of the investment portfolio. The critical focus for Stage 3
maturation is to establish a consistent, well-defined perspective on the
IT investment portfolio and to maintain mature, integrated selection (and
reselection), control, and evaluation processes. These processes will be
evaluated during postimplementation reviews (PIR). Once IT projects have
been selected and are meeting their scheduled performance expectations- as
outlined in Stage 2-the organization needs to develop an IT investment
portfolio using an investment process that is consistent with its EA and
employs sound selection criteria.
The development and use of portfolio selection criteria enable the
organization to expand its focus from being primarily project-oriented to
including the broader portfolio perspective. The portfolio perspective
drives the organization to focus on the benefits gained from the synergies
to be found among the investments in the entire collection, rather than
just from the sum of the individual investments. Instead of focusing
exclusively on the balance between the costs and benefits of individual
investments, in
Section 2: Overview of ITIM
Stage 3 decision makers also must consider the interaction among
investments and the contribution to organizational mission goals and
strategies that could be made by alternative portfolio selections. The
development of the portfolio selection criteria communicates
organizational priorities to the IT project management community and
ensures that each investment submitted for funding supports the
organization's mission, strategies, and goals, as well as project-specific
outcomes. The critical process for Creating the Portfolio describes how
the organization should use the portfolio selection criteria to develop an
IT investment portfolio. Individual investments are reviewed and evaluated
following their implementation in order to compare actual results with
performance expectations.
An organization's policies and procedures should provide for specifying
the relationship between its architecture and its investment
decision-making authority. The links between the EA and the investment
portfolio should be explicitly defined. In addition, when operating at
this stage, organizations should be working to align their EA with their
IT portfolio selection criteria.
Stage 4: Improving the Investment Process
An organization at Stage 4 maturity is focused on using evaluation
techniques to improve its IT investment processes and portfolio(s) while
maintaining mature control and selection processes. At this stage, the
organization should also regularly analyze its investment portfolio(s) to
ensure that its investments continue to be aligned with the most current
version of its architecture, since small changes in either an investment
itself or in the EA may have occurred over time without being recognized
in periodic selection/reselection decisions. As described in Stage 3,
postimplementation reviews typically identify lessons learned from an
investment and determine whether the benefits anticipated in the business
case for the investment have been achieved. Analyzing a number of PIRs
serves as a basis for creating recommendations for changing and improving
IT investment processes.
Section 2: Overview of ITIM
Portfolio categories are used to organize the lessons learned and the
recommendations gleaned both from PIRs conducted during Stage 3 and from
other sources of process or investment information. The information within
these categories is then used to fine-tune the investment processes and
the portfolios. Additionally, at Stage 4 maturity the organization has the
capacity to conduct IT succession activities and thus can plan and
implement the "deselection" of obsolete, high-risk, or low-value IT
investments.
Stage 5: Leveraging Information Technology for Strategic Outcomes
Once an organization has mastered the selection, control, and evaluation
processes, it seeks to shape its strategic outcomes by (1) using its EA as
a critical frame of reference to ensure alignment with the target
architecture, (2) learning from other organizations, (3) continuously
improving the manner in which it uses IT to support and improve its
business outcomes, and (4) focusing on flexibility and becoming a more
agile organization that relies on its architecture for its vision of the
future and the ITIM as a critical means for implementing it. Thus, an
organization with Stage 5 maturity benchmarks its IT investment processes
relative to other "best-in-class" organizations and conducts proactive
monitoring for breakthrough information technologies that will allow it to
significantly change and improve its business performance.
Progressing through Within ITIM, lower maturity stages provide the
foundation for higher maturity stages. Thus, an organization increases its
IT investment maturity
the Stages of Maturity and management capability as it progresses through
the ITIM maturity stages. The following section describes the critical
maturation steps that occur as an organization moves from one stage to the
next (see fig. 3).
Section 2: Overview of ITIM
Figure 3: Critical Maturation Steps Required to Move to the Next Stage
Source: GAO.
Moving from Stage 1 to Stage 2
Investment control processes are the essential proficiencies that an
organization establishes as it moves from ITIM Stage 1 to Stage 2. As
investment control processes become better established,
o one or more investment board(s) is created to oversee and select IT
projects;
o investment information such as costs, benefits, schedule, risk
assessments, performance metrics, and system functionality is collected to
support executive decision making;
o the organization gains a better perspective on the IT projects in which
it is investing;
Section 2: Overview of ITIM
o communicating the status of ongoing projects improves organizationwide
system acquisition, development, and management practices;
o the organization creates and maintains better project-level cost
information; and
o key customers (or end users) and business needs for each IT project are
identified, and the users are engaged in this process.
Critical to maturing project-level IT investment control processes is the
ability to recognize the need for and to take swift corrective action when
a project is having trouble meeting its schedule expectations and cost
estimates. As it moves through Stage 2, an organization develops robust
methods to collect data from the project-level management processes and
aggregate it appropriately to provide executive management with the
information it needs to execute its oversight responsibilities. As the
organization matures, it also learns from past decisions and better
manages the causal factors that created past problems, thus improving the
performance results of ongoing projects.
Beyond investment control processes, the organization also begins to
implement basic selection processes. The core business needs for each IT
project are identified and the basic portfolio development processes are
used to select new IT proposals.
Moving from Stage 2 to Stage 3
Creation of a mature IT process for selecting investments is the major
accomplishment that an organization demonstrates as it moves from Stage 2
to Stage 3 maturity. In addition, well-developed investment control
processes lead to greater certainty about future IT investment outcomes
and greater confidence that IT investments, when they are selected, will
achieve their expected cost, schedule, and performance goals, as well as
their expected functionality. Thus, once the investment control processes
have been established, an organization can build on these fundamental
investment processes to create mature portfolio selection processes.
Mature selection processes include
o the creation and maintenance of portfolio selection criteria,
o the analysis associated with examining the merits of each IT investment
in the context of the portfolio,
Section 2: Overview of ITIM
o the use of an EA to help align IT investments with strategic
objectives, and
o the grouping of similar investments together and the development of the
portfolio.
Beyond the creation of a mature selection process, the organization now
refines the elements of benefit and risk management in its investment
control process, because it has installed the supporting tools for doing
so as its selection process matures. Individual investments are reviewed
and evaluated following their implementation and are judged based on how
well they meet their performance expectations.
Moving from Stage 3 to Stage 4
As an organization reaches Stage 4 maturity, it has created mature IT
investment evaluation processes and established a complete IT investment
management process. In this stable environment, the organization can take
the lessons it has learned from evaluating its investment processes (i.e.,
based on postimplementation reviews in Stage 3) and change these processes
with predictably beneficial results. By doing so, it also creates the
environment and the mechanisms for continuous improvement in Stage 5. In
addition to improving its investment processes, an organization operating
in Stage 4 can manage resource succession-that is, "deselecting" current
IT investments-by migrating to successor investments or retiring obsolete
and low-performing ones and by making these decisions in the context of
the portfolio created in Stage 3 and a well understood EA sequencing plan
and "to-be" architecture. Together, the portfolio, sequencing plan, and
"to-be" architecture provide a full picture of the current state of an
organization's investments, its vision of the future, and its plan for
getting there. In this context, the obsolescence of systems can be
anticipated, and the declining benefits of specific systems can be viewed
in the light of alternative investments.
Section 2: Overview of ITIM
Moving from Stage 4 to Stage 5
An organization that is moving from Stage 4 to Stage 5 has mature
selection, control, and evaluation processes in place. It now seeks ways
to (1) institutionalize the continuous improvement of these processes and
(2) improve its strategic business outcomes. It accomplishes these goals
by examining and learning from other organizations by means of
benchmarking. Benchmarking is used because there may be external
organizations with specific processes that are more innovative or more
efficient than its own processes. Beyond benchmarking, the organization
leverages IT to significantly change and improve its business performance
and outcomes.
Section 2: Overview of ITIM
Section 3: Components of ITIM
ITIM Hierarchy Like other maturity models, ITIM is subdivided into a
hierarchy. Each maturity stage consists of critical processes that are
composed of a number of key practices. These hierarchical components are
described below.
Maturity Stages Each of the four maturity stages beyond Stage 1 is a
plateau of well-defined critical processes. The five maturity stages
represent the steps toward achieving a mature, comprehensive IT investment
management process.
Critical Processes With the exception of Stage 1, each maturity stage is
composed of multiple critical processes, such as the processes used to
create an IT investment portfolio. Each critical process contains a set of
key practices that, when fulfilled, implement the critical process needed
to attain a given maturity stage.
Key Practices The key practices are the tasks that must be performed by an
organization in order to implement and institutionalize a critical process
effectively. Key practices fall into three categories: organizational
commitments, prerequisites, and activities. An explanation and a
description of the relationship among these different types of key
practices is shown in figure 4. In Section 5, each key practice is listed,
followed by commentary and additional information that may assist an
organization in understanding or interpreting how it could be implemented.
Section 3: Components of ITIM
Figure 4: The Components of an ITIM Critical Process
Source: GAO.
Section 4: Uses of ITIM
ITIM identifies critical IT investment processes, establishes the presence
or absence of these critical processes in an organization, assesses an
organization's IT investment management capability and maturity, and
offers recommendations for improvement. Used in this way, ITIM can be a
valuable tool that (1) supports organizational self-assessment and
improvement and (2) provides a standard against which an evaluation of an
organization can be conducted.
Principles Guiding the Regardless of the specific reason for using ITIM,
the following principles17 Use and Interpretation should guide each
interpretation and use of this framework.
of the Framework o The ITIM is a generic framework intended for broad
use. The way in which an organization implements the framework will vary,
depending on its needs for improving its investment processes and its
managerial and professional judgment.
o The ITIM is a road map for improvement and describes the
characteristics of an IT investment management process that one would
expect to see at each maturity stage. The maturity stages prescribe the
order in which to improve the processes, but not how an organization is to
improve its processes.
o The ITIM may not exhaustively describe the necessary conditions for
successful investment management in all organizations. Other components of
the investment management process may exist and could be considered for
addition to this framework as greater context sensitivity develops to the
issues surrounding the process of IT investment management.
o Each ITIM critical process will generally go through a step-by-step
evolution-consisting of introduction, adoption, development, and finally
full implementation-within an organization as that organization changes
over time, modifies necessary functions and operations, and reaches a
particular maturity stage. ITIM does not address all factors that can
affect investment success. For example, organizational processes and other
factors-such as strategic planning, availability of
17These principles were derived from the principles found in SEI's
Software Acquisition Capability Maturity Model.SM
Section 4: Uses of ITIM
funding, risk assessments, and specific technology implementations- can
strongly influence an organization's investment success.
o There is no one right way to implement the ITIM, because the framework
describes the characteristics of mature and successful IT investment
management processes, not specific implementation techniques. Because of
this, the framework is technology independent. For example, no specific
tools, methods, or technologies are mandated by its use. Appropriate
tools, methods, and technologies should be made available to support the
processes that an organization develops within ITIM.
Tool for Organizational Improvement
ITIM offers organizations a road map for improving their IT investment
management processes in a systematic and organized manner. These process
improvements are intended to
o improve the likelihood that investments will be completed on time,
within budget, and with the expected functionality,
o promote better understanding and management of related risks,
o ensure that investments are selected based on their merits by a
wellinformed decision-making body,
o implement ideas and innovations to improve process management, and
o increase the business value and mission performance of investments.
ITIM can be implemented as a tool for organizational improvement in a
variety of ways. For example, an organization can create a separate
improvement program, employ external assistance and support, or use the
framework as a managerial support tool. Regardless of the implementation
technique, the following important factors should be considered when using
ITIM as an organizational improvement tool:
o Many organizations will have a variety of selection, control, and
evaluation processes in place. ITIM can help these organizations
understand the relationships among these processes and determine the key
opportunities for immediate improvements.
Section 4: Uses of ITIM
o The framework uses a structured approach that identifies the key
practices for creating and maintaining successful investment management
processes. However, it describes what to do, not how to do it. Thus,
specific implementation methods can and will vary by organization, based
on specific attributes of the organization, such as size, complexity, and
culture.
o The developmental nature of a maturity model means that process
maturation is cumulative. Lower-stage processes provide the foundation for
upper-stage processes. As additional critical processes are introduced
into the organization and implemented, the organization attains greater
process capabilities and maturity. As the organization incorporates
additional processes at each successive stage of maturity, it must
maintain the lower-stage critical processes that it has previously
implemented.
o The framework depends on good project management to form the foundation
of good performance measurement and the project-level control processes
that underlie mature investment control processes.
o Where one exists, the use of an EA is a critical frame of reference for
making investment decisions, and only investments that move the
organization toward its target architecture-as defined by its sequencing
plan-should be approved unless a waiver is provided and/or a decision is
made to modify the EA.
o Critical processes initially may be implemented and practiced within
individual bureaus or divisions before they are implemented and are mature
across the organization.
o Business process improvement initiatives are usually not themselves
considered to be IT investments; they are considered to be parallel
efforts that may or may not be linked to investments. Thus, ITIM
assessments do not evaluate individual initiatives. However, if such
initiatives include IT investments, then the investments should be subject
to the organization's investment management process.
o Change management should be a cornerstone of process improvement,
because culture affects the nature of investment decisions. Investment
decisions are about change, and change affects an organization's culture.
For example, a decision can be creative or cautious, strategic or
tactical. Culture emanates from the values of the organization.
Section 4: Uses of ITIM
Tool for Assessing the Maturity of an Organization
Just as ITIM can be used as a tool for organizational improvement, it can
also be used as a standard against which to judge the maturity of an
organization's IT investment management process. For example, ITIM can be
used to support assessments to help ensure compliance with industry
standards or acceptable practices, independent reviews of organizational
maturity by oversight bodies, or other external IT process reviews.
Regardless of the specific use, however, the following important factors
should be considered when using ITIM as an organizational assessment tool:
o An assessment using the framework can be conducted for an entire
organization (e.g., an executive branch department) or for one of its
lower-level divisions (e.g., a branch, bureau, or agency). However, the
unit or scope of analysis (e.g., branch, bureau, agency, or department)
must be defined before an ITIM assessment is conducted. Additionally, the
assessed maturity stage for a lower-level division is not necessarily
indicative of the maturity stage of a higher-level division or of the
organization as a whole.
o The use and interpretation of ITIM by organizations may vary with their
size, culture, and organizational structure-as well as other factors. The
overriding objective of the framework is to enable senior managers to
systemically maximize the benefits of IT investments through the use of a
structured investment process. In achieving this objective, different
organizations may choose different specific implementations of the ITIM,
which may be influenced by the factors mentioned above. For example,
although ITIM addresses the organizational need to align and coordinate
multiple investment boards, an organization with only one IT investment
board would not need to perform the key practices associated with board
alignment. Also, small organizations-or those with highly centralized IT
management-may not require as extensive written guidance as large
organizations, because their investment management processes are executed
by a small, cohesive cadre of managers. Ultimately, each organization must
use its best judgment in determining how to implement ITIM within its own
context.
o An organization may be concurrently implementing key practices that are
associated with several maturity stages. In fact, key practices associated
with higher stage critical processes are frequently initiated while the
organization as a whole is at a lower stage of maturity. However,
organizational maturity is determined by assessing at what
Section 4: Uses of ITIM
maturity stage the organization implements all of the key practices for
all of the critical processes associated with a given stage of maturity-in
addition to all of those associated with lower maturity stages. For
example, performing key practices in only some of the Stage 3's critical
processes does not mean that the organization has attained Stage 3
maturity.
Limitations and Boundaries
The purpose of ITIM is to describe and improve an organization's IT
investment management processes so that the strategic plans and decisions
that it makes can and will be supported by highly effective investments.
However, like other assessment tools, the framework has its limitations
and boundaries. For example, while strategic planning and executive
decision making can greatly influence an organization's performance, the
framework does not evaluate these. If IT plans and business plans are
linked, there is a high likelihood that investment decisions will be
closely aligned with the business.
Similarly, performance measures that are created and used to guide the
organization and its activities are an integral part of controlling the
expenditures on an investment and can be viewed as maturing in parallel
with the IT investment management processes. However, this guide does not
describe in detail18 the development or implementation of these measures.
In addition, the framework does not address IT acquisition (e.g., which
type of contract to use or how best to conduct price negotiations, etc.)
as a separate investment management step. While they are important, the
primary purpose of acquisition-related activities is to support the
execution of the investment decisions that are made by the IT investment
board(s)19 Thus, one would expect that the acquisition aspects of project
development would be embedded in the project proposal and analysis steps
within the framework. Alternatively, the acquisition strategy might be
18For additional guidance on developing performance measures, see U.S.
General Accounting Office, Executive Guide: Measuring Performance and
Demonstrating Results of Information Technology Investment, GAO/AIMD-98-89
(Washington D.C.: March 1998).
19For more information on procurement within the context of a capital
budget, see OMB's Capital Programming Guide, Version 1.0 (July 1997).
Section 4: Uses of ITIM
part of the project's risk assessment (i.e., the risks of pursuing various
acquisition alternatives).
Finally, organizations selecting ITIM as an assessment tool should
o become proficient with the related GAO and OMB guidance on IT
investment.20 This is particularly important for those seeking to apply
ITIM in the federal government. Understanding this guidance provides
greater insight into the developmental history, key issues, and critical
success factors associated with the IT investment approach.
o become familiar with generally accepted capital decision-making
approaches and associated analytical tools;
o become familiar with the concepts associated with EA management;
o receive training to become familiar with the basic concepts behind
maturity models; and
o have experience using standardized assessment tools to assess
organizations.
For further guidance on how to conduct an ITIM evaluation, refer to
appendix II of this document.
20U.S. General Accounting Office, Assessing Risk and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision-making,
GAO-AIMD-10.1.13 (Washington, D.C.: February 1997); U.S. General
Accounting Office, Assessing Risk and Returns: A Guide for Evaluating
Federal Agencies' IT Investment Decision-making, GAO-AIMD-10.1.13
(Washington, D.C.: February 1997); Evaluating Information Technology
Investments, A Practical Guide, Executive Office of the President, Office
of Management and Budget, November 1995.
Section 5: Critical Processes for the ITIM Stages
Figure 5: The ITIM Stages of Maturity with Critical Processes
Source: GAO.
The following subsections describe each maturity stage in greater detail.
The first subsection describes only the attributes of Stage 1 because no
critical processes are associated with this stage. Each subsequent
subsection describes one of the stages. In each subsection, the stage is
briefly introduced and its associated critical processes are identified,
along with a list of applicable criteria. For each critical process, a
brief introduction and purpose is presented, along with a map showing the
associated key practices (organizational commitments, prerequisites, and
activities) that make up the critical process and a discussion and
interpretation of the key practice. For easy reference, each page heading
in section 5 indicates which stage and critical process are being
discussed on that page.
Section 5: Critical Processes for the ITIM Stages o Stage 1: Creating
Investment Awareness
Stage 1: Creating Investment Awareness
Figure 6: The ITIM Stages of Maturity with No Stage 1 Critical Processes
Source: GAO.
The following section provides a description of the conditions and
characteristics associated with an organization operating at ITIM Stage 1.
Within ITIM, Stage 1 is different from the other maturity stages because
o there are no critical processes associated with Stage 1; and
o it is typified by the absence of an organized, executable, and
consistently applied IT investment management process.
The following description of an ITIM Stage 1 organization is not intended
to be comprehensive; rather, it provides an overview of the general
conditions and problems that typically confront a Stage 1 organization.
Generally, an ITIM Stage 1 organization has ad hoc or undisciplined IT
investment management processes. This often contributes to escalating
project costs, unmitigated risks, frequent slippages in project schedules,
and low-value mission or business benefits. Furthermore, while the
organization may have "pockets of excellence" in IT investment
Section 5: Critical Processes for the ITIM Stages o Stage 1: Creating
Investment Awareness o o Select Process
management, the variability in these processes across the organization may
lead to inconsistency in IT project outcomes.
Select Process The Stage 1 organization's focus is more often on a
project's funding requirements and lower level organizational requirements
rather than on (1) its value toward achieving the organization's mission
goals, (2) its technical and economic risks, (3) its performance problems,
or (4) cost and schedule overruns. IT is treated as an expense item in
most organizations' budgets, and it may be intertwined with other
administrative and management support funding needs. Also, multiyear IT
projects that are "in the budget pipeline" are reviewed each year largely
in terms of marginal increases or decreases to the previous year's funding
base, regardless of cost, schedule, and performance results to date.
In short, while some IT projects within a Stage 1 organization may be
funded because they link to a defined business or mission purpose, many
projects are funded despite the absence of critical information that
demonstrates expected and achieved improvements in program, business, or
mission performance.
Control Process Stage 1 organizations typically have unstructured,
ill-timed, and inconsistent IT investment management controls. Senior
executives and line managers may rarely review IT projects' performance
data, and thus the organization lacks an early warning method for quickly
detecting and rectifying major problems. Instead, project crises are
handled as they arise, focusing only on quick fixes rather than
considering possible systemic causes of the problems. As a result, the
success of individual projects is unpredictable and may often be the
result of extraordinary efforts by individuals or the project team.
Additionally, a Stage 1 organization rarely would have an up-to-date and
complete collection of investment information. For example, although it
might have an IT hardware (equipment) inventory, it might lack a
comprehensive list of systems, software applications and tools, and
licensing agreements. Without a complete inventory of IT information, an
organization cannot develop an adequate investment control process.
Section 5: Critical Processes for the ITIM Stages o Stage 1: Creating
Investment Awareness o o Evaluate Process
Evaluate Process Finally, a Stage 1 organization rarely, if ever, (1)
evaluates IT investment outcomes or (2) identifies lessons learned from
its projects. If such evaluations are conducted, they often are triggered
only in response to outside pressures (e.g., an audit or a budget
oversight review), and they tend to be poorly staffed and conducted
without a formal process that delineates method, scope, and
responsibilities.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation
Stage 2: Building the Investment Foundation
Figure 7: The ITIM Stages of Maturity with Stage 2 Critical Processes
Source: GAO.
Stage 2 builds the foundation for current and future IT investment success
by establishing basic IT selection and control processes. This stage is
defined by five critical processes. Each critical process is described
below, followed by a set of "Criteria," and a listing of documents that
establish criteria supporting the use of the critical process in ITIM.
o Instituting the Investment Board is the process for creating and
defining the membership, guiding policies, operations, roles,
responsibilities, and authorities for one or more IT investment boards
within the organization.
Criteria: Assessing Risks and Returns: A Guide for Evaluating Federal
Agencies' IT Investment Decision-making (hereafter referred to as IT
Assessment Guide) (AIMD-10.1.13), 32, (CCA, OMB M-97-0(2)); Executive
Guide: Improving Mission Performance Through Strategic Information
Management and Technology (hereafter referred to as SIM Executive Guide)
(AIMD-94-115), Practices 2, 10; Evaluating Information Technology
Investments, version 1.0 (hereafter referred to as OMB IT Investment
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Instituting the Investment Board
Guide), Office of Management and Budget, 3; Capital Programming Guide,
version 1.0, Office of Management and Budget, ii.
o Meeting Business Needs is the process for developing a business case
that identifies the key executive sponsor and business customers (or end
users) and the business needs that the IT project will support.
Criteria: IT Assessment Guide (AIMD-10.1.13), 15, 16, 17; SIM Executive
Guide (AIMD-94-115), Practices 4, 9; OMB M-97-16.
o Selecting an Investment introduces a defined process that an
organization can use to select new IT project proposals and reselect
ongoing projects.
Criteria: IT Assessment Guide (AIMD-10.1.13), 23-25, (CCA, PRA, EO 13011,
OMB A-11, OMB A-130, OMB A-109, OMB A-94, OMB M-97-0(2))
o Providing Investment Oversight is a pivotal process whereby the
organization monitors projects against cost and schedule expectations as
well as anticipated benefits and risk exposure.
Criteria: IT Assessment Guide (AIMD-10.1.13), 52, (CCA, PRA, FASA, EO
13011, OMB A-11, Part 3); OMB IT Investment Guide, 10.
o Capturing Investment Information is the process by which specific
details about a particular investment are captured and maintained to
provide asset-tracking data to executive decision makers.
Criteria: IT Assessment Guide (AIMD-10.1.13), 8, 19; PRA; E.O. 13103;
Capital Programming Guide, ii.
Instituting the Investment Board
The IT investment board is a key component in the investment management
process. This critical process defines the membership, guiding policies,
operations, roles, responsibilities, and authorities for each designated
board and, if appropriate, each board's support staff. This definition
provides the basis for each board's investment selection, control, and
evaluation activities. The organization may choose to make this board the
same board that provides executive guidance and support for the EA. This
overlap of responsibilities may enhance the ability of the board to ensure
that investment decisions are consistent with the architecture and that it
reflects the needs of the organization.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Instituting the Investment Board
Depending on its size, structure, and culture, an organization may have
more than one IT investment board. This critical process is based on the
assumption that, for managerial reasons, the key practices in this
critical process will be implemented consistently across each of these
boards and that the organization will tailor each board's operations as
part of this implementation.
Figure 8: Instituting the Investment Board
Source: GAO.
Purpose To define and establish an appropriate IT investment management
structure and the processes for selecting, controlling, and evaluating IT
investments.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Instituting the Investment Board
Organizational Commitments Commitment 1: An enterprisewide IT investment
board composed of senior executives from IT and business units is
responsible for defining and implementing the organization's IT investment
governance process.
The enterprisewide investment board is created to (1) define the
investment board's structure and accompanying processes and (2) implement
the processes as they are defined. This board is comprised of senior
executives, including the organization's head or a designee,21 the Chief
Information Officer (CIO) or other senior executive representing the CIO's
interests, and heads of business units and supporting units such as
financial management. When the CIO is represented on the board by another
senior executive, this executive must have knowledge of the CIO's
management responsibilities and be able to fully represent the technical
criteria that are being applied in the investment decision process. In
cases where lower-level investment boards, comprised of individuals from
across the organization, are chartered to carry out the responsibilities
of the enterprisewide IT investment board within their own business units,
the enterprisewide IT investment board still must maintain ultimate
responsibility for the lower-level boards' activities. These subordinate
boards should have the same broad representation as the enterprisewide
board, though at the subordinate unit's level.
The enterprisewide IT investment board is responsible not only for major
systems that affect multiple departments and users. These enterprisewide
investments should be elevated to the enterprisewide IT investment board
to ensure buy-in from senior executives and users representing various
departments. The enterprisewide IT investment board should be actively
involved in all IT investments and proposals that are high cost or high
risk or have significant scope and duration.
21The organization head typically has ultimate responsibility for
submitting a budget request to the organization's authorizing budget
office (e.g., OMB in the federal government) and thus may rework or adjust
the IT budget recommendations made by the investment board. An effective
investment environment exists when the organization head with the senior
executives and the CIO exhibit a corporate responsibility and serve as
corporate officers on the investment board, instead of competing for their
individual interests.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Instituting the Investment Board
Commitment 2: The organization has a documented IT investment process
directing each investment board's operations.
The organization uses the available IT investment process guidance22 and
defines the unique manner in which the guidance will be implemented. The
guidance should lay out the roles of key boards, working groups, and
individuals involved in the organization's IT investment processes, and it
should explain the procedures for assigning responsibility for decision
making for a given investment or proposal. The guidance should specify
that individual business or operational units retain decision-making
authority for unit-specific IT decisions while still following
enterprisewide standards and procedures, and it should outline the
significant events and decision points within the processes; identify
external and environmental factors that will influence the processes
(i.e., legal constraints, the behavior of key suppliers or customers, or
industry norms); and specify the manner in which IT investment-related
processes will be coordinated with other organizational plans, processes,
and documents-including, at a minimum, the strategic plan, budget, and EA.
In IT organizations that have multiple IT investment boards, the
enterprisewide investment process guide should document the policies and
procedures that define each IT investment board's span of authority and
describe how investment board activities are to be coordinated.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, are provided for supporting the operations of each IT
investment board.
Executive management is typically responsible for creating the investment
board(s), defining their scope and resources, and specifying their
membership. Establishing an investment management working group can
benefit both the IT investment boards and IT project managers by
22U.S. General Accounting Office, Assessing Risks and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision-making
GAO/AIMD-10.1.13 (Washington D.C.: February 1997); U.S. General Accounting
Office, Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD94-115, May 1994);
Evaluating Information Technology Investments, A Practical Guide,
Executive Office of the President, Office of Management and Budget,
November 1995. Capital Programming Guide, version 1.0, Office of
Management and Budget, (July 1997). E-Government Act of 2002, Public Law
107-347 (Dec. 17, 2002).
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Instituting the Investment Board
coordinating requests for information and verifying and providing
responses.
Prerequisite 2: The board members understand the organization's IT
investment management policies and procedures and the tools and techniques
used in the board's decision-making process.
Members of the investment board should have an understanding of the
board's policies and procedures and the experience and skills to carry
them out. Thus, the organization should consider introducing investment
concepts to board members with little or no investment decision-making
experience or relevant education in this area. Orientation sessions might
be provided to board members in areas such as economic evaluation
techniques, capital budgeting methods, performance measurement strategies,
and risk management approaches. In addition, board members should be made
aware of the specific processes for which they are responsible.
Knowledge building and/or orientation sessions might include
o briefings specifically designed for new board members,
o educational forums,
o formal seminars, and
o executive training programs offering in-depth courses.
Prerequisite 3: Each board's span of authority and responsibility is
defined to minimize overlaps or gaps among the boards.
When multiple boards execute the organization's IT investment governance
process, criteria aligning these boards must be defined such that there
are no overlaps or gaps in the boards' authorities and responsibilities.
These criteria can be based on cost, benefit, schedule, and risk
thresholds, the number of users affected, the function of the business
unit (e.g., CIO, human resources, or program office), the life cycle phase
of an IT investment (e.g., proof of concept, full scale development, or
operations and maintenance), or other comparable and useful measures. An
example would be to manage investments with less than a $100,000 life
cycle cost at the lowest departmental level, but to have investments with
more than $100 million in life cycle costs managed by the enterprisewide
investment board.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
Activities Activity 1: The enterprisewide investment board has oversight
responsibilities for the development and maintenance of the organization's
documented IT investment process.
As the board responsible for defining and implementing the organization's
IT investment management process, the enterprisewide IT investment board
should also have responsibility for developing the organizationspecific IT
investment guide. The board's work processes and decisionmaking processes
(i.e., schedules, agendas, authorities, decision-making rules, etc.) are
described and documented in the guidance. In addition, after the guide has
been developed, the enterprisewide investment board must actively maintain
it, making sure that it always reflects the board's current structure and
the processes that are being used to manage the selection, control, and
evaluation of the organization's IT investments.
Activity 2: Each investment board operates in accordance with its assigned
authority and responsibility.
For the whole IT investment management process to function smoothly and
effectively, each investment board must operate within its assigned
authority and responsibility, so that investments are properly aligned
with the organization's objectives and are reviewed by the appropriate
board.
Activity 3: The organization has established management controls for
ensuring that investment boards' decisions are carried out.
Establishing management controls helps to ensure that management will
carry out the decisions made by the IT investment board. Without these
controls in place, decisions made by the investment board might not be
implemented because of conflicting priorities. To ensure adherence to
management controls, the structure of the relationship between upper
management and the investment board must be documented and agreed to by
both parties,. The investment board must have the confidence of upper
management when selecting new proposals and ongoing projects for funding.
Meeting Business Needs IT projects and systems should be tightly aligned
with the business needs of the organization, providing support for highly
visible core business processes. These strategically aligned IT projects
and systems provide the highest value and most obvious investment benefits
to an organization and are hallmarks of successful return on investment.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
To achieve such a robust level of support, the organization must
continually identify the business necessity for its IT projects and
systems. Periodic identification of the business needs ensures that the
correct and appropriate IT projects and systems are funded and that they
directly support the organization's strategic plan. The frequency of this
business verification may range from every quarter for an R&D project to
every 3 years for systems in operations and maintenance; the appropriate
interval depends upon the pace of functional changes in the system and the
evolution of users' needs. Identifying business needs ensures that IT
projects and systems will maintain an alignment with the organization's
strategic plans and its business goals and objectives.
To the extent that the organization has planning documents-such as a
strategic plan or a target enterprise architecture-these documents should
be used as a source of agreed upon business needs. In addition, other
business needs may surface through the investment process itself. In all
cases, these business needs should be aligned with specific strategic
objectives of the organization.
The essence of identifying business needs is for the business case for
every IT project and system to be periodically reviewed and verified with
respect to the business need(s) it is supporting. If an IT project or
system is out of alignment with its strategic plan, then the IT investment
needs to be resynchronized with the strategic plan or the overall
strategic plan needs to be changed. Based upon the business case review,
the most promising IT projects and systems are identified for continued
investment. The investment board addresses whether business and user needs
continue to be met in a cost-effective and risk-insured manner.
This critical process establishes a mechanism for verifying the business
case (such as business requirements and rules, congressional mandate, and
the organizational users) that drives continued support for each IT
system. Ensuring that an essential link exists between the organization's
business objectives and its IT strategy and that a defined partnership
exists between the sponsoring unit and the IT solution providers
strengthens and institutionalizes the organization's investment management
process.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
Figure 9: Meeting Business Needs
Source: GAO.
Purpose To ensure that IT projects and systems support the organization's
business needs and meet users' needs.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for identifying IT projects or systems that
support the organization's ongoing and future business needs.
The organization has policies and procedures that outline a systematic
process for identifying, classifying, and organizing its business needs
and the IT projects used to support these needs. In many cases, this can
be covered in the internal guidance that is used for documenting business
cases for IT investments. These policies and procedures typically specify
that
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
o a systematic process for identifying, classifying, and organizing
business needs is linked to the business planning process,
o business needs or opportunities should be stated in functional terms or
in terms of desired business improvement and not in product-or
technology-specific terms,
o each IT project or system fits within the organization's EA and
established security standards
o IT projects or resources that do not support an identified business
need (and the associated customers or end users) are further examined for
possible termination,
o there is a procedure by which similar needs or opportunities within
different operating units are reconciled, and
o meeting business needs occurs regularly as part of the strategic
planning cycle.
Prerequisites Prerequisite 1: The organization has a documented business
mission with stated goals and objectives.
The business mission, containing the stated goals and objectives is
typically identified in
o strategic management or business plans (e.g., agency strategic plans
prepared for GPRA),
o business process architecture documents,
o process improvement initiatives, or
o performance measurement plans.
Defining these goals and objectives, however, is largely outside the scope
of ITIM. (See also Section 4: Limitations and Boundaries of ITIM.)
Prerequisite 2: Adequate resources, including people, funding, and tools,
are provided for ensuring that IT projects and systems support the
organization's business needs and meet users' needs.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
These resources typically involve
o funding for these activities;
o managerial attention to this process;
o an executive sponsor for the project;
o staff support for carrying out these activities; and
o supporting methods, analytical tools, and processes.
Activities Activity 1: The organization defines and documents business
needs for both proposed and ongoing IT projects and systems.
Each IT project is directly or indirectly linked to at least one of the
organization's business needs or mission goals; a direct link is of
greater value than an indirect link. This link can be established in a
variety of ways. For example, an organization can
o identify a project's business purpose as part of the project's
initiation activities,
o define an executive sponsor for each project, or
o obtain validation from external groups supporting the business value of
the project.
The business needs for each IT project will generally be documented in the
business case for the project.
Activity 2: The organization identifies specific users and other
beneficiaries of IT projects and systems.
Each major IT project or system will have end users or customers who will
benefit from the system. A given project or system may address the needs
of multiple sets of end users or customer groups. The primary end users or
customers will be formally identified by the organization.
Identifying the end users early in the process assists the IT staff
developing the IT project or system in focusing on the specific,
well-defined goals of delivering value to end users. So that they may
accomplish their particular
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Meeting Business Needs
work, end users depend directly on the IT staff to deliver a project's
capability and to provide a system's functionality.
Activity 3: Users participate in project management throughout an IT
project's or system's life cycle.
End user involvement will vary during the different stages of a project's
system life cycle. During the project's conception, end users should be
heavily involved in developing the business case and in defining how the
system will help to meet business needs or opportunities. They will be
heavily involved again during user acceptance testing. During other phases
of development, they will play a more limited role.
In the final phases of the system's life cycle, especially during the
operational phase of the system, end users should play a major role in
helping to identify and document any benefits that are realized from the
system's implementation. Users should also participate in the operational
analysis of the system. The analysis should involve collecting information
about the system's performance and comparing it with the initial
performance baseline.
Activity 4: The investment board periodically evaluates the alignment of
its IT projects and systems with the organization's strategic goals and
objectives and takes corrective actions when misalignment occurs.
This activity permits the investment board to assess a project's or
system's outcomes and its value in comparison to predefined expectations,
in preparation for determining whether or not and how well the IT project
or system is meeting the organization's expectations. After deployment, a
system's success is measured by its ability to continually meet a business
or user need.
The length of the period for collecting IT system data prior to review and
analysis varies from one organization to another. An organization could,
for example, annually review one-third or one-half of its operational IT
systems. Another organization could decide to review all operational IT
systems every 3 years. The essential point is that operational IT systems
are investments that need to be reviewed on a regular basis to ensure that
they are still providing value to the organization in a cost-effective and
riskinsured manner.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Selecting an Investment
Using historical data, system expectations, and other factors as criteria,
the investment board evaluates every IT system to determine its value to
the organization. The review cycle should reflect the risk and volatility
of the project or system being evaluated. Periodic evaluation of each IT
project or system permits the investment board to determine the ongoing
value that each investment is providing to the organization and its end
users. These periodic evaluations are critical to determining whether or
not to continue to fund an IT system.
When an investment is found to be out of alignment with the organization's
strategic goals and objectives, immediate action must be taken at the
project level, with oversight provided by the investment board, to realign
the project or system. But even a successful system will eventually begin
to provide diminishing returns as it becomes more expensive to maintain.
In addition, changing business requirements also can make a system
obsolete.
Selecting an Investment The purpose of this critical process is (1) to
predefine a method for selecting new IT proposals and (2) use this method
to select new proposals. Within ITIM, "new" proposals include both (1)
previously submitted IT proposals that were not originally selected for
funding and (2) IT proposals that have never been submitted.
Defining and implementing a selection process is a basic step toward
implementing the mature IT critical processes for proposal and project
selection in Stage 3. The key activities implemented within this critical
process include (1) concurrent review of IT proposals by the
organization's executives, (2) the use of predefined selection criteria to
analyze the proposals, and (3) decision making by executives to fund some
proposals and not others. The EA, where it exists, should be reflected in
the selection criteria. Investments may come up outside of the EA, in
which case their value must be considered under the same criteria as all
other investments. Investments that are not consistent with the current EA
should either be assimilated into the EA or be provided a waiver.
Reselection of ongoing projects is a very important part of this critical
process. If a project is not meeting the goals and objectives that were
established in the original selection, the investment board must make a
decision on whether to continue to fund it.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Selecting an Investment
Source: GAO.
Purpose To ensure that a well-defined and disciplined process is used to
select new IT proposals and reselect ongoing investments.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for selecting new IT proposals.
The organization has policies and procedures that outline a structured
method for identifying, evaluating, prioritizing, and selecting its new IT
proposals.
Using a structured method to select new IT projects accomplishes several
objectives. First, a structured method provides the organization's
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Selecting an Investment
investment board, business units, and IT developers (whether they are
internal IT staff or contractors) with a common understanding of the
process and the cost, benefit, schedule, and risk criteria that will be
used to select IT projects. Second, whether a business unit identifies a
business need and develops an IT proposal itself or the organization's IT
group develops the proposals, organizational roles and responsibilities
will be defined for each participating unit involved in the project
selection process. Lastly, the data required for decision making and the
decisionmaking procedures should be predefined.
A documented selection process can help to ensure consistency when an
organization is considering multiple investments for funding. Transparency
in the process can help to create an environment that is objective, fair,
and rational. Thus, potential investments will be judged solely on the
merits of their contributions to the strategic goals of the organization
without undue influence from outside the process.
Commitment 2: The organization has documented policies and procedures for
reselecting ongoing IT investments.
The organization has policies and procedures that outline a structured
method for identifying, evaluating, prioritizing, and reselecting ongoing
projects.
A policy-driven, structured method for reselecting ongoing projects for
further funding can also accomplish several objectives. A structured
method provides the organization's investment board with a common
understanding of how ongoing projects will be reselected for continued
funding. Each ongoing project should be judged based on its success in
meeting the investment outcomes that were stated in the policies and
procedures for reselection. The information needed for decisions on
project reselection should be predefined.
A documented reselection process ensures consistency when an organization
is considering multiple investments for additional funding. Again,
transparency in the process will create an environment that is objective,
fair, and rational. Thus, ongoing investments will be judged solely on the
merits of their current contributions to the strategic goals of the
organization without undue influence from outside the process.
Commitment 3: The organization has policies and procedures for integrating
funding with the process of selecting an investment.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Selecting an Investment
The process of selecting investments is not feasible unless the policies
and procedures for selection and reselection take into account how much
funding is available for IT investments. No decision to fund a project can
be considered valid without considering what funds are available. It is
therefore vitally important to include procedures for project funding in
the documented policies and procedures for selecting investments.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, are provided for identifying and selecting IT projects
and systems.
These resources typically involve:
o managerial time and attention to the process, including project
sponsorship;
o staff support, including, at a minimum, a designated official to manage
the process; and
o supporting tools, methods, and equipment for organizing and analyzing
the proposals.
Prerequisite 2: Criteria for analyzing, prioritizing, and selecting new IT
investment opportunities have been established.
The organization has created a process for comparing projects within the
portfolio of IT investments. Any decision-support process should be based
on predetermined criteria. In order to maintain consistency, the criteria
should include quantitative or qualitative measures for comparing
projects. Projects are compared with one another based on criteria such as
investment size, project longevity, technical difficulty, project risk,
business impact, customer needs, cost-benefit analysis, organizational
impact, and expected improvement. The results of such a comparison will
help the investment board analyze the potential risk and return of
investing in a particular project and prioritize the portfolio of projects
using a scoring mechanism that considers strengths and weaknesses. After a
careful analysis of the various projects vying for funding, senior
executives should be able to prioritize the list of IT investment
proposals based on supporting documentation.
Prerequisite 3: Criteria for analyzing, prioritizing, and reselecting IT
investment opportunities have been established.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Selecting an Investment
The organization has created a process for analyzing and prioritizing
ongoing projects within its IT investment process. Any decision-support
process for analyzing ongoing operations and maintenance projects should
be based on predetermined criteria. There should be consistent
quantitative or qualitative measures for analyzing projects for
reselection or, if necessary, termination. If corrective actions cannot be
implemented to maintain the desired investment outcome, the project should
be identified, based on developed criteria, for termination. The results
of such an analysis will help the investment board determine the potential
risk and return of continuing to fund an ongoing project and to prioritize
the projects based on decision criteria. After a careful analysis of the
various ongoing projects competing for continued funding, senior
executives should be able to prioritize the list of existing IT
investments for reselection based on supporting documentation.
Prerequisite 4: A mechanism exists to ensure that the criteria continue to
reflect organizational objectives.
The organization has created a process for ensuring that the criteria
change as organizational objectives change. During project selection,
decision makers use various criteria to help them assess a system's
projected outcomes, resource allocations, and benefits and costs. Because
criteria are usually presented in a hierarchical structure, decision
makers are able to apply judgments based on the criteria/objectives deemed
important to achieving specific goals. As organizational goals and
objectives change- and the criteria for selecting projects changes with
them-decision makers need to have management structures and tools in place
to help them reassess their decision criteria and the effects of those
criteria on decisions, results, and outcomes.
Activities Activity 1: The organization uses its defined selection
process, including predefined selection criteria, to select new IT
investments.
The organization uses a structured process for submitting IT proposals
that require funding or organizational support. This activity typically
occurs within the context of the organization's cyclical budgeting
process. A designated official manages the data submission and screening
activities that are associated with the process.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
Activity 2: The organization uses the defined selection process, including
predefined selection criteria, to reselect ongoing IT investments.
The part of the process during which organizations tend to need the most
help is in determining which projects to reselect and which to terminate.
Competing priorities and differing objectives make it extremely difficult
for IT decision makers to determine where to allocate their scarce IT
funds. Faced with a changing laundry list of important and potential IT
projects that exceeds budget parameters, managers need a predefined
selection process that will help them choose among new and ongoing
projects. To help ensure the selection and continuation of the most
promising projects, ongoing projects should be reviewed continually along
with new projects and go/no-go decisions should be made using predefined
selection criteria.
Activity 3: Executives' funding decisions are aligned with selection
decisions.
The organization's executives have discretion in making the final funding
decisions on IT proposals. However, their decisions should be based upon
the analysis that has taken place in the previous activities.
Additionally, as part of the decision-making process, there should be
evidence that some proposals are judged less meritorious than others and
thus do not get funded.
Providing Investment Oversight
The purpose of this critical process is to ensure that the organization
provides effective oversight for its IT projects throughout all phases of
their life cycles. While the board should not micromanage each project in
order to provide effective control, it should maintain adequate oversight
and observe each project's performance and progress toward predefined cost
and schedule expectations as well as each project's anticipated benefits
and risk exposure. The board should expect that each project development
team will be responsible for meeting project milestones within the
expected cost parameters that have been established by the project's
business case and cost/benefit analysis. The board should also employ
early warning systems that enable it to take corrective actions at the
first sign of cost, schedule, and performance slippages.
The investment board has ultimate responsibility for the activities within
this critical process. However, in larger organizations, the board may
authorize designated subgroups to carry out some of these activities. The
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
investment board must ensure that projects maintain alignment with the EA,
where one exists.
Source: GAO.
Purpose To review the progress of IT projects and systems, using
predefined criteria and checkpoints, in meeting cost, schedule, risk, and
benefit expectations and to take corrective action when these expectations
are not being met.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for management oversight of IT projects and
systems.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
These policies and procedures typically specify
o each investment board's responsibilities when providing investment
oversight within its domain,
o the procedural rules for the investment board's operation and for
decision making during project oversight,
o the threshold criteria that the investment board(s) uses when analyzing
project performance as part of its oversight function (threshold is
typically based on cost or schedule measures-for example, currently more
than 10 percent over expected cost-and will be a major factor in
determining whether to take remedial actions),
o that corrective actions are required when the project deviates or
varies significantly from the project management plan,
o that changes to the project's commitments to meet cost, schedule,
performance, or other expectations be made with the involvement of
affected groups, including
o enterprise architecture,
o system engineering,
o software engineering (including all subgroups, such as software
design),
o hardware engineering,
o project planning and estimating,
o information assurance,
o project stakeholders and champions,
o business units, and
o customers and end users.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
o that each investment board oversee all changes to new and existing
project commitments that it has made to individuals and groups external to
the organization,
o the procedures for escalating/elevating unresolved and/or significant
issues,
o the conditions under which a project would be terminated and the funds
redirected to other "successful" projects.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, are provided for IT project oversight.
The organization performs an assessment of the resources needed to oversee
its IT projects and systems. These resources should include
o managers and staff who are assigned specific responsibilities for
monitoring IT projects and systems,
o tools to support board(s)' oversight operations, which may include
project summary reports on various metrics and decision support
applications.
Prerequisite 2: IT projects and systems, including those in steady state
(operations and maintenance),23 maintain approved project management plans
that include expected cost and schedule milestones and measurable benefit
and risk expectations.
23"Steady state" means maintenance and operation costs at current
capability and performance level, including costs for personnel,
maintenance of existing information systems, corrective software
maintenance, voice and data communications maintenance, and replacement of
broken IT equipment.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
Each IT project management team creates and maintains a project management
plan24 for the project or system for which it is responsible. This plan
documents a variety of project decisions, assumptions, and expectations,
including project performance.25 These expectations could include a cost
and schedule baseline control system, such as the earned value management
system, milestone-based accomplishment expectations, or other such control
systems as are commensurate with the project's size, importance, cost, and
risk.26
Each project that is in its operations and maintenance (O&M) phase should
have its own distinct project management plan, one that is different from
plans for new investments. This requirement is due in large part to the
differences in how each investment is managed. O&M projects typically do
not have milestones, and their cost structure is more predictable.
Activities Activity 1: Data on actual performance (including cost,
schedule, benefit, and risk performance) are provided to the appropriate
IT investment board.
For an organization to establish control of projects in Stage 2, it is
essential that all performance data including cost, schedule, benefits,
risks, and system functionality (both expected and actual) for each IT
project are collected and distributed to the appropriate IT investment
boards. In addition, to monitor the long-term value of a project or
system, the organization needs to collect and distribute this information
to the appropriate IT investment board during agreed-upon stages of the
project's life cycle.
These performance data may be collected by the board itself or collected
and distributed in some other manner (e.g., through a centralized third
24See IEEE 1058 Standard for Software Project Management Plans for an
example of additional guidance on creating a project management plan.
25See U.S. General Accounting Office, Executive Guide: Measuring
Performance and Demonstrating Results of Information Technology
Investments, GAO/AIMD-98-89 (Washington D.C.: March 1998) for additional
guidance on performance measurement.
26For additional guidance on earned value management, see the Defense
Department's Earned Value Management Web site at http://www.acq.osd.mil/pm
and Capital Programming Guide Supplement to Part 7 of Circular No. A-11
(Office of Management and Budget Circular No. A-11, July 2003).
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
party). These data will be key to assisting each IT board in its decision
making.
IT projects in development, by definition, provide little current benefit,
but they may provide benefits to the organization upon completion. The
potential benefits of an IT project are enumerated in the project's
business case; they are used to conduct an [expected] benefit/cost
analysis and to persuade executives to select the project as a good
investment. These potential benefits will be realized after implementation
is complete. Measuring the actual benefit of a project while it is in
development is a challenge. One way to measure the benefit of development
work is to approximate it. Measuring a project's actual cost and schedule
progression (i.e., evaluating earned value, which is a measure of the
amount of preplanned work that is actually performed in relation to the
funds expended) renders an approximate value of the project to the
organization.
Activity 2: Using verified data, each investment board regularly reviews
the performance of IT projects and systems against stated expectations.
The board typically oversees the project's performance by conducting
reviews at predetermined checkpoints and/or major milestones, in order to
interpret the data on project cost and schedule with respect to historic
project data and expectations.
Project oversight
o is conducted at least at the major life cycle milestones for each
project;
o is managed to limit changes in scope, such as increasing functionality
requirements (scope creep);
o differs in its degree of depth depending on the size, cost, and
importance of the project;
o must compare estimated schedule time frames to actual schedules,
including schedule slippages and/or compressions;
o must compare estimated costs with funds spent or obligated to date, any
changes in funding, and the impact of these changes; and
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Providing Investment Oversight
o ensures that project information and data are valid and that corrective
actions are verified by qualified and independent audit teams, quality
assurance groups, or internal verification and validation (IV&V)
contractors.
Project oversight should also address each of the following project
management issues:
o Development/Acquisition. Problems (e.g. contractor management) stemming
from the selection of a specific project development and implementation
approach.
o Technical. Technical issues or problems concerning such components as
hardware, software, or telecommunications.
o Benefits. Evaluation of benefits delivered to date and the relationship
of the project to specific business objectives.
o Risks. Assessment of the risks encountered to date and how expected
risks are to be managed.
Activity 3: For each underperforming IT project or system, appropriate
actions are taken to correct or terminate the project or system in
accordance with defined criteria and the documented policies and
procedures for management oversight.
Using estimated and actual cost and schedule data, the organization should
identify projects that are not meeting their cost and/or schedule
performance expectations. The following are examples of data that could be
compared:
o actual cost data to planned cost data;
o the current number and scope of requirements to the original
requirements established for the project;
o the current conditions and assumptions to the projects' initial
assumptions and context; and
o the actual performance of the software development organization to its
specified deliverables (e.g., schedule, costs, functionality, technical
solutions).
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
Senior executives should ensure that there is a support and reward
structure in place for identifying issues and raising them to the
appropriate decision-making level and that there are no incentives for
covering up significant problems. Go/no-go criteria can be a helpful tool
in supporting management oversight.
Activity 4: The investment board regularly tracks the implementation of
corrective actions for each underperforming project until the actions are
completed.
The investment board ensures that
o corrective actions and related efforts are executed by the project
management team and tracked by the investment board until the desired
outcomes occur, and
o if the corrective actions are significant enough, an independent review
is conducted before returning to the original project plan (i.e.,
reinstatement of funding) to ensure that all corrective actions have
achieved their intended results and to determine whether additional
changes or modifications are still needed.
Capturing Investment Information
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format, to be used in future investment
decisions. During this critical process the organization identifies its IT
assets and creates a comprehensive repository of investment information.
This repository of IT investment information is used to track the
organization's IT resources to provide insights and trends about major IT
cost and management drivers. The information in the repository serves to
highlight lessons learned and to support current and future investment
decisions. This critical process may be satisfied by the information
contained in the current EA, augmented by additional information (e.g.,
financial information, risks, benefits, etc.) that the investment board
may require to ensure that informed decisions are being made.
This repository can take many forms (e.g., a catalog, a list, IT system
and software inventories, or a balance sheet), but regardless of form, the
collection method should identify each IT investment and its associated
components. An organization's "as-is" architecture, along with its
sequencing plan, can provide a resource for developing a list of existing
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
investments. In addition, the EA tool may provide an opportunity for
gathering all of the necessary information in one place. This information
does not have to be centrally located; it can be managed on a distributed
basis. The guiding principle for developing the information source is that
it should be accessible where it is of the most value to those making
decisions about IT investments. The information is particularly important
when executing the critical processes for Providing Investment Oversight,
Selecting an Investment, Creating the Portfolio, and Managing the
Succession of Information Systems. Additionally, beyond serving as a tool
to aid in IT investment decision making, the IT information can also
assist the organization with software licensing management, hardware life
cycle management, and system architecture plans.
Source: GAO.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
Purpose To make available to decision makers information to evaluate the
impacts and opportunities created by proposed (or continuing) IT
investments.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for identifying and collecting information about
IT projects and systems to support the investment management process.
These policies and procedures typically specify
o that responsibility for submitting, updating, and maintaining relevant
inventory information for each project or asset is explicitly assigned;
o the process to be followed for the collection of information, access to
the information, and support for maintaining the information; and
o the data elements required for each IT-related item, including
o the cost (e.g., history of actual development costs, annual operating
and maintenance costs, and expected life cycle costs) of each item;
o the owner of each item;
o the physical location of each item; and
o the logical (e.g., architectural) location of each item.
For systems, specific IT data elements could be part of the organization's
configuration management process. These data elements could include
schedule data, such as dates of installation, last upgrade, last
maintenance, and last security patch.
As in other critical processes of the ITIM, large and small organizations
may implement this key practice differently. For example, the amount of
administration and supporting infrastructure needed to collect information
on projects and systems depends in large part on the size of the
organization. A smaller organization that has a limited number of systems
may be able to utilize systems that were created for other purposes,
creating reports on an ad hoc basis. Larger organizations, however-in
which IT-related information might be expected to be more extensive and
decentralized-may require a dedicated system to acquire the relevant
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
information and make it available to decision makers in a more structured
manner. In a large, decentralized organization the collection and
reporting of investment information on an ad hoc basis would likely be
unmanageable.
Commitment 2: An official is assigned responsibility for ensuring that the
information collected during project and systems identification meets the
needs of the investment management process.
A designated official is needed to adequately manage this process. The
official will ensure that a process is developed and maintained for
collecting IT investment information so that assets can be accurately
tracked. Staff or external advisors may be assigned to assist the official
in conducting IT asset tracking and in verifying and validating IT
investment data.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, are provided for identifying IT projects and systems
and collecting relevant investment information about them.
These resources typically involve
o managerial attention to the process;
o staff support including, at a minimum, a designated official to manage
the process; and
o supporting tools and equipment for tracking IT assets which may include
o an IT information database;
o IT data reporting, updating, and query tools; and
o a method for communicating changes in IT information to affected
parties.
Activities Activity 1: The organization's IT projects and systems are
identified, and specific information is collected to support decisions
about them.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
A standard, documented procedure is used so that developing and
maintaining the information is a repeatable event, producing IT data that
are timely, sufficient, complete, and comparable. The information may be
prepared by the information systems support component of an organization,
and the verification and validation may be performed by a designated
official or by another organizational unit, depending on the needs of the
organization.
An IT project and system data repository typically includes an inventory
listing of software licenses, planned IT projects, and existing systems
with their own unique identifiers. The repository may also include
information on
o how the project or system fits into the EA;
o the organizational unit that is responsible for the project;
o interfaces and dependencies with other projects and systems;
o the current life cycle phase of the project or system (e.g., being
prototyped, under development, being operated and maintained, etc.) and
associated life cycle events (e.g., current development, modernization, or
enhancement efforts under way);
o the costs to date for the project or system and anticipated future
costs;
o the general category of the project or system (e.g., infrastructure,
software application, hardware replacement); and
o anything else that would be relevant to investment decision making
about the project or system.
For example, a large project could be implemented using an incremental
investment approach. Such an approach would require that the project's
increments or useful segments be identified as part of the repository.
This information would help decision makers select and prioritize the
project's useful segments and align them with other projects and systems.
Activity 2: The information that has been collected is easily accessible
and understandable to decision makers and others.
Section 5: Critical Processes for the ITIM Stages o Stage 2: Building the
Investment Foundation o o Capturing Investment Information
The repository of information about the IT investment is of value only to
the extent that decision makers and stakeholders can and do use it.
Knowledge of the contents of the repository by staff and managers
throughout the organization can help them to avoid duplication of effort
and reconcile overlapping resources. For example, a report in the
repository can be used to better manage the licensing of an organization's
application software by showing individually licensed applications that
may be candidates for group licensing.
Activity 3: The information repository is used by investment decision
makers and others to support investment management.
In order to continue to make informed investment decisions, it is
important to maintain up-to-date information. Maintaining the integrity of
the repository is important to ensuring that it remains a useful
decision-making tool. As projects and systems change (i.e., additions,
updates, and/or deletions), this information should be documented in the
repository. An individual or organizational unit should be designated to
maintain the repository.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio
Stage 3: Developing a Complete Investment Portfolio
Source: GAO.
During Stage 3, the investment board enhances the IT investment management
process by developing a complete investment portfolio. Taking a portfolio
perspective enables the organization to consider its investments in a
comprehensive manner, so that the investments address not only the
strategic goals, objectives, and mission of the organization, but also the
impact that projects have on one another. The organization develops its IT
investment portfolio by combining all IT assets, resources, and
investments that it owns, considering new proposals along with previously
funded investments, and identifying the appropriate mix and synergies of
IT investments that best meet its mission needs, organizational needs,
technology needs, and priorities for improvement. This maturity stage is
comprised of the following four critical processes:
o Defining the Portfolio Criteria is the process of developing
quantitative or qualitative factors such as cost, benefit, schedule, and
risk in order to compare and select projects for inclusion in the
investment portfolio(s).
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
Criteria: IT Assessment Guide (AIMD-10.1.13), 27-29, 45-46 (CCA); OMB IT
Investment Guide, 7-9.
o Creating the Portfolio is the process of comparing worthwhile
investments and then combining the investments selected into a funded
portfolio.
Criteria: IT Assessment Guide (AIMD-10.1.13), 32-35 and 52, (CCA, OMB
A-94, OMB A-130, OMB M-97-0 (2), Capital Programming Guide, 16-17; (CCA,
OMB M-97-0 (2), OMB IT Investment Guide, 6-7.
o Portfolio Review is the process that builds upon the Providing
Investment Oversight critical process from Stage 2 by adding the element
of portfolio performance to the organization's control process activities.
Criteria: IT Assessment Guide (AIMD-10.1.13), 52-55, (CCA, PRA, FASA, EO
13011, OMB A-11, Part 3); Information Technology Investment (AIMD-96-64),
65; IT Assessment Guide (AIMD-10.1.13), 6162, (CCA, GPRA, CFO, OMB A-127,
OMB A-123).
o Conducting Postimplementation Reviews (PIR) is the process for
reviewing IT projects in order to learn from past investments and
initiatives by comparing actual results to estimates. PIRs also serve as
vehicles for evaluating the entire ITIM process.
Criteria: IT Assessment Guide (AIMD-10.1.13), 70-72 (CCA, PRA, EO 13011,
GPRA, CFO, OMB A-130); OMB IT Investment Guide, 12; Information Technology
Investment (AIMD-96-64), 66.
Defining the Portfolio Criteria
Portfolio selection criteria are a necessary part of an IT investment
management process. Developing an IT investment portfolio involves
defining appropriate IT investment CBSR criteria to ensure that the
organization's strategic goals, objectives, and mission will be satisfied
by the selected investments. If an EA, including a sequencing plan,
exists, it should be used as the foundation for developing and updating
the portfolio selection criteria. Portfolio selection criteria reflect the
strategic and enterprisewide focus of the organization and build on the
criteria that are used to select individual IT projects. When IT projects
are not considered in the context of a portfolio, criteria based on
narrow, lower-level requirements may dominate enterprisewide selection
criteria. IT projects
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
sometimes are selected on the basis of an isolated business need, the type
and availability of funds, or the receptivity of management to a project
proposal. Portfolio selection criteria build on the criteria that are used
to select individual projects. The portfolio criteria focus on alignment
with the organization's mission, organizational strategy, and
line-of-business priorities. In Stage 3, portfolio selection criteria are
used by the organization's investment board to select IT investments in
the context of all other investments. These criteria should also be
applied as uniformly as possible throughout the organization to ensure
that decision making is consistent and that processes become
institutionalized. When an organization's mission or business needs and
strategies change, these criteria should be re-examined.
Source: GAO.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
Purpose To ensure that the organization develops and maintains IT
portfolio selection criteria that support its mission, organizational
strategies, and business priorities.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for creating and modifying IT portfolio selection
criteria.
The organization has policies and procedures that outline a systematic
process for creating and modifying the selection criteria. In smaller or
highly centralized organizations, there may not be as critical a need to
institute elaborate polices and procedures to manage portfolio criteria.
If the investment decision-making process is managed for the entire
organization by a compact group, and if the objectives for the
organization's IT investments are well understood and stable, portfolio
selection criteria might be established once and then modified
incrementally year-to-year by this same small group. In large,
decentralized organizations with diverse and evolving objectives, it is
much more critical to solicit input to the development of portfolio
selection criteria and to have a documented process for doing so.
For larger organizations, policies and procedures would typically specify
o the objectives for the portfolio management process;
o a link to the organization's strategic plans, budget processes, and
enterprise IT architecture;
o the key information elements required to create or modify the selection
criteria;
o a description of the roles and responsibilities for creating,
modifying, and prioritizing the selection criteria;
o suggested investment and proposal selection criteria;
o a record of previous selection criteria, their weights and rankings,
and how they were developed;
o triggers for initiating a change in the selection criteria; and
o a list of people to whom the selection criteria should be distributed.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
Commitment 2: Responsibility is assigned to an individual or group for
managing the development and modification of the IT portfolio selection
criteria.
An individual or a working group shall be assigned the responsibility of
developing the selection criteria and any subsequent modifications to
those criteria. The assignment of responsibility is critical because it
creates a point of focus for the successful implementation of this
critical process. Those individuals who are assigned the task of
developing and modifying the criteria should have a good working knowledge
of investment management. Past experience in investment management can be
beneficial when developing the selection criteria. Developing the right
criteria with which to analyze a portfolio of projects is essential for
making sound investment decisions.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, have been committed for portfolio selection criteria
activities.
These resources typically involve
o the time and attention of the executives involved in the process,
o staff to support the activities within this process, and
o supporting tools and equipment.
Prerequisite 2: A working group has been designated to be responsible for
developing and modifying the IT portfolio selection criteria.
A working group is designated to develop and modify the selection
criteria. This group should incorporate the organization's mission,
strategy, and priorities into the criteria. Thus, this group might be the
IT investment board or a subset of the board that includes the CIO or some
other member of the executive management team. While a working group may
develop draft criteria, final approval should fall to the investment board
or to an individual or group that has been designated by the board.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
Activities Activity 1: The enterprisewide investment board approves the
core IT portfolio selection criteria, including cost, benefit, schedule
and risk (CBSR) criteria, based on the organization's mission, goals,
strategies, and priorities.
The selection criteria should be linked directly to the organization's
broader mission, goals, strategies, and priorities. This ensures that the
selected IT investments will support these larger organizational tenets
and purposes. It is important that the criteria also take into account the
organization's IT architecture in orders to (1) avoid unwarranted overlap
across investments, (2) ensure maximum systems interoperability, and (3)
increase the assurance that investments align with strategy as captured in
the EA.
An organization often chooses to establish multiple portfolios to
facilitate the investment process. This grouping of investments with
similar characteristics can enable the organization to clarify the value
of certain types of investments-such as infrastructure or e-government
systems-by developing criteria that focus on the contribution each type of
investment makes to the organization. Also, the organization can determine
beforehand how to distribute funding across the portfolios. Ultimately,
the investment board should assess each investment as part of the single
enterprise portfolio-that is, the aggregation of all of the smaller
portfolios. The selection criteria used for assessing and ranking
individual investments and proposals should generally include the four
essential investment elements: cost, benefit, schedule, and risk. The
assessment may also include other criteria, which serves to enhance the
evaluation of each investment's strategic alignment and synergy with other
projects. Organizations typically focus on these four areas and develop
multiple measures under each broad element.
o Cost may include life cycle costs broken apart into initial costs,
ongoing development costs, and indirect costs.
o Benefit may include tangible benefits and intangible benefits estimated
using a variety of techniques (e.g., cost/benefit analyses using net
present value, return on investment calculations).
o Schedule may include the life cycle schedule and the schedule of
benefits.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Defining the Portfolio Criteria
o Risk may include investment, organizational, funding, and technical
risks.
The organization must determine how these criteria are to be used to
select IT investments for the portfolio. Costs and benefits are both
affected by risks. A risk-adjusted return on investment could combine all
of these categories. The selection criteria also may include a description
of an investment's or proposal's minimum or maximum acceptable CBSR
thresholds (e.g., a minimum acceptable return on investment hurdle rate or
a maximum acceptable schedule length).
An organization could use a weighting schema when creating the selection
criteria. The organization would then derive weights for each of the broad
categories, as well as any subelements related to each category. This
would help the organization prioritize those subelements that it considers
the most significant (e.g., an organization that has limited experience
developing systems may give technical risk a greater weight than projected
cost). Alternatively, other risk analysis methods might incorporate the
same "weighting" effect.
The mixture of weights among the ranking criteria will vary from
organization to organization. The weighting schema used should take into
account the organization's unique mission, capabilities, and limitations.
The organization may also create different weighting schemas for different
kinds of investments (e.g., operational, infrastructure, applications
development investments, R&D). These weights may need to be refined over
time as the organization gains more operational experience using the
weighting schema. Additionally, as a starting point, the organization may
want to borrow selection criteria used by other comparable organizations.
Ultimately, the criteria should reflect the priorities of the
organization. Often, the most senior investment decision makers are
involved in the development of these criteria.
Activity 2: Project management personnel and other stakeholders are aware
of the portfolio selection criteria.
The criteria should be distributed to each IT investment board and all of
the IT project managers, organizational planners, and any other interested
parties. The selection criteria should be clearly addressed in funding
submissions for IT projects.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
In a large organization with multiple IT investment boards, a lower-level
board may add its own criteria that would deal with lower-level
requirements, but the portfolio-level criteria would always take
precedence.
Activity 3: The enterprisewide investment board regularly reviews the IT
portfolio selection criteria, using cumulative experience and event-driven
data, and modifies the criteria as appropriate.
The IT criteria for selecting investments may be changed based on (1)
historical experience; (2) changes in the organization's strategic
direction, business goals, or priorities; or (3) other factors, such as
increased IT management capabilities or technological changes. Ultimately,
however, the task of modifying the criteria will be based on the
experience and judgment of the enterprisewide investment board.
Creating the Portfolio Individual IT investments vary in type and purpose.
Some investments may involve purchasing hardware, others developing
software, and still others operating or maintaining IT systems. The
organization may choose to organize its investment process by considering
investments within smaller portfolios (as described in Defining the
Portfolio Criteria). These subordinate portfolios can help facilitate the
prioritization of investments within business or service categories. The
development of the portfolio is an ongoing process that includes decision
making, prioritization, review, realignment, and reprioritization of
projects that are competing for resources and funding. The process for
creating the portfolios should ensure that each IT investment board
manages investments according to an organizational, strategic-planning
perspective. The boards should collectively analyze and compare all
investments and proposals to select those that best fit with the strategic
business direction, needs, and priorities of the entire organization. This
is the fundamental process through which investments are selected into the
portfolio.
Additionally, each organization has practical limits on funding, the risks
it is willing to take, and the length of time for which it is willing to
incur costs for a given investment before benefits are realized. To
address these practical limits, the process of creating the portfolio
primarily uses categorization to aid in investment comparability and CBSR
oversight. Categorization involves grouping investments and proposals into
predefined logical categories. Once this is accomplished, investments and
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
proposals can be compared to one another within and across the portfolio
categories, and the best overall portfolio can then be selected for
funding.
Fundamental to the comparison of investments is an appropriate analysis of
each investment. During Stage 2, the primary basis for comparison is CBSR,
and each investment's performance is compared with those dimensions.
However, in Stage 3 the basis for comparison expands to include more
factors related to alignment, such as the degree of correlation to the
organization's planning, market position, financial objectives, and
business environment. Also, characteristics of each investment that could
potentially influence the value of other investments in the portfolio-and
at the same time be influenced by other investments-should be taken into
consideration. This process may be greatly aided by establishing EA
compliance as a fundamental requirement for selection and by ensuring that
the final portfolio is consistent with the EA as a whole.
Source: GAO.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
Purpose To ensure that IT investments are analyzed according to the
organization's portfolio selection criteria and to ensure that an optimal
IT investment portfolio with manageable risks and returns is selected and
funded.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for analyzing, selecting, and maintaining the
investment portfolio.
As part of the process for selecting an investment portfolio, each IT
investment board should have policies and procedures in place to help them
select the most promising proposals and to ensure that the most feasible
investments are considered. These policies should include specific
screening criteria to help identify and expedite the selection of the most
promising projects. To the extent possible, in order to help minimize
risk, the organization should have a policy in place to ensure that
projects are proposed in useful segments or "modules" that are short in
duration, small in scope, and useful, even though the project may, at some
point, be discontinued There should also be a documented process for
reconciling differences between the IT investment portfolio and the
organization's EA. Reconciliation may include an EA waiver or modifying
the EA to include the delinquent investment. Also, as part of the process
for selecting the portfolio, a structured and proven investment analysis
(e.g. Return on Investment and Benefit/Cost Analysis) should be required.
The results from the analysis should be used to help support portfolio
decisions and ensure that the organization is aware of the financial as
well as other internal and external effects.
The organization's policies and procedures for analyzing and developing IT
investment portfolios typically
o provide common definitions for IT investment portfolio categories,
o apply to each IT investment board as each develops its comprehensive IT
investment portfolio, and
o stipulate conditions that should be met for investment funding
decisions where exceptions are made.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, are provided for the process of creating the
portfolio.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
These resources typically involve
o managerial time and attention to focus on creating the portfolio,
o staff support for carrying out activities within this critical process,
and
o supporting tools and equipment to be used by the staff in creating the
portfolio.
Prerequisite 2: Board members are knowledgeable about the process of
creating a portfolio.
Understanding the principles behind the portfolio creation process is
critical to successfully executing this process. Thus, it may be necessary
to train board members to ensure that they are familiar with the goals of
the process and can carry out their responsibilities competently.
Knowledge building and/or training may be provided ranging from
o in-depth courses for new members to
o a mandatory annual overview for all board members of the investment
process, current process modifications, operational procedures for
selecting investments, control, and evaluation.
Prerequisite 3: The investment board is provided with information
comparing project and system performance with expectations.
The organization has defined the common portfolio categories that will be
used across the organization when each IT board creates its portfolio of
IT investments (if the organization has more than one board). The creation
of these common categories (1) aids in the comparison of similar
investments across the organization and (2) helps the boards create a
common set of definitions.
Common portfolio categories should enhance decision making during the
portfolio creation process. The organization should use categories that
mirror its business strategy and goals. Organizations also need to
consider their EA when developing their IT portfolio. By using the
organization's EA framework to identify and establish the "as-is"
environment, the "to-be" environment, and the transition plan, decision
makers have an explicit and
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
meaningful structural frame of reference for making better IT decisions.
For example, the portfolio categories might be established by
o aligning IT spending with the strategic goals of the organization-which
types of projects, across which groups and which service lines;
o defining spending levels for the portfolio categories, for example, XX
percent to technology development, XX percent to new services, XX percent
to infrastructure projects, XX percent to technology enhancements and
improvements; and
o prioritizing IT projects within the portfolio categories.
Establishing portfolio categorization allows projects to be prioritized
within their own portfolio categories. Moreover, it keeps dissimilar
projects from competing against each other (for example, O&M projects do
not compete against new services projects). At the end of the budget
cycle, resource spending should be more consistent with planned or desired
IT budgets.
The organization may also want to define a set of thresholds for each
common portfolio category. These thresholds should be meaningful to the
organization and useful when making investment decisions, and they should
differentiate the categories. A small organization with relatively few
investments may want to use a simple set of portfolio categories. An
organization using functional categories could define thresholds for each
category, such as
o the maximum investment cost variances, both annually and in total;
o the minimum benefit that a given investment is expected to deliver,
such as return on investment;
o the maximum length of time an investment should take; and
o a maximum risk score derived using an industry-accepted risk assessment
tool.
Activities Activity 1: Each IT investment board examines the mix of new
and ongoing investments and their respective data and analyses and selects
investments for funding.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
After the investments have been assigned to portfolio categories, the
investment board completes the selection process by examining the
portfolio's mix of investments and making final investment decisions that
are justified by sound management principles. To provide decision makers
with an understanding of the relative costs, benefits, schedules, and
risks of each investment and proposal compared to the others, the
organization may use a scoring model or decision support tool. Typically,
such a model or tool compares the costs, benefits, schedules, and risks of
each investment or proposal against the organizational investment criteria
and assigns each investment proposal a score. These scores may then be
used to rank all investments. This ranked list of investments may then
provide a starting point for the decision-makers to apply their judgment
and knowledge of the organization's imperatives as they select investments
for the portfolio. (See also GAO's Executive Guide: Measuring Performance
and Demonstrating Results of Information Technology Investments
(AIMD-98-89, March 1998) for additional guidance on performance
measurement.)
The investment board may have to reconcile imbalances between total IT
funding expectations and funds required for the qualified IT investments
within each portfolio category. For example, the investment board may find
that the funding requests for investments within the O&M category are
higher than expected and that the funding requests for investments within
the R&D category are lower than expected. The investment board can address
this problem by (1) leaving the outcome as it is, (2) modifying the mix of
investments, (3) modifying investment-level funding, or (4) using some
combination of these options.
The investment board can also use other applicable sources of information
when comparing investments and determining each investment's funding.
While the investment board should strongly consider the organizational
priorities created by the selection criteria, it may also want to take
into account
o the qualifications, abilities, and achievements of the project team;
o the unique or significant links between the investment and the
organization's mission, strategies, and plans;
o the historical data, data on similar investments, or their own
investment management experiences; or
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Creating the Portfolio
o other organizational objectives.
Activity 2: Each investment board approves or modifies the performance
expectations for its selected IT investments.
The board modifies or approves annual performance expectations, including
CBSR for each investment. Because some investments may span multiple
years, and most organizations select investments on an annual or biennial
cycle, the investment board needs to approve each investment's performance
expectations on an annual basis (e.g., performance expectations for a
particular investment are to meet or exceed the performance goals by the
end of the first year). Investment expectations should also take into
account each investment's past performance, in addition to serving as the
basis for future board reviews, control process activities, and
postimplementation reviews.
Activity 3: Information used to select, control, and evaluate the
portfolio is captured and maintained for future reference.
The organization creates a repository for capturing information (e.g.,
investment performance expectations and portfolio category thresholds)
related to the portfolio creation process. This repository can be a part
of a larger IT investment management information system or a component of
the information collection and may be centrally located or distributed
within the organization. Storing the information facilitates its use as
part of control process activities, during investment evaluations, future
selection decision making, and future training for board members.
The ability to effectively capture information on past and present IT
decisions can enable an organization to make better decisions on IT
investments during control process activities, as well as during the
evaluation and selection process. In theory, IT investment decisions are
only as good as the information that supports them. Data should be
validated before they are used in the decision-making process. An IT
information system that delivers information that is up to date,
encompassing, and presented in a useful format enhances the decision
process. Organizations sometimes base IT decisions more on judgment,
intuition, partial data, and ad hoc studies than on objective, systematic,
ITrelated information that is routinely collected and analyzed.
Organizations should focus on identifying effective approaches for
collecting, analyzing, and utilizing IT information, as well as other
strategic information.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
Evaluating the Portfolio This critical process builds upon the Stage 2
critical process, Providing Investment Oversight, by adding the elements
of portfolio performance to an organization's investment control capacity.
Compared to less mature organizations, Stage 3 organizations will have the
foundation they need to control the risks faced by each investment and to
deliver benefits that are linked to mission performance. In addition, a
Stage 3 organization will have the benefit of good performance data that
have been generated by Stage 2 processes. Executive-level oversight of
risk management outcomes and incremental benefit accumulation provides the
organization with increased assurance that each IT investment will achieve
the desired results. Expanding this focus to the entire portfolio provides
the organization with longer-term assurances that the IT investment
portfolio will deliver mission value at acceptable cost.
The investment board's role is not to micromanage each investment, but
instead to ensure appropriate executive-level involvement and
participation in monitoring each investment's progress toward achieving
performance expectations. As part of its oversight responsibilities, the
board should also ensure that the investments in the portfolio continue to
maintain alignment with the EA, where one exists. These investment (and
portfolio) expectations are the baseline for periodic performance reviews
that examine the costs incurred; the benefits attained; the current
schedule; the accuracy of project reporting; and the risks that have been
mitigated, eliminated, or accepted to date. For this reason, this critical
process does not simply focus on, for example, the size and attributes of
the benefits for a given investment because benefit expectations were
defined during the process for selecting investments. Instead, this
process focuses on how the investment board monitors and controls the
investment portfolio to ensure that the overall portfolio provides the
maximum benefits at a desired cost and at an acceptable level of risk. One
way the investment board performs this executive-level involvement is by
reviewing the adequacy of the risk management reviews conducted by the
investment board's working group.
Organizations have different approaches for managing and controlling the
performance expectations of projects. In some organizations the review
board designates a working group to review and make decisions on projects
based on their performance at specified project checkpoints or gates. At
these checkpoints, funding and go/no-go decisions are made and projects
are prioritized based on their investment categories. The checkpoints
provide for an in-depth review of each project by the working group. This
approach focuses on designated senior officials reviewing the
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
projects throughout the year. The review board periodically evaluates the
portfolio-once or twice a year-to ensure that the working group's
decisions are sound. These reviews should be conducted from an
enterprisewide portfolio perspective, not at the individual project level.
Source: GAO.
Purpose To review the performance of the organization's investment
portfolios at agreed upon intervals and to adjust the allocation of
resources among investments as necessary.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for reviewing, evaluating, and improving the
performance of its portfolio(s).
The documented policies and procedures typically specify that the IT
investment board is responsible for reviewing, evaluating, and ultimately
improving the performance of the investment portfolio. The policies and
procedures could designate a working group, composed of the project
manager, the executive sponsor, and members of the affected business unit,
to first review performance and then report the results to the investment
board. Actual investment data should be used as the basis for the review.
Other potential policies and procedures might include having the project
manager maintain information on the current status of the investment and
its performance outcomes to date and the scope and frequency of portfolio
performance reviews.
The investment board or designated working group should use a
predetermined performance threshold when analyzing actual versus expected
performance. This threshold is typically defined on the basis of the
measures (e.g., more than XX percent over expected cost). However, it can
include some other significant organization-specific factors (e.g., the
scope of an investment has grown to reach mission-critical importance).
This predetermined threshold will be a major factor in defining the
remedial action for underperforming investments. Changes to the
investment's expectations and commitments are made with the involvement
and agreement of the stakeholders and in concurrence with the investment
board.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools have been provided for reviewing the investment
portfolio and its projects.
These resources typically include
o staff members for managing information associated with tracking
investment performance and
o tools to support the staff members' activities.
Prerequisite 2: Board members are familiar with the process for evaluating
and improving the portfolio's performance.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
Board members should be familiar with how to evaluate and improve
portfolio performance. This knowledge is essential to successfully
executing this process. If board members do not possess prior experience
in evaluating performance, then training may be necessary to ensure that
they are familiar with the evaluation process and can carry out their
responsibilities competently.
Training may be provided, ranging from
o in-depth courses for new board members to
o a mandatory annual overview for all board members of the investment
process, current process modifications, and operational procedures for
investment selection, control, and evaluation.
Prerequisite 3: Results of relevant Providing Investment Oversight reviews
from Stage 2 are provided to the investment board.
Throughout the control phase of investment management, it is important
that decision makers have all relevant data on current projects available
to review. This is a tenet of good portfolio management. The data could
include any results stemming from periodic project oversight. The
investment board oversees project performance through routine reviews at
predetermined checkpoints and/or major milestones. The board or a
designated third party performs the review. The investment board review
compares actual results with predetermined performance expectations for
items such as project cost and adherence to schedule.
Prerequisite 4: Criteria for assessing portfolio performance are
developed, reviewed, and modified at regular intervals to reflect current
performance expectations.
During the control phase, projects are periodically reviewed to assess
their performance. Criteria for assessing portfolio performance must also
be reviewed at regular intervals to reflect current performance
expectations. Criteria that were developed to assess the original
investment portfolio might no longer reflect the organization's strategic
objectives. The IT working group on portfolio selection criteria should
assess the relevance of the selection criteria at least once a year; among
other things, the performance of the portfolio selected should be assessed
against the current criteria.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
Activities Activity 1: IT portfolio performance measurement data are
defined and collected consistent with portfolio performance criteria.
The IT investment board is responsible for monitoring each investment's
performance. The IT investment board or its designated third party
examines actual investment performance to date with each investment's
expectations using the collected investment data. The board is notified of
and reviews any differences between actual outcomes and expectations.
Guidelines for executing this activity include
o using exception reporting techniques to better manage the volume of
data produced by this activity,
o designing performance management systems to collect information used in
this process,
o conducting this review during a formal project review activity,
o documenting annual and life cycle expectations as a basis for the
comparison, and
o using historical organizational performance data and industry baseline
data as a basis for comparison.
(See Stage 3: Creating the Portfolio for a description of portfolio-level
CBSR expectation setting.)
Activity 2: Adjustments to the IT investment portfolio are executed in
response to actual portfolio performance.
It is quite common for adjustments to be made to the investment portfolio
based on actual performance results. The portfolio should be reevaluated
on a continuing basis and adjustments made if necessary. The investment
board identifies IT investments in its portfolio that are underperforming
by comparing each investment's actual performance to stated expectations.
If an investment in the portfolio has been identified as underperforming,
the investment board must take action to address its poor performance. If
the board is unable to improve the performance of the individual
investment, it must consider other options, including terminating the
project early because the goal of the investment board is to maintain the
optimal performance of the portfolio.
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Evaluating the Portfolio
The "balanced scorecard" approach is a tool that the IT investment board
can use to measure the performance of its IT investments. This approach
creates a measurement balance across the overall performance management
framework. A balanced approach to measuring the contribution of IT to
mission outcomes and performance improvement recognizes the broad impact
of IT's supporting role. By measuring IT performance across the four major
goal areas (strategic, customer satisfaction, internal business
performance, and IT innovation and learning), the scorecard forces
managers to consider measurement within the context of the whole
organization. This limits the possibility of overemphasizing one area of
measurement at the expense of others. In addition, measuring IT
performance from different perspectives helps strengthen the analysis of
both the tangible and the intangible benefits that are attributable to
technology. (For more information, see GAO's Executive Guide: Measuring
Performance and Delivering Results of Information Technology Investments,
GAO/AIMD-97-163, September 1997.)
Another tool to consider when measuring performance is the Earned Value
Project Management27 method. This tool measures the actual work being
performed against a detailed plan in order to accurately predict the final
costs and schedule results for a given project. The tool requires that a
plan for project performance measurement be created. The earned value, or
work performed, is then measured against the actual costs of accomplishing
the work, providing a measure of the project's true cost performance. The
method provides project managers with a type of "early warning" system,
allowing them to take corrective action should project spending outpace
the physical work being accomplished. (Earned Value Project Management,
Quentin W. Fleming, Project Management Institute, 2000.)
As a communication tool, organizations may choose to use a spreadsheet or
graphic illustration such as a management "stoplight," with green, yellow,
and red identifiers, to summarize performance metrics for each investment.
The stoplight should be backed up by rigorous controls and measures to
ensure the reliability and validity of the project metrics. A graphic
illustration provides a simple way for board members to quickly
27For additional guidance on earned value management, see the Defense
Department's Earned Value Management Web site at http://www.acq.osd.mil/pm
and Capital Programming Guide Supplement to Part 7 of Circular No. A-11
(Office of Management and Budget Circular No. A-11, July 2003).
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
understand the status of each investment and any potential emerging
problem areas.
Some investments that the board reviews may exceed performance
expectations (e.g., at lower cost, in less time, and with better benefits
than expected). In these cases, the board may wish to accelerate an
investment's funding or schedule, reallocate resources within the overall
portfolio, or make some other type of adjustment.
Conducting Postimplementation Reviews
The purpose of a postimplementation review (PIR) is to evaluate an
investment after it has completed development (i.e., after its transition
from the implementation phase to the O&M phase) in order to validate
actual investment results. This review is conducted to (1) examine
differences between estimated and actual investment costs and benefits and
possible ramifications for unplanned funding needs in the future and (2)
extract "lessons learned" about the investment selection and control
processes that can be used as the basis for management improvements.
Similarly, PIRs should be conducted for investment projects that were
terminated before completion, to readily identify potential management and
process improvements.
When conducting a PIR, an organization should measure a project's success
not only against the CBSR presented in the business case, but also against
the organizational objectives that are associated with the project. In
certain instances, this is difficult because project objectives are not
always defined clearly and expressed in quantifiable terms that are agreed
to by all stakeholders. Even when the objectives are precise and are
agreed to by all stakeholders, there still might not be a unified view of
the value generated by undertaking the project. Uncontrollable variables
can also affect the outcome of a project, affecting the criteria by which
success is judged. Therefore, sound project and investment management are
essential when assessing the results of a project. In order to judge the
project's results fairly, the PIR team may have to consider what the
objectives were at the time the project was initiated.
The timing of a PIR can be problematic-a PIR conducted too soon after an
investment has been implemented may fail to capture the full benefits of
the new system. In contrast, if a PIR is conducted too late, institutional
knowledge about the investment can be lost. As a general guideline, PIRs
should be conducted within a range of 6 to 18 months after the investment
begins its operational phase. However, this guideline should be adjusted
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
depending upon the nature of the project and the organization's
expectations for when the benefits that are documented in the project
plans should be realized.
Source: GAO.
Purpose To compare the results of recently implemented investments with
the expectations that were set for them and to develop a set of lessons
learned from these reviews.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for conducting PIRs.
The postmplementation review is used to evaluate the overall effectiveness
of the organization's capital planning and acquisition process. The PIR is
used to evaluate an investment following its implementation in order to
validate actual investment results. An organization will have difficulty
performing an effective PIR unless it has established policies and
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
procedures to assess the benefits and performance of its investments. Some
of the more common policies and procedures governing an effective PIR
include
o who conducts and participates in a PIR;
o types and sizes of investments for which a PIR is conducted;
o when it is appropriate to conduct a PIR;
o what information is presented in a PIR;
o the criteria and procedures for tailoring the standard PIR process;
o how conclusions, lessons learned, and recommended management action
steps are to be disseminated to executives and others;
o where PIR information and documents are stored (electronically or
otherwise) for later use; and
o when a PIR-like study should be conducted for other IT-related
initiatives (such as a strategic shift in technology).
A PIR should generally cover
o investment expectations;
o actual investment results (e.g., end-user satisfaction, technical
capability, mission and program impact, unanticipated benefits);
o cost and schedule deviations, such as additional "hidden" costs related
to investments that have been made to enable the primary investment;
o environmental changes that affected the investment;
o a review of the assumptions that were made during the decision-making
period;
o expected next steps for the investment;
o general conclusions and lessons learned; and
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
o recommendations to executives.
Prerequisites Prerequisite 1: Adequate resources, including people,
funding, and tools, have been provided for conducting PIRs.
These resources typically involve
o assigning a team to prepare and conduct each PIR,
o assigning one team member responsibility for leading the PIR, and
o tools to support each PIR, such as
o investment documentation in an asset library,
o spreadsheet programs and templates,
o investment planning and scheduling programs, and
o risk and benefit assessment methods and tools.
In most cases, the project team should actively assist the PIR team in
conducting the PIR.
Prerequisite 2: Individuals assigned to the investment board to conduct
PIRs should be familiar with both the policies and the procedures for
conducting such reviews.
A PIR's success will largely depend on the credibility and competence of
the individuals conducting the review. They should be familiar with
established policies and procedures and be objective, well trained, and
experienced in conducting PIRs. Training might include a class on
assessing projects based on some developed criteria. Also, the team leader
should have past experience conducting similar investment reviews.
Activities Activity 1: The investment board identifies which projects will
have a PIR conducted.
While it may not be necessary to conduct PIRs on all investments, an
organization should review a mix of successful and less successful
projects. By capturing this information and sharing it among those
involved
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
in the investment process, the organization can enhance the success of the
portfolio as a whole.
In accordance with organizational policy, an IT investment board will
identify and designate the projects for which a PIR will be conducted.
Projects need to be selected based on some type of selection criteria.
Then one or more examining teams will conduct the PIR(s) on the selected
projects. The standard PIR process may be tailored to the specific
investment being reviewed.
Often a centralized group (e.g., quality assurance, audit committee, etc.)
will conduct PIRs under the direction of the enterprisewide IT investment
board. Owners or users of the investment under review can participate in
its PIR. This approach enhances the consistency of the data collected and
ensures coverage of the appropriate projects. However, there are other
acceptable approaches that may be used. The organization should employ the
approach that best meets its needs as constrained by its resources.
Activity 2: Quantitative and qualitative investment data are collected,
evaluated for reliability, and analyzed during the PIRs.
As part of the objective analysis of the investment, quantitative PIR data
are collected. These data should largely arise from the selection and
control process activities that have been conducted previously. Specific
types of quantitative data can include
o performance expectations and actual outcomes;
o updated performance data and explanations for changes;
o measures of business or mission objectives, such as operating costs,
schedule, and product cycle time;
o measurements of improved technical capability; and
o contribution of the investments toward achieving both the strategy and
the objectives of the organization's IT strategic plan.
In addition to quantitative investment data, qualitative information, such
as the perspectives and insights of project participants and end users,
may serve to validate or raise questions about the quantitative
information and
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
the existing investment management processes used by the organization.
Qualitative data can include
o surveys and interviews of end users, customers, project management,
project staff, contractors, and developers;
o project management and staff interviews; and
o interviews with senior decision makers involved in investment
oversight.
Some common techniques for performing analyses during a PIR can include
o conducting trend analysis using historical investment data,
o conducting a review of how benefits are realized,
o conducting means-end analysis to compare results with known causal
factors, and
o performing force field analysis to understand the effects on the
investment of major decisions that were made.
Activity 3: Lessons learned and recommendations for improving the
investment process are developed during the PIR, documented, and then
distributed to all stakeholders.
The investment board, with the assistance of the project team, conducts a
review of the investment selection and control process in order to extract
"lessons learned." Based on these lessons learned, recommendations can
then be developed to (1) improve the investment process (i.e., select,
control, and evaluate) and (2) improve the management of individual
investments. Additional recommendations could include improving the
development of the business case, refining the project/portfolio selection
criteria, or determining whether or not the objectives of the project are
being met.
Once the PIR has been documented, it should be made available to all
stakeholders and then archived for future reference. A positive outcome of
identifying lessons learned and developing recommendations is having the
Section 5: Critical Processes for the ITIM Stages o Stage 3: Developing a
Complete Investment Portfolio o o Conducting Postimplementation Reviews
information available for future reference. Sharing experiences, both good
and bad, promotes improvements to the overall investment process.
Using the collective results of PIRs from IT projects during different
stages of their life cycles, organizations can learn valuable lessons and
gain insights. The results from one project will often not provide enough
information to allow significant modifications to be made to the
investment decision-making process. PIRs help organizations focus their
evaluation efforts on areas they deem important and avoid pitfalls and
problems that others have experienced. Furthermore, PIRs help
organizations address specific issues that may be impeding progress toward
the effective management of an IT investment or the development of an
assessment framework to fine-tune the IT investment process.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process
Stage 4: Improving the Investment Process
Source: GAO.
The primary focus of Stage 4 is on improving the overall performance of an
organization's IT portfolio. In addition, the critical processes
associated with this stage help the organization to manage the succession
of low-value operating IT systems to higher-value, follow-on investments.
Thus, this stage comprises the following two critical processes:
o Improving the Portfolio's Performance is the process for evaluating the
performance of the portfolio and using this information gained from the
portfolio to improve both current IT investment processes and the future
performance of the investment portfolio.
Criteria: IT Assessment Guide (AIMD-10.1.13), 73, 78, 80 (CCA, GPRA, OMB
A-130, OMB A-127, OMB A-123).
o Managing the Succession of Information Systems is the process for
analyzing and managing the replacement of identified IT investments and
assets with their higher-value successors.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Improving the Portfolio's Performance
Criteria: SIM Executive Guide (AIMD-94-115); Year 2000 Computing Crisis:
An Assessment Guide (AIMD-10.1.14), 10; Capital Programming Guide, 54-55.
Improving the Portfolio's Performance
Ultimately, an organization needs to know how well its collected pool of
investments in information management and technology is contributing to
improvements in mission performance. Improving the Portfolio's Performance
is, at the level of the investment portfolio, the equivalent of Stage 3's
PIR for an investment. This critical process seeks to determine how well a
portfolio of IT investments is (1) helping to achieve the strategic needs
of the enterprise, (2) satisfying the needs of individual units and users
with IT products and services, and (3) improving IT business performance
for users and for the enterprise as a whole. To determine these things,
performance information for an organization's entire portfolio of
investments should be compiled and analyzed and trends examined.
Key input for these reviews includes PIRs, the IT investment board's
experiences, and results to date for major investments, extracted from
control process activities. These data are generally project or
investmentspecific and often are not aggregated for general trend
analysis.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Improving the Portfolio's Performance
Source: GAO.
Purpose To assess and improve the performance of the IT investment
portfolio and the investment management process.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for evaluating and improving the performance of
its portfolio(s).
These policies and procedures typically specify
o that each IT investment board be responsible for managing a
comprehensive portfolio evaluation and improvement process,
o that access to portfolio data be provided and confidential/sensitive
data be appropriately controlled,
o that each portfolio be evaluated at least annually to assess its
performance,
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Improving the Portfolio's Performance
o a mechanism for assembling and aggregating the investment performance
data,
o the key measures and methods used to assess portfolio performance
(e.g., a "balanced scorecard" approach),
o methods for analyzing the performance data,
o methods for comparing portfolio performance and portfolio expectations,
and
o a mechanism for reporting the results of the analysis.
Prerequisites Prerequisite 1: Adequate resources are provided for
evaluating and improving the portfolio's performance.
These resources can include
o support staff for executing the activities in this critical process,
o methods and tools to aid the teams conducting the PIRs, and
o current and historical portfolio data.
Prerequisite 2: Board members who are responsible for evaluating and
improving the investment processes and investment portfolio(s) exhibit
core competencies in evaluating and improving portfolios.
These board members should be familiar with the IT investment management
approach. Training for this critical process may also include
familiarizing executives with techniques for economic and process
management analysis. Training in quality management analysis and tools may
also be helpful.
Knowledge building and/or training may be provided, ranging from
o in-depth courses for new members to
o an annual overview, for all board members, of the investment process,
current process modifications, and operational procedures for investment
selection, control, and evaluation.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Improving the Portfolio's Performance
Activities Activity 1: Comprehensive performance measurement data for IT
portfolios are defined and collected using agreed-upon methods.
The portfolio of investments should be evaluated on its ability to meet
the strategic needs of the organization, provide general user satisfaction
with product and service delivery and management, and deliver effective
and efficient IT business functions (e.g., applications development,
infrastructure availability, project performance). A combination of
quantitative data and supporting qualitative information can be used to
construct a picture of the organization's IT portfolio performance. This
can be analogous to developing a balanced scorecard for IT investments.
(For more information, see GAO's Executive Guide: Measuring Performance
and Delivering Results of Information Technology Investments,
GAO/AIMD-97-163, September 1997.)
Data collection and information synthesis should focus on answering key
portfolio performance questions, such as the following:
o Is IT spending in line with expectations?
o Are we consistently producing cost-effective results?
o How well is the portfolio being managed?
o Are users satisfied with the products and services that are being
delivered?
o Are IT projects delivering their expected share of process
improvements?
o How well are integrated project teams being used on major investment
projects?
o Are quality IT products and services being delivered within general
industry standards?
o Are accepted methods and tools being used on major systems investment
projects?
o Is the IT infrastructure providing reliable and needed support for the
organization?
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
Measures should be constructed to help objectively determine performance
outcomes in these areas. In addition, the results of individual PIRs, as
well as internal and external audits or reviews, should be examined. Other
types of analyses, such as total cost of ownership, can also provide
useful performance data on specific IT portfolio categories, such as
infrastructure O&M.
Activity 2: Aggregate performance data and trends are analyzed.
Trend analysis and reports can help provide evidence that the investments
in the IT portfolio have helped to achieve expected improvements in the
effectiveness and efficiency of operations or service delivery. To make
this a meaningful exercise, it is critical to develop baseline performance
data.
Activity 3: Recommendations for improving the investment process and
portfolio are developed and implemented.
Based on its analysis of the data, the organization develops
recommendations for improving the investment process and the portfolio.
After the board approves the recommendations, a plan should be put in
place for implementing them. There are always problems in implementing
change, but there are opportunities as well. Addressing problems or
opportunities usually involves
o developing recommendations for the IT investment board;
o documenting the decision criteria, justification, and rationale for
investment decisions;
o defining the expected benefits of the action recommended;
o making a decision on implementing each recommendation; and
o tracking the recommended action as it is implemented.
Managing the Succession of This critical process develops the capability
for (1) planning and managing
Information Systems the migration of IT investments to their successors
(i.e., replacement systems, software applications, and hardware) and (2)
retiring low-value or high-cost IT investments. This critical process also
enhances the organization's ability to forecast, plan, and manage the
migration to new system investments. The target EA and sequencing plan can
be useful
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
guides in evaluating which investments should be phased out and which ones
should be retained by the organization. Using these guides allows an
organization to make investment decisions that align with its EA. Such a
strategy ultimately helps the organization plan for the future by aligning
its investment decisions with its mission goals and objectives.
This critical process is significant because some IT investments can
outlive their usefulness and yet acquire organizational inertia or
entrenchment, consuming resources that begin to outweigh their benefits
and obscuring the full cost of operations and maintenance. This inertia or
entrenchment can often occur because these assets (1) have created
important constituencies within the organization, (2) have a number of
popular user features, even though the total system cost exceeds the total
system benefits, or (3) have not had an alternative IT analysis performed.
Organizations at this maturity stage develop investment "exit criteria"
such that investments can be "deselected" appropriately. This critical
process supports a migration to a forward-looking, solution-oriented view
of IT investments.
This process differs from the reselection activity in Stages 2 and 3 in
that it occurs solely in the O&M phase, whereas reselection can be
considered during the implementation phase of the system development life
cycle.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
Source: GAO.
Purpose To ensure that IT investments in operation are periodically
evaluated to determine whether they should be retained, modified,
replaced, or otherwise disposed of.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for managing the IT succession process.
The organization has documented policies and procedures that define how IT
investments are identified, evaluated, and selected for succession. These
policies and procedures typically specify
o that each investment board make IT replacement decisions within its
business unit,
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
o that the enterprisewide investment board have final authority for
making replacement decisions,
o the coordination of replacement decisions across multiple investment
boards,
o the procedures for managing the migration of IT systems to their
successors, and
o the procedures for disposing of retired IT systems.
Commitment 2: An official is designated to manage the IT succession
process.
An official is designated to manage this process. While the IT investment
board decides which investments to continue, change, replace, or retire,
this official is responsible for managing the succession process and
ensuring that the board's plans are executed.
Prerequisites Prerequisite 1: Adequate resources are provided for
conducting IT succession.
These resources typically involve
o the attention of executives involved in this process;
o staff, such as programmers and O&M, to support this process; and
o supporting tools and equipment for the staff to use.
Prerequisite 2: Investment board members exhibit core competencies in
activities involving the succession of information systems.
To make competent replacement decisions, board members must have
sufficient training to carry out their role. Because this critical process
is similar in its core concepts to the project selection process, the IT
succession training can be tied to selection-related training.
Knowledge building and/or training may be provided, ranging from
o in-depth courses for new members to
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
o an annual overview for all board members of the investment process,
current process modifications, and operational procedures for investment
selection, control, and evaluation.
Prerequisite 3: Investment information from the repository is used by the
investment board.
A repository of investment information helps to ensure that each board is
aware of all of the investments and resources for which it is responsible
and of which system owner/manager(s) will be affected by system
replacement decisions.
(See also Stage 2: Capturing Investment Information for a description of
the activities associated with developing a collection of IT investment
information.)
Activities Activity 1: The investment board develops criteria for
identifying IT investments that may be ready for replacement.
Each investment board develops the criteria that determine which types of
investments are candidates for replacement. In an organization with
multiple boards, the enterprisewide board should formulate the criteria
first. The criteria should then cascade down to the lower boards. A lower
level board may have separate criteria for investments strictly within its
domain.
These candidates for replacement might include investments
o at, near, or exceeding their planned life cycles;
o in their O&M phases;
o that have encountered significant data conversion problems;
o that are based significantly on assumptions that are no longer valid
(e.g., investments that were based on a type of technology that is now
obsolete); and
o for which a replacement application or hardware technology is imminent
or planned.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
(See also Stage 2: Instituting the Investment Board for a description of
the manner in which multiple investment boards interact.)
Activity 2: IT investments are periodically evaluated and appropriate
investments are identified as candidates for replacement.
The defined criteria are applied to the IT portfolio to identify the
replacement candidates. The analysis will generally be done case by case,
looking at the continuing business case and mission benefits surrounding
each candidate and the emerging technologies that are possible successors.
The analysis should be based on the performance factors for each candidate
under consideration (e.g., the ongoing costs of O&M, the risk of hardware
loss due to unavailability of spare parts). This analysis may require
managerial judgment to determine the merits of each particular case or the
prospects for a particular candidate. Also, it is imperative that the
investment's sponsor, manager, and/or owner be involved with this
activity.
Beyond the normal process of retiring older systems, this activity may be
triggered by a variety of other events. For example, after undergoing a
significant strategic realignment or shift in its underlying IT
architecture, the organization will probably want to engage in this
activity to ensure that its IT resources are being utilized efficiently.
(See also Stage 2: Capturing Investment Information for a description of
the activities associated with creating an inventory.)
Activity 3: The interdependency of each investment with other investments
in the IT portfolio is analyzed.
Some of the investments that are identified as candidates for replacement
may be interdependent with other investments and projects. The purpose of
this activity is to identify potential interdependencies among investments
and to assess the effects that replacing any of these investments would
have on the others. Contingency plans should be prepared to mitigate any
negative impact that replacing an investment might have on remaining
investments. The board may find it necessary to revise the replacement
plans for some investments based on an analysis of the effects replacement
could have on other investments.
Section 5: Critical Processes for the ITIM Stages o Stage 4: Improving
the Investment Process o o Managing the Succession of Information
Systems
Activity 4: For each IT investment that has been identified as a candidate
for replacement, the investment board decides whether or not to replace
it.
Decisions on whether to replace an investment will usually fall into the
following categories:
o Retain/continue. Take no replacement actions and continue to operate
and maintain the current investment.
o Fix. Propose repairs to the investment so that it once again meets a
predefined level of performance or business need.
o Enhance/improve. Propose modifications to the investment so that it
provides greater functionality, lasts longer, or costs less.
o Replace. Propose replacing the investment with a new or different
investment.
o Combine or disaggregate. Propose combining the functionality or
technical attributes of one or more investments or break an investment
apart and manage each piece individually.
o Retire/dispose. Terminate the investment and dispose of it.
Succession plans are implemented as needed to ensure timely and effective
replacement of investments within the context of the IT investment
management process.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes
Stage 5: Leveraging Information Technology for
Strategic Outcomes Figure 21: The ITIM Stages of Maturity with Stage 5
Critical Processes
Source: GAO.
At Stage 5, an organization leverages its IT investment capabilities both
to anticipate the effects of next-generation information technologies and
to significantly drive strategic business transformation. As organizations
harness the capability to run effective management processes for
constantly selecting, controlling, and evaluating IT investment, they can
more effectively examine how best to institute major business
transformations to better achieve their missions. These transformations no
doubt will include fundamental changes made possible through the
application of new information technologies to support major innovation in
customer interaction, service delivery mechanisms, and more effective
knowledge management. One essential success factor is to institute
effective processes capable of analytically sorting through technology
choices of increasing complexity.
Organizations at Stage 5 are focused on continuous improvement and
strategic decision making aimed at anticipating and utilizing technology
options to drive desired business transformation outcomes. Two critical
processes are central to this stage.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Optimizing the
Investment Process
o Optimizing the Investment Process is the process used to exploit IT
decision making to improve the value of an IT investment management
process. Best practices of other organizations are captured to improve the
IT investment process-leading to world-class outcomes. The focus of these
activities is cross functional, broad, and strategic in nature.
Criteria: CCA, Section 5123 (5); Benchmarking course material from
CCI, Inc.; Best Practices in Information Technology: How Companies
Get the Most Value From Exploring Their Digital Investment, James
Cortada; The Information Paradox: Realizing the Business Benefits
of Information Technology, John Thorpe; Business Process
Improvement: The Breakthrough Strategy for Total Quality,
Productivity, and Competitiveness, H. James Harrington; Better
Change: Best Practices for Transforming Your Organization,
PricewaterhouseCoopers.
o IT-Driven Strategic Business Change is the process for using
information technology to strategically renovate and transform work
processes and push the organization to explore new and better ways to
execute its mission. The CIO should be directly involved in this process
to ensure that IT strategy is tightly linked to the business strategy and
that senior management is "on board" with the IT strategy so that funding
does not become an issue.
Criteria: CCA, Section 5123 (5); Breakthrough Process Redesign: New
Pathways to Building Customer Value, Charlene Adair and Bruce Murray;
Transforming the Public Sector, David Osborne and Ted Gaebler; The
Innovator's Dilemma, Clayton M. Christensen; Quality is Free: The Art of
Making Quality Certain, Philip B. Crosby.
Optimizing the Investment Process
The purpose of this critical process is to measurably improve IT
investment processes by learning from and adopting the tools, techniques,
or methods used by best-in-class external organizations. Improvements can
include using innovative investment oversight tools and techniques,
changing the mechanics of investment management, or improving the "lessons
learned" feedback mechanism. This process is part of an effort to
continually improve the value of the organization's IT investments.
Aspects of this process, such as measurement of the IT investment
management process, can be implemented in earlier stages; at Stage 5,
process measurement becomes an absolute necessity.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Optimizing the
Investment Process
Process-based benchmarking-the first step in this critical process-is a
structured technique for measuring an organization's IT investment
management processes. It is different from traditional measurement-based
benchmarking where an organization compares its performance, cost, and
cycle time to those of competitors, to industry averages, or to a
consultant's proprietary data. Once they are benchmarked, an
organization's IT investment management processes can be modified and
improved using the tools, techniques, or methods learned from
"best-in-class" organizations. The performance gains resulting from
implementing these process modifications can be measured and should result
in IT investment management processes that meet or exceed those of the
"best-in-class" organizations.
Source: GAO.
Purpose To identify and implement measurable improvements in the
organization's IT investment management processes so that the processes
meet or exceed those used by best-in-class organizations.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Optimizing the
Investment Process
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for improving its IT investment management process
using benchmarking.
These policies and procedures typically specify the following:
o As part of the benchmarking activity, performance measurements for the
IT investment management process are collected and analyzed to form a
process baseline. This baseline should include
o the current, documented IT investment management process,
o performance measurement definitions, and
o the expected performance measurement range.
o Historical data are used to analyze current performance.
o Leading practices are identified in external organizations to be used
as benchmarks for process improvements.
o Significant changes to business processes are approved by senior
management.
o The baselines and benchmarks are revisited and updated periodically.
Commitment 2: An official is designated to manage the benchmarking
activities.
The organization designates an official to manage this process. This
official is responsible for managing the benchmarking activities, ensuring
that team members are well trained, and serving as the focal point for
this critical process. Whether this official has benchmarking as his or
her sole duty may depend on the specific circumstances of the
organization. A smaller organization, with relatively simple investment
processes, may not require extensive external benchmarking, and so the
effort to carry out this critical process may be limited. In a larger
organization, with complex and multifaceted investment activities,
benchmarking activities may be extensive and require more focused
attention by the responsible official.
Prerequisites Prerequisite 1: Adequate resources are provided for
conducting process benchmarking activities.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Optimizing the
Investment Process
These resources can include
o external organizations or individuals who are responsible for measuring
investment process performance and
o tools to support investment process measurement.
Prerequisite 2: Organizational managers and staff with responsibilities in
this area are trained in process benchmarking techniques or are
experienced in using these techniques.
For the benchmarking results to be valuable and useful, benchmarking team
members must know how to conduct benchmarking studies. To ensure that they
are competent, team members must either have recent benchmarking expertise
or receive training.
Activities Activity 1: Baseline data are collected for the organization's
current IT investment management processes.
The board tasks a group with measuring the current state of the investment
management process in order to provide a baseline for evaluating expected
and actual process changes. Creating this baseline usually involves
identifying and gathering process data on the components of the investment
management process. These data typically include
o the level of resources an organization expends in conducting IT
investment activities;
o quantitative process results, such as returns on investment and
tangible benefits achieved;
o qualitative process results, such as measures of customer satisfaction
and contributions to mission achievement; and
o the predefined range of values expected from the performance
measurement.
Activity 2: External comparable best-in-class IT investment management
processes are identified and benchmarked.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
The purpose of this activity is to find and learn from organizations with
more efficient and effective investment management processes. Tasks for
doing this include
o identifying best-in-class organizations;
o collecting data from internal, private, and public sources about
best-inclass organizations;
o visiting several best-in-class organizations;
o developing working relationships with one or more of these
organizations; and
o benchmarking components of the best-in-class organizations' investment
management processes.
Activity 3: Improvements are made to the organization's investment
management processes.
Once an organization has learned from the best-in-class external
organizations, it must apply this knowledge to its own processes. Thus,
the organization should
o decide on improvement goals and expectations;
o develop appropriate target activities that will result in measurable
process improvement; and
o analyze, rank, and choose process improvement activities.
The organization then creates and executes an improvement action plan.
This plan will vary with the type and scope of the benchmarking studies.
Executives should review and approve the action plan before implementing
it so that (1) they are aware of the process changes and (2) other parties
who may be interested in the research and process changes can learn from
these initiatives.
Using IT to Drive Strategic In the earlier ITIM maturity stages, the
organization invested in information
Business Change technologies, making certain that a good business case had
been defined within the context of the IT investment management process
and its
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
enterprisewide investment portfolio. In Stage 5, the organization evolves
its investment thinking toward managing IT-driven change of the overall
business process. IT can provide the opportunity to change business
processes and leverage the organization's human capital.
Information technologies can also provide opportunities for an
organization to move dramatically in new directions in order to meet its
goals.
Citizens and countries are using widely available computer encryption
tools to secure their communications. These tools can be used for creating
"digital signatures," which can support legally binding electronic
transactions and help prevent fraud.
The Internet has created opportunities for (1) organizations to "move
closer" to their customers; (2) business partners to reduce or eliminate
the need for a third-party distribution network; and (3) government
agencies to present one common and integrated service provider "face" for
service requests and service delivery to the citizen (thus reducing the
need for local offices despite the diversity of functions being executed
at the agency). In addition, "smart munitions"-which can find their
targets in any weather, be reprogrammed in flight, or be controlled in
real time by a human far away from the target-are changing the way war is
fought for some components of the military services.
Once an organization can competently manage its IT investments, it must
proactively manage the potential of information technologies to profoundly
influence the organization's strategic direction and outlook. The
organization must develop mechanisms to actively scan its environment for
new opportunities to utilize technology. This critical process describes
the activities associated with employing IT investments strategically to
change the organization's core processes.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
Source: GAO.
Purpose To dramatically improve business outcomes by employing IT
investments strategically.
Organizational Commitments Commitment 1: The organization has documented
policies and procedures for conducting IT-driven strategic business change
activities.
The purpose of these policies and procedures is to define the activities
and tasks to be carried out, the roles of the various parties when
executing this critical process, and how these activities relate to the
organization's ongoing business activities. Because business managers may
be resistant to changing current business processes based on the promises
of new technology, these policies should include incentives for
management's participation in this critical process.
Commitment 2: An official is designated to manage the activities within
this critical process.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
An official is designated to initiate and manage a program to enhance
awareness of state-of-the-technology and new information technologies, and
to encourage the use of selected technologies to plan and manage changes
to the organization's business processes.
Prerequisites Prerequisite: Adequate resources are provided for conducting
ITdriven strategic business change activities.
These resources typically include
o funding support for an IT state-of-the-technology laboratory, test
center, or library;
o technical information and research;
o funding for employing external experts or reviewers;
o staff support for executing this critical process; and
o supporting tools and equipment.
Activities Activity 1: The organization creates and maintains a knowledge
base of state-of-the-technology IT products and processes.
The organization creates the capacity to follow and understand major
technological events and trends. This capacity can be generated using one
of several organizational structures (e.g., an advanced technology group,
a cross-departmental group of experts, a group of external experts, or
technology centers of excellence). A designated official is charged with
managing this group, maintaining the knowledge base, and keeping the
organization up to date on emerging technologies.
Activity 2: Information technologies with strategic businesschanging
capabilities are identified and evaluated.
Emerging trends, events, and technologies that have the potential to
improve the organization's business are identified for further study
(e.g., the growth of the Internet and the World Wide Web or the
proliferation of wireless forms of communication). Particular attention
should be paid to breakthrough technologies that have the capacity to
radically improve the current working environment, business processes,
products or services, or the organization's relationship with its
customers (e.g., permitting staff to
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
telecommute or to create "virtual communities" across the Internet). Also,
to ensure that this activity focuses on applicable information
technologies, the organization should ensure that individuals with
business knowledge and experience are involved as stakeholders in this
activity.
Activity 3: Strategic changes to the business processes are planned and
implemented based on the capabilities of identified information
technologies.
Once it has been concluded that a set of technologies offers a significant
opportunity, senior managers must make the decision to plan for and engage
in the change to the business processes. If the change is significant
enough, managers might wish to create a separate organizational entity
that is (1) uniquely positioned to take advantage of the set of
technologies and (2) independent of the current way of doing business.
As part of planning these changes to the business processes, the
organization should engage in risk-reducing activities such as pilots,
simulations, or the development of prototypes. These risk-reducing
activities are particularly important for large, complex, expensive, or
important process change initiatives. The organization may also want to
seek external review or expertise when conducting these process change
activities. When planning the change, the organization should involve
stakeholders from business, IT support, oversight, and customer groups.
Section 5: Critical Processes for the ITIM Stages o Stage 5: Leveraging
Information Technology for Strategic Outcomes o o Using IT to Drive
Strategic Business Change
Appendix I
Glossary
acquisition: The acquiring by contract, with appropriated funds, of
supplies or services (including construction) by and for the use of the
federal government through purchase or lease, whether the supplies or
services are already in existence or must be created, developed,
demonstrated, and evaluated. Acquisition begins at the point when agency
needs are established and includes the description of requirements to
satisfy agency needs, the solicitation and selection of sources, the
awarding of contracts, contract financing, contract performance, contract
administration, and those technical and management functions that are
directly related to the process of fulfilling agency needs by contract.
action plan: A plan derived from recommendations that identifies the
specific actions that will be taken to improve a process or a project and
outlines a schedule for implementing those actions.
activity: An ITIM key practice that describes the actions necessary to
implement a critical process. An activity occurs over time and has
recognizable results. It typically involves establishing plans and
procedures, performing the work, tracking it, and taking corrective
actions as necessary.
alignment: The degree of agreement, conformance, and consistency among
organizational purpose, vision, and values; structures, systems, and
processes; and individual skills and behaviors.
assessment: An appraisal by a trained team of professionals to determine
the state of an organization's current processes and to identify the
highpriority process-related issues facing an organization. An assessment
may also result in organizational support for process improvement.
asset: Property, funding, technical knowledge, or other valuable item
owned by an organization. Investments typically create assets.
benchmarking: A structured approach for identifying the best practices
from industry and government and comparing and adapting them to an
organization's operations. Such an approach is aimed at identifying more
efficient and effective processes for achieving intended results based on
outstanding practices of other organizations.
benefit: A term used to indicate an advantage, profit, or gain attained by
an individual or organization. Tangible benefits include benefits that can
be explicitly quantified. Such benefits may include reductions in cost,
Appendix I Glossary
increases in productivity, decreases in cycle time, or improvements in
quality of service. Intangible benefits include benefits that may be easy
to identify but that can be difficult to quantify. These benefits may
include more efficient decision making, greater data accuracy, improved
data security, reduced customer burden, or increased organizational
knowledge.
business case: A structured method for organizing and presenting a
business improvement proposal. Organizational decision makers compare
business cases when they are deciding to expend resources. A business case
typically includes an analysis of business process performance and
associated needs or problems, proposed alternative solutions, assumptions,
constraints, and a risk-adjusted cost/benefit analysis.
business process: A collection of related structured activities-a chain of
events-that produces a specific service or product for a particular
customer or customers.
business process improvement: A systematic disciplined approach that
critically examines, rethinks, and redesigns mission-delivery processes
and subprocesses within a process management approach.
Capability Maturity ModelSM: A descriptive model of the stages through
which an organization progresses as it defines, implements evolves, and
improves its organizational processes. This model serves as a guide for
selecting process improvement strategies by facilitating the determination
of current process capabilities and the identification of the issues that
are the most critical to quality and process improvement.
change management: Those activities involved in (1) defining and
instilling new values, attitudes, norms, and behaviors within an
organization that support new ways of doing work and overcome resistance
to change; (2) building consensus among customers and stakeholders on
specific changes designed to better meet their needs; and (3) planning,
testing, and implementing all aspects of the transition from one
organizational structure or business process to another.
cost: A term used to indicate the expenditure of funds for a particular
investment alternative over an expected time period. Cost may include
direct and indirect initial costs plus any periodic or continuing costs
for operation and maintenance.
Appendix I Glossary
cost/benefit analysis: A technique used to compare the various costs
associated with an investment with the benefits that it proposes to
return. Both tangible and intangible factors should be addressed and
accounted for in the analysis.
critical process: A structured set of key practices that, when performed
collectively, contributes to the attainment of a maturity stage.
customer: Individual or organizational entity for whom the product or
service is rendered. The customer may also be the end user.
end users: The individuals or groups who will operate the system for its
intended purpose when it is deployed.
enterprise architecture (EA): An integrated framework for evolving or
maintaining existing IT and acquiring new IT to achieve the organization's
strategic and business goals. A complete enterprise architecture should
consist of both logical and technical components. The logical architecture
provides the high-level description of the organization's mission,
functional requirements, information requirements, system components, and
information flows among the components. The technical architecture defines
the specific IT standards and rules that will be used to implement the
logical architecture.
executive sponsor: The executive who champions the investment, sets the
terms of reference, guides the project team, and receives the progress
reports and recommendations.
failure: The inability of a system or component to perform its required
functions within specified performance parameters.
information system: The organized collection, processing, transmission,
and dissemination of information in accordance with defined procedures,
whether automated or manual.
information technology (IT): The computers, ancillary equipment, software,
firmware, and related procedures, services (including support services),
and other resources that are used by an organization to accomplish a
function.
Appendix I Glossary
institutionalization: The building of corporate culture that supports
methods, practices, and procedures so that they are the ongoing way of
doing business.
inventory: An organized and itemized list of assets (e.g., IT products,
services, or contracts).
IT investment: The expenditure of resources on selected information
technology or IT-related initiatives with the expectation that the
benefits from the expenditure will exceed the value of the resources
expended.
IT investment board: A decision-making body, made up of senior program,
financial, and information managers, that is responsible for making
decisions about IT projects and systems based on comparisons and
trade-offs among competing projects, with an emphasis on meeting mission
goals.
IT investment portfolio: The combination of all IT assets, resources, and
investments owned or planned by an organization in order to achieve its
strategic goals, objectives, and mission.
IT management: The processes and procedures used by IT project managers to
direct, control, administer, and regulate a project team creating an IT
asset in such a way that the resultant product meets its requirements upon
delivery.
IT project: An organizational initiative that employs or produces IT or
ITrelated assets. Each project has or will incur costs, expects or will
realize benefits, has a schedule of project activities and deadlines, and
has or will incur risks.
key practices: The infrastructures and activities that contribute most to
the effective implementation and institutionalization of a critical
process.
succession management: An approach for determining when and how to replace
current investments with other investments that provide greater benefits
at lower costs.
maintenance: The process of modifying a system or component after delivery
to correct faults, improve performance or other attributes, or adapt to a
changed environment.
Appendix I Glossary
maturity model: A model of the stages through which organizations progress
as they define, implement, evolve, and improve their processes. This model
serves as a guide for selecting process improvement strategies by
facilitating the determination of current process capabilities and the
identification of the issues that are most critical to achieving quality
and process improvement.
maturity stage: A well-defined evolutionary plateau toward achieving
mature processes.
milestone: A scheduled event for which some individual is accountable. A
milestone is typically used to measure progress.
mission: The enduring, chartered, long-term goal(s) of an organization.
modification: A change made to a system or component to improve
performance or some other attribute or to adapt the system or component to
function in a changed environment.
need: A capability shortfall such as those documented in a mission needs
statement, deficiency report, or engineering change proposal. A new
technology application or breakthrough may create a new expressed need by
the customer.
organizational commitment: An ITIM key practice that describes the
management actions that are necessary to ensure that the critical process
is established and will endure. An organizational commitment typically
involves establishing organizational policies and senior management
sponsorship that drive the activities for the critical process.
outcome: The actual result, effect, or impact of a business initiative,
program, or support function. Typically, actual outcomes are compared to
expected outcomes.
performance measurement: The process of developing measurable indicators
that can be systematically tracked to assess the progress that has been
made in achieving predetermined goals and using such indicators to assess
progress in achieving these goals.
policy: A guiding principle, typically established by senior management,
that is adopted by an organization to influence and determine decisions.
Appendix I Glossary
portfolio: see IT investment portfolio.
portfolio management: The combination of practices, tools, and techniques
that are used to measure, control, and increase the return on individual
IT investments as well as on an aggregate enterprise level.
prerequisite: An ITIM key practice that describes the conditions that must
exist within an organization to successfully implement a critical process.
A prerequisite typically involves resources, organizational structures,
and training. Along with organizational commitment, a prerequisite must be
in place before a critical process's activities can be undertaken
successfully.
procedure: A documented description of a sequence of actions to be taken
in order to perform a given task.
process: A sequence of steps performed for a given purpose.
process maturity: The extent to which a specific process is explicitly
defined, managed, measured, controlled, and implemented effectively.
Maturity implies a potential for growth in capability and indicates the
sophistication of an organization's processes and the consistency with
which the organization conducts these processes.
project manager: The individual with business responsibility for an entire
project. This individual typically directs, controls, administers, and
regulates a project that involves developing or acquiring an information
system.
project plan: A document that describes the technical and management
approach to be followed for a project. The plan typically describes the
work to be done, the resources required, the methods to be used, the
procedures to be followed, the schedules to be met, and the way that the
project will be organized.
project team: A group of people, each with assigned responsibilities, who
work closely together to achieve the shared objective of delivering,
operating, or maintaining an information system. The project team may work
together on tasks that are highly interdependent. The project team may
vary in size from a single individual assigned part time to a large
organization assigned full time.
purpose: The desired outcome for a critical process.
Appendix I Glossary
Return on Investment (ROI): A financial management approach that is used
to explain how well a project delivers benefits in relation to its cost.
Several methods are commonly used to calculate a return on investment,
including: Economic Value Added (EVA), Internal Rate of Return (IRR), Net
Present Value (NPV), Payback, and the use of nominal qualitative measures.
risk: A term used to define the class of factors which (1) have a
measurable probability of occurring during an investment's life cycle, (2)
have an associated cost or effect on the investment's output or outcome
(typically an adverse effect that jeopardizes the success of an
investment), and (3) have alternatives from which the organization may
chose.
risk management: An approach for addressing the risks associated with an
investment. Risk management includes identification, analysis,
prioritization, and control of risks. Especially critical are those
techniques that (1) help define preventive measures to reduce the
probability of negative events' occurring and (2) ensure that appropriate
countermeasures are deployed to deal successfully with the consequences of
these events.
schedule: A term used to define the time period corresponding to the life
of the investment. The investment schedule typically contains associated
phases and milestones that include: planning, proposal generation,
acquisition or development, implementation, operations and maintenance,
and succession.
selection criteria: Factors that are identified for use by an investment
review board to identify and discriminate among investments for subsequent
funding.
stakeholder: An individual or group with an interest in the success of an
organization in delivering intended results and maintaining the viability
of its products and services. Stakeholders influence programs, products,
and services.
strategic plan: A document used by an organization to align its structure
and budget with its priorities, missions, and objectives.
threshold: The limiting acceptable value of a measurement or technical
parameter, typically a performance requirement.
Appendix I Glossary
validation: The process of determining whether or not the product
delivered at the end of the development process satisfies predefined
requirements.
verification: The process of determining whether or not the products of a
given phase of development fulfill the requirements established at the
start of the phase.
Appendix II
Conducting an ITIM Assessment
As described in the early sections of this guidance, ITIM may be used as a
framework for the continuous improvement of an organization's IT
investment management processes. It may also be used to evaluate the
maturity of an organization's investment capabilities. This appendix
describes the assessment process that individuals and teams may use when
conducting an organizational review using the Information Technology
Investment Management (ITIM) framework.
In the ITIM framework, maturity stages are achieved through implementation
of critical processes. These critical processes were derived from our
research on leading organizations, our reviews of federal agencies, and
comments that we received from external reviewers during the ITIM
development process.
Using ITIM to Assess IT Investment Decision-Making Processes
This framework is designed to complement either a formal audit methodology
or an organization's self-assessment process. Using this assessment
process, the review team completes the following phases:
o prepares both itself and the organization for the assessment,
o collects evidence of organization activities,
o evaluates the organization based on the ITIM framework, and briefs the
organization on its findings.
Using this assessment approach provides a widely accepted, repeatable,
criteria-based assessment process for organizations to use when they are
conducting self-assessments of their IT investment management
capabilities. It also provides the organization with an understanding of
any gaps in its investment management processes that are identified during
the assessment. However, before engaging in an assessment, individuals and
teams should do the following:
o Become proficient with the ITIM framework.
o Review the related GAO and OMB IT investment guidance (see
GAO/AIMD-10.1.13; AIMD-99-32; AIMD-98-89; AIMD-94-115 and OMB A130; A-11,
M-97-12; M-97-02). Understanding this past guidance provides greater
insight into the developmental history, key issues, and critical success
factors associated with the IT investment approach.
Appendix II
Conducting an ITIM Assessment
o Become familiar with generally accepted capital decision-making
approaches and associated analytical tools.
o Gain an understanding, through training or experience, of the basic
concepts behind the development, maturation, and evolution of
organizational management skills and capabilities (i.e., maturity models).
o Have experience assessing organizations using standardized assessment
process and tools.
Summary of ITIM Figure 24 summarizes the three phases of the ITIM
assessment process. Each phase is necessary to ensure that the assessment
team and
Assessment Process organization management have sufficient understanding
of the process and the ITIM approach, that appropriate evidence is
collected to support the assessment, and that the conclusions are founded
on the ITIM framework.
Source: GAO.
Appendix II
Conducting an ITIM Assessment
Phase 1: Prepare for Assessment
Present ITIM Overview Briefing to the Organization
The assessment process begins with the assessment team (hereafter referred
to as the team) defining the scope of the assessment (i.e., department,
agency, or bureau). The scope of the assessment will influence the
location of the assessment (i.e., the physical place where most of the key
people and activities are located), who will attend the team's briefing,
and the extent of documentation required. After the scope of the
assessment has been defined, the team conducts an overview briefing for
the organization being assessed (hereafter referred to as the
organization). This briefing covers the ITIM framework in general, the
assessment process, and any organization-specific factors that are
relevant to the job. The purpose of this briefing is to ensure that the
organization understands
o ITIM and the assessment process (including some techniques for
efficiently and effectively performing an ITIM assessment),
o the anticipated schedule of events,
o the importance of involving the right people,
o the general rules of data collection and evidence, and
o the expected reporting process.
In order for the assessment to return value to the organization, it is
important that senior managers understand the ITIM framework and the
assessment process. This understanding will facilitate the provision of
appropriate information during the assessment and will demonstrate
management commitment to the process. The key factors to consider are the
role(s) of the participants in the organization's IT investment activities
and its decision-making process.
The following organizational stakeholders should be considered:
o the Chief Information Officer (CIO),
o the Deputy CIO,
o representatives from the organization's IT investment board(s),
o representatives from the office of the Chief Financial Officer (CFO),
Appendix II
Conducting an ITIM Assessment
o representatives from the organization's budget and planning offices,
and
o various IT managers.
This overview briefing should take place sufficiently early in the
assessment process to allow the organization to learn from the
presentation and prepare for the assessment. This means that the briefing
should occur at least a month before the start of on-site assessment
activities. As a result of the briefing, the organization should be able
to expedite the assessment by collecting the expected documentation,
identifying the management processes for observation, and providing access
to appropriate, relevant staff for interviews.
Evidence of an IT Investment Management Process
A central component of any ITIM assessment is the team's collection of
evidence about the organization's IT investment management process. The
ITIM framework guides the team's collection efforts by listing examples of
physical, documentary, and testimonial evidence for each ITIM critical
process. The team should evaluate the variety of material with respect to
the standards of evidence (sufficient, competent, and relevant) found in
GAO's Government Auditing Standards, 2003 Revision (GAO-03-673G)- also
known as the "Yellow Book").
Besides collecting documents, a typical ITIM assessment may include
interviews with as many as 25 to 50 people as well as numerous group
discussions and briefings. Even more people can participate through the
use of assessment instruments such as case studies, questionnaires, and
surveys (see Conduct Case Study Reviews below). The number of participants
will depend on the scope of the assessment and the organization's size.
Obtain IT Management Overview and Background Information
In order for the assessment team to have a basis for understanding the
existing investment processes, and to begin to understand the possible
mapping of organizational processes on the framework, the organization's
management should provide the team with an overview of their IT investment
management process. It is incumbent upon the organization to ensure that
its representatives have sufficient knowledge and experience in managing
IT investments within the organization to accurately represent the
organization and answer questions. The team should consider using an
organizational liaison for the duration of the assessment to assist in
identifying and gaining access to knowledgeable staff, providing access to
Appendix II
Conducting an ITIM Assessment
and delivering copies of requested documentation, and facilitating access
to physical evidence.
The organization's overview briefings should provide a high-level
perspective on how the organization manages its IT investments. The
briefings are intended to provide the team with the following:
o An overview of the organization's IT investment management process
(i.e., what the organization does-especially how it selects, controls, and
evaluates its IT investments);
o An explanation of the organization's structure (who does what as
documented in current organizational charts-especially any recent or
anticipated changes);
o A description of how responsibility, accountability, and authority for
the IT investment management process are distributed; and
o An index of relevant documents (the IT investment management processes
contained in documented policies, procedures, and guidance, etc.). The
index should describe how the organization's documents are laid out and
how they relate to one another other.
The organization should also supply other documents and background
information to the team to increase the team's efficiency and prevent
misunderstandings during the assessment process. The following information
may also be included:
o a list of current IT investments (often referred to as the investment
portfolio);
o examples of the data, information, and analyses upon which investment
decisions are based;
o descriptions of the decisions that are made during the investment
process;
o an overview of the organization's mission and business processes (this
may be contained in the organization's current strategic plan);
o a list, with definitions, of terminology unique to the organization;
and
Appendix II
Conducting an ITIM Assessment
o the organization's current investment performance and process
improvement plan.
Refine the Assessment Plan
Based on the initial information it receives from the organization, the
team may refine its assessment plan. For instance, the team should reach
consensus on which critical processes and/or maturity stages are
applicable.
Phase 2: Collect Evidence Attend Briefings, Conduct Interviews, and
Collect Documentary Evidence The purpose of this set of activities is to
obtain supporting evidence in greater depth regarding the organization's
implementation of the key practices and critical processes and to follow
up on issues or questions arising from other evidentiary sources to date.
The amount of additional information to be collected, and the level within
the organization from which it must be obtained, will depend upon many
factors, including
o the evidence obtained to date,
o the maturity of the organization's management processes,
o the organization's size and complexity, and
o the scope of the assessment.
A detailed, revised data collection plan should be developed based on the
information required and the information that was received in the initial
overview and background briefings. The team should focus on the gaps that
remain.
Rather than proceeding sequentially through the critical processes, the
team may find it more effective and efficient in some situations to use
other approaches to collect evidence. These alternative approaches can
include
o collecting evidence from one organizational component at a time for
multiple critical processes (e.g., collect and review all of the IT
investment-related policies from a central policy review committee);
Appendix II
Conducting an ITIM Assessment
o collecting evidence for a single stage from multiple organizational
components (e.g., collect and review all evidence for Stage 2 at one
time); or
o collecting evidence for one ITIM component across all organizational
components (e.g., collect and review all evidence relating to
Organizational Commitments).
If the organization states that it is implementing a critical process
using some set of practices other than the ones described in ITIM, then
these practices should be
o clearly delineated,
o formally approved by the organization, and
o convincingly supportive of the intent of the critical process that it
is supposed to support.
The organization may also provide for the team an in-depth walkthrough of
specific key practices within a critical process. This would provide the
team with evidence of a critical process and would also support the
documentary evidence associated with an assessment of a critical process.
Obtain Information on the Investment Process
Briefings at this point should be focused on those critical processes and
key practices that were insufficiently documented following the initial
background briefings. Processes and practices that are known to be missing
in the organization may be skipped. Presenters should be encouraged to
bring documentation to the briefings for distribution. In many instances
the briefings may evolve into discussions as the team focuses on the
supporting evidence of existing investment management processes.
Conduct Interviews
The purpose of these interviews is to collect supporting evidence from
organization staff who directly participate in the IT investment
management process (e.g., executives, managers, support personnel).
Interviewing a variety of organization staff assists the team in
determining the extent to which the investment process policies and
procedures have been communicated throughout the organization. These
interviews should also point the team to other documentary evidence
(probably located
Appendix II
Conducting an ITIM Assessment
within investment projects) and guide the collection of evidence. (Also
see Conduct Case Study Reviews below.)
Collect and Review Documentary Evidence
The purpose of this step is to review the documentary evidence of how
investment management processes are actually implemented and determine how
well the evidence correlates to the ITIM key practices. This activity is
repeated for each key practice that is being carried out by the
organization.
The team will typically begin by collecting broad, organization-level
evidence (e.g., policy planning documents). This evidence will lead the
team to lower-level, implementation-oriented documentation (e.g., meeting
notes and working papers). In this step the team will
o determine what documentary evidence is available, based on information
provided at briefings and interviews,
o request or collect documentary evidence,
o evaluate the documentary evidence, and
o organize the evidence according to the key practices within the ITIM
framework.
Consolidate Evidence and Collect Follow-Up Evidence
Before the team can make rating judgments of the key practices, critical
processes, and maturity stages under consideration, it must complete the
following:
o determine whether or not the evidence provides a sufficient, competent,
and relevant basis for making a rating judgment;
o assemble, organize, and analyze the collected evidence and consolidate
it into a manageable summary of evidence according to the ITIM framework;
and
o determine what follow-up evidence the team requires to make a rating
judgment and develop a method to collect this evidence. The team must also
decide how to proceed if (1) there is no other evidence available or (2)
the available evidence is ambiguous and/or inadequate.
Appendix II
Conducting an ITIM Assessment
Invariably the team will identify the need for additional analyses or
followup evidence to complete the assessment. To collect this evidence,
the team can either send written questions or requests for specific
evidence, or it can conduct follow-up interviews.
Determine the Sufficiency, Competency, and Relevance of the Evidence
In order to achieve accurate and reliable ratings in the assessment
process, the following evidence guidelines must be met:
o There should be sufficient evidence collected from two or more
(preferably independent) sources to support a rating.
o The evidence must be corroborative and directly relevant or logically
linked to the key practice and the critical process.
o The evidence must provide adequate coverage and be competent. More
specifically,
o testimonial evidence must be from interviews with or presentations by
the staff who perform the related investment management process;
o original documentary evidence must be a direct result of executing the
investment management process; and
o physical observations must be made by team members or other credible,
unbiased third parties.
Under some circumstances, the team may decide that confirmation from three
or more separate evidentiary sources is needed. For example, the team may
realize that a particular individual's interview is significant enough
that it may cause a critical process to be rated as "not implemented." In
this case, the team may decide that this interview, as a single source of
evidence, warrants corroboration from other interviews. As a general rule,
if there is any doubt about whether a rating is valid, the team should
initiate additional information collection efforts.
Consolidate the Evidence
Consolidation helps the team to review and organize the large quantity of
evidence that is typically acquired during an assessment. Evidence
consolidation also provides an opportunity for the team to share
Appendix II
Conducting an ITIM Assessment
interpretations of the collected evidence and enables the team members to
develop a consensus on rating.
During evidence consolidation, the team assesses its progress toward its
goals and reviews the evidence it has collected up to that point. While no
particular format is mandatory, these steps are typically followed (often
they are repeated multiple times):
o Team members index, review, and assess the evidence collected to date.
o Team members identify key practices that require further clarification.
o Team members share opinions of the sufficiency of the evidence and
develop preliminary ratings based on team consensus.
If the team cannot reach consensus, it must (1) identify the evidence
needed to resolve the outstanding issues and (2) generate a plan for
collecting the needed evidence.
Conduct Case Study Reviews
The team may choose specific IT investment projects for in-depth reviews
to validate organization-level evidence and to better understand the
organization's IT investment management process. The decision to conduct
case studies will depend on additional evidence is required to document
investment processes. By evaluating the actual investment processes used
with a variety of investment projects, the team obtains a clearer picture
of
o the investment processes as they have actually have been implemented,
o the consistency with which the investment processes are executed,
o evidence of whether the organization's policies and procedures have
been communicated to the project level,
o the commitment that the organization has to its investment processes,
and
o the beneficial effects that improvements in these processes might have
on the performance of the organization.
Select Investment Projects
The team should select one or more investment projects in each major life
Appendix II
Conducting an ITIM Assessment
cycle phase (e.g., R&D, full-scale development, and O&M). At least one of
the cases should include a high-cost and/or high-risk investment project.
For each project, the team should follow the history of the investment
project as it has cycled through the organization's IT investment process.
Projects may be selected on the basis of whether the required
documentation is available, though this approach may bias the conclusions
that can be drawn from the evidence.
Select Participants
The team also needs to determine whom they expect to participate in these
project-level reviews. In all cases, participants should come from the
investment projects selected and the organizational groups that support
those investment projects. It may also be necessary to include people from
other organizational components (e.g., IT investment oversight staff).
Execute the Review(s)
These reviews will typically cover the following dimensions:
Process conformance-the degree to which the project being reviewed has
gone through the organization's IT investment decision-making process.
Data sufficiency, quality, and completeness-the type, accuracy, and value
of the data used to make investment decisions about the project.
Decisions implemented-the type of decision made and the degree to which it
has been implemented.
Reconcile Differences
In some situations, the results of the case studies may contradict the
preliminary ratings that the team has developed during the assessment of
the organization. In this case, the team should investigate the
contradiction(s), determine their root cause, and modify the preliminary
rating(s) if necessary. As mentioned before, the purpose of the case
studies or surveys is to provide additional corroborative evidence for the
organizational ratings and conclusions that are reached during the
organization-level ITIM assessment.
Phase 3: Final Ratings and Determine the Final Ratings
Assessment Once evidence collection is complete, the team must assess the
consolidated evidence and decide whether each key practice, critical
process, and maturity stage has been successfully executed. The team
Appendix II
Conducting an ITIM Assessment
makes final rating judgments as a group. Developing a consensus, so that
the majority agrees and no one is opposed, ensures that the decision is
fair and that all the evidence has been considered.
ITIM is a hierarchical framework, so the rating of each higher-level
component is entirely dependent on the components below it. That is, if
any key practice is not executed satisfactorily, its corresponding
critical process is not implemented satisfactorily, and the corresponding
maturity stage cannot be considered complete. Because of this hierarchical
prioritization, the team must begin by rating key practices and work its
way up the hierarchy. The sequence of ratings is as follows:
o First, key practices are rated.
o Next, critical processes are rated.
o Last, the ITIM maturity stage is determined.
The team members should devise a method and a mechanism for tracking and
documenting the rating judgments as they are being made. Besides creating
a reproducible "audit trail," these supporting documents are useful when
delivering summary results.
Determine a Rating for Each Key Practice
Key practices are rated as
o "executed" or
o "not executed."
An ITIM key practice is successfully "executed" if (1) the team judges
that the key aspects of the practice are being carried out by the
organization or (2) the organization presents the team with convincing
evidence that an alternative practice achieves the same outcome. An ITIM
key practice is "not executed" if there are significant weaknesses in the
organization's execution of the practice and if no adequate alternative is
in place. If the team has found no evidence of a practice during the
assessment process, that result would support a key practice rating of
"not executed."
If the team rates a key practice as "not executed," the organization
should be given an opportunity to produce evidence that might mitigate or
refute
Appendix II
Conducting an ITIM Assessment
the evidence that indicated this rating. By double-checking, the team
avoids making ratings based on incorrect information.
Determine a Rating for Each Critical Process
Critical processes are rated as
o "implemented,"
o "not applicable,"
o "not implemented," or
o "not implemented, but improvements under way."
An ITIM critical process is "implemented" if all of its underlying key
practices are successfully executed or if a satisfactory alternative is in
place. An ITIM critical process is "not implemented" if there are
significant weaknesses in the organization's execution of the underlying
key practices and no adequate alternative is in place. An ITIM critical
process is "not implemented, but improvements under way" if more than
half, but not all, of its underlying key practices are rated as executed.
For example, if welldefined policies and procedures have been developed,
but no training has been established, the critical process would be rated
as "not implemented, but improvements underway." This rating is intended
to indicate that the organization has made progress in addressing the
critical process, but the work has not been completed.
A critical process, like a key practice, can be implemented by alternative
means. The crucial point in assessing an alternative approach is that the
techniques used to fulfill the purpose of the critical process must be
defined, implemented, and institutionalized. These are the same criteria
that are described in the ITIM framework and used to assess the adequacy
of an organization's execution of a key practice.
Determine the Investment Management Stage
All of the critical processes within a particular investment management
maturity stage-and within each lower stage-must be rated as "implemented"
or "not applicable" in order for the organization to achieve that stage.
For example, for an organization to be rated as an ITIM Stage 3
organization, all of the critical processes within both Stage 2 and Stage
3 must be rated as being "implemented" or "not applicable" by the team.
Appendix II
Conducting an ITIM Assessment
Deliver the Draft Summary Assessment
The final step in the assessment process is the delivery of draft results
to the organization. In addition, these draft results can form the basis
for the development of a full audit report, if one is requested. The draft
assessment, typically in the form of a briefing, contains
o an itemization of ITIM critical processes that have been assessed and
rated;
o an identification of implemented critical processes, an identification
of the achieved investment management stage, and graphical or summary
representations of the above information;
o a rating of each key practice for each critical process that was
assessed; and
o an evidence-based rationale for each rating determination.
The team can use the case study reviews to illustrate the ratings and
conclusions that the team reached as a result of the assessment. In order
to focus on the key practices needing improvement, the team typically will
deliver draft ratings only for key practices that it judges to be "not
executed." This approach optimizes time overall and ensures maximum time
is spent corroborating investment management weaknesses and collecting
additional evidence about them or about other areas.
Appendix III
Acknowledgments
A number of experts in the field of IT investment management commented
on the draft revision of this framework. They provided us with a wide
range
of insights and ideas for improvement, and we have sought to acknowledge
as many of these as possible. We wish to thank the reviewers for their
encouragement and for sharing their time and expertise with us.
Thom Arnsperger
Director, Strategic Consulting Services
The Morgan-Franklin Corporation
Former consultant with MITRE at DOD
Dr. Scott Bernard
Director of Innovation and Strategy, DigitalNet Government Solutions
Assistant Professor and Director, Washington, D.C., Programs, School of
Information Studies, Syracuse University
Dr. John Christian
Professor, Information Resources Management College
National Defense University
Author of ITIM Exposure Draft
Hank Conrad
Consultant, Eagle One
Former CIO of McDonald's Corporation
Michael Farber
Principal, Booz Allen & Hamilton
Jim Grant
President, C. G. James & Associates
Former Executive Vice President for Systems and Technology at the Royal
Bank of Canada
Maj. Gen. Donald Lasher
Senior Executive and Consultant
U.S. Army (Ret.) and former CIO at USAA and Department of Interior
Bill McVay
Vice President for E-Government Solutions, DigitalNet Government
Solutions
Former Deputy Branch Chief for Information Policy and Technology in the
Office of Electronic Government at the Office of Management and Budget
Appendix III Acknowledgments
Paul Rummell
Management Consultant
Former CIO of Canada
Debra Stouffer
Vice President for Strategic Consulting Practice, DigitalNet Government
Solutions
Former Deputy CIO at HUD and former CTO at the Environmental
Protection Agency
Bill Taylor
Director, Project Management and Evaluation
Office of the Chief Information Officer
Department of Housing and Urban Development
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***