Precollege Math and Science Education: Department of Energy's Precollege Program Managed Ineffectively (Letter Report, 09/13/94, GAO/HEHS-94-208). Public policymakers and U.S. industrial leaders have expressed grave concern about precollege students in other industrialized countries significantly outperforming American students on recent international mathematics and science tests. In recognition of the Energy Department's (DOE) world-class scientists and engineers as well as its state-of-the-art research facilities, Congress made mathematics and science education a major mission for DOE in fiscal year 1991. DOE's precollege mathematics and science budget has grown about 1,250 percent--reaching about $27 million in fiscal year 1993. In the early 1990s, DOE did not effectively manage this program. First, DOE jeopardized the program's success by not using a risk management strategy to run the program's projects. Second, DOE forfeited an invaluable management tool by taking a lax approach to program evaluation. Third, DOE greatly reduced its chances of helping achieve National Education Goal 5--"By the year 2000, U.S. students will be first in the world in mathematics and science achievement"--by launching a variety of projects that did not clearly seek to improve student achievement. DOE has announced steps to substantially improve the program's management and evaluation functions; however, the depth of executive support for these measures and their subsequent staying power are uncertain. --------------------------- Indexing Terms ----------------------------- REPORTNUM: HEHS-94-208 TITLE: Precollege Math and Science Education: Department of Energy's Precollege Program Managed Ineffectively DATE: 09/13/94 SUBJECT: Secondary education Agency missions Energy research Education program evaluation Mathematics Scientific research Elementary education Teacher education Mission budgeting Students IDENTIFIER: National Education Goals National Education Longitudinal Study of 1988 Goals 2000 ************************************************************************** * This file contains an ASCII representation of the text of a GAO * * report. Delineations within the text indicating chapter titles, * * headings, and bullets are preserved. Major divisions and subdivisions * * of the text, such as Chapters, Sections, and Appendixes, are * * identified by double and single lines. The numbers on the right end * * of these lines indicate the position of each of the subsections in the * * document outline. These numbers do NOT correspond with the page * * numbers of the printed product. * * * * No attempt has been made to display graphic images, although figure * * captions are reproduced. Tables are included, but may not resemble * * those in the printed version. * * * * A printed copy of this report may be obtained from the GAO Document * * Distribution Facility by calling (202) 512-6000, by faxing your * * request to (301) 258-4066, or by writing to P.O. Box 6015, * * Gaithersburg, MD 20884-6015. We are unable to accept electronic orders * * for printed documents at this time. * ************************************************************************** Cover ================================================================ COVER Report to the Chairman, Committee on Governmental Affairs, U.S. Senate September 1994 PRECOLLEGE MATH AND SCIENCE EDUCATION - DEPARTMENT OF ENERGY'S PRECOLLEGE PROGRAM MANAGED INEFFECTIVELY GAO/HEHS-94-208 Precollege Math and Science Education Abbreviations =============================================================== ABBREV AMES - Ames Laboratory ANL - Argonne National Laboratory AWU - Associated Western Universities BNL - Brookhaven National Laboratory FCCSET - Federal Coordinating Council for Science, Engineering, and Technology CEHR - Committee on Education and Human Resources LANL - Los Alamos National Laboratory LBL - Lawrence Berkeley Laboratory LLNL - Lawrence Livermore National Laboratory NSF - National Science Foundation ORISE - Oak Ridge Institute for Science and Education ORNL - Oak Ridge National Laboratory PNL - Pacific Northwest Laboratory SNL-A - Sandia National Laboratory-Albuquerque SNL-L - Sandia National Laboratory-Livermore Letter =============================================================== LETTER B-251474 September 13, 1994 The Honorable John Glenn Chairman, Committee on Governmental Affairs United States Senate Dear Mr. Chairman: Our nation's ability to remain the economic world leader depends on its citizens' strong mathematics and science skills. Understandably, public policymakers and industrial leaders have expressed grave concern about precollege students in other industrialized countries significantly outperforming American students on recent international mathematics and science tests. In response to educational reform and competency concerns, former President Bush and the nation's governors developed six National Education Goals to be achieved by the year 2000.\1 In recognition of the Department of Energy's world-class scientists, engineers, and technicians, as well as its state-of-the-art laboratories and research facilities, the Congress made mathematics and science education a major mission for the Department in fiscal year 1991. Consequently, Energy's precollege mathematics and science program budget has grown approximately 1,250 percent--from approximately $2 million in fiscal year 1990 to approximately $27 million in fiscal year 1993 (see app. I). This report responds to your questions about how effectively Energy manages its precollege mathematics and science program. On the basis of discussions with your office, we agreed to determine (1) the appropriateness of Energy's precollege program implementation priorities, (2) the role of project evaluations in ensuring rational budget decisions, and (3) whether its precollege program helps achieve National Education Goal 5--"By the year 2000, U.S. students will be first in the world in mathematics and science achievement." -------------------- \1 In 1994, the Goals 2000: Educate America Act expanded the National Education Goals from six to eight. The goals address (1) school readiness; (2) school completion; (3) student achievement and citizenship; (4) teacher education and professional development; (5) mathematics and science achievement;(6) adult literacy; (7) safe, disciplined, and alcohol- and drug-free schools; and (8) parental participation. The National Education Goals were originally developed in 1989. BACKGROUND ------------------------------------------------------------ Letter :1 Over the last 10 years, numerous reports have charged that many U.S. students complete high school scientifically and technologically illiterate. According to these reports, not only are U.S. students less well educated than their predecessors, they are also less well trained in mathematics and science than their peers in other industrialized countries. Reported decreasing student enrollments in science courses, declining achievement test scores, and the continuing decline in the number of high-quality mathematics and science teachers have highlighted problems in precollege mathematics and science instruction.\2 To remedy this perceived crisis in education, the Congress conferred mathematics and science education responsibilities on the Department of Energy and 13 other federal agencies--in addition to the Department of Education and the National Science Foundation (NSF). To improve mathematics and science education, the Federal Coordinating Council for Science, Engineering, and Technology (FCCSET)\3 Committee on Education and Human Resources (CEHR) was charged with coordinating the efforts of these 16 agencies. FCCSET developed broad implementation priorities for precollege education, including (1) standards for curriculum, teaching, and assessment; (2) curriculum, course, and instructional materials; (3) systemic reform; and (4) teacher preparation and enhancement. These priorities were formalized in January 1993 as part of a 5-year strategic plan. -------------------- \2 Science, Engineering, and Mathematics Education, Congressional Research Service Issue Brief, (Washington, D.C.: 1992), p. 1. \3 In November 1993, President Clinton established by executive order a cabinet level National Science and Technology Council (NSTC) to coordinate science, space, and technology policies throughout the federal government. The establishment of NSTC consolidated the responsibilities previously carried out by a number of agencies, including FCCSET. ENERGY'S PRECOLLEGE MATHEMATICS AND SCIENCE PROGRAM ---------------------------------------------------------- Letter :1.1 The Department of Energy Science Education Enhancement Act authorized Energy to undertake a wide range of precollege education and training activities. Some of Energy's activities include making loans of equipment and staff to schools; allowing Energy employees to provide education-oriented community services; and participating in joint programs with schools, businesses, museums, and other community partners. Energy manages its precollege program with a decentralized organizational structure: Energy's nine national laboratories and 22 of its research facilities were given the flexibility to design and implement projects using the broad implementation priorities provided by FCCSET. In designing projects, each Energy facility considered FCCSET's broad implementation priorities, as well as its individual areas of specialization and local needs. Although the projects vary, most emphasize hands-on experiences and fall within three implementation categories: Teacher enhancement--These projects attempt to further the content knowledge, skills, and experiences of teachers already in the workforce. Teacher enhancement projects typically provide teachers with opportunities to (1) work on a variety of scientific and technical subjects as a member of an Energy laboratory research team, (2) train with mentors who assist them with in-school science experiments, or (3) obtain sophisticated computer training. Student support--These projects seek to reward students for outstanding achievement, afford them enrichment experiences, and furnish them with supplementary educational services such as tutoring and mentoring. Student support projects typically allow students opportunities to (1) participate in cutting-edge research at an Energy laboratory, (2) attend scientific lectures and demonstrations, and (3) study emerging topics such as environmental energy. Systemic reform--These projects aim to improve education by changing all aspects of an educational system. Systemic reform projects typically involve key education stakeholders--students, teachers, administrators, policymakers, and parents--in collaborative efforts to (1) create goals and standards for all students, (2) develop related curricula and instructional materials, and (3) provide professional development for teachers. RESULTS IN BRIEF ------------------------------------------------------------ Letter :2 Although Energy invested more than $50 million in precollege education in fiscal years 1990-93, it did not effectively oversee or direct the program. For example, although research findings indicate that systemic reform may have the greatest potential to improve student learning, projects in this implementation category constituted the smallest share of Energy's precollege budget, about 11 percent. In contrast, Energy used about 70 percent of its precollege budget to finance teacher enhancement projects, even though research suggests that these projects may be ineffective at increasing student achievement. To compound problems, Energy did not link budget decisions to project evaluation results. As a result, Energy had not evaluated almost half of its 17 most resource-intensive projects at the time of our review; for those projects with evaluation reports, all were inadequate. Nonetheless, Energy substantially increased funding for most of these projects--in one case by over 1,700 percent. In addition, it is doubtful whether Energy's precollege program will help achieve National Education Goal 5. In this regard, Energy's projects typically do not focus on student achievement, which is central to achieving this goal. In fact, more than half of Energy's most resource-intensive projects did not directly include improving student achievement as an objective. During our review, Energy indicated recognition of the need to pay closer attention to managing its precollege program. To correct this situation, Energy recently drafted an agency-specific strategic plan. In addition, officials said they plan to restructure their program to ensure that all projects are evaluated and linked more clearly to National Education Goal 5. SCOPE AND METHODOLOGY ------------------------------------------------------------ Letter :3 Our review of Energy's precollege mathematics and science program focused primarily on its 288 fiscal year 1992 projects. To accomplish our objectives, we conducted site visits at Energy's headquarters in Washington, D.C., and eight laboratories and research facilities in California, Illinois, New Mexico, Tennessee, and Utah, which administered the most resource-intensive precollege projects.\4 During these site visits, we interviewed laboratory personnel as well as school administrators, teachers, and students involved in the precollege program. To obtain an overview of the program, we collected general information on Energy's entire array of fiscal year 1992 precollege mathematics and science projects and reviewed the literature on the relationship between teacher quality and subsequent student achievement. For the 17 most resource-intensive projects, we collected and analyzed budget data for fiscal years 1990-93 (see app. III) and, when available, evaluation reports. Although constituting less than 6 percent of Energy's program portfolio, these 17 projects accounted for 50 percent ($11 million) of total program dollars in fiscal year 1992. To determine whether the evaluation reports provided reliable information on project effectiveness, we also conducted a technical review of the nine project evaluation reports Energy officials submitted to our staff (see app. IV). We conducted our work between November 1992 and July 1994 in accordance with generally accepted government auditing standards. -------------------- \4 To identify the most resource-intensive projects, we examined the extent to which both financial and personnel resources were used. We selected 16 projects with the largest budgets and 1 project that involved considerably more Energy personnel than other projects. FOCUS ON TEACHER ENHANCEMENT PROJECTS IS QUESTIONABLE ------------------------------------------------------------ Letter :4 Energy's decision to concentrate its precollege program resources on teacher enhancement projects, which account for more than two-thirds of the program budget, is a questionable implementation strategy. In an earlier report, we found no evidence that training programs to upgrade existing science and mathematics teachers' skills will improve teaching effectiveness.\5 In reviewing more recent studies, we found mixed results: in some instances, researchers found small yet statistically significant positive correlations between teacher knowledge and student achievement; in others, researchers failed to demonstrate any significant correlations. Conversely, current literature suggests that systemic reform measures, such as high-quality curriculum development, may hold the most promise for improving academic achievement and realizing the national math and science goals. -------------------- \5 New Directions for Federal Programs to Aid Mathematics and Science Teaching (GAO/PEMD-84-5, Mar. 6, 1984). PRECOLLEGE PROGRAM HEAVILY WEIGHTED TOWARD TEACHER ENHANCEMENT PROJECTS ---------------------------------------------------------- Letter :4.1 Both in budget dollars and project numbers, Energy devoted most of its precollege program resources to teacher enhancement projects during fiscal year 1992. Regarding the program budget, Energy spent about 70 percent ($15.4 million) of all precollege program dollars to upgrade teachers' mathematics and science skills--45 percent exclusively for teacher enhancement and another 25 percent for teacher enhancement combined with a student support component. For the residual, about 19 percent ($4.2 million) of Energy's precollege budget focused exclusively on student support; systemic reform efforts constituted just 11 percent ($2.4 million). Regarding the total number of projects, 157 of 288 (55 percent) had a substantial teacher enhancement component, another 113 projects (39 percent) focused exclusively on student support, 14 projects (5 percent) involved systemic reform efforts, and 4 projects (1 percent) included activities that did not specifically involve teachers and students (see table 1). Table 1 Precollege Program Heavily Weighted Toward Teacher Enhancement Projects Budget (in thousa Percen Percen Implementation priorities nds) t Number t ---------------------------- ------ ------ ------ ------ Teacher enhancement $9,911 45 100 35 Teacher enhancement and 5,524 25 57 20 student support Student support 4,187 19 113 39 Systemic reform 2,448 11 14 5 Other\a 95 \b 4 1 ============================================================ Total $22,16 100 288 100 5 ------------------------------------------------------------ \a "Other" includes activities that do not specifically involve students and teachers, such as efforts by Energy's staff to develop a catalog listing precollege physics projects. \b Less than 1 percent. NO STRONG RELATIONSHIP BETWEEN TEACHER ENHANCEMENT AND STUDENT ACHIEVEMENT ---------------------------------------------------------- Letter :4.2 Research has failed to show conclusively a relationship between teacher enhancement and student achievement.\6 We based our finding on the results of a 1984 GAO report and a review of recent studies that examined the effect of teacher quality on student achievement (see bibliography, p. 25). In our earlier report, we found no evidence that training programs to upgrade mathematics and science teachers' skills improved student achievement.\7 That analysis was based first on an NSF study that compared the achievement of eighth and eleventh grade students whose teachers participated in NSF institutes to those who did not. The study showed that teacher participation in NSF institutes had a positive effect on eleventh grade students' science and mathematics achievement. However, institute participation did not have a statistically significant effect on eighth grade student achievement in either science or mathematics. Second, several general studies from the 1960s and 1970s as a group failed to show a consistent relationship between teacher knowledge and student achievement. More recent studies have also failed to demonstrate a strong relationship between teacher enhancement/knowledge and student achievement. Two of the most prominent studies conducted between 1984 and 1994 that examined this relationship reported mixed results: A 1992 NSF study, which analyzed teacher transcript and student test data from the National Education Longitudinal Study of 1988 (NELS:88),\8 showed a statistically significant relationship between eighth grade students' mathematics achievement and their teachers' preparation.\9 Specifically, students whose teachers had majored in mathematics performed slightly better than those whose teachers had majored in education only. However, no statistically significant relationship existed for science. A 1994 Chicago Academy of Sciences study, which analyzed the effect of teachers' participation in 4-week summer science workshops, reported a small, but statistically significant, increase in the level of science achievement for seventh grade students of participants.\10 For these students, the average science achievement score increased from 47.1 to 49.6. However, no statistically significant change occurred for either sixth or eighth grade science students. At least four explanations for the weak relationship between teachers' knowledge or participation in training programs and student achievement exist. First, most training programs generally involve one-time, relatively short events with little or no follow-up. For example, a single 4- to 8-week summer research experience probably would not produce a dramatic change in a teacher's effectiveness. Second, teacher training programs are subject to self-selection bias; that is, in-service training often attracts exemplary or highly motivated teachers. Exposing such teachers to short-term workshop training may not significantly add to their teaching effectiveness. Third, the most knowledgeable teachers may not be the best teachers. Other factors besides teacher knowledge--such as enthusiasm, confidence, and organization of class time--may determine student achievement. Fourth, some researchers suggest that current student assessments inadequately measure the higher order, problem-solving skills, which could be affected by teacher training and knowledge. -------------------- \6 Although research in this area has been limited, the studies we identified continue to be cited in current research, and the findings remain unchallenged. \7 New Directions for Federal Programs to Aid Mathematics and Science Teaching. \8 NELS:88 is a nationally representative sample of 26,435 eighth-grade students clustered within 1,052 schools. \9 Senta Raizen and Theodore Britton, "Science and Mathematics Teachers," Indicators of Science and Mathematics Education in 1992, National Science Foundation (Washington, D.C.: 1993), pp. 85-113. \10 Jon D. Miller, Enriching Middle School Science: A Final Evaluation of the 1991-92 Columbia College Workshops Utilizing an Innovative Approach to the Teaching of Science. Chicago Academy of Sciences (NSF grant TPE 89-55128), (Chicago: 1994), pp. 30-31. SYSTEMIC REFORM CONSIDERED PROMISING APPROACH FOR INCREASING STUDENT ACHIEVEMENT ---------------------------------------------------------- Letter :4.3 Energy has implemented few systemic reform projects, even though current educational literature suggests systemic reform may have the greatest potential for improving student learning.\11 Systemic reform is promising because it (1) attempts to stimulate change in many or all components of the educational system simultaneously; (2) establishes clear standards for what students should know and be able to do; and (3) involves key educational stakeholders--students, teachers, administrators, teacher educators, textbook publishers, policymakers, and parents--at all levels of the education system--national, state, district, and school. Under systemic reform, teacher training ideally takes place in an environment that supports new curricula or teaching techniques learned during training. However, many educators believe that retraining individual teachers will have little measurable impact on student achievement if the education system is not prepared to absorb improvements. For example, although teacher enhancement programs may provide teachers the knowledge, skills, and enthusiasm essential to implement new curricula, teachers can rarely implement and sustain a new program if their school support systems and attitudes of administrators, colleagues, and parents have not changed. -------------------- \11 Much of this literature is cited in Marshall Smith and Jennifer O'Day, "Systemic School Reform," Politics of Education Association Yearbook, (1990), pp. 233-267. ADDITIONAL PROGRAM DIVERSITY COULD REDUCE PROGRAM RISK ---------------------------------------------------------- Letter :4.4 Given the evidence cited, Energy's heavy investment in teacher enhancement projects substantially increases its risk of not improving student achievement in mathematics and science. However, Energy could reduce this risk by changing its mix of projects to balance the program, much like financial advisers do by diversifying investment portfolios. In managing uncertainty, financial advisers minimize the risk of loss by acquiring a variety of investment vehicles; thus, good returns from one investment counterbalance poor returns from another. Building on this analogy, Energy could view its precollege program as a collection of investment vehicles assembled to meet an investment goal--improved student achievement in mathematics and science. Thus, given the uncertainty of its projects' educational payoffs, a more diverse program portfolio would enhance the likelihood of Energy's achieving its program goal. To address these concerns, Energy developed a strategic plan, which identifies agency-specific precollege goals and objectives in March 1994. Energy envisions using the strategic plan, which should be in place by November 1994, to help it create a program strategy that supports the best mix of projects and minimizes risk. In addition, Energy plans to look at the feasibility of eliminating projects that do not support its strategic plan and restructuring all teacher enhancement projects to include systemic reform elements, such as follow-up support beginning in fiscal year 1995. BUDGET DECISIONS NOT LINKED TO PROJECT EVALUATIONS ------------------------------------------------------------ Letter :5 Energy did not link budget decisions to project evaluations. Until recently, Energy neither required project evaluations nor ensured their adequacy when its research facilities conducted them. For example, Energy elected not to evaluate eight of its 17 most resource-intensive projects. When conducted, evaluations were of poor quality. For instance, all projects with evaluations contained discrediting technical flaws--such as insufficient sample sizes, the absence of statistical tests, and insufficient supporting data--that potentially invalidated any evaluation findings (see app. IV). Energy's limited use of program evaluation reflected its management priorities. According to Energy officials, the Department did not emphasize effectiveness evaluations because program expansion was its primary objective. These officials also said that Energy lacked the capacity--staff, funds, and expertise--to design and monitor evaluations for such a vast array of projects. Consequently, in the absence of sufficient evaluation results, Energy substantially increased project budgets on the basis of self-reported data, such as customer satisfaction surveys, or anecdotal data such as participant testimonials, requests to participate in particular projects, or other popularity indicators. Consider the following examples: Energy increased the budgets for four projects with no evaluations, with increases ranging from 62 to 912 percent from fiscal year 1991 to 1992 (see fig. 1).\12 Figure 1: Project Budgets Increased Substantially Without Evaluations (See figure in printed edition.) Notes: \a The Science/Math Carnival (SNL-L) project budget was increased from approximately $3,000 to $26,000 or by 912 percent. \b The Bay Area Science and Technology Education Collaboration (LBL) project budget was increased from approximately $288,000 to $466,000 or by 62 percent. \c The Environmental Management Precollege Analytical Chemistry (AWU) project budget was increased from approximately $225,000 to $615,000 or by 173 percent. \d The National Science Explorers (ANL) project budget was increased from approximately $318,000 to $693,000 or by 118 percent. Energy increased the budgets for five projects with inadequate evaluations, with increases ranging from 113 to 1,731 percent from fiscal year 1990 to 1992 (see fig. 2). Figure 2: Project Budgets Increased Substantially Despite Inadequate Evaluations (See figure in printed edition.) Notes: \a The Environmental Education Outreach for Minorities (BNL) project budget was increased from approximately $93,000 to $199,000 or by 113 percent. \b The Teacher Research Associates (ORISE) project budget was increased from approximately $49,000 to $231,000 or by 372 percent. \c The OPTIONS (PNL) project budget was increased from approximately $50,000 to $915,000 or by 1,731 percent. \d The Teacher Research Associates (AWU) project budget was increased from approximately $526,000 to $1,779,000 or by 238 percent. \e The Science Advisors (SNL-A) project budget was increased from approximately $545,000 to $2,500,000 or by 359 percent. On the basis of our findings, Energy needs to change the way it views the relationship between project implementation and evaluation. Energy should view program evaluation as an integral part of program management; rather than perceiving the two as mutually exclusive. Because quality program evaluation is expensive, Energy must accept the unavoidable trade-off that it must fund fewer projects with stronger evaluation components. To address its program evaluation shortcomings, Energy began a partnership with the National Center for Improving Science Education to jointly develop a system for ongoing evaluation of its precollege program in May 1992. In addition, Energy established an evaluation guidance committee responsible for developing a long-range implementation plan for project evaluation. Energy officials also said that beginning in fiscal year 1995, the Department will require each precollege project to have an evaluation component as a prerequisite for funding. -------------------- \12 Projects shown in figure 1 began in fiscal year 1991. PROGRAM UNLIKELY TO CONTRIBUTE TO ACHIEVING NATIONAL EDUCATION GOAL 5 ------------------------------------------------------------ Letter :6 Energy has greatly diminished the prospect of its program's helping to achieve National Education Goal 5--making American students first in the world in mathematics and science--by not emphasizing student achievement. Although it is the essence of Goal 5, increasing student achievement is the key objective in only 7 of the 17 most resource-intensive program projects. The remaining 10 projects generally seek to improve students' attitudes toward mathematics and science and motivate them to eventually pursue science careers by improving their perceptions of scientists. Moreover, Energy's evaluation process did not focus on student achievement: only one of the nine projects that were evaluated tried to measure student achievement. In fact, Energy's projects are seldom clearly linked to National Education Goal 5. For example, two of seven program managers we interviewed were unfamiliar with the National Education Goals. In addition, none of the program managers could demonstrate how their projects helped improve student achievement. Generally, these managers told us their projects aim to increase mathematics and science literacy and promote science as a career, not improve student achievement. CONCLUSION ------------------------------------------------------------ Letter :7 In the early 1990s, Energy did not effectively manage its precollege mathematics and science program. First, the Department jeopardized the program's success by not using a risk management strategy to administer the program's projects. Second, Energy forfeited an invaluable management tool by taking a lax approach to program evaluation. Third, Energy greatly reduced its probability of helping achieve National Education Goal 5 by implementing a variety of projects that did not clearly seek to improve student achievement. In response to concerns raised during our review, Energy officials announced plans to undertake several initiatives to substantially improve this program's management and evaluation functions. These initiatives constitute an important step toward effective program management. However, the depth of executive support for these initiatives and their subsequent staying power were uncertain at the time our review was completed. If ongoing changes in Energy's management philosophy are fully implemented, needed program improvements could ultimately result. RECOMMENDATIONS TO THE SECRETARY OF ENERGY ------------------------------------------------------------ Letter :8 In continuing the Department's efforts to improve management of the precollege mathematics and science education program, we recommend that the Secretary of Energy strengthen its management role. Specifically, the Secretary should place greater emphasis on balancing the program by increasing the proportion of systemic reform projects; strengthen its evaluation component so that it serves as a basis for (1) improving projects; (2) making informed budget decisions about terminating, retaining, and expanding projects; and (3) measuring gains in student achievement; and restructure or discontinue all projects that do not clearly support National Education Goal 5--increasing students' mathematics and science achievement. ---------------------------------------------------------- Letter :8.1 Department of Energy officials who reviewed a draft of this report generally agreed with our findings. As requested, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the appropriate House and Senate Committees, the Secretary of Energy, and other interested parties. If you or your staff have any questions about this report, please call me on (202) 512-7014 or Cornelia Blanchette, Associate Director, on (202) 512-8403. The major contributors to this report are listed in appendix V. Sincerely yours, Linda G. Morra Director, Education and Employment Issues PRECOLLEGE PROGRAM BUDGET INCREASED SIGNIFICANTLY (FISCAL YEARS 1990-93) =========================================================== Appendix I (See figure in printed edition.) Note: Actual budget totals shown for fiscal years 1990-92. Appropriated budget totals shown for fiscal year 1993. OVERVIEW OF 17 MOST RESOURCE-INTENSIVE PROJECTS (FISCAL YEAR 1992) ========================================================== Appendix II Fiscal year Energy 1992 Implementation facility Project name Project objective budget priority -------------- ----------------- ----------------- ---------- -------------- Projects with no evaluation (eight projects) -------------------------------------------------------------------------------- Ames Adventures in Introduce $315,888 Teacher Laboratory Supercomputing teachers to the enhancement (Iowa) use of high- performance computers in mathematics and science instruction. Argonne National Science Provide teachers 693,000 Teacher National Explorers with science enhancement Laboratory videos, (Illinois) instructional guides, and training workshops to improve science teaching. Associated Environmental Provide students 615,014 Teacher Western Management with an enhancement Universities Precollege opportunity to and student (Utah) Analytical take a college- support Chemistry level analytical chemistry course. Lawrence Bay Area Science Provide hands-on 466,000 Systemic Berkeley & Technology activities, reform Laboratory Education curriculum (California) Collaboration development, instructional materials, and training districtwide to enhance classroom teaching and learning of science, mathematics, and technology. Oak Ridge Science & Provide teachers 244,000 Systemic Institute for Mathematics and students with reform Science and Action for research Education Revitalized opportunities, (Tennessee) Teaching instructional materials, and technical support to increase the effectiveness of mathematics and science education districtwide. Oak Ridge Adventures in Provide teachers 398,000 Teacher National Supercomputing and students with enhancement Laboratory access to and and student (Tennessee) training on high- support performance computers to improve mathematics and science instruction. Oak Ridge Preparation & Provide teachers 260,000 Teacher National Enhancement with summer enhancement Laboratory research (Tennessee) opportunities and training to improve mathematics and science instruction. Sandia Science/Math Provide 26,306 Teacher National Carnival scientific enhancement Laboratory- demonstrations to Livermore improve (California) mathematics and science teaching. Subtotal $3,018,208 Projects with evaluation (nine projects) -------------------------------------------------------------------------------- Argonne Chicago Science Provide teachers $531,000 Teacher National Explorers and students with enhancement Laboratory hands-on and student (Illinois) activities, field support trips, and videos to improve science instruction and learning. Associated Teacher Research Provide summer 1,779,269 Teacher Western Associates research enhancement Universities opportunities to (Utah) teachers. Brookhaven Environmental Provide students 199,211 Student National Education with an support Laboratory Outreach for opportunity to (New York) Minorities take a college- level environmental science course. Brookhaven Northeast Provide students 229,381 Student National Consortium for with an support Laboratory Minorities opportunity to (New York) take college- level science courses. Los Alamos Students Watching Provide 1,016,900 Teacher National Over Our Planet materials, enhancement Laboratory Earth teacher training, and student (New Mexico) and student support instruction on environmental concerns. Lawrence National Provide access to 726,000 Teacher Livermore Education and training on enhancement National Supercomputer high-performance and student Laboratory Program computers to support (California) students and teachers. Oak Ridge Teacher Research Provide summer 230,600 Teacher Institute for Associates research enhancement Science and opportunities for Education teachers. (Tennessee) Pacific OPTIONS in Provide students 915,370 Systemic Northwest Science with high- reform Laboratory quality (Washington) mathematics and science education by enhancing teachers' instructional strategies and ability to develop curriculum through a statewide systemic reform effort. Sandia Science Advisors Provide a 2,500,000 Teacher National scientist in the enhancement Laboratory- school who offers and student Albuquerque technical support (New Mexico) assistance to teachers and students and participates in activities to support science (i.e., science fairs). Subtotal $8,127,731 Total $11,145,93 9 -------------------------------------------------------------------------------- PERCENT CHANGE IN BUDGET FOR ENERGY'S 17 MOST RESOURCE-INTENSIVE PROJECTS (FISCAL YEARS 1990-93) ========================================================= Appendix III Per cen Perc t ent Percen Bud cha chan t Energy facility Project name get Budget nge Budget ge Budget change ------------------ ---------------- --- ---------- --- ------------ ---- ------------ ------ Projects with no evaluation (eight projects) ---------------------------------------------------------------------------------------------------- Ames Laboratory Adventures in $315,888 N/A $650,000 106 (Iowa) Supercomputing Argonne National National Science $318,000 N/ 693,000 118 507,000 -27 Laboratory Explorers A (Illinois) Associated Western Environmental 225,117 N/ 615,014 173 601,870 -2 Universities Management A (Utah) Precollege Analytical Chemistry Lawrence Berkeley Bay Area Science 288,300 N/ 466,000 62 515,000 11 Laboratory & Technology A (California) Education Collaboration Oak Ridge Science & 24, 279,100 1,0 244,000 -13 260,000 7 Institute for Mathematics 100 58 Science and Action for Education Revitalized (Tennessee) Teaching Oak Ridge National Adventures in 398,000 N/A 920,000 131 Laboratory Supercomputing (Tennessee) Oak Ridge National Teacher 260,000 N/A 260,000 N/A Laboratory Preparation & (Tennessee) Enhancement Sandia National Science/Math 2,600 N/ 26,306 912 29,000 10 Laboratory - Carnival A Livermore (California) Subtotal $24 $1,113,117 4,5 $3,018,208 171 $3,742,870 24 ,10 19 0 Projects with evaluation (nine projects) ---------------------------------------------------------------------------------------------------- Argonne National Chicago Science $61 $570,000 -7 $531,000 -7 $750,000 41 Laboratory Explorers 3,0 (Chicago) 00 Associated Western Teacher Research 525 1,105,519 110 1,779,269 61 1,731,326 -3 Universities Associates ,79 (Utah) 1 Brookhaven Environmental 93, 125,358 34 199,211 59 168,500 -15 National Education 351 Laboratory (New Outreach for York) Minorities Brookhaven Northeast 122,503 N/ 229,381 87 249,700 9 National Consortium for A Laboratory (New Minorities York) Los Alamos Students 85, 750,000 782 1,016,900 36 289,000 -72 National Watching Over 000 Laboratory (New Our Planet Mexico) Earth Lawrence Livermore National 340,000 N/ 726,000 114 403,000 -45 National Education A Laboratory Supercomputer (California) Oak Ridge Teacher Research 48, 139,700 186 230,600 65 193,000 -16 Institute for Associates 900 Science and Education (Tennessee) Pacific Northwest OPTIONS in 50, 629,762 1,1 915,370 45 374,033 -59 Laboratory Science 000 60 (Washington) Sandia National Science Advisors 545 1,400,000 157 2,500,000 79 2,200,000 -12 Laboratory - ,00 Albuquerque (New 0 Mexico) ==================================================================================================== Subtotal $1, $5,182,842 164 $8,127,731 57 6,358,559 -22 961 ,04 2 ==================================================================================================== Total $1, $6,295,959 217 $11,145,939 77 $10,101,429 -9 985 ,14 2 ---------------------------------------------------------------------------------------------------- Note: N/A represents not applicable. EVALUATION REPORT RESULTS FOR NINE PROJECTS WITH EVALUATIONS ========================================================== Appendix IV (See figure in printed edition.) MAJOR CONTRIBUTORS TO THIS REPORT =========================================================== Appendix V Wayne B. Upshaw, Assistant Director Valerie Giles-Reynolds, Assignment Manager, (313) 256-8000 Lemuel Jackson, Evaluator-in-Charge Ella Cleveland Joel Grossman Revae E. Steinman Yelena K. Thompson BIBLIOGRAPHY =========================================================== Appendix 0 Begle, Edward. Teacher Knowledge and Student Achievement in Algebra. School Mathematics Study Group. (Palo Alto, Calif.:1972). Darling-Hammond, Linda. "Enlarging the Knowledge Base--Are Our Teachers Ready To Teach?" National Council for Accreditation of Teacher Education, 1991. Darling-Hammond, Linda, and Lisa Hudson. Indicators for Monitoring Mathematics and Science Education: A Sourcebook. ed. Shavelson, McDonnell, and Oakes. RAND and the National Science Foundation, 1989. Druva, Cynthia A., and Ronald D. Anderson, "Science Teacher Characteristics by Teacher Behavior and by Student Outcome: A Meta-Analysis of Research." Journal of Research in Science Teaching, Vol. 20, No. 5 (1983), pp. 467-479. Fitzsimmons, Stephen J. A Study of NSF Teacher Enhancement Program (TEP) Participants and Principal Investigators: 1984-1989 (DRAFT) Vol. I: Summary Report. Abt Associates, Inc. for the National Science Foundation. Cambridge, Mass.: 1993. Juster, Thomas, F. Precollege Science and Mathematics Teachers: Monitoring Supply, Demand and Quality. National Research Council, Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, 1990. Lawrenz, Frances. "The Relationship Between Science Teacher Characteristics and Student Achievement and Attitude," Journal of Research in Science Teaching, Vol. 12, No. 4 (1975), pp. 433-37. McKinney, Kay. Improving Math and Science Teaching. U.S. Department of Education, Office of Educational Research and Improvement. Washington, D.C.: 1992. Miller, Jon. Enriching Middle School Science: A Final Evaluation of the 1991-92 Columbia College Workshops Utilizing an Innovative Approach to the Teaching of Science. Chicago Academy of Sciences. Chicago: 1994. Murnane, Richard J., and Senta A. Raizen, Improving Indicators of the Quality of Science and Mathematics Education in Grades K-12. National Research Council, Committee on Indicators of Precollege Science and Mathematics Education, Commission on Behavioral and Social Sciences and Education, 1988. Raizen, Senta, and Theodore Britton, "Science and Mathematics Teachers," Indicators of Science and Mathematics Education--1992. National Science Foundation. (NSF 93-95). Washington, D.C.: 1993. Rothman, Arthur I. "Teacher Characteristics and Student Learning," Journal of Research in Science Teaching, Vol. 6 (1969), pp. 340-48. Smith, Marshall, S., and Jennifer O'Day, "Systemic School Reform." Politics of Education Association Yearbook, 1990, pp. 233-267. U.S. General Accounting Office. New Directions for Federal Programs to Aid Mathematics and Science Teaching, GAO/PEMD-84-5. Washington, D.C.: 1984. U.S. General Accounting Office. Systemwide Education Reform: Federal Leadership Could Facilitate District-Level Efforts, GAO/HRD-93-97. Washington, D.C.: 1993.