Elections: Results of GAO's Testing of Voting Systems Used in
Sarasota County in Florida's 13th Congressional District
(08-FEB-08, GAO-08-425T).
In November 2006, about 18,000 undervotes were reported in
Sarasota County in the race for Florida's 13th Congressional
District (Florida-13). After the election results were contested
in the House of Representatives, the task force unanimously voted
to seek GAO's assistance in determining whether the voting
systems contributed to the large undervote in Sarasota County. In
October 2007, GAO presented its findings on the review of the
voting systems and concluded that while prior tests and reviews
provided some assurance that the voting systems performed
correctly, they were not enough to provide reasonable assurance
that the voting systems in Sarasota County did not contribute to
the undervote. GAO proposed that a firmware verification test, a
ballot test, and a calibration test be conducted. The task force
requested that GAO proceed with the proposed additional tests.
GAO also verified whether source code escrowed by Florida could
be rebuilt into the firmware used in Sarasota County. To conduct
its work, GAO conducted tests on a sample of voting systems used
in Sarasota County during the 2006 general election. GAO
witnessed the rebuild of the firmware from the escrowed source
code at the manufacturer's development facility. GAO reviewed
test documentation from Florida, Sarasota County, and the voting
system manufacturer and met with election officials to prepare
the test protocols and detailed test procedures.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-08-425T
ACCNO: A80609
TITLE: Elections: Results of GAO's Testing of Voting Systems
Used in Sarasota County in Florida's 13th Congressional District
DATE: 02/08/2008
SUBJECT: Calibration
Elections
Electronic equipment
System vulnerabilities
Systems analysis
Systems evaluation
Systems management
Systems monitoring
Systems testing
Voting
Voting records
Florida
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-08-425T
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Statement:
Before the Task Force for the Contested Election in the 13th
Congressional District of Florida, Committee on House Administration,
House of Representatives:
For Release on Delivery:
Expected at 10:00 a.m. EST:
Friday, February 8, 2008:
Elections:
Results of GAO's Testing of Voting Systems Used in Sarasota County in
Florida's 13th Congressional District:
Statement of Nabajyoti Barkakati, Ph.D: Acting Chief Technologist:
Applied Research and Methods:
GAO-08-425T:
GAO Highlights:
Highlights of GAO-08-425T, a statement before the Task Force for the
Contested Election in the 13th Congressional District of Florida,
Committee on House Administration, House of Representatives.
Why GAO Did This Study:
In November 2006, about 18,000 undervotes were reported in Sarasota
County in the race for Florida�s 13th Congressional District (Florida-
13). After the election results were contested in the House of
Representatives, the task force unanimously voted to seek GAO�s
assistance in determining whether the voting systems contributed to the
large undervote in Sarasota County. In October 2007, GAO presented its
findings on the review of the voting systems and concluded that while
prior tests and reviews provided some assurance that the voting systems
performed correctly, they were not enough to provide reasonable
assurance that the voting systems in Sarasota County did not contribute
to the undervote. GAO proposed that a firmware verification test, a
ballot test, and a calibration test be conducted. The task force
requested that GAO proceed with the proposed additional tests. GAO also
verified whether source code escrowed by Florida could be rebuilt into
the firmware used in Sarasota County.
To conduct its work, GAO conducted tests on a sample of voting systems
used in Sarasota County during the 2006 general election. GAO witnessed
the rebuild of the firmware from the escrowed source code at the
manufacturer�s development facility. GAO reviewed test documentation
from Florida, Sarasota County, and the voting system manufacturer and
met with election officials to prepare the test protocols and detailed
test procedures.
What GAO Found:
GAO conducted three tests on the iVotronic Direct Recording Electronic
(DRE) voting systems in Sarasota County and these tests did not
identify any problems. Based on its testing, GAO obtained increased
assurance that the iVotronic DREs used in Sarasota County during the
2006 general election did not contribute to the large undervote in the
Florida-13 contest. Although the test results cannot be used to provide
absolute assurance, GAO believes that these test results, combined with
the other reviews that have been conducted by the State of Florida,
GAO, and others, have significantly reduced the possibility that the
iVotronic DREs were the cause of the undervote.
GAO�s firmware verification test showed that the firmware installed in
a statistically selected sample of 115 machines used by Sarasota County
during the 2006 general election matched the firmware certified by the
Florida Division of Elections. The statistical approach used in
selecting these machines lets GAO estimate with a 99 percent confidence
level that no more than 60 of the 1,499 iVotronic DREs that recorded
votes in the 2006 general election were using different firmware.
Consequently, GAO is able to place more confidence in the results of
other tests conducted on a small number of machines by GAO and by
others, which indicated that the iVotronic DREs did not cause the
undervote. GAO also confirmed that when the manufacturer rebuilt the
iVotronic DRE firmware from the source code that was held in escrow by
the Florida Division of Elections and previously reviewed by GAO and
others, the resulting firmware matched the version certified by the
Florida Division of Elections.
For the ballot test, GAO cast predefined test ballots on 10 iVotronic
DREs and confirmed that each ballot was displayed and recorded
accurately. GAO conducted the calibration test by miscalibrating two
iVotronic DREs and casting ballots on them to validate that the
machines recorded the information that was displayed on the touch
screen. Based on the results of the ballot and calibration tests, GAO
found that (1) the machines properly displayed, recorded, and counted
the selections for all test ballots cast during ballot testing
involving 112 common ways a voter may have interacted with the system,
and (2) the deliberately miscalibrated machines, though difficult to
use, accurately recorded the ballot selections as displayed on screen.
At this point, GAO believes that adequate testing has been performed on
the voting machine software and does not recommend further testing in
this area. Given the complex interaction of people, processes, and
technology that must work effectively together to achieve a successful
election, GAO acknowledges the possibility that the large undervote in
Florida�s 13th Congressional District race could have been caused by
factors such as voters who intentionally undervoted, or voters who did
not properly cast their ballots on the iVotronic DRE, potentially
because of issues relating to interaction between voters and the
ballot.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-425T]. For more information, contact
Nabajyoti Barkakati at (202) 512-6412 or [email protected].
[End of section]
Mr. Chairman and Members of the Task Force:
I am pleased to appear before the task force today to present the
findings on our testing of the voting equipment used in the 2006
general election in Florida's 13th Congressional District (Florida-13).
I would like to thank the task force for its overall support of our
efforts and specifically for the assistance provided in obtaining
resources from the House Recording Studio that were critical to
successfully completing our testing efforts.
In November 2006, about 18,000 undervotes were reported in Sarasota
County in the race for Florida's 13th Congressional District.[Footnote
1] After the election results were contested in the House of
Representatives, the task force met and unanimously voted to seek GAO's
assistance in determining whether the voting systems contributed to the
large undervote in Sarasota County. In our October 2, 2007, statement
for the task force, we presented the findings of our review of the
voting systems and stated that while prior tests and reviews provided
some level of assurance that the voting systems in Sarasota County--
iVotronic direct recording electronic (DRE) voting systems manufactured
by Election Systems and Software (ES&S)--functioned correctly, they
were not enough to provide reasonable assurance that the iVotronic DRE
voting systems did not contribute to the undervote.[Footnote 2]
Specifically, we found that assurance was lacking in three areas and
proposed to the task force that additional tests--firmware
verification, ballot, and calibration--be conducted to address these
areas. We stated that successful accomplishment of these tests would
provide increased, but not absolute, assurance that the iVotronic DREs
used in Sarasota County during the 2006 general election did not cause
the undervote. The task force requested that we proceed with the
proposed additional tests. Our objectives were to (1) verify that
firmware installed in a statistical sample of iVotronic DREs was
identical to the firmware certified by the State of Florida, (2)
perform ballot testing using 112 ways to cast a ballot for the Florida-
13 contest to ensure that the voting machines would properly record and
count the ballots, and (3) deliberately miscalibrate voting machines
and then cast ballots on those machines to ensure that the voting
machines would properly record the ballots. As part of the first
objective, we also validated that the source code, which was held in
escrow by the Florida Division of Elections, would produce the firmware
used by Sarasota County during the 2006 general election.
To conduct our tests, we developed test protocols and detailed test
procedures. We met with officials from the Sarasota County Supervisor
of Elections, the Florida Department of State and Division of
Elections, and ES&S to obtain the necessary details about the voting
systems and prior tests to document our test procedures. We also
reviewed voting system documentation to develop a testing approach and
the test procedures. To ensure that the certified firmware held in
escrow by the Florida Division of Elections corresponded to the source
code that was reviewed by a team from Florida State University and us,
on November 19, 2007, we visited ES&S's development facility in
Rockford, Illinois, and witnessed the rebuild of the firmware from the
escrowed source code.
Further details on our test methodology are included in the following
sections on each of the three tests. Appendix I outlines the process
used to select machines for testing, and appendix II lists the
iVotronic DREs that we tested. We coordinated with the Florida Division
of Elections and the Sarasota County Supervisor of Elections to obtain
access to the iVotronic DREs and other necessary test equipment to
conduct our testing. We conducted the firmware verification, ballot,
and calibration tests at the Sarasota County Voting Equipment Facility
(VEF) in Sarasota, Florida. We established the test environment on
November 26, 2007, and conducted the tests from November 27, 2007, to
December 4, 2007. During this time, we completed the steps necessary to
conduct the tests and collected the test data. In addition, we video
recorded the tests. One camera was used to capture a wide angle shot of
the test room. Other cameras recorded the conduct of the firmware
verification, ballot, and calibration tests.
We provided a draft of this statement to the Florida Department of
State and ES&S for their review and comments. We briefed the Sarasota
County Supervisor of Elections on the contents of our statement. The
Florida Department of State and ES&S also conducted a sensitivity
review to ensure that business proprietary information is not disclosed
in this statement. We conducted our work from October 2007 to February
2008 in Washington, D.C.; Tallahassee and Sarasota, Florida; and at
ES&S facilities in Rockford, Illinois, and Omaha, Nebraska.
Results in Brief:
We conducted three tests on the iVotronic DRE voting systems used in
Sarasota County and these tests did not identify any problems that
would indicate that the machines were responsible for the undervote in
the Florida-13 race in the 2006 general election. In our firmware
verification test, we extracted the firmware from a random probability
sample of 115 iVotronic DREs out of the 1,499 iVotronic DREs used in
Sarasota County's 2006 general election and found that each machine's
firmware matched the certified version of firmware held in escrow by
the Florida Division of Elections. The statistical approach used in
selecting these machines enables us to estimate with a 99 percent
confidence level that at least 1,439 of the 1,499 machines used the
same firmware that was certified by the Florida Division of Elections.
Consequently, we have more confidence in the results of other tests
conducted on a small number of machines by GAO and by others, which
indicated that the iVotronic DREs were not the cause of the undervote.
We witnessed the rebuild of the iVotronic DRE's firmware from the
source code that was held in escrow by the Florida Division of
Elections and that was previously reviewed by Florida State University
and by us. At ES&S's software development facility, we observed that
rebuilding the firmware from the escrowed source code resulted in the
same firmware that was certified and held in escrow by the Florida
Division of Elections. This validation provides greater confidence in
the results of prior source code reviews by Florida State University
and us.
For the ballot test, we cast predefined test ballots on 10 iVotronic
DREs and confirmed that each ballot was displayed and recorded
accurately. The test ballots represented 112 common ways a voter may
have interacted with the iVotronic DRE to select a candidate in the
Florida-13 race and cast a ballot. These tests were performed on nine
machines configured as election day machines and then repeated on one
machine configured as an early voting machine.
Finally, we conducted the calibration test by miscalibrating two
iVotronic DREs and casting ballots on them to validate that the
machines recorded the information that was displayed on the touch
screen. Our tests, involving a total of 10 different miscalibration
patterns and capturing 39 ballots, found that the machines correctly
displayed the selection in the Florida-13 race on the review screen and
correctly recorded the ballot. Although the machines were more
difficult to use, the selections shown on the screen were the same
selections captured by the machine when the ballot was cast.
Based on the results of these tests, we have obtained increased
assurance, but not absolute assurance that the iVotronic DREs used in
Sarasota County's 2006 general election did not contribute to the large
undervote in the Florida-13 contest. Absolute assurance is impossible
to achieve because we are unable to recreate the conditions of the
election in which the undervote occurred. Although the test results
cannot be used to provide absolute assurance, we believe that these
test results, combined with the other reviews that have been conducted
by the State of Florida, GAO, and others, have significantly reduced
the possibility that the iVotronic DREs were the cause of the
undervote. At this point, we believe that adequate testing has been
performed on the voting machine software to reach this conclusion and
do not recommend further testing in this area. Given the complex
interaction of people, processes, and technology that must work
effectively together to achieve a successful election, we acknowledge
the possibility that the large undervote in Florida's 13th
Congressional District race could have been caused by factors such as
voters who intentionally undervoted, or voters who did not properly
cast their ballots on the iVotronic DRE, potentially because of issues
relating to interaction between voters and the ballot.
Background:
The 13th Congressional District of Florida comprises DeSoto, Hardee,
Sarasota, and parts of Charlotte and Manatee Counties. In the November
2006 general election, there were two candidates in the race to
represent the 13th Congressional District: Vern Buchanan, the
Republican candidate, and Christine Jennings, the Democratic candidate.
The State of Florida certified Vern Buchanan the winner of the
election. The margin of victory was 369 votes out of a total of 238,249
votes counted. Table 1 summarizes the results of the election and shows
that the results from Sarasota County exhibited a significantly higher
undervote rate than in the other counties in the congressional
district.
Table 1: Results from 2006 General Election for Florida Congressional
District 13:
County: Charlotte;
Buchanan: 4,460;
Jennings: 4,277;
Undervotes: 225;
Total ballots cast: 8,962;
Percentage undervote: 2.51.
County: DeSoto;
Buchanan: 3,471;
Jennings: 3,058;
Undervotes: 142;
Total ballots cast: 6,672;
Percentage undervote: 2.13.
County: Hardee;
Buchanan: 2,629;
Jennings: 1,686;
Undervotes: 269;
Total ballots cast: 4,584;
Percentage undervote: 5.87.
County: Manatee;
Buchanan: 50,117;
Jennings: 44,432;
Undervotes: 2,274;
Total ballots cast: 96,828;
Percentage undervote: 2.35.
County: Sarasota;
Buchanan: 58,632;
Jennings: 65,487;
Undervotes: 18,412;
Total ballots cast: 142,532;
Percentage undervote: 12.92.
County: Total;
Buchanan: 119,309;
Jennings: 118,940;
Undervotes: 21,322;
Total ballots cast: 259,578;
Percentage undervote: [Empty].
Source: GAO analysis of Florida Division of Elections, Charlotte
County, DeSoto County, Hardee County, Manatee County, and Sarasota
County data.
Note: Numbers do not add up because of overvotes-where voters select
more than the maximum number of candidates allowed in a race; in this
case, an overvote was a ballot that had votes for both Buchanan and
Jennings.
[End of table]
As seen in table 1, about 18,000 undervotes were reported in Sarasota
County in the race for Florida's 13th Congressional District. After the
election results were contested in the House of Representatives, the
task force met and unanimously voted to seek GAO's assistance in
determining whether the voting systems contributed to the large
undervote in Sarasota County. On June 14, 2007, we met with the task
force and agreed upon an engagement plan. We reported on the status of
our review at an interim meeting held by the task force on August 3,
2007.[Footnote 3]
On October 2, 2007, we reported that our analysis of election data did
not identify any particular voting machines or machine characteristics
that could have caused the large undervote in the Florida-13
race.[Footnote 4] The undervotes in Sarasota County were generally
distributed across all machines and precincts. We found that some of
the prior tests and reviews conducted by the State of Florida and
Sarasota County provided assurance that certain components of the
voting system in Sarasota County functioned correctly, but they were
not enough to provide reasonable assurance that the iVotronic DREs did
not contribute to the undervote. We proposed three tests--firmware
verification, ballot, and calibration--to provide increased assurance,
but not absolute assurance, that the iVotronic DREs did not cause the
large undervote in Sarasota County. We stated that the successful
conduct of the tests could reduce the possibility that the voting
systems caused the undervote and shift attention to the possibilities
that voters intentionally undervoted or voters did not properly cast
their ballots on the iVotronic DRE, potentially because of issues
relating to interaction between voters and the ballot.
Overview of the Voting Systems Used in Sarasota County in the 2006
General Elections:
In the 2006 general election, Sarasota County used voting systems
manufactured by ES&S. The State of Florida has certified different
versions of ES&S voting systems. The version used in Sarasota County
was designated ES&S Voting System Release 4.5, Version 2, Revision 2,
and consisted of iVotronic DREs, a Model 650 central count optical scan
tabulator for absentee ballots, and the Unity election management
system. It was certified by the State of Florida on July 17, 2006. The
certified system includes different configurations and optional
elements, several of which were not used in Sarasota County.[Footnote
5]
The election management part of the voting system is called Unity; the
version that was used was 2.4.4.2. Figure 1 shows the overall election
operation using the Unity election management system and the iVotronic
DRE.
Figure 1: Overview of Election Operation Using the Unity Election
Management System and iVotronic DRE:
[See PDF for image]
This figure is an illustration of the Overview of Election Operation
Using the Unity Election Management System and iVotronic DRE. The
following data is depicted:
Unity:
Election Data Manager:
Program, store, and format election data for a jurisdiction;
Ballot Image Manager:
Create paper ballot layouts and create information for personalized
electronic ballot (PEB) to be used in iVotronic DREs;
Hardware Programming Manager:
Copy election definitions for iVotronic DREs into PEBs;
Precinct Count, iVotronic DRE:
iVotronic DREs are used to capture votes and tabulate votes for a
precinct;
Copy precinct-level votes from PEBs into Election Reporting Manager and
ballot images and event logs from compact flash cards.
Upload via modem to: Data Acquisition Manager;
Data Acquisition Manager: sends to: Election Reporting Manager.
Election Reporting Manager:
Outputs Election reports.
Source: GAO.
[End of figure]
Sarasota County used iVotronic DREs for early and election day voting.
Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware
version 1.1 with firmware version 8.0.1.2.[Footnote 6] Some of the
iVotronic DREs are configured to use audio ballots, which are often
referred to as Americans with Disabilities Act (ADA) machines. The
iVotronic DRE uses a touch screen--a pressure-sensitive graphics
display panel--to display and record votes (see fig. 2).
Figure 2: The iVotronic DRE Voting System and Its Components:
[See PDF for image]
This figure is a photograph of the iVotronic DRE Voting System and its
components. Four components are labeled in the photograph:
Touch screen;
Personalized electronic ballot;
Vote button;
Connection to communications pack and printer (removed during voting).
Source: GAO.
[End of figure]
The machine has a storage case that also serves as the voting booth.
The operation of the iVotronic DRE requires the use of a personalized
electronic ballot (PEB), which is a storage device with an infrared
window used for transmission of ballot data to and from the iVotronic
DRE. The iVotronic DRE has four independent flash memory modules, one
of which contains the program code--firmware--that runs the machine;
the remaining three flash memory modules store redundant copies of
ballot definitions, machine configuration information, ballots cast by
voters, and event logs (see fig. 3). The iVotronic DRE includes a VOTE
button that the voter has to press to cast a ballot and record the
information in the flash memory. The iVotronic DRE also includes a
compact flash card that can be used to load sound files onto iVotronic
DREs with ADA functionality. The iVotronic DRE's firmware can be
updated through the compact flash card. Additionally, at the end of
polling, the ballots and audit information are to be copied from the
internal flash memory module to the compact flash card.
Figure 3: Inside View of the iVotronic DRE Showing the Flash Memory
Modules:
[See PDF for image]
This figure is a photograph of an Inside View of the iVotronic DRE
Showing the Flash Memory Modules. Two components are labeled in the
photograph:
Firmware flash memory module;
Redundant flash memory modules with ballot data and event log.
Source: GAO.
[End of figure]
To use the iVotronic DRE for voting, a poll worker activates the
iVotronic DRE by inserting a PEB into the PEB slot after the voter has
signed in at the polling place. After the poll worker makes selections
so that the appropriate ballot will appear, the PEB is removed and the
voter is ready to begin using the system. The ballot is presented to
the voter in a series of display screens, with candidate information on
the left side of the screen and selection boxes on the right side (see
fig. 4).
Figure 4: Second Ballot Page Showing the Congressional and
Gubernatorial Races in Sarasota County's 2006 General Election:
[See PDF for image]
This figure is a screen capture of the Second Ballot Page Showing the
Congressional and Gubernatorial Races in Sarasota County's 2006 General
Election. The following data is depicted on the page:
U.S. Representative in Congress, 13th Congressional District (Vote for
One):
Vern Buchanan, Republican;
Christine Jennings, Democrat.
State: Governor and Lieutenant Governor (Vote for One):
Charlie Crist, Jeff Kottkamp, Republican;
Jim Davis, Daryl L. Jones, Democrat;
Max Linn, Tom Macklin, Ref;
Richard Paul Dembinsky, Dr. Joe Smith, NPA;
John Wayne Smith, James J. Kearney, NPA;
Karl C.C. Behm, Carol Castagnero, NPA;
Write-in.
Previous Page;
Page 2 of 15 (Public Count: 0);
Next Page.
Source: Sarasota County Supervisor of Elections.
[End of figure]
The voter can make a selection by touching anywhere on the line, and
the iVotronic DRE responds by highlighting the entire line and
displaying an X in the box next to the candidate's name. The voter can
also change his or her selection by touching the line corresponding to
another candidate or by deselecting his or her choice. "Previous Page"
and "Next Page" buttons are used to navigate the multipage ballot.
After completing all selections, the voter is presented with a summary
screen with all of his or her selections (see fig. 5). From the summary
screen, the voter can change any selection by selecting the race. The
race will be displayed to the voter on its own ballot page. When the
voter is satisfied with the selections and has reached the final
summary screen, the red VOTE button is illuminated, indicating the
voter can now cast his or her ballot. When the VOTE button is pressed,
the voting session is complete and the ballot is recorded on the
iVotronic DRE. In Sarasota County's 2006 general election, there were
nine different ballot styles with between 28 and 40 races, which
required between 15 and 21 electronic ballot pages to display, and 3 to
4 summary pages for review purposes.
Figure 5: First Summary Page in Sarasota County's 2006 General
Election:
[See PDF for image]
This figure is a screen capture of the First Summary Page in Sarasota
County's 2006 General Election. The following data is depicted on the
page:
Instructions:
Return to any contest by touching the contest title. To cast your
ballot now, press the Vote button.
United States Senator:
No selection made.
U.S. Representative in Congress:
No selection made.
Governor and Lieutenant Governor:
No selection made.
Attorney General:
No selection made.
Chief Financial Officer:
No selection made.
Commissioner of Agriculture:
No selection made.
State Representative:
No selection made.
Charter Review Board District:
No selection made.
Charter Review Board District:
No selection made.
Charter Review Board District:
No selection made.
Charter Review Board District:
No selection made.
Charter Review Board District:
No selection made.
Previous Page.
Summary Ballot, Page 1 of 3.
Next Page.
Source: Sarasota County Supervisor of Elections.
[End of figure]
Election Systems Involve People, Processes, and Technology:
An election system is based upon a complex interaction of people
(voters, election officials, and poll workers), processes (controls),
and technology that must work effectively together to achieve a
successful election. The particular technology used to cast and count
votes is a critical part of how elections are conducted, but it is only
one facet of a multifaceted election process that involves the
interplay of people, processes, and technology.
As we have previously reported, every stage of the election process--
registration, absentee and early voting, preparing for and conducting
Election Day activities, provisional voting, and vote counting--is
affected by the interaction of people, processes, and
technology.[Footnote 7] Breakdowns in the interaction of people,
processes, and technology may, at any stage of an election, impair an
accurate vote count. For example, if the voter registration process is
flawed, ineligible voters may be allowed to cast votes. Poll worker
training deficiencies may contribute to discrepancies in the number of
votes credited and cast, if voter information was not entered properly
into poll books. Mistakes in using the DRE systems could result from
inadequate understanding of the equipment on the part of those using
it.
As noted in our October statement, we recognize that human interaction
with the ballot layout could be a potential cause of the undervote, and
we noted that several suggestions have been offered as possible ways to
establish that voters are intentionally undervoting and to provide some
assurance that the voting systems did not cause the undervote.[Footnote
8] For instance,
* A voter-verified paper trail could provide an independent
confirmation that the touch screen voting systems did not malfunction
in recording and counting the votes from the election. The paper trail
would reflect the voter's selections and, if necessary, could be used
in the counting or recounting of votes. This issue was also recognized
in the source code review performed by the Security and Assurance in
Information Technology (SAIT) laboratory at Florida State University as
well as the 2005 and draft 2007 Voluntary Voting Systems Guidelines
prepared for the Election Assistance Commission. We have previously
reported on the need to implement such a function properly.[Footnote 9]
* Explicit feedback to voters that a race has been undervoted and a
prompt for voters to affirm their intent to undervote might help
prevent many voters from unintentionally not casting a vote in a race.
On the iVotronic DREs, such feedback and prompts are provided only when
the voter attempts to cast a completely blank ballot, but not when a
voter fails to vote in individual races.
* Offering a "none of the above" option in a race would provide voters
with the opportunity to indicate that they are intentionally
undervoting. For example, the State of Nevada provides this option in
certain races in its elections.
We reported that decisions about these or other suggestions about
ballot layout or voting system functions should be informed by human
factors studies that assess such measures' effectiveness in accurately
recording voters' preferences, making voting systems easier to use, and
preventing unintentional undervotes.
Tests Confirm Sarasota County iVotronic DREs Used Same Firmware
Certified by Florida:
We previously reported that having reasonable assurance that all
iVotronic DREs that recorded votes in the 2006 general election were
running the same certified firmware would allow us to have more
confidence that the iVotronic DREs will behave similarly when
tested.[Footnote 10] Consequently, if we are reasonably confident that
the same firmware was running in all 1,499 machines, then we are more
confident that the results of other tests, conducted both by GAO and by
others, on a small number of machines can be used to obtain increased
assurance that the iVotronic DREs did not cause the undervote. We also
reported that there was a lack of assurance that the source code that
was held in escrow by the Florida Division of Elections and that was
previously reviewed by Florida State University and by us, if rebuilt,
would corresponded to the firmware that was certified and held in
escrow by the Florida Division of Elections. We found that the firmware
on a statistically selected sample of 115 iVotronic DREs was the same
as that certified by the Florida Division of Elections. We also found
that the escrowed source code, when rebuilt into executable firmware,
corresponded to the 8.0.1.2 firmware that was certified by the Florida
Division of Elections.
Methodology for Firmware Verification Testing:
Our methodology to obtain reasonable assurance that the firmware used
on Sarasota County's iVotronic DREs during the 2006 general election
was the same as that certified by the State of Florida was broken down
into two basic steps: (1) selecting a representative sample of
machines, and (2) verifying that the firmware extracted from the voting
machines was the same as the escrowed firmware that had been certified
by the Florida Division of Elections. Appendix I details the
methodology for selecting the representative sample of machines.
Appendix II contains a list of the serial numbers of the tested
iVotronic DREs.
To ensure that we would be testing with the iVotronic firmware
certified by the Florida Division of Elections, on October 18, 2007, we
and officials from the Florida Division of Elections made two copies of
the escrowed iVotronic 8.0.1.2 firmware on compact discs (CD) and
placed them in two tamper-evident bags with serial numbers. The bags
were subsequently hand-delivered by a Florida Division of Elections
official for our use in the firmware verification test and for the
rebuilding of the firmware from the source code.
In order to extract the firmware from an iVotronic DRE, the machine was
placed on an anti-static mat and the case was opened using a special
screwdriver. After lifting the case, a special extraction tool was used
to remove the flash memory module that contains the firmware. The flash
memory module was then inserted in the socket of a Needham Electronics'
EMP-300 device that was connected to the universal serial bus (USB)
port of a personal computer (PC). The EMPWin application running on
that PC was used to read the firmware from the flash memory module and
save the extracted firmware on the PC. The Florida Division of
Elections loaned us the EMP-300 and EMPWin application for use in
extracting firmware from the flash memory module.
To compare the extracted firmware with the escrowed version, we relied
on two commercially available software programs. First, we acquired a
license for PrestoSoft's ExamDiff Pro software that enables comparison
of files. The ExamDiff Pro software is a commercially available program
designed to highlight the differences between two files. For each
selected iVotronic DRE, the extracted firmware was compared with the
escrowed version with any differences highlighted by the program.
Second, to further ensure that the extracted firmware matched the
escrowed firmware, we compared the SHA-1 hash value of the extracted
firmware to the hash value of the comparable certified
firmware.[Footnote 11] We computed the SHA-1 hash by using the
Maresware hash software that was provided by the Florida Division of
Elections. In order to ensure that the commercial Maresware hash
software properly calculated the SHA-1 hash value, we (1) created four
files and obtained a fifth file that contained executable code, (2)
obtained hash values for each file by either using an external program
that generated the hash values using the same hashing algorithm as the
commercial product or using known hash values,[Footnote 12] and (3)
used the commercial program acquired for testing the firmware to ensure
that the hash values it generated for these five files were identical
to the expected hash values for those files. In each case, the hash
values generated by the commercial program were identical to the
expected values. Accordingly, reasonable assurance for the purposes of
our review was obtained that the commercial program produced its hash
values in accordance with the NIST algorithm.
At the end of each day, we (1) used the commercial Maresware software
to compute hash values for each of the firmware programs that had been
unloaded during that day and all previous days, and (2) compared each
hash created by this program to the expected value that was calculated
from the firmware that had been escrowed by the Florida Division of
Elections. This comparison provided further assurance that the
extracted firmware was (1) identical to the version escrowed by the
Florida Division of Elections when the hashes agreed, or (2) different
if the hashes did not agree.
We also verified that sequestered machines were not used since the 2006
general election. For each of these sequestered machines, we used an
audit PEB to copy the audit logs onto a compact flash card and then
used the Unity election reporting manager to generate event log
reports. We examined the event logs for the date and time of occurrence
of activities that would indicate whether the machine had been used.
Lack of such activities since the 2006 general election provided
reasonable assurance that the machines had not been used since they
were sequestered.[Footnote 13]
In addition, to verify that the source code for iVotronic DRE firmware
version 8.0.1.2 previously examined by the Florida State University
SAIT source code review team and by GAO corresponded with the version
certified by the Florida Division of Elections, ES&S officials stated
that it still had the development environment that could be used to
compile, or rebuild, the certified firmware from the source code
retained in escrow by the Florida Division of Elections.[Footnote 14]
As we previously noted, a software review and security analysis of the
iVotronic DRE firmware was conducted by a team led by Florida State
University's SAIT laboratory.[Footnote 15] The software review team
attempted to confirm or refute many different hypotheses that, if true,
might explain the undervote in the race for the 13th Congressional
District. In doing so, they made several observations about the source
code, which we were able to independently verify.
The rebuilding of the firmware was conducted by ES&S at its Rockford,
Illinois, facility on November 19, 2007, and witnessed by us. Prior to
the rebuild, the Florida Division of Elections provided an unofficial
copy of the source code to ES&S so that ES&S could prepare the
development environment and test the rebuild steps. Using the official
sealed copy of the source code CD, ES&S rebuilt the firmware in front
of GAO representatives. ES&S described the development environment and
we inspected it to satisfy ourselves that the firmware was faithfully
rebuilt using the escrowed source code. After the rebuilding of the
firmware, the certified version of 8.0.1.2 firmware was compared with
the rebuilt version using PrestoSoft's ExamDiff Pro.
Results of Firmware Verification Testing:
While the Florida audit team had previously confirmed that the firmware
running on six iVotronic DREs matched the certified version held in
escrow by the Florida Division of Elections, we found that the sample
size was too small to support generalization to all 1,499 iVotronic
DREs that recorded votes during the 2006 general election. Accordingly,
we conducted a firmware verification test on a statistically valid
sample of 115 iVotronic DRE machines used by Sarasota County during the
2006 general election. The selected machines fell into two groups--
machines that had not been used since the 2006 general election
(referred to as sequestered machines) and machines that had been used
in subsequent elections. For each machine, we extracted the firmware
from a flash memory module in that machine and then compared the
extracted firmware with the escrowed version using commercially
available file comparison tools to determine whether they agreed. We
found that the firmware installed in the flash memory module of each
machine matched the escrowed firmware that had been certified by
Florida. The statistical approach used to select these machines lets us
estimate with a 99 percent confidence level that at least 1,439, or 96
percent, of the 1,499 machines used in the 2006 general election used
the firmware that was certified by the State of Florida.
We witnessed the rebuild of the iVotronic DRE's firmware from the
source code that was held in escrow by the Florida Division of
Elections and that was previously reviewed by Florida State University
and by us. At ES&S's software development facility, we observed that
rebuilding the firmware from the escrowed source code resulted in the
same firmware that was certified and held in escrow by the Florida
Division of Elections. The comparison of the escrowed firmware to the
version that was rebuilt by the vendor identified no differences and
provides us reasonable assurance that the escrowed firmware
corresponded to the escrowed source code. The successful rebuilding of
the firmware from the escrowed source code enables us to have greater
confidence in the conclusions derived from prior source code reviews by
Florida State University and us.
Ballot Testing Showed That Machines Accurately Recorded and Counted
Ballots:
In our October 2007 statement, we noted that there were 112 common ways
a voter may interact with the system to select a candidate in the
Florida-13 race and cast the ballot, and that prior testing of the
iVotronic DREs covered only 13 of these 112 possible ways. We developed
224 test ballots to verify that the iVotronic DRE could accurately
capture ballots using each of these 112 common ways a voter may
interact with the system; 112 test ballots were cast on one machine
configured for early voting, and another 112 ballots were cast on nine
machines configured for election day voting. Our tests showed that for
each of the 224 test ballots, the iVotronic DRE correctly captured each
vote as cast for the Florida-13 race. We also conducted firmware
verification tests on these machines and verified that they were
running the certified firmware.
Methodology for Ballot Testing:
The methodology for ballot testing can be broken into two major areas-
-development of the test ballots and execution of the test using those
ballots. The following sections discuss these areas.
Development of Test Ballots:
In examining how the system allowed voters to make a selection in the
Florida-13 race, we found at least 112 different ways a voter could
make his or her selection and cast the ballot in the Florida-13 race,
assuming that it was the only race on the ballot. Specifically, a voter
could (1) initially select either candidate or neither candidate (i.e.,
undervote), (2) change the vote on the initial screen, and (3) use a
combination of features to change or verify his or her selection by
using the page back and review screen options. Accordingly, we tested
these 112 ways to select a candidate on the early voting machine and on
the election day machines (224 test ballots in total).
The 112 standard test ballots cover all combinations of the following
types of voter behavior:
* Voter makes selection on the initial ballot screen and makes no
changes or takes any other action to return to the contest to review or
change selection.
* Voter makes selection on the initial ballot screen and decides before
leaving that screen to change the selection because of an error in
selecting the candidate or for some other reason.
* Voter makes selection on the initial ballot screen and then decides
to use the page back option to review or change selection.
* Voter makes selection on the initial ballot screen and continues to
the review screen and then decides to use the review screen option to
review or change selection.
* Voter makes selection on the initial ballot screen and uses a
combination of page back and review screen options to review or change
selection.
In each instance where a selection could be made, three choices were
possible for the Florida-13 race: a selection for one of the two
candidates, or no selection (i.e., an undervote).
In developing the standard test ballots, we did not consider all
combinations of some other types of voter behavior that would have
significantly increased the number of test cases without providing
significant benefits. In most cases, such behavior are variants of the
primary voter behavior that we examined. The following are examples of
voter behavior that were not included in the standard test set in order
to reduce the number of test cases to practicable levels:
* Using a one-touch or two-touch method to make changes on a ballot
page.[Footnote 16]
* Varying the number of pages a voter may go back ("page backs") to
return to the page containing the Florida-13 race to change or review a
selection.
* Casting a ballot from the review screen selection. The VOTE button is
not activated until the voter reaches the last review screen. However,
once the VOTE button has been activated, a ballot may be cast from any
screen. For example, a voter may activate the VOTE button and then
return to a contest to review or change the selection using the review
screen option. Once the voter goes to the contest from the review
screen and makes any desired changes, the voter can then cast the
ballot from that screen rather than going back to the last page of the
review screen or even the review screen that was used to return to the
selection.
Although we did not consider all combinations of these types of voter
behavior when developing the standard test ballots, we included some of
these user interactions in the execution of applicable test ballots to
provide increased assurance that the system would handle these voter
behaviors. For each applicable test ballot, we randomly determined the
test procedure that should be used for the following attributes:
* Initial change method - The standard test ballots address voters
making changes on the initial ballot screen. Where possible, the method
used to change (one-touch or two-touch) the selection was randomly
selected.
* Number of page backs - The ballots used by Sarasota County included
the page back function. After reviewing the ballots, it appeared
reasonable to expect that voters who may have used the page back option
would probably decide that they had missed the race by the time they
went one or two pages beyond the page with the Florida-13 race.
Therefore, when a standard test ballot contained a page back
requirement, the number of page backs was randomly selected to
determine whether one or two page backs should be used.
* Page back change method - Some test ballots required a change after
the page back option was selected. As with the initial change method,
where possible, the method of changing (one-touch or two-touch) the
selection was randomly assigned.
* Review screen change method - The system displays a review screen
that shows the voter's selections (or lack of selections) after the
voter has progressed through all contests. On the review screen, the
voter can select a race to go directly to that contest and (1) review
the selection made, and (2) make any desired corrections. The standard
test ballots were designed to cover this type of event. Where possible,
the method used to make the change (one-touch or two-touch) was
randomly selected.
* Activate VOTE button and cast ballots from the review screen - In
order to test casting ballots from locations other than the last review
screen, the VOTE button must be activated prior to going to a screen
where the ballot is cast.[Footnote 17] In order to determine which test
ballots should be used for this test, a two-step approach was adopted.
First, a random selection of the ballots that use the review screen
option was made to determine which test ballots should have the VOTE
button activated. Then a random selection of these test ballots was
made to determine whether the ballot should be cast from the review
screen selection.
Besides those attributes that directly affect the selection in the
Florida-13 race, we varied the other attributes on the ballot in order
to complete the ballot test. For each of the 224 test ballots, we used
random values for other attributes, including the following:
* Ballot style - Each ballot was randomly assigned one of the nine
ballot styles used in the election.
* Write-in candidate - All ballot styles includes write-in options in
at least 2 races --United States Senate and State Governor/Lieutenant
Governor. To verify that the iVotronic DRE accurately recorded the
selection in the Florida-13 race for each test ballot, we needed a way
to identify each test ballot in the ballot image log. To accomplish
this, we randomly selected one of these two races, selected the write-
in candidate for the race, and entered a unique value (i.e., the test
ballot number) in the write-in field.
* Candidates and selections in other races on the ballot - Each ballot
style had between 28 and 40 contests on the ballot. The values for the
contests besides the Florida-13 race and the write-in field were also
randomly selected. For example, most items had three possible choices-
-candidate 1 (or Yes), candidate 2 (or No), and undervote. Which of
these three values was used for a given contest was randomly
determined.
The values used for these attributes were independently determined for
the election day and early voting test ballots. For example, Test
Ballot 2 (election day) and Test Ballot 202 (early voting) were
designed to test the same standard condition described by one of the
112 standard test ballots.[Footnote 18] Table 2 illustrates some of the
similarities and differences between the two test ballots that result
from the random selection process used to determine the other aspects
of the ballot.
Figure 8: Examples of Differences between Test Ballot 2 and Test Ballot
202:
Test item: Precinct;
Test Ballot 2: 142;
Test Ballot 202: 143.
Test item: Ballot style;
Test Ballot 2: 6;
Test Ballot 202: 7.
Test item: Contest used to contain unique value used to identify the
test ballot during the review process.;
Test Ballot 2: Governor/Lieutenant Governor;
Test Ballot 202: U.S. Senate.
Test item: Method used to make change on initial screen for contest;
Test Ballot 2: Two-touch;
Test Ballot 202: One-touch.
Test item: Number of page backs to return to contest;
Test Ballot 2: 2;
Test Ballot 202: 1.
Test item: Method used to make change after paging back to contest;
Test Ballot 2: Two-touch;
Test Ballot 202: Two-touch.
Test item: Activate Vote button prior to using the review screen to
return to the contest;
Test Ballot 2: No;
Test Ballot 202: Yes.
Test item: Selection for Attorney General;
Test Ballot 2: McCollum;
Test Ballot 202: Campbell.
Test item: Selection for Constitutional Amendment 1;
Test Ballot 2: No;
Test Ballot 202: Undervote.
Test item: Selection for Constitutional Amendment 8;
Test Ballot 2: No;
Test Ballot 202: No.
Test item: Method used to make change using the review screen approach;
Test Ballot 2: Two-touch;
Test Ballot 202: Two-touch.
Test item: Cast ballot from contest selection;
Test Ballot 2: No;
Test Ballot 202: Yes.
Test item: Return to review screen and then cast ballot;
Test Ballot 2: Yes;
Test Ballot 202: No.
Source: GAO.
[End of table]
Finally, we selected 10 random machines to be used for the ballot
testing.[Footnote 19] One machine was selected from those that were
used in early voting in the 2006 general election. The other nine were
selected from those that used each of the ballot styles on election day
in the 2006 general election.[Footnote 20] For each election day
machine, the assigned precinct was the same as the precinct where the
machine was used during the 2006 general election. For the early voting
machine, we needed to assign precincts for each ballot style. We used
the precinct associated with the back-up machine used for election day
testing as the precinct for that ballot style.[Footnote 21] If the
first back-up machine was assigned the same precinct number as the
primary election day machine, then we used the precinct associated with
the second back-up machine. This approach was taken to maximize the
number of precincts used in the testing efforts.
Process Used in Executing the Ballot Test:
A two-person test team conducted the ballot testing. One tester read
out aloud the steps called for in the test ballot while the other
tester performed those actions. In order to ensure that all of the
actions relating to the Florida-13 congressional race were performed as
laid out in the test ballots, a two-person review team observed a video
display of the test and compared the actions taken by the tester to
those called for in the test ballot. Furthermore, after the testing was
completed, another team reviewed the video recording of these tests to
validate that the actions relating to the Florida-13 contest taken by
the tester were consistent with those called for by the test
ballots.[Footnote 22]
The criteria used to determine whether the test produced the expected
result was derived from the Florida Voting System Standards.[Footnote
23] Specifically, among other things, these standards require the
system to allow the voter to (1) determine whether the inputs given to
the system have selected the candidates that he or she intended to
select, (2) review the candidate selections made by the voter, and (3)
change any selection previously made and confirm the new selection
prior to the act of casting the ballot. Furthermore, the system must
communicate to the voter the fact that the voter has failed to vote in
a race (undervote) and require the voter to confirm his or her intent
to undervote before casting the ballot. During the ballot test, the
actual system response was compared to the expected results by a review
team and after the testing was completed another review team compared
the video records to the test ballots to validate that the tests had
been performed in accordance with test scripts for the Florida-13
contest.
At the beginning of testing on each iVotronic DRE, the machine was
opened for voting and a zero tape was printed. After the casting of all
test ballots on the machine, the machine was closed and a results tape
was printed. The closing of the machine also writes the audit data to
the compact flash card, including event data and ballot images. We
examined the results tapes and compared the total votes cast for the
Florida-13 contest against what was expected from the test ballots. We
also kept track of the total number of ballots handled by the machine,
called the "protective count" of an iVotronic DRE, before and after the
test and confirmed that the increase in protective count matched the
number of test ballots cast on that machine.[Footnote 24]
Using the Unity election reporting manager, we read the compact flash
cards and processed the audit data on each ballot test machine. We
generated the ballot image log and examined the individual test ballots
in the ballot image log. We looked for the unique identifier that was
used for each test ballot and then confirmed that the ballot image
reflected the correct selection for the Florida-13 race as called for
by the test ballot. For example, the test script for Test Ballot 1
required the tester to (1) select a write-in candidate for U.S. Senate
and (2) enter the value of "TB1" in the write-in field. Because only
this test ballot used this value, we could review the ballot image log
to determine what selection the voting machine recorded for the Florida-
13 contest for the ballot showing "TB1" as the write-in candidate for
U.S. Senate.[Footnote 25]
Finally, using the process discussed previously for firmware testing,
the firmware on all machines used for ballot testing was validated to
ensure these machines used the same firmware that had been certified by
the Florida Division of Elections.
Results of Ballot Testing:
After executing the ballot tests on the election day and early voting
machines, we found that all 10 iVotronic DREs captured the votes for
the Florida-13 race on the test ballots accurately. We used a unique
identifier in a write-in field in each test ballot and verified that
the iVotronic DRE accurately captured the tester's final selections in
the Florida-13 race for each test ballot.
Testing 112 ways to select a candidate on a single machine also
provided us some additional assurance that the volume of ballots cast
on election day did not contribute to the undervote. We noted that
casting 112 ballots on a single machine was more than the number of
ballots cast on over 99 percent of the 1,415 machines used on election
day.
Deliberately Miscalibrated iVotronic DREs Accurately Recorded Displayed
Ballots:
Because little was known about the effect of a miscalibrated machine on
the behavior of an iVotronic DRE, we deliberately miscalibrated two
iVotronic DREs using 10 different miscalibration methods to verify the
functioning of the machine. Although the miscalibration made the
machine more difficult to use, the 39 ballots used in this test
confirmed that the system correctly recorded the displayed vote for the
Florida-13 contest and did not appear to contribute to the undervote.
Methodology for Calibration Testing:
For the calibration testing, we judgmentally selected five different
miscalibration patterns and repeated each pattern twice--once with a
small amount of miscalibration and the second time with a large amount
of miscalibration. The amount of miscalibration was also subjective--
roughly 0.25 to 0.5 inch for a small amount and about 0.7 to 1 inch for
a large miscalibration.
The miscalibration patterns are shown in the following figures.
Figure 6: Miscalibration Pattern 1: For Each Calibration Point, the
Tester Touches a Point Shifted Diagonally Inward:
[See PDF for image]
This figure is an illustration of Miscalibration Pattern 1.
Source: GAO.
[End of figure]
Figure 7: Miscalibration Pattern 2: For Each Calibration Point, the
Tester Touches a Point Shifted Horizontally Inward:
[See PDF for image]
This figure is an illustration of Miscalibration Pattern 2.
Source: GAO.
[End of figure]
Figure 8: Miscalibration Pattern 3: For Each Calibration Point, the
Tester Touches a Point Shifted Vertically Inward:
[See PDF for image]
This figure is an illustration of Miscalibration Pattern 3.
Source: GAO.
[End of figure]
Figure 9: Miscalibration Pattern 4: For Each Calibration Point, the
Tester Touches a Point Shifted Horizontally to the Right:
[See PDF for image]
This figure is an illustration of Miscalibration Pattern 4.
Source: GAO.
[End of figure]
Figure 10: Miscalibration Pattern 5: For Each Calibration Point, the
Tester Touches a Point Shifted Horizontally to the Left:
[See PDF for image]
This figure is an illustration of Miscalibration Pattern 5.
Source: GAO.
[End of figure]
We conducted calibration testing on two different machines that were
used for ballot testing.[Footnote 26] As with ballot testing, at the
beginning of testing of each machine, we opened the machine for voting
and printed a zero tape. During the opening process, we calibrated the
machine with one of the miscalibration patterns. After the machine was
miscalibrated, we then executed at least three of the test ballots that
were used during ballot testing on that machine for each test.[Footnote
27] The test ballots were rotated among the miscalibration patterns.
For example, one of the machines had eight different ballot test
scripts. The first three were used on one miscalibration pattern, the
next three on another miscalibration pattern, and the final two plus
the first one would be used on another miscalibration pattern. After
the ballots were cast for one miscalibration pattern, the machine would
be miscalibrated with another pattern. After the needed miscalibration
patterns were tested on a machine, the iVotronic DRE was closed and a
results tape was printed. The closing of the iVotronic DRE also wrote
the audit data to the compact flash card.
During the testing, the tester was instructed to take whatever actions
were necessary to achieve the desired result. For example, if the
script called for the selection of Candidate A, then the tester would
keep touching the screen until Candidate A was selected. A review team
monitored the testing to ensure that (1) the proper candidate for the
Florida-13 congressional race was ultimately selected and (2) the
review screen showed this candidate selection when it was first
presented.
As with the ballot test, we used the Unity election reporting manager
to read the compact flash cards and processed the audit data or each
ballot test machine. We generated the ballot image log and examined the
individual test ballots in the ballot image log. We looked for the
unique identifier that was used for each test ballot and then confirmed
that the ballot image reflected the correct selection for the Florida-
13 race as called for by the test ballot. After the testing had been
completed, the expected results shown in the test ballot scripts were
compared to the actual results contained in the ballot image log and
the results tape using the same process discussed in the ballot testing
methodology.
Results of Calibration Testing:
The 39 ballots used in this test confirmed that the system correctly
recorded the displayed vote for the Florida-13 contest. We also noted
that the miscalibration clearly made the machines harder to use and
during an actual election these machines would have probably been
either recalibrated or removed from service once the voter brought the
problem to the precinct's attention, according to a Sarasota County
official who observed the tests.[Footnote 28]
Figure 11 shows an example of effects of our miscalibration efforts on
the screen that is used to confirm the calibration results.
Specifically, the stylus points to where the tester is touching the
screen while the "X" on the screen shows where the machine indicated
the stylus was touching the screen.[Footnote 29] In a properly
calibrated machine, the stylus and the "X" are basically at the same
point.
Figure 11: Example of the Effects of a Miscalibrated Machine on the
Calibration Screen:
[See PDF for image]
This figure is an illustration of the effects of a Miscalibrated
Machine on the Calibration Screen.
Source: GAO.
[End of figure]
Figure 12 shows an example of where the tester is touching the screen
to make a selection and how this "touch" is translated into a
selection. As can be seen, the finger making the selection is touching
a position that in a properly calibrated machine would not result in
the selection shown. However, the machine clearly shows the candidate
selected and our tests confirmed that for the 39 ballots tested, the
candidate actually shown by the system as selected (in this example,
the shaded line) was the candidate shown on the review screen, as well
as the candidate that received the vote when the ballot was cast.
Figure 12: Example of the Effects of a Miscalibrated Machine on a
Candidate Selection:
[See PDF for image]
This figure is an illustration of the effects of a Miscalibrated
Machine on a Candidate Selection.
Source: GAO.
[End of figure]
Conclusions:
Our tests showed that (1) the firmware installed in a statistically
selected sample of machines used by Sarasota County during the 2006
general election matched the firmware certified by the Florida Division
of Elections, and we confirmed that when the manufacturer rebuilt the
iVotronic 8.0.1.2 firmware from the escrowed source code, the resulting
firmware matched the certified version of firmware held in escrow, (2)
the machines properly displayed, recorded, and counted the selections
for all test ballots cast during the ballot testing involving the 112
common ways a voter may interact with the system to cast a ballot for
the Florida-13 race, and (3) the machines accurately recorded the test
ballots displayed on deliberately miscalibrated machines. The results
of these tests did not identify any problems that would indicate that
the iVotronic DREs were responsible for the undervote in the Florida-13
race in the 2006 general election.
As we noted when we proposed these tests, even after completing these
tests, we do not have absolute assurance that the iVotronic DREs did
not play any role in the large undervote. Absolute assurance is
impossible to achieve because we are unable to recreate the conditions
of the election in which the undervote occurred. Although the test
results cannot be used to provide absolute assurance, we believe that
these test results, combined with the other reviews that have been
conducted by Florida, GAO, and others, have significantly reduced the
possibility that the iVotronic DREs were the cause of the undervote. At
this point, we believe that adequate testing has been performed on the
voting machine software to reach this conclusion and do not recommend
further testing in this area. Given the complex interaction of people,
processes, and technology that must work effectively together to
achieve a successful election, we acknowledge the possibility that the
large undervote in Florida's 13th Congressional District race could
have been caused by factors such as voters who intentionally
undervoted, or voters who did not properly cast their ballots on the
iVotronic DRE, potentially because of issues relating to interaction
between voters and the ballot.
Comments:
We provided draft copies of this statement to the Secretary of State of
Florida and ES&S for their review and comment. We briefed the Sarasota
County Supervisor of Elections on the contents of this statement and
asked for their comments. The Florida Department of State provided
technical comments, which we incorporated. ES&S and the Sarasota County
Supervisor of Elections provided no comments.
Mr. Chairman, this completes my prepared statement. I would be happy to
respond to any questions you or other Members of the Task Force may
have at this time.
Contact and Acknowledgments:
For further information about this statement, please contact Naba
Barkakati at (202) 512-6412 or [email protected]. Contact points for
our Office of Congressional Relations and Public Affairs may be found
on the last page of this statement. Other key contributors to this
statement include James Ashley, Stephen Brown, Francine Delvecchio,
Cynthia Grant, Geoffrey Hamilton, Richard Hung, Douglas Manor, John C.
Martin, Jan Montgomery, Daniel Novillo, Deborah Ortega, Keith Rhodes,
Sidney Schwartz, Patrick Tobo, George Warnock, and Elizabeth Wood. We
also appreciate the assistance of the House Recording Studio in the
video recording of the tests.
[End of section]
Appendix I: Methodology for Selecting IVotronic DREs for GAO Testing:
Each of the three tests--firmware verification, ballot, and
calibration--was conducted on a sample of the 1,499 iVotronic DREs that
recorded votes during the 2006 general election in Sarasota County,
Florida. We selected 115 iVotronic DREs for the firmware test, 10 for
the ballot test, and 2 for the calibration test. Appendix II contains
the serial numbers of the iVotronic DREs that were tested.
Firmware Test Sample:
We selected a stratified random probability sample of iVotronic DREs
from the population of 1,499. The sample was designed to allow us to
generalize the results of the firmware sample to the population of
iVotronic DREs used in this election. We stratified the population into
two strata based on whether the machines had been sequestered since the
2006 general election. There were a total of 818 machines that were
sequestered and 681 machines that had been used in subsequent
elections. The population and sample are described in table 3.
We calculated the sample size in each stratum using the hypergeometric
distribution to account for the relatively small populations in each
stratum. We determined each sample size to be the minimum number of
machines necessary to yield an upper bound of 7.5 percent, at the 99
percent confidence level, if we observed zero failures in the firmware
test. Assuming that we found no machines using an uncertified firmware
version, these sample sizes allowed us to conclude with 99 percent
confidence that no more than 7.5 percent of the machines in each
stratum were using uncertified firmware. Further, this sample allowed
us to conclude that no more than 4 percent of the 1,499 iVotronic DREs
were using uncertified firmware, at the 99 percent confidence level.
Table 2: Description of the Stratified Population and Sample Sizes for
the Firmware Test:
Stratum: Sequestered machines;
Population size: 818;
Sample size: 58.
Stratum: Non-sequestered machines;
Population size: 681;
Sample size: 57.
Stratum: Total;
Population size: 1,499;
Sample size: 115.
Source: GAO based on analysis of Sarasota County voting data.
[End of table]
An additional five sequestered machines and five non-sequestered
machines were selected as back-up machines should there be problems in
locating the selected machines or some other problem that prevented
testing them.
Ballot Test Sample:
We randomly selected a total of 10 machines from the population of
1,384 machines that were not selected in the firmware test sample. This
sample size is not sufficient to allow us to make direct
generalizations to the population. However, if we are reasonably
confident that the same software is used in all 1,499 machines, then we
are more confident that the results of the other tests on a small
number of machines can be used to obtain increased assurance that the
iVotronic DREs did not cause the undervote. We randomly selected one
machine from each of the nine ballot styles used during the general
election and one machine from the machines used for early
voting.[Footnote 30] In case of problems in operating or locating the
machines, we also selected randomly selected two additional machines
for each ballot style and for early voting.
Calibration Test Sample:
The two iVotronic DREs selected for calibration testing were selected
from those tested in the ballot test. Because the machines used for the
ballot tests included an ADA machine and "standard" machines, we
selected one of each for calibration testing. Although we did not test
the ADA capabilities of the ADA machine (e.g., the audio ballots), we
found that the on-screen appearance of selections on the ADA machine
differed slightly from that on non-ADA machines. For example, the
standard non-ADA machine displayed a blue bar across the screen and an
X in the box next to the candidate's name when a selection was made,
while an ADA machine only showed an X in the box next to the
candidate's name.
[End of section]
Appendix II: List of Machines Tested by GAO:
[End of section]
Table 4 table lists the iVotronic DREs that were tested by GAO. For
each machine, the table shows whether the machine was sequestered and
what type of testing was conducted on the machine.
Table 4: List of iVotronic DREs Tested by GAO:
Serial number: V0105178;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105203;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105222;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105255;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105305;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105351;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105379;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105390;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105396;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105422;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105481;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105499;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105500;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105524;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105526;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105563;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105573;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105607;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105613;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105623;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105651;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105656;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105661;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105664;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105743;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105848;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0105873;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105874;
Machine sequestered: No;
Type of testing conducted:
Firmware testing.
Serial number: V0105894;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105903;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105906;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105923;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105964;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105971;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105992;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106001;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106016;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106024;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106025;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106034;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106064;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106068;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106069;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106084;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106087;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106126;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106156;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106191;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106203;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106254;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106264;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106265;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106274;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106282;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106343;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106368;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106377;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106396;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106445;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106461;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106475;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106478;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106486;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106507;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106522;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106525;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106531;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106552;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106585;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106586;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106588;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106602;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106615;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106656;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106658;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106661;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106667;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106681;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106711;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106718;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106740;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106744;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106833;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106840;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106864;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106865;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106878;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106881;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106883;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106907;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0106933;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106936;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106949;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0106965;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107000;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107011;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107020;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107042;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107045;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107053;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107077;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107082;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107094;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107108;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107138;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0107143;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0107147;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0110355;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0111064;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0113816;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0114087;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0114415;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0117658;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0118183;
Machine sequestered: No;
Type of testing conducted: Firmware testing.
Serial number: V0118293;
Machine sequestered: Yes;
Type of testing conducted: Firmware testing.
Serial number: V0105386;
Machine sequestered: Yes;
Type of testing conducted: Early voting ballot testing.
Serial number: V0105266;
Machine sequestered: Yes;
Type of testing conducted: Election day ballot testing.
Serial number: V0105694;
Machine sequestered: No;
Type of testing conducted: Election day ballot testing.
Serial number: V0106082;
Machine sequestered: Yes;
Type of testing conducted: Election day ballot testing.
Serial number: V0106145;
Machine sequestered: Yes;
Type of testing conducted: Election day ballot testing.
Serial number: V0106247;
Machine sequestered: Yes;
Type of testing conducted: Election day ballot testing.
Serial number: V0106509;
Machine sequestered: No;
Type of testing conducted: Election day ballot testing and calibration
testing.
Serial number: V0106671;
Machine sequestered: Yes;
Type of testing conducted: Election day ballot testing.
Serial number: V0117861;
Machine sequestered: No;
Type of testing conducted: Election day ballot testing and calibration
testing.
Serial number: V0117951;
Machine sequestered: No;
Type of testing conducted: Election day ballot testing.
Source: GAO.
[End of table]
[End of section]
Footnotes:
[1] Undervotes occur when the number of choices selected by the voter
is fewer than the maximum allowed for that contest. In this case, it
means ballots that did not record a selection for either candidate in
the congressional contest.
[2] GAO, Elections: Further Testing Could Provide Increased but Not
Absolute Assurance That Voting Systems Did Not Cause Undervotes in
Florida's 13th Congressional District, GAO-08-97T (Washington, D.C.:
Oct. 2, 2007).
[3] GAO, Elections: Status of GAO's Review of Voting Equipment Used in
Florida's 13th Congressional District, GAO-07-1167T (Washington, D.C.:
Aug. 3, 2007).
[4] GAO-08-97T.
[5] In May 2007, the State of Florida enacted legislation requiring, in
general, the use of optical scan voting equipment that provides a paper
trail. These requirements are effective July 1, 2008. There is an
exemption from these requirements for voting by persons with
disabilities.
[6] The certified version of ES&S Voting System Release 4.5, Version 2,
Revision 2, specifies the use of iVotronic hardware version 1.0.
According to Florida Division of Election officials, hardware version
1.1 of the iVotronic DRE has been available since at least 2004 and
should have been included as a part of the certification for ES&S
Voting System Release 4.5, Version 2, Revision 2. According to ES&S
officials, iVotronic firmware version 8.0.1.2 runs in exactly the same
manner on hardware versions 1.0 and 1.1.
[7] GAO, Elections: The Nation's Evolving Election System as Reflected
in the November 2004 General Election, GAO-06-450 (Washington, D.C.:
June 6, 2006).
[8] GAO-08-97T.
[9] GAO, Elections: Federal Efforts to Improve Security and Reliability
of Electronic Voting Systems Are Under Way, but Key Activities Need to
Be Completed, GAO-05-956 (Washington, D.C.: Sept. 21, 2005).
[10] GAO-08-97T.
[11] The National Institute of Standards and Technology (NIST) has
issued a Federal Information Processing Standard (FIPS) that describes
four hashing algorithms that are iterative, one-way hash functions that
can process a file and produce a condensed representation called a
message digest or "hash." These algorithms enable the user to validate
a file's integrity since any change to the file will, with a very high
probability, result in a different message digest. The technical
details of this process are contained in FIPS 180-2. The algorithm
selected for this testing effort is commonly referred to as SHA-1 and
is the same algorithm used by the Florida Division of Elections during
its audit.
[12] Two of the files and the expected values used came from FIPS 180-
2.
[13] We verified that sequestered machines were not used since the 2006
general election by (1) verifying that the seals placed on these
machines agreed with Sarasota County's records, and (2) checking the
event logs maintained on the machine to determine whether the machines
had been used since the machine had been sequestered. In every case, we
found that the seal numbers agreed with Sarasota County's records. We
were able to check the event log for 57 of the 58 sequestered iVotronic
DREs. We were unable to power up the remaining iVotronic DRE and were
consequently unable to extract the needed audit data.
[14] In our October 2007 statement, we reported that according to ES&S,
firmware compiled from the Florida escrowed source code may not be
exactly identical to the firmware certified by the Florida Division of
Elections because the embedded date and time stamp in the firmware
would be different. We found that the date and time was not embedded in
the firmware and that an identical version could be created.
[15] Security and Assurance in Information Technology Laboratory,
Florida State University, Software Review and Security Analysis of the
ES&S iVotronic 8.0.1.2 Voting Machine Firmware (Tallahassee, Florida:
Feb. 23, 2007).
[16] The iVotronic DREs used in Sarasota County allow the user to make
changes using two methods. The first method allows the user to simply
touch the other candidate; e.g., Candidate A is initially selected and
the voter decides to select Candidate B by touching the name of
Candidate B. We referred to this as the "one-touch method." The other
method, referred to as the "two-touch method," involves the user first
deselecting the initial choice and then making another selection; e.g.,
Candidate A is initially selected and the voter decides to select
Candidate B by (1) touching the name of Candidate A, which deselects
Candidate A, and then (2) touching the name of Candidate B to select
it.
[17] The actual procedure is to (1) go to the last review screen, which
activates the VOTE button, (2) page back to the contest (normally 2 or
3 page backs depending on the ballot style), and (3) selecting the
contest on the review screen that should be revisited. We assumed that
voters would cast such ballots using this procedure instead of using
the page back option because it did not appear reasonable that a voter
would page back at least 17 screens to reach the Florida-13 race, which
was the focus of the testing.
[18] The standard actions taken in these two test ballots called for
the tester to (1) make a selection on the initial screen and then
change the selection, (2) page back to the initial selection screen and
change the selection, and (3) use the review screen option to change
the selection again.
[19] Details on the random selection can be found in appendix I.
[20] We excluded machines from one precinct that used two ballot styles
instead of one.
[21] In order to ensure that we could complete our tests even if a
machine selected for testing failed to operate, our statistical
sampling methodology generated a list of machines that could be used as
replacements and still maintain the integrity of the testing process.
These are referred to as "back-up" machines.
[22] These two reviews identified two early voting and seven election
day test ballots where the specified scripts were not followed exactly
for the Florida-13 contest. Because these test ballots had not followed
the test script for the Florida-13 contest exactly, they were retested.
Accordingly, the testing efforts resulted in 233 actual ballots being
cast.
[23] Florida Department of State, Florida Voting System Standards, Form
DS-DE 101 (Jan. 12, 2005).
[24] The iVotronic DRE is designed to maintain a count of all ballots
cast on a given machine and functions much like an automobile's
odometer. The protective count can be used to help ensure that the
election process did not lose any votes. For example, before a machine
is sent to a precinct, the protective count is recorded. Accordingly,
if the precinct's voting register show that 100 individuals voted, then
the increase in the protective counts for all machines assigned to that
precinct should increase by 100. This value can then be compared to the
actual votes recorded in the election to ensure that the values are
consistent; i.e., the results tape for the election shows that 100
votes have been accounted for during this election using this example
precinct.
[25] In some cases, a test ballot had to be reentered because the
original test did not follow all of the desired actions associated with
the Florida-13 contest. In these cases, the value entered was made
unique by adding a letter to the value, e.g., "TB1A".
[26] The approach used to select these machines is described in
appendix I.
[27] In the testing of the first two miscalibration patterns for the
first machine, all the test ballots used in the ballot testing for that
machine were repeated. However, the individual performing the testing
soon recognized the changes that were needed to compensate for the
miscalibration. Accordingly, the tester did not make as many attempts
to perform the desired function in the later cases as with the first
three cases. Therefore, for the remaining eight miscalibration test
patterns, we executed three test ballots per pattern because these
cases produced the greatest likelihood of generating spurious touches
before obtaining the desired selection.
[28] Our review of the election day records identified two reported
cases on election day where the miscalibration of the iVotronic DRE led
to its closure and discontinued use for the rest of the day.
[29] While votes are normally cast using fingers on the touch screen, a
stylus is normally used during the calibration process.
[30] We also excluded those election day machines from one precinct
that supported two different ballot styles.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: [email protected]:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, [email protected]:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, [email protected]:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
*** End of document. ***