Georgia’s Hand Count of 2020 Ballots was No Risk-Limiting Audit, Says Audit Creator
(Photo by Steven Rosenfeld)
On Friday, Georgia certified its 2020 election results. Joe Biden’s 12,284-vote victory over President Trump was more than 1,000 votes smaller than what was reported days after the November 3 election, where 5 million Georgians voted for president.
The official results slightly changed because Georgia did something not seen in any other state. It double checked its results, which had been compiled electronically, by conducting a hand count of every presidential ballot. Election officials did not plan on the hand count. Following a 2019 law, they expected to conduct a “risk-limiting” audit, which assesses the accuracy of the unofficial results by randomly sampling paper ballots. That 2019 law required counties to conduct the audit before certifying the winners.
Risk-limiting audits, of which there are several varieties, were invented by Philip Stark, a University of California Berkeley statistician. In general, once a county has its inventory of all of its paper ballots, statistics can be used to find and pull ballots to grade the reported results. Typically, a risk-limiting audit will look at enough ballots to achieve a 90 percent confidence level that the totals were accurate. The process is not intended to assess any other issue.
What unfolded in Georgia wasn’t a risk-limiting audit, Stark said, although that was what it was called by top state officials. It was closer to a statewide ballot inventory check, which turned up several thousand uncounted votes. And it was closer to what is called the canvass, which is where local officials reconcile ballots and totals. It was also not a recount, which is a different legal process. President Trump is expected to file for a recount in coming days.
Here is their conversation.
Steven Rosenfeld: You have some concerns about Georgia’s audit.
Philip Stark: There’s this term flying around called “risk-limiting audit.”
SR: You created it. This exercise in Georgia is not that.
PS: People are defining it to suit their convenience. Let’s not worry about the terminology, but instead focus on what voters actually would like to have. I would claim that voters would like to have confidence that the reported winners really won. That’s what this is all about. And that the risk that they care about is the risk that an incorrect outcome will be certified.
SR: Right, for all kinds of reasons.
PS: When I invented risk-limiting audits, that is the risk I was talking about, and that was the risk that was being limited [by this kind of audit]. And in order to do that, you need a trustworthy paper trail of votes. Among other things, even if can trust that every piece of paper accurately reflects what the voters said, they need to keep track of the paper and keep it secure. And a precursor to doing a statistical risk-limiting audit that you have a complete inventory of the paper, so that you can draw a random sample of the paper.
SR: I understand. It’s not an easy thing to do. And Georgia didn’t do that.
PS. The way that Gabe [Sterling, the state’s elections operations manager] is being incredibly misleading about all of this is saying, ‘Look how great it was that we audited and we uncovered the fact that some batches of ballots were never scanned, and some memory cards [from ballot scanners with vote totals] were never uploaded and what not. All of that has nothing to do with the risk-limiting audit. That is all a precursor to starting a risk-limiting audit.
SR: You’re saying that the lapses they found should have been found and fixed earlier.
PS: Yes. Those are all ballot accounting measures. They’re standard canvass activities. They’re reconciliation measures like checking the number of poll book signatures against the number of ballots, so that you have a separate physical count of the number of ballots against the reported tallies from the voting machines. That is all stuff you have to do before you start the audit.
If you don’t have control over how many ballots there are, you would never notice if there were 100,000 ballots missing from the machine totals. Right? So, one fundamental flaw in what Georgia is doing is they’re relying on the voting system to tell them how many ballots there are, rather than relying on other procedures and cross checks to tell them how many ballots there are. Because some ballots were never scanned and some memory cards were never uploaded, the voting system doesn’t know about those ballots, and a random sampling based RLA would not have had any chance of selecting those ballots. The sample needs to be drawn from a comprehensive list of ballots, not just the ballots the voting system happens to have a record of.
SR: That’s what I saw. Elections are narratives that unwind step by step. We can follow the chronology forward, but there were lapses. The chain of custody around ballots. The data that’s reported about those ballots.
PS: It’s clear that Georgia’s process has been anything but transparent. Observers have been kept away from the ability to actually verify the counts. Even observers with party credentials couldn’t verify that what’s being input to ARLO [the vendor’s counting software] accurately reflects those counts, or [examine] the inner workings of ARLO—which has had a number of changes in the course of this. ARLO was never designed for this kind of thing. They [Georgia] would have been better off using Google Sheets. They could have let the rest of the world watch in real time, with a read-only version to see as they enter the data. They could have given separate logins to everyone in the county election offices who were entering data, so that they can trace the edit history of any cell in the [overall vote counting] spreadsheet. Instead, they’ve got people sharing logins. They’ve got an opaque system that nobody can watch.
Maybe this whole thing gives election officials more confidence that they got right answer [from the state’s new vote count scanners], but there’s no reason it should give the public more confidence that they got the right answer because too much of it [this audit] was done out of public view. First and foremost, the public has no reason to believe that election officials have accounted for every ballot–since we already have several examples where they didn’t. Georgia’s canvas [ballot-accounting] procedures aren’t providing the foundation needed for a risk-limiting audit.
SR: It was Harvey Branscomb [a Colorado activist and audit expert] who pointed out that this is called a hand count, but all the tabulations are being done electronically after what could be data entry errors. He also said that a bunch of local officials appear to have figured out that they could compare their electronic counts and the audit’s hand count. If their figures were off, they could find and fix what was wrong before entering the audit’s results into ARLO.
PS: The other thing that Gabe kept saying is this [audit] is uncovering human errors in the count, not machine errors. We know that the scanner settings on the Dominion scanners erase voter marks [because lightly marked ballots are not read]. We know the majority vote-by-mail voters were Biden voters. I would expect their hand count to pick up [missed] votes and to get more votes for Biden than for Trump. And we’re not seeing any of that. There’s no reason to presume that it’s the people who are wrong and it’s the machines that are right.
SR: I’ve been thinking about the pros and the cons. On the positive side, the audit showed that full hand counts can be done and done quickly after Election Day and before certifying winners. But you’ve identified more negatives than what I’ve heard.
PS: Let me raise one more thing which is this notion; that because the margin [between Biden and Trump] was 0.3 percent, that a risk-limiting audit (RLA) requires a full hand count. That is not true. It is a result of the fact that the state contracted with Voting Works to conduct the RLA. VotingWorks software only implements one of the less efficient ways of conducting an RLA, called a ballot-polling RLA. It’s true that in a ballot-polling RLA with a margin of about 0.3 percent [in a race with 5 million votes cast], they would have expected to have to look at more than a million ballots by hand to confirm the outcome to a 10 percent risk limit, on the assumption that everything else was right. But had they set themselves up to do a ballot-comparison RLA instead of a ballot-polling RLA, the sample size would be about 2,000 ballots. That’s how Colorado does it.
SR: That’s a very big difference. Examining millions of ballots verses 2,000.
PS: Let me tell you what that would have required. Colorado was set up to do that because they’re almost an entirely vote by mail state. They do a lot of central scanning [to count votes]. So, the issue is that to use that [ballot comparison] method, the voting system has to tell you how it interpreted individual ballots in such a way that you can tie that interpretation back to the corresponding paper ballot.
In precinct-count optical scan systems [counting ballots], that typically isn’t possible because the system will deliberately shuffle the cast vote records [a spreadsheet of all the tallied votes] compared to the order in which the ballots were scanned, to try to protect voter privacy. You can’t tell the order the ballots went into the box; whose ballot is whose. Now, Georgia had about a quarter of their ballots that were hand-marked paper ballots scanned centrally, so those were taken care of. The Dominion system would let them make that linkage between cast records and paper. Had they simply rescanned the other 75 percent, they would have had the ability to do a ballot-level level comparison audit. Moreover, if Trump’s campaign demands a full recount as expected, they’d be 75 percent of the way done with that exercise, since Georgia recounts are electronic re-scans, not manual counts.
SR: What should they have done with catching mistakes during the audit? Should they have tried to have fixed discrepancies before they sent their audit data to the state?
PS: There’s all kinds of potential discrepancies that should have been addressed earlier. There’s reconciling the number of registered voters against the number of poll book signatures, against the number of physical pieces of paper [all of the ballots], against the number of tabulated votes. That all should have happened before the audit started. The kinds of stuff that they’re uncovering where somebody forgot to scan a batch of ballots, or somebody didn’t upload a memory card, that all should have been taken care of by processes before RLA started. That’s all part of the standard process of generating an accurate ballot manifest, from which you draw the sample ballots [when conducting a statistical-based risk-limiting audit].
They basically skipped step zero. Instead, the counties evidently are constructing a ballot manifest, which the audit relies on, from what the computers report, rather than from ground truth on the amount of paper ballots there are. Your inventory of ballots should not rely at all on the voting system to tell you how many ballots you have. You should have control over that from external means. You know how many ballots went to a precinct. You know how many came back voted, spoiled or blank. You should be physically counting these things. That baseline shouldn’t be a function of the voting system. It starts with inventory control.
SR: All of this is why what Georgia did wasn’t a risk-limiting audit or even close.
PS: Right. You are trying to confirm the winner. You do not confirm the tally, you confirm the winner. You not use the sample-based results to alter the official totals. The only circumstance under which you alter the official totals is if you do a full hand count. They aren’t doing that—and Georgia’s audit law prohibits that.
You understand a legal situation, right? If this was an audit done under Georgia’s audit law, it can’t change the outcome. Therefore, by definition, it is not a risk-limiting audit, because it cannot limit the risk of certifying a wrong outcome unless it can correct the outcome. By definition, their process did not address the risk people that care about. Second, if it’s a recount, then it can’t be done by hand, it has to be done by machine [scanners] according to Georgia law.
So rather than an audit, recount, and re-canvass, it [the presidential re-tally] seems to have been their first canvass, because it’s the first time they looked to make sure that they actually had all their ballots accounted for. If they’re going to just keep counting until you get the same answer the machine did, that’s obviously biased procedure. That’s nowhere close to a risk-limiting audit.
