Inside the Iowa Democratic Party’s ‘Boiler Room’ Meltdown

(Photo by Steven Rosenfeld)

The app and software that failed to report and count the Iowa Democratic Party’s 2020 presidential caucus results were not the only miscalculations by the IDP and its vendor.

Its “boiler room” or secret operations center was unprepared to handle the chaos that ensued. Fewer than 100 computers and phones were set up for IDP volunteers to receive the results and complaints. IDP help desk materials anticipated problems with the app, including that some caucus chairs would end up calling in their results. The IDP’s instructions were complex, but they were also lacking some basics, according to experts in voting technology systems.

“In general, there is no documentation of what to do if various other problems arise,” said David Jefferson, a voting systems software expert who has studied and critiqued electronic election infrastructure since the 1990s. He is an executive board member of Verified Voting, a national advocacy group founded by computer scientists and academics.

“The universal answer seems to be, ‘If you have a problem, call in your results,'” he continued. “That might make sense if the app and the server were so robust that there were almost never any problems, and if the training of the caucus chairs was thorough so that they were all confident that they knew how to handle it. But apparently that was not the case.”

New revelations about the IDP “boiler room” offer important lessons. Nevada’s 2020 Democratic Party presidential caucus starts with early voting on Saturday, February 15. Its party officials have ditched their plans to use the same reporting app and system that failed in Iowa’s 2020 presidential caucus. While the Nevada Democratic Party has said that it might use paper-based voting in precinct caucuses, it has not released further specifics.

Meanwhile, DNC Chairman Tom Perez has called on Iowa’s state party to recount its results amid complaints from campaigns—a process that the IDP’s 2020 rules only anticipated if a campaign filed a formal request and paid for it upfront.

(On Thursday, Bloomberg.com reported that Donald Trump’s supporters apparently “flooded a hotline used by Iowa precinct chairs” after finding its number online. This was a development that nobody in state or national party circles overseeing the caucuses had voiced beforehand. Instead, they had focused on more complex cybersecurity threats.)

Inside IDP’s boiler room

The vote counting and help desk operations center was tucked into two conference rooms in the mezzanine above Veterans Memorial Hall at the Iowa Events Center—the opposite end of the same complex where a media filing center hosted TV networks whose analysts spent hours Monday night trashing the IDP for failing to release the caucus results. As the night unfolded, the party announced that irregularities in the counting, software glitches and jammed phones all collided and pushed the IDP to abandon using its results-reporting app and switch to compiling results from paper summary sheets from nearly 1,700 precincts.

Voting Booth found the boiler room on Wednesday. While IDP officials were continuing to compile results in one room, this reporter saw the other room. It was empty but still set up as an operations center. It had computers, phones, routers; training materials on desks, help desk scripts and forms for taking caucus results and other complaints; an “app FAQ” sheet for caucus chairs who experienced problems and other instructions on wall posters.

The training materials were for a session that occurred on Sunday—one day before the 2020 caucus. An agenda said it included a “brief description of the caucus night set up,” “boiler room set up” work shifts, rules in the room, phone security and questions. Voting Booth took photos and shared them with Verified Voting’s Jefferson for context.

Jefferson concluded that the boiler room had less than half the staff that it needed at peak hours to field calls; that its FAQ and other help desk materials omitted key information—even as it anticipated that caucus chairs would encounter problems logging in and using passwords. He also noted what appeared to be additional missing elements that added to the congestion and reporting meltdown, such as the app not saying that results had been successfully received, which prompted caucus chairs to call in to confirm that they had been received—further clogging the phone lines.

The operations center was a windowless rectangular room filled with a half-dozen rows of long desks covered in black tablecloth. On each desk sat two or three laptop computers and a phone, all hardwired by Ethernet cables to a server in the room. The front desks had piles of a “2020 Precinct Caucus Guide” that summarized procedures, piles of forms to report results and other problems, and red and blue folders with these documents and others—such as app FAQs and phone scripts.

The front-facing wall had hand-lettered posters. One said, “NO photos can be taken in this room!!! NO social media usage allowed at all!!!!” There also were reminders for volunteers “to confirm [precinct] ID,” to tell caucus chairs where to find a “Precinct PIN” in their materials (to be able to log in using the app), and an email address where the chairs were to send photos of the “caucus math worksheets.” That last document had the key results from two rounds of caucus voting in each of Iowa’s 1,678 precincts. It was what the party scrambled to collect to verify and redo the statewide count, after finding tabulation errors from the app and data-entry mistakes by boiler room volunteers (as reported by the New York Times).

Together, if both conference rooms were similarly equipped (this reporter was escorted out when approaching the second room), that meant there were fewer than 100 computer stations ready to receive precinct results and handle complaints. All calls came in via the same telephone number. Once a caucus chair with results got through, IDP volunteers were told to fill out a form “completely… [and] confirm each number by reading it back to the precinct leader.” If the call was reporting something else, labeled “general public issues” or “temp chair issues,” volunteers took briefer notes and were told to “tell the caller [that] they will be called back ASAP.”

The party knew the app would be problematic. Their materials listed at least 10 different problems with the reporting app, including: “Can’t Sign In,” “Wrong Pin,” “Password,” “2 Factor Authentication,” “Error Message,” “Calculation Issues,” “Screen is Frozen,” “Can’t Take a Picture” and “Satellite Caucus Reporting [from outside the state].”

The “App FAQ” anticipated more specific problems, such as caucus chairs not finding their precinct IDs or PIN numbers to log in, and PIN numbers not working. The FAQ said there were two different PINs that the chairs had to use, one to identify the precinct and another to report the results. If they cannot get into the system’s backend with the PINs, the FAQ said, “They’ll need to close out of the app and restart the login process.”

It described additional steps “if they’re on a two-factor authentication screen.” If they could not log in, it said, “If for some reason they’re not receiving the code via text after a couple of tries, they should call in their results.”

The FAQ also said the caucus chairs could not go backward through the screens if any error was made. “Once they move on from a screen, they cannot go back to a previous screen and they cannot restart the caucus as they could in the test mode,” it said. “Every screen will ask them to confirm their numbers before moving on. If they made an error on any step, they should call in their results.” (Emphasis in original.)

A complex process without snafus

Verified Voting’s Jefferson called the app and these details “a human interface disaster.”

“One thing that sticks out like a sore thumb is the discussion under ‘Can I go back while reporting in the app?’ It says that no, you cannot go back to previous screens when using the app. It says, ‘If they’ve made an error on any step, they should call in their results,'” he said. “There is no recovery from error! No ‘undo.’ No ‘go back.’ They cannot even start over and ‘restart the caucus.'”

“This, of course, is a human interface disaster,” he said. “It is hard to fathom how anyone could design and field such a rigid, fragile app, [so] unforgiving, and expect hundreds of people who have never used the app before to be successful at it. This feature alone would likely cause most of the caucus chairs to call in their results instead of using the app, adding to frustration and phone-in volume. This alone should have caused the IDP to completely reject the app.”

Jefferson also said that the FAQ omitted basics, such as including different answers for people using Apple and Android operating systems. It did not mention how caucus chairs were to download the app, which would be different on Apple and Android phones.

But the larger point was the IDP was understaffed to handle the breakdown that ensued when it had to shift from an app-based reporting system that it knew was problematic.

Jefferson, doing rough calculations, said that at best, each call should have taken 5 minutes. Of course, once the apps started failing and the calls cascaded (apart from any interference intentionally clogging lines from Trump supporters), the volume went up quickly and a backlog developed that lasted as long as 90 minutes. 

Under the best of circumstances, it would have taken five or six hours to efficiently handle the telephone reporting, he said, noting that the volume of information collected in 2020 was much more than in past years. The IDP was seeking to be more transparent. It was collecting and planned to announce four figures for every caucus: the participant numbers, first alignment results, realignment results and delegates awarded.

Jefferson suggested that the IDP had half as many help desk stations as it needed. But the IDP never expected that it would have to abandon its app and software backend. On the Friday before Monday’s caucus, IDP Executive Director Kevin Geiken was upbeat about the system. When pressed by this reporter about the “worst-case scenario,” he said that it would take one-to-two days to count the paper “caucus math worksheets” submitted in person or by emailed photos from across the state.

The IDP has not responded to Voting Booth’s inquiries since Monday’s caucus.

Up next: Nevada’s caucuses

In response to the Iowa meltdown, Nevada State Democratic Party officials have said that they were abandoning the use of two different apps from the same vendor that failed in Iowa, and were looking at reverting to a paper-based voting and counting process. The NDP did not respond to Voting Booth’s inquiries about its likely shift to paper, which would begin with early voting that starts on February 15.

The party had planned to use 1,000 party-owned digital tablets for those participants to vote online, according to the Nevada Independent. It had also planned on using the same app as in Iowa to send its early voting results to its caucus chairs at 250 sites at the start of the regularly scheduled caucus process on February 22, and to transmit the results once the caucuses were finished.

The Nevada Democratic Party, like that of Iowa, intended to have every caucus participant fill out a paper presidential preference card, and for caucus chairs to compile results summary sheets that every candidate’s precinct captain would sign to affirm the results. But the system to file those results, and then compile and report statewide totals, is yet to be seen.

Also Available on: www.salon.com