Berkeley Talks transcript: Barbara Simons on election hacking and how to avoid it in 2020

Tsu-Jae King Liu: Welcome everyone. I’m Tsu-Jae King Liu, the Dean of the College of Engineering. It is my pleasure to welcome you to this year’s Minner Distinguished Lecture in Engineering Ethics, featuring Dr. Barbara Simons. She’s the board chair of Verified Voting, which is a nonprofit organization. It’s a nationwide organization that advocates for best practices in voting.

Before I go further, I’d like to just acknowledge today’s sponsors, the Society of Women Engineers and The Women in Computer Science and Engineering or WICSE. These two student organizations are cohosting today’s event. So thank you very much for doing that. So, recognition in the back there. It’s a full house today, which is wonderful. I’d also like to welcome members of the Dean’s Society, who might be here in person or online, on the web. So thank you for joining us today.

I hope everybody here knows that in the College of Engineering, we strive not only to uphold our university’s strong traditions in excellent and access, in research and education, but we also strive to be the exemplar for transforming our students to become socially engaged and inclusive leaders, as this is to help ensure that we are innovating new technologies that address the grand challenges of a society in a fair and equitable manner, so that every citizen of our global society can thrive in the safe, secure and sustainable world.

Now, key to achieving this transformation, our programs such as the Minner Distinguished Lecture. So, let me tell you a little bit about program. In the year 2011, Warren and Marjorie Minner establish this Minner endowment which supports the college in our mission to instill in our students a strong sense of social responsibility, ethics and leadership. We do this by sparking conversations and critical thinking and societal issues facing engineering students, researchers and practitioners and society at large.

Today’s speaker, Dr. Simons, who I’m proud to note is a Berkeley engineering alumna, exemplifies the value of social responsibility in engineering and computer science. In fact, in recognition of her outstanding technical contributions and her leadership in promoting inclusion in the field of computer science, we honored Dr. Simons here earlier this year with the 2019 WITI@UC Anthea Award for a lifetime achievement. So at this point, I just wanted to mention that WITI stands for Women in Technology Innovation at the University of California.

We sponsor annual awards to recognize outstanding leaders who promote diversity in tech. And so, we actually are soliciting, or we’re opening. We actually are welcoming nomination right now for award winners for next year. So if you know of any worthy recipients, please visit the website and submit your nominations. All right, so coming back to Dr. Simons. When she was a graduate student here at Berkeley, she cofounded the Women in Computer Science and Engineering group, or WICSE. And this is a networking advocacy and outreach organization, which has a stellar tradition of attracting and supporting female graduate students. So I’m really delighted that we have here today, the copresident of WICSE, Cecilia Zhang. She’s going to help introduce and welcome Barbara to the stage. Cecilia is fifth-year Ph.D. student, working in Professor Ren Ng’s group on computational photography and machine learning. And she’s been a member of WICSE since the year 2015.

Cecilia Zhang: Thank you, Dean Liyu. So I’m honored to be here today representing WICSE, Women in Computer Science and Engineering and also across College of Engineering. On behalf of our students, faculty and staff, it’s such a pleasure to welcome Barbara Simons to campus. I’d like to share a highlight of her career with you.

In the 1970s, Barbara started taking computer science courses at the State University of New York in Stony Brook. The field was still in its infancy, and it wasn’t long before she advanced to graduate-level coursework. She was admitted to Berkeley’s graduate program with Turing Laureate Dick Karp who’s also sitting in the audience as her advisor. In 1981, Barbara earned her PhD in electrical engineering and computer science with her dissertation on deterministic scheduling theory.

After graduating from Berkeley, Barbara began a 17-year career at IBM, starting off in the research division before transitioning to a policy as a senior technology advisor for IBM Global Services. After retiring from IBM in 1998, Simons served for two years as the president of the Association for Computing Machinery, the largest computer society in the world. During her tenure, she was invited to participate in a study of internet voting that have been requested by then president Bill Clinton. Barbara’s interest in voting practices was born.

As a member of the national workshop on internet voting, she helped conduct one of the first studies of internet voting in 2001. In addition to serving on the board of Verified Voting, she has served on the board of advisors of the US Election Assistance Commission and co-authored the report that led to the cancellation of Department of Defense’s internet voting project known as SERVE in 2004 because of security concerns. She co-authored the July 2015 report of the U.S. Vote Foundation entitled, “The Future of Voting” and to end verifiable internet voting. In 2012, she co-authored with fellow computer scientist, Douglas Jones, a book about electronic voting. Electronic voting machines called, “Broken Ballads: Will Your Vote Count?” Suffice it to say, Barbara has lots to say for today’s Minner Lecture topic. Can we recover from an attack on our elections? Please join me in welcoming Barbara Simons.

Barbara Simons: Thank you so much. This is an amazing audience. Thank you all for showing up and being concerned about this really critical issue. So, I’m gonna start, kind of, in a negative way. But we’re gonna end up somewhere positive. So that’s an incentive to stay on.

You’ve all, I’m sure, have seen what Robert Mueller said, when he testified before Congress, where he said there were multiple systemic efforts to interfere in our election. And this was actually, I think, the most emotional part of his testimony, is when he really seemed to care, talking about the threats to our elections. He’s not the only person who’s made these comments.

James Madison, the former Secretary of Defense, said that Putin tried to much around in our election this last month — this was in reference to the midterms — and we are seeing continued efforts around those lines. And Christopher Wray, the FBI director said Russia attempted to interfere with the last election and continues to engage in malign influence operations to this day.

So, we do have something to be concerned about. The intelligence communities also have been pretty consistent along these lines. The Senate intelligence committee, by the way I think is the most functional of the Senate committees. They really do do bipartisan work. And I think it’s because they’ve been briefed, and both parties understand how serious the threat is. The Department of Homeland Security says that the Russian searches were done alphabetically. The probes included all 50 states. Now we were told initially, that there was a small number and then with the 29 and 39. They’re saying all 50 states were probed, and it consisted of research on general election-related webpages, voter ID information, election system software and collection service companies.

Now, probing doesn’t mean that anything was actually hacked. But as the computer scientists in this audience, I’m sure appreciate if you probe a system, who knows what you’ve done to it. You might of implanted malware. You might’ve implanted a backdoor. You might be planning to come back later and do something. So, we don’t know what these probes meant.

The report also said, and people have been consistently saying, that there’s no evidence that any votes were changed. Note, they don’t say no votes were changed. They just say, there’s “no evidence” that no votes were changed. And the reason that the say that, is that we don’t know. Paperless voting machines can’t be checked. And even in states that have paper, they frequently don’t do an adequate of the paper because the paper is tabulated by computers, and those computers might have been hacked.

We also know that many countries are capable of attacks. Russia, as we’ve already talked about. China, as you also know, they have a state-sponsored hacking group that has attacked U.S. utility companies among others. And then, of course, there’s the famous North Korea attack on Sony because they didn’t like the film, The Interview. And Iran has also been engaged in a lot of malicious hyperactivity recently. Not surprisingly, given the tension between the U.S. and Iran.

Now, these are some of the countries we know have the capability of attacking, of committing cyber attacks. There are many others. It really doesn’t take a lot in the way of resources to do this kind of thing. So there are a number of myths about elections that we’ve been hearing, saying that they are secure. And I want to shoot down two of those key myths.

The first is, that because voting machines are never connected to the Internet, they can’t be hacked. They’re secure. And again, as the computer scientists in this audience know, that’s the definition of security. For one thing, in the case of voting machines, yet they program these machines to tell them who’s running for each election, who is running, their places on the ballot, what their prepositions are and so on. This programming is done by separate computers. And these computers typically are attached to the internet some time, or might be done by two- or three-person companies. And who knows what kind of security they use. So, if any of these machines that are used to program the voting machines has malware inserted into it, that malware could then be transported to the voting machines when the information from these computers is inserted into the voting machines. So the fact that the voting machines themselves may not be attached to the internet does not prevent them from being hacked. And, of course, some of you may remember the Stuxnet virus, which brought down the Iranian’s centrifuges. Those centrifuges also were not connected to the internet, but they were brought down nonetheless.

The second myth is that, there are so many types of voting systems that it’s impossible to rig an election. And anybody who stops to think a minute about the electoral college, realizes that you don’t have to rig the whole country. Because we know that, at least in terms of presidential races, how certain states are likely to go. What matters of course, is the swing states. So, if you really want to change election results, you’re gonna focus on the swing states. And even there, you may not have to focus on the entire state. You may focus simply on a few swing districts. And that, in a close race might be sufficient to change the outcome of a race. So, the fact that there are a great variety of systems out there, really doesn’t protect us.

So the question is, how did we get here? And since I think this is supposed to be a lecture on ethics, I’m not sure if it’s an ethical thing or not, but basically this is a story of the inappropriate use of technology. And it’s really come back to haunt us. So computer scientists have been involved from the very early days.

In the 1990s, Peter Neumann, some of you may know. He’s at SRI, just down the road, and Rebecca Mercury, warned of paperless systems. But, of course, their warnings were ignored. And, as was mentioned in the intro, I was on a panel in 2000 that was put together by the National Science Foundation at President Clinton’s request. That panel contained election officials, social scientists and computer scientists. And when I started on that panel, I thought internet voting sounded like a really cool idea. I really liked the idea, you know, everybody says, “Oh, you can vote from home in your pajamas.” Well, you know, that sounded appealing to me.

Fortunately, there were security experts on that panel, in particular Auviel Rubin, some of you may know who very quickly disabused all of us of that notion that internet voting might be a cool thing to do. And as a result, the panel recommended against internet voting for the foreseeable future. In 2002, the Help America Vote Act, was passed with minimal input from technologists. And I’ll talk more about that in a minute.

And then, in 2003, California started purchasing paper voting machines. And David Dill at Stanford was appalled to hear about this, and he started an online petition. A number of us signed it. In fact, he got a large number of signatures, which of course didn’t matter whatsoever. And there was this dramatic hearing in Santa Clara County of the Santa Clara County Board of Supervisors that I attended, NPR was there. I mean, this was really in the early days. NPR was there. And there was a row, the front of the hearing room, there were about six or seven computer scientists. A number of them prominent computer scientists. A few students and a few people like me who were freshly minted.

Anyway, a number of computer scientists were there. And the vendors were given a large amount of time to push their wares, and each of the computer scientists was given 90 seconds. David and I, before hand, had gone down to the Santa Clara County Board of Supervisors and talked to one of the members of the Board of Supervisors and convinced him that this was a bad idea. But we made the mistake of not talking to all five of them. And so we lost that vote, two to three.

Now, an interesting thing happened. Because of our pressure, they did say if, at some point, paper is required by the state, that the vendor would have to retrofit the system to provide paper. Subsequently, paper was acquired by the state. And so, the vendor had to pay for that. So they got something from our efforts, but they didn’t really listen to us. And then as a result of this, David started Verified Voting. I was on the first board. The board consisted of David, his wife and me. We had no money, no staff. We didn’t know what we were gonna do. We just decided we had to do something. So, that was how Verified Voting was born.

In 2004, the Hopkins-Rice report came out on Diebold voting machines. And this is the first time that somebody had gotten access to the software. When an independent computer security expert had gotten access to the software to examine it. And the results were really shocking. I mean, these systems were … it was very badly written. And an example of what they did: They encrypted the data, the votes in the machines. They encrypted it using a single encryption key, which was in the source code in plain text. So anybody who got ahold of one of these machines could figure out what the goals were in the machines and they could change them.

And these machines, Diebold machines were widely used starting in Georgia in 2002 and up to and including 2018 in Georgia and in a number of other states. Georgia especially, was particularly egregious. They finally decided to get rid of them. But it’s been a long time. And we don’t know if any of these systems have been hacked. We do know how to these machines, even remotely. That’s something that Alex Halderman has shown. It’s on their website. If you want to see a five-minute New York Times video showing how to hack a Diebold machine remotely, just go to the Verified Voting website.

So, computers were introduced into our elections without an analysis of the risks. And this was triggered first by Florida 2000, with the hanging chads. But the law that did it, the Help America Vote Act, wasn’t passed until 2002, and it was pushed by the midterm problems that happened in Florida in 2002. So, Florida really contributed significantly to where we are today.

So, the Help America Vote Act allocated almost $4 billion for new voting machines. Now, the vendors of course, came out with lots of assurances. The machines are secure. And something, which is very appealing, of course, to election officials, is you just push a button at the end election and you get the results, it’s me you can go home at a decent hour. Believe me, running an election is an exhausting job.

And so, election officials really loved that idea. I mean, I would too if I were an election official. They’re federally certified. I’ll talk about certification in a minute. It was pretty meaningless in this case. And there was a deadline for spending the money. So that created a gold rush mentality. Everybody wanted the latest and the greatest. They wanted the new shiny voting machine so they could show the people who are voting in their area that they’ve done the right thing. They are modern. They’ve got the best stuff.

So, the earliest of computers in voting, initially there were a lot of these paperless machines, like this Diebold machine I was just about. They’re called Direct Recording Electronic or DRE. And what DRE means is that the result is stored directly in the electronics. So basically, the votes are stored in the memory of the machine. Nowhere else. At least with the paperless ones. So, you probably have seen pictures of people touching the touch screen to vote for candidate A. And it votes for candidate, and B lights up. People have complained about that a lot. If I were gonna rig an election, I wouldn’t do that because that’s so obvious. There are much better ways to rig elections. But in this case, with it almost certainly means, if the screens were out of calibration, which happens about these machines. And that also means that there’s an element of non-determinism here. Because if I touch A and B lights up, I don’t know which one actually got my vote. Could’ve been A. Coulda been B. Who knows.

As I say, they were also badly engineered. Anytime any independent experts got to the software, they found major problems. And these machines, when they would fail, or if there were a insufficient number of them, would generate long lines at the polls, because you have to vote on the machine in the case of these paperless DREs. So, there was a big push for what we call paper trails. And I’ll have to say, as somebody who was involved with a lot of this stuff, there were many stages where we were incredibly naive, and this was one of them.

So, in response to our call for paper trails, the vendors retrofitted these machines with something called voter-verified paper audit trails. These were supposed to hardcopy store the results of your vote, as a backup in case there was a problem with memory or if you wanted to do a recount. What they came up with, was continuous thermal rolls. These are like, the gas station receipts that you get. They easily fade, and they’re hard to count. They were often small fonts, which made them hard to read. And they were typically other transparent glass, transparent plastic which also made them hard to read.

MIT did a study on the systems to see if people actually checked them. And what they found out is that most people didn’t bother to validate their votes on these voter-verified paper audit trails. And in many cases, they didn’t understand why they should, because they didn’t know what they were there for. There wasn’t a motivation to validate them, so they didn’t.

Also, in the early days, there were some voter-marked paper ballot systems, and that would be a case where the voter would manually mark the ballot. In most cases were counted by scanners, which again, as you all know, are computers. But I think a lot of people who aren’t technical don’t appreciate the fact that scanners are computers. And therefore are vulnerable to everything that a computer is vulnerable to. And the scanners could be at the polling place or they could be centrally located. So they’re two different kinds of systems.

Some of these early scanners had calibration problems also. If they were too sensitive, they might think that a stray mark mark is an extra vote, which would be an over-vote and would disqualify the vote. If they weren’t sensitive enough, they might not pick up the vote. So, they had problems. If there were long lines, if the polling place scanner was down, voters to deposit their ballots in a ballot box at the polling place in the ballots to be scanned later, this was a big plus compared to these DREs for sure. Unfortunately, in some cases, the volunteers didn’t even realize that this could be done. And the long lines continued, even though they were ballot boxes where people could’ve deposited their ballots.

So, testing and certification. There are voluntary voter guidelines, voting system guidelines at the federal level. They’re voluntary, although a lot of states have adopted them as requirements in their state. California, for one, at least did initially. That’s been changed slightly. When they were originally created, they had minimal security and accessibility testing. Accessibility is the buzzword for something that’s easy to use by people with disabilities. And computer security experts are not involved. I mean, these things were sort of like standards you might come up with for a toaster or some other product. If you drop it, it should still work. It should be able to withstand heat and cold. Things like that. But not the security that was so critical to these systems.

So the first significant independent testing was done here in California. Secretary of State Bowen did it, and she ran on this. She did something called a top-to-bottom review. And by the way, a number of UC faculty and students were involved with this study. And the study tested all aspects of the three systems that were being used in California, the security and accessibility among others. And this is the first time I know of when there was a meaningful test of accessibility of any of these machines. They were being sold on the basis that they were accessible. But the accessibility testing that was done show that they actually pretty poor accessibility in many, many ways. And that everything else was bad too. The security was bad. And these machines were just badly engineered in every respect.

And then in Ohio, the Everest study followed the California study. And it confirmed all the problems discovered in California and found additional ones. As further studies have been done, they tend to support the fact that these machines were just badly engineered. So, here’s what we should not do. Should not internet voting, including cell phone voting and blockchain. What do these things have in common?

They’ve all been hacked. This doesn’t include, for example, the attack on the DMC. It doesn’t include John Podesta’s emails. It doesn’t include a lot of things that we know about that happened recently or at least in the 2016 race. So I think it’s kind of obvious when you think about, these things, that local election officials are really not well-equipped to stand up to a massive attack from a foreign country, from political operatives, or even a rogue hacker. They are underfunded, under resourced. They don’t have good access to computer security expertise. It’s a hard job for a well-funded major corporation to protect themselves. And to expect local election officials to be able to do that is just … it won’t work.

Internet voting. So we define internet voting as the return of a voted ballot over the internet. And I want to distinguish this from placing a blank ballot on the internet. So that’s something that is used, as I’ll mention later. We don’t consider that internet voting. There are issues with that, too. But most of us feel that the risks are worth it to avoid actual internet voting. So, you can vote over the internet via the web or as an email attachment. As they say here, email voting can be even more dangerous than web-based voting. There is some confusion about whether or not email voting is internet voting. Again, sorta like scanners are computers.

Email voting of course, is internet voting. You can use personal computers, smart phones, smart tablets and so on. There is research on using crypto to do internet voting. What’s interesting is, that the major. The people who have been doing this research, most of them say, we are nowhere near ready for that. Ron Rivest, some of you know, has basically come out very strongly against internet voting, even though he understands how to use crypto to do it. Benedita, a former student of Ron’s who actually wrote a system for internet voting, also said this is not the time to do it and is working on a different system that he hopes will make voting safer and more secure, and facilitate risk-limiting audits, which I’ll talk about momentarily.

So, vulnerabilities of internet voting. Authentication. It’s very hard to authenticate the voter. You have malware on the voter’s devices that can change the vote without the voter’s knowledge. And again, what you see on the screen may not be what is sent out over the internet — yet another thing that I think a lot of people don’t understand. So just because your vote is presented properly on screen, does not mean that’s what’s going out. It doesn’t mean that’s what stored in memory.

Denial of service attacks to prevent ballots from reaching the election official. Penetration attacks on the vote server can change votes. The votes, of course, cannot be audited. And you can have vote buying and selling or voter coercion now. I should say, vote buying and selling and coercion are issues of any kind of remote voting. Not just internet voting. There are no regulations for internet voting. No independent standards. No independent testing. No government oversight, no legal accountability. No ability to conduct a recount. The National Institute Standards and Technology was asked to develop standards. They basically said, “We don’t know how to do it.” They throw up their hands. They said malware on voter’s personal computers pose a serious threat that could compromise the secrecy or integrity of voter’s ballots.

Nonetheless, internet voting is used in about 30 states in the United States. For military and overseas voters, fortunately that’s a small group. Now, the MOVE Act was passed in 2009 as a way of getting around this push. And it provides online blank ballots 45 days in advance, at least, so that voters can download the ballots, print them out, mark them and then mail them back in standard mail. And if you are a military voter overseas, you can use expedited mail. So almost all military voters, if the MOVE act is followed, you can get your ballot back in plenty of time.

And, I like to think of internet voting as a solution in search of a problem. A lot of people think that internet voting will increase work participation. There is no evidence to that effect. There was a major British Colombia study, that impact found that there was no appreciable increase in either group. What they found was the same people who would’ve voted already, voted over the internet.

Blockchain voting. So, Blockchain voting is sort of the latest buzzword. Blockchain is a distributed data structure. That would be multi-owner or single owner chains. The owners have to be in agreement. You could have collusion among owners. That’s one of the risks. You could have outside attackers to penetrate the server. There’s no central authority to police the activities. With voting, a blockchain. You almost certainly are gonna have a single blockchain, because it’s going to be owned by the vendor or the local election official. And with Blockchain voting, all of the other internet vulnerabilities are still present. Blockchain voting, whatever you been hearing about it does not solve the unit voting problem.

As a National Academies of Science said in a 2008 study, it does not solve security problem. Nonetheless, Voatz is a company which is pushing Blockchain voting. There’s no federal or state certification. They claim they don’t have to be certified because they’re not really a voting system. But, in fact, they’re collecting votes. There’s no disclosed source. No open testing by third parties. No testing in mock elections. They claim to have done security audits, but nothing was made public. And they are nonetheless being used in West Virginia for overseas voters in 2018 primaries and midterms. The city and County of Denver just basically brought them on to run their military and overseas voting. They’re basically funded by Tusk Philanthropies, which has some links, there are people involved with Tusk who have some pretty strong links to the Democratic party. And they may be used in the Alaskan Democratic caucuses, so some of us are fighting the use of them in the caucuses.

We have a solution, so that’s the good news. We have a solution. You need voter-marked paper ballots. You need a strong chain of custody, and you need to physically sound, manually post-election ballot audits called risk-limiting audits.

So we need well-designed paper, obviously. The text has to be easily readable by the voter. And we have examples where badly designed ballots may have changed the outcome of an election. Perhaps most notoriously was the Butterfly ballot. But in Broward County, Florida, in 2018, there was an unusually large undercount for the Senate race. And the ballot design there was very poor. I actually have slides I can show later. I’m not gonna include it right now. But if someone asks I can show you pictures of the Broward County ballot, which was just badly designed and may have changed the outcome of that Senate race.

So, voter-marked paper ballots. There’s been a move towards them. That’s the good news. In fact, Louisiana will be the only state after all these years, that will be totally paperless in 2020. And I have to say, that’s a big change from when we first started doing this work. Nonetheless, there is some paperless jurisdiction still. I’ve listed them here: Kansas, Kentucky, New Jersey, Mississippi, Tennessee and Texas. Pennsylvania’s in parentheses because through 2016 and 2018, 83% of the population of Pennsylvania voted on paperless machines. Now Pennsylvania is finally going to replace these machines. The governor issued a $90 million bond issue to replace them. And the question is, will all of the districts in Pennsylvania actually do this for the 2020 election? And that’s why there’s a question mark there.

So hand-marked paper ballots are widely used and inexpensive. There are ballot-marking devices which are relatively new. I mean, there was some old ones. But, there’s a new batch of them come on the market. They use a computer, like a touchscreen, to produce a voter-marked paper ballot. They should. I say, they should have good accessibility features. I don’t know if they do or not, because I don’t know of any testing that’s been done. Well, except Philip, you say was negative for one of them.

Philip Stark: Pennsylvania and Texas both failed.

Barbara Simons: Yeah. Again, it depends on the ballot, on the actual machine. They are both types of paper ballots and scanners, and the ballots produced by the ballot marking devices are counted by scanners. So there are potential problems with both systems. Hand-marked paper ballots, the voter might inadvertently undervote. And, for example, in California we have these horrendous ballots which have races on both sides. And I know of at least one person who forgot to turn his ballot over, and therefore missed a whole bunch of races. Also, if you vote by mail, there’s no corrected feedback from scanner if it’s scanned locally. For example, if you’ve done overvoting. Again, this is a vote-by-mail issue, I think, more than the paper ballot issue. But, that is an issue.

But ballot-marking devices, they print the voter selection in human readable text. But some of them only print out with the voter voted on. This says nothing about the races where the voter didn’t cast a vote. And this seems like a particularly bad way of doing things because the voter might have forgotten to vote for something or might have voted for something and it’s not recorded and doesn’t notice that. Again, this is a bad design.

Some of these systems also have security issues.They typically have a barcode for reading by scanners. And that is an area of contention. But in all cases, they must have human-readable text that’s used for audits and recounts. And again, it’s critical the voter verify all the ballot selections. And again, that’s a question. Will they verify them all? And there’s ongoing research in that area. So, strong chain of custody. There’s just a few example things you have to worry about. You can have inspectors from both major parties. Hard to forge seals for ballot boxes. It is important to check for the tampered. But you have to check the seals afterwards. The custody logs, surveillance videos. And you have to the check that everything matches at every step of the way at the polling place, in transit, while in storage, while tabulated and while audited and recounted.

Post-election ballot audits. So, this is really the crux of what we want to talk about. Preliminary results must be reported before the audit is done. Now again, remember, the reason we need to do audits is that we can’t trust the computers to tabulate the votes. So we need to audit these computers, even if they’re not called computers. If they’re called scanners, or whatever they’re called. The audit must be completed before the certification results because the audit could find that there was a problem. It must be done manually and the ballot selection must be random. And basically, the gold standard of audits is what’s called risk-limiting audits.

As you can see, recommended by Presidential Commission, National Academies, the Senate Intelligence Committee, and was developed by UC Berkeley statistics Professor Philip Stark, who’s sitting right there. I have to say, Philip’s contribution was huge. Before Phil got involved with this, we didn’t know how to do the audits properly. So, California in 1965 did a 1% manual recount. But the law didn’t say what you were supposed to look for. It didn’t say what happens if you find a problem. It didn’t find a problem. There was no notion of escalating the count. I mean, it was still for its time, ahead of its time. But, it didn’t really solve the problem.

In more recent times, in the 2000s, as we started thinking about this problem more seriously, people came up with different schemes such as a tiered audit — 1%, 3%, 5% — depending on how close the results are. So, if the results are very close, you have to look at more, which makes sense intuitively. But it still didn’t solve the problem. So, basically, it wasn’t until Philip looked at this and came up with a mathematical way of doing it, that we know how to do it. Basically, risk-limiting audits have a guaranteed large prespecified chance of correcting the wrong reported outcome.

So, what is an outcome? What is a wrong outcome? It’s wrong if what the computer says differs from what a manual count would’ve said. And basically, the largest chance that a wrong outcome will not be corrected by the audit, is the risk limit of that audit. So, if the risk limit is 10%, then there’s a 90% chance. This is a lower bound. It’s at least 90% chance that the audit will lead to a full recount that corrects it.

So basically, with risk-limiting audits, there are two factors, and that is: How close is the race and what is the risk limit? And that determines the initial sample size that you start with. The random samples continue. The random sampling continues until sufficient evidence exists to confirm the computer declared outcome or results in a complete manual recount, which typically will happen if the outcome was wrong. I mean, that’s when you do want to do a manual recount. So there’s various types of risk limiting audits. The ballot level comparison audit is the best, is the most efficient kind. That’s where you basically randomly select a ballot and compare it with its cast vote record. That’s representation in the computer. And you check to see if they match.

There also comparison audits at the ballot level, which are less efficient. But you may need to do that at the batch level. But you may need to do that if you can’t do the ballot matching. And the reason you may not be able to do the ballot matching is that some of these computers are designed to make it difficult to do the ballot matching. So get to do something like a batch level. There’s also a ballot-polling audit, which doesn’t really do the comparison at all. Instead it pulls the ballot, sort of like an exit poll. But you’re pulling the ballots instead of people. And the ballots don’t like, unlike people. And basically, as I say, these are developed. These different systems are developed because the voting machines just weren’t designed for that.

There is research ongoing to develop voting systems that will facilitate risk limiting audits. That can’t come soon enough. So they have been a number of pilots, of risk-limiting audits in the United States. The state of Colorado passed a law, requiring risk-limiting audits for the state. And they did in 2018 for the full state. Rhode Island passed a similar law. They’ll be influencing there’s in 2020. There have been highly risk-limiting audits in a variety of states that I’ve listed here. There are a number of groups.

Many of us are working on it. Philip has been very hands-on in this effort. And then Verified Voting, again the organization that I’m board chair of, and Philip is on the board, we have something called an audit roadshow. We’re doing outreach to election officials. Basically, in order to do these pilot audits, unless we can get someone on high to mandate it, we need to get the cooperation of election officials.

Unfortunately, when these pilots audits are done, the election officials tend to like them, because they can then say, “We can prove to our voters that the results are correct.” So once you get them to do it, they like it. The hard part is getting them to do it. And that’s where a lot of work has to be done. There’s also a lot of work that needs to be done to help election official duties audits, because they involve statistics. And most election officials are not deeply immersed in statistics. I think that’s a fair statement.

So, here’s how we can validate 2020: We focus on the swing states. We can’t do the whole country to begin with, because parts of the country are still voting on paperless machines. I mean, we can audit them at all. But in addition, we don’t need to audit everybody because with the electoral college, you kind of know how certain states are gonna go. What really matters are the swing states. So we need to focus on the swing states. And I’ve listed some of three key ones, I think, Wisconsin, Michigan and Pennsylvania. Fortunately, they all have or will have paper ballots. Although, as I say, Pennsylvania hopefully will be completely paper by 2020.

We, ideally, would like to conduct statewide risk-limiting audits. If we don’t get the top election official in the state to require it … it’s too late to get laws passed, by the way. It’s really too late. But if we can get the top election official to require it, that’s great. But if we can’t, ideally, we can still do what we call a pilot audit. If we can do a pilot statewide audit, that’s not officially sanctioned. But we’re working with election officials to do this. We will be able to show would result was, if it was correct or not. Because we’ve done the work, even if it’s not called an official state audit. So, that’s the only way we can think of to get around the absence of laws, aside from maybe having the secretary of state.

In the case of Michigan, I think we might get the secretary of state to actually mandate this, which would be fantastic. But in most states, that’s not the case. And if we can’t do statewide, let’s at least what we can do. Let’s try to focus on swing districts. Let’s try to focus. Do it wherever we can in critical areas.

This has two benefits: First of all, if there is an attack or just a software bug. I mean, because these things are computers, there could be bugs. If something like that happens, there’s a good chance we’ll find it. And also, it can discourage someone from doing this in the first place, from attacking in the first place, if they know that there’s a good chance that the attack will be discovered. The third reason is, it could prevent the loser from claiming the election was rigged if you can show that the results were perfect. So, there are lots of good reasons for focusing on risk limiting audits. That’s what I’m devoting my efforts to. That’s what Philip is the one is efforts to.

We need the cooperation of local election officials and we need a national campaign, because this is a really huge undertaking. It’s late and there’s a lot of outreach that needs to be done. There’s a lot of training that needs to be done. We need to get volunteers and staff in these places to help the local election officials do these audits. They have to do them. We can’t do them, I mean, legally. This is something that’s done by election officials, not by volunteers. But we can help them. And that’s what we need. And if we can do that, I think there’s a good chance that we can avoid the hacking of the 2020 election. But it’s a big if. So that’s it.