Berkeley Talks: Nate Cohn on polling and the 2024 election

[Music: “Silver Lanyard” by Blue Dot Sessions]

Intro: This is Berkeley Talks, a Berkeley News podcast from the Office of Communications and Public Affairs that features lectures and conversations at UC Berkeley. You can follow Berkeley Talks wherever you listen to your podcasts. New episodes come out every other Friday. Also, we have another podcast, Berkeley Voices, that shares stories of people at UC Berkeley and the work that they do on and off campus.

[Music fades out]

David Broockman: Thank you so much for coming. So I know that we’re a little bit early by Berkeley standards, but where the room is full enough already that I feel like we might as well get going and leave more time for our distinguished awardee to deliver his talk.

My name is David Broockman. I’m an associate professor of political science here in the Department of Political Science at UC Berkeley. I’m also the current faculty director of the Citrin Center for Public Opinion Research. The Citrin Center was started about six years ago by the friends and former students and colleagues of Professor Jack Citrin, who we have here today, to focus on the study of public opinion largely in the United States. We support student research, we support events like this, public conversation, and my favorite part of my role as the faculty director of the Citrin Center is today’s event and being able to recognize some just fantastic contributions to the study of public opinion.

So when we were brainstorming this year, who would deserve the Citrin award this year, and it’s an impressive list of people who have earned the award in the past of the leading lights of public opinion research and polling, someone mentioned the name Nate Cohn. And as soon as I heard that name I said, “Ah, that is who deserves this award.” I will admit that, realizing that Nate is my age, I did feel a little bit uncomfortable. I was like, “Oh, wow, someone who’s this accomplished in my age, I’ve been slacking.” But then I was relieved to learn he’s actually three months older than me, so I still have some time to catch up.

Nate really does need no introduction. You have, no doubt, I’m sure read his articles. There’s very few, I think, actually really, probably, any journalist other than Nate who, whenever I hear there’s a new Nate Cohn article, I immediately drop everything and I know I have to read it.

He manages to continue to have some of the most valuable and I think an important commentary on American politics. When I think of data journalism, when I think of polling, he’s the first name that comes to mind. And I’m sure you’ve also seen him not only on the New York Times, but he’s also, as I’m sure he’ll talk about today, helps run what has been rated the most accurate poll, the New York Times Siena College Upshot poll. And that is no easy feat. We are in a time when polling is increasingly difficult but it’s also increasingly important, as there’s increasing division around what even Americans want from government, where they stand on major issues.

I think having accurate public opinion data has never been more important. It’s something that’s always been really important at Berkeley, and there’s no one, I think, pushing that forward in a more important way than our awardee.

So with that, I will turn it over to Nate Cohn. And Nate will be speaking for about 40 minutes, then we’ll do Q&A, and then we’ll give him the award officially. And then please stick around because we will then have a reception. So if you don’t get your question answered during the Q&A, you can try to buttonhole Nate, get a photo, get him to sign your hat, whatever it’s you want. Great. OK, with that, I’ll turn it over to Nate. Thank you so much.

Nate Cohn: Hi, everyone. I’m Nate. As David just mentioned, I’m the chief political analyst at the New York Times. I’m responsible for our survey methodology, our election night projections, and I also write with some regularity including for a newsletter called The Tilt, which you’re welcome to subscribe to. It’s always a pleasure to get to speak with people in the real world, but it’s a real honor to be here today to receive this award. I want to thank Jack Citrin and the Citrin Center, David, for putting everything together for this. And I also want to thank the New York Times and Siena College.

As David just alluded to, I think I’m reasonably confident that I am here by the margin of the success of the Times/Siena Poll. I do think it set the standard for rigor and transparency in public polling, and it’s really only possible because of the support of the New York Times. And there have been a lot of opportunities for the New York Times to stop supporting public polling, whether it’s due to challenges in the industry, whether it’s due to troubling economic times, or due to questions about its continued accuracy.

And I’m certain that it wouldn’t be possible at another organization that didn’t have the resources or the commitment to … I don’t know if I was responsible for this or not, but if so, I’m sorry … that didn’t have the resources or the commitment to this kind of work. And I also want to thank Siena College for being an incredible partner in being willing to go along with crazy ideas, including things that are not fun for them. We take steps that make their life more difficult, we call more people who are less likely to respond to polls. It’s a consequence their interviewers have to call for two hours before they reach even a single respondent, and they put up with that. I also … well, what do we think?

All right. I’ll skip ahead to a little anecdote here. Beyond the New York Times and Siena College, I do think there’s a third reason why the Times/Siena Poll has been successful, at least with respect to the competition. This is a little bit of a hot take, and it might seem a little insulting, but I do think we care more about getting accurate political survey results than much of the competition.

I don’t mean to say that much of the competition doesn’t care about getting accurate political survey results, but it’s not their highest priority, there are other things that they value more. I was at Cornell earlier this year at a conference on a NSF funded study of new innovative methods in polling, and panelists after panelists complained that they were there to talk about polling elections, every single one of them, it was incredible. They were just like, “You know what would really be great, is if we could get all this money to poll anything else.”

And I was really astonished, but I think they really believe it. I think that it’s not just complaining that sometimes they look wrong, or that they get attacked, or that the news media doesn’t want them to do, I don’t think that they really consider doing a good job in political election polling to be their highest calling, to use one of their words for it. And I think that the clearest proof of that isn’t simply their words, but also their actions.

There are small things, like the order of questions. Is the first question in your survey about Biden vs. Trump, or are you going to ask about a bunch of other questions first that you might think is more important but has the potential to bias your result about Biden or Trump? Many other surveys put other questions first. There are larger methodological choices, like, should you design a poll that represents the whole American population, which includes many non-voters and even people who are ineligible to vote, or do you focus on likely voters?

Many pollsters focus on adults. That may not be the way to generate the best and most accurate results when it comes to elections. And I think the very best proof of all, though, is that many public pollsters don’t even pull the horse race at all anymore. And I know that you … “Oh, we are really way off in our order here.” I know that you won’t be able to read all of these words, but both Gallup and the Pew Research Center, two of the most prominent organizations in public opinion research, no longer release presidential results pitting the two major party candidates against each other.

That’s, to me, pretty good proof that they really don’t value this stuff as much as you might think. Monmouth University, this may be you can read from the back, “The pollster who wants to quit horse-race polling,” and actually did. And I think the use of the word horse-race says it all, it’s a sporting event, it’s ephemeral, it’s devoid of any meaning, it only matters to horse junkies, I guess, horse race fans, as Eitan Hersh might put it, horse hobbyists. I think that it’s pretty clearly a diminishing way to describe the effort, and I don’t agree with it. And before we talk about the state of polls, I think it’s worth talking about the importance of accurate and representative election polling, not just to feed the appetites of political junkies, but because of the effect it ultimately has on the functioning of our democracy.

And by failing to take it seriously, we have a political process that can be less representative and less responsive to the electorate, with potentially serious consequences for the way our country is going right now. What I just said I don’t think is necessarily the view of most pollsters, as I just mentioned, the Gallup and Pew position is rooted in a deeply held view of the role of public opinion in our democracy. And the view goes all the way back to the dawn of modern social science, it goes back to 19th century progressives, and it holds that the will of the people should be this guiding light that our elected officials should follow. And the consequence of that is that there’s a quasi-religious commitment to trying to learn what the public wants in hope that politicians will do exactly that, to add the voice of the people to political conversations. And this is their calling, as they would put it.

And I won’t stay philosophical for too long before I get to the nitty-gritty of polling, but it’s worth contrasting this with other ways of looking at our democracy. There’s a republican vision, a more elitist vision, that’s lowercase R, republican, by the way, that holds the people like the representatives, and the representatives should do what’s in the public interest but not necessarily what is in line with public opinion. And if you take that view, then you’re like George W. Bush, and you say, “Oh, I don’t follow the polls. I just do what’s right.” And regardless of whether you follow the progressive or republican view, there’s not much room for accurate political polling. At best, it’s a distraction from measuring the true thing that matters, which is what the American people’s attitudes on the issues. And at best, I guess, it pays the bills, so that they can end up getting grants from the NSF for that conference I alluded to at Cornell.

But I think that while that may be all true in principle, that our democracy works in a pretty different way, our politicians don’t just naively follow what they believe is best for the country, and they don’t naively follow the public will either. Instead, there’s this other big thing that affects what they do, and that’s electoral competition.

In order to exercise power, they have to win it, in our democracy, they do it through winning elections. And the decision-making of all political actors in our country is powerfully shaped by who will win or lose elections, and they decide who to support and what policy to support in no small part based on what they think will help them win or lose an election. And polling has come to play a really important role in that process. I want to note, by the way, it’s not polling’s fault that politicians are calculating. I was reading a book recently about the 1860 election and it turns out that Lincoln was the electability candidate.

They thought that he was the Midwestern moderate who could win Pennsylvania. It sounds very familiar, and that’s even at a time when Civil War was at stake, when slavery was at stake. Of all the times in American history, were going with what was righteous and morally right, that would be it, and yet the Republican Party nominated the relative moderate in hopes of winning the election, and even though there was no polling to inform them. And I think it’s worth pondering the stakes of that kind of decision. Imagine if that was wrong, what if, hypothetically, they could have elected an abolitionist as the president in 1860 and people could have been freed years earlier. It’s an incredibly high stakes decision. And conversely, what if they had believed an abolitionist could have won and they have lost, and slavery would’ve last longer?

These are really important decisions that are being made based on calculations about what can and can’t be done. And today, polling is, I don’t know how to rate it among all the different ways that politicians make choices about what can or can’t be done politically, but it’s definitely on the list, and the stakes today, I think it’s fair to say, are smaller than slavery, but they are still quite real.

And in recent decades, I fear that many of these calculations, these political choices, have been made based on bad polling and bad data. I don’t think it’s a coincidence that we have a crisis of polling at the same time we have a crisis of democracy. I don’t think it’s a coincidence that Trump mobilized a so-called silent majority of voters who felt that they were unrepresented in our political system, and who turned out to be underrepresented in polls by an order of magnitude for decades.

Just think about all of the choices that politicians made from the ’80s onward, that in each one of those decisions, they were doing it, in part, based on data that underrepresented the number of white working class Americans by tens of millions. I think it added up, and I think I’ll start by proving that to you, and I think it offers a nice launching point for where polling is today. Because although it’s tempting to think the problems in polling are recent to Trump, I think it’s probably fair to say that Trump exposed issues in polling that had existed for a very long time before that.

And we could probably start in 1980 or something, but for concision, I’m going to start in 2012. And it’s a little hard to remember today, but the Obama reelection was heralded as marking the arrival of a new coalition of young, diverse, and highly educated voters who were creating this emerging democratic majority.

And it was a democratic majority that was anchored in demographic changes. The country was much more diverse, much more highly educated than it used to be, and there was a new generation of young people, and polling was at the center of the construction of this narrative. The exit polls in that election found that Obama won 39% of white voters, which was the lowest since Walter Mondale in 1984. From the Republican point of view, that proved that they had done all that they could realistically do among white voters and that they faced political peril, instead, they had lost the election because of demographic forces, Latino voters rose to be 10% of the electorate, Romney won just 27% of Hispanic voters. And I think that the consequence of this story was really immense because it suggested that the GOP had no choice but to appeal to diverse, young, and highly educated cohorts.

And I really do think it’s difficult to overstate how much this affected Republicans. These are two stories from Nov. 9, 2012. On the left, this is a New York Times article. I know some of you are far away, but it reads, “The prospects for immigration reform overhaul next year improved with stunning speed after a presidential election, which Latino voters rewarded President Obama while punishing Republicans.” On the right, “Sean Hannity embraces immigration reform. He declared that he had evolved on the issue.” And this is all because of the sense of crisis that Republicans felt due to Obama’s reelection victory, and not just that he won, but the way that they believed he had won it based on surveys.

The catch of course is that this story was basically wrong. On the left, this is what the exit polls that were used to construct the narrative I just described supposed for the makeup of the 2012 electorate, and on the right, is what the census showed eight months later. And as you can see, the exit polls showed that there were fewer white voters, fewer people without a college degree, fewer people over age 45 than it turns out there were in reality. And by an order of magnitude, when we’re talking about, again, I mentioned tens of millions of people, but on that college share of the electorate, that’s a 10 percentage point difference. And if you do some quick math here, 130 million people voted in the 2012 election, it’s 13 million people without college degrees that are somehow being missed by a gold standard survey that was the basis of almost all of the political analysis in the aftermath of the 2012 election.

And the consequences of that are, again, they’re quite striking, because if you go and try and understand the 2012 election with these numbers instead of the numbers we had after the election, you get a totally different picture. It turns out that Obama would’ve won in 2012 even if Republicans had won 40% of the Hispanic vote, which is what they hoped for. It turns out that Obama would’ve won even if white voters had represented the same share of the electorate they had in 2004. He would’ve won even if both turnout and the demographics were all the same as they were in 2004. And as you can probably piece together out of all of this, what that actually means is that the reason Obama won the election is because he actually did pretty well among white people, and much better among white people than John Kerry did.

Now, granted, in order to win an election, you need voters from all demographic groups. I don’t mean to say that young people, or college educated people, or white working class voters, or any given demographic group was not central to his victory, what I mean to say though is that even if the Republicans achieved all of the things that they hoped to do among the groups that they thought cost them the election, they still would’ve lost because of the group they thought they had in the bag. It’s worth pausing to ask ourselves how much all I just said mattered. I think I’ve demonstrated to you with some satisfaction that the exit polls were wrong, and that the narrative created by the exit polls probably promoted a certain approach by Republicans in the days after the election.

Marco Rubio, Jeb Bush, all of these people embraced comprehensive immigration reform, Sean Hannity did for a minute. And look, it’s worth imagining what would’ve happened if it had been the other way. Would the Republican Party have raced to support comprehensive immigration reform? I don’t know. Would Obama have raced to support comprehensive immigration reform, gun legislation, climate legislation, the TPP, if he believed that there were tens of millions of more white working class voters than they were, and that it was white working class voters in the Midwest who provided him with the decisive margin in the states that ultimately put him over the top in the electoral college?

Maybe. I don’t know. What I do know, though, for sure, is that four years later, Trump won the Republican nomination, championing opposition to immigration reform and free trade against Republican candidates who had just endorsed immigration reform by their account, in no small part because of their interpretation of polling data. I know that he won the presidency by making gains among a group of voters who are mostly, colored in blue here, white working class northerners who voted for Obama.

I think we can say in retrospect, because of his economic policies, that they opposed Mitt Romney’s willingness to cut the social safety net, that you may remember all the attacks on Mitt Romney over Bain Capital, you may remember that Romney would support outsourcing, you may remember that Romney supported tax cuts, all of these things that allowed Obama to paint Mitt Romney as a rapacious plutocrat. And before the election, it was generally understood that that was the reason why Obama would win the election, until the exit poll showed something very different.

And I also know that the polling in 2016 failed to represent these voters yet again. They systematically underestimated the voters without a college education in that election as well. And that’s the major reason, maybe not the only reason, but a major reason why the polls showed Hillary Clinton with a considerable lead. I think I just used the term “waiting by education,” this is a little bit arcane, but it basically just means making sure your poll has the right number of people without a college degree in its overall result, regardless of how many white working class voters you may have actually reached. And up until 2016, most polls did not make sure that they had the right number of voters without a college degree. That was true for the exit polls that you saw earlier. Those exit polls did nothing to make sure that they had the right number of voters without a college degree.

They actually used these fun tricks where the interviewer will use their eyes to be like, “Oh, I think you are a Black person,” or, “Oh, I think you’re a young person.” And then after the fact, they’re like, “Well, the interviewer thought 50 people who voted here were Black, but we only have 45 respondents who said they were Black. Therefore, we need to give more weight to the people who are actually Black to match the visual characteristics of the respondents.” That’s actually what they do. And as you can imagine, they can’t do that for education. Doing the race and age like that, it’s a little sketchy, but education, you definitely can’t do. And so they had no ability to ensure that they were properly representing the voters who, just a few years later, would form the basis of Trump’s support.

And this was probably true … well, it was certainly true for years. For years, pollsters were not representing voters without a degree. It’s just that in 2016, it started to matter a lot more. That’s the chart that you see on the right, and although I’ve lost the y-axis here, unfortunately, that’s my fault, it shows how much it mattered whether a poll made sure they had the right number of less educated respondents. In 2012, didn’t make any difference, Obama did just as well among voters with or without a college degree. So there’s no consequence to whether the poll properly represented them. In 2016, however, the gap between college educated, non-college educated voters had grown so big that if polls didn’t do anything about it, they would now underestimate Republicans by four percentage points, which by the y-axis is what you would see there.

So that’s basically what goes wrong in 2016. But I would argue, it may show up for the first time in 2016, but it was happening in substantive ways to our politics for years before that.

And I think in the 2012 election, exit poll is one example, but again, who knows how many other times decisions were made by the margin of whether this group of voters was properly represented in a political survey or not. This is also the world that the Times/Siena poll enters. I haven’t really talked much about what most pre-election polls were doing around this time.

The main thing I guess I would tell you is that from my point of view, they were fundamentally sound, but they failed in three specific ways. They failed to represent less educated and less engaged voters. This being one example. But it’s not just that low-turnout voters are also really unlikely to take a survey, that makes sense. You can imagine someone really excited to take a survey if they’re a political junkie, but if you don’t even care about politics, what are you doing taking this 30 question survey?

I also felt that they weren’t taking advantage of the best data that was available on the makeup of the electorate. I just gave you one example of how many people were relying on data that may not have been the best that you could use to try and figure out the makeup of the electorate to make sure polls are representative, but it’s not the only source of data on that. Another source of data that pollsters in the world of partisan politics use, it’s called the voter file. And the voter file is the basis for most partisan polling but not public polling, and it contains data like this on every voter. This is a little sample of me, this is my voter file, a snippet of my voter file data. Well, it shows you my name. I took out my address, but I gave you my lat, long, if you want to see where I live. You can see then in Washington DC my precinct voted 93 to 5 for Biden, that white people are 81% of my census tracked … sorry, 93% of people in my census track have a college degree.

It doesn’t know my race, even though it knows I’m Jewish. For some reason, that seems fixable. Although maybe they’re making a political statement there, actually. It estimates the value of our apartment at about $700,000, which should be pretty good. This is the same data you can get from Trulia. You all can figure out the estimate of your place. They can use that same data to match up with my address. It doesn’t know if I’m married, I am now married. Partisan description is other. I’m registered as an independent. You’ll also know I have not voted in any of the elections that you see on the left that are zeros, I didn’t vote in either 2020 or 2022, journalistic integrity.

You can also see that, here, we have our own estimates, that I have an 86% chance in my own view of supporting Joe Biden in 2020. It’s about the same as the chance that we gave Hillary Clinton of winning in 2016, so it’s just enough to keep things interesting, I guess. And I’m giving myself a 23% chance of voting in 2024 as well. I think that’s basically about right, I don’t know. I haven’t done it, but I could change my mind. So anyway, this is a rich data set that we basically have on every American who’s registered to vote. And the way this works is that when you register to vote and you fill out that form, it goes to your state, state compiles it, and then commercial vendors buy it from each of the states. They append all of this extra information to it, and then we get to use it for polling, because although I blanked it out, your telephone number is on here too. If you provided it on your voter registration form, I got your phone number.

If you didn’t put your phone number on there, then a commercial vendor can go and get a list of all people’s telephone numbers and names and then match it up and say, “Oh, Nate Cohn, we have him over here as having this telephone number.” So that must be the same one, same zip code, same name, and so on. I think this worked pretty well for our time. You mentioned that we have a good track record of accuracy, at least got us through a couple of elections. In 2016, we actually did better than a lot of people by making use of this data. Just as one example, we were one of a very small number of pollsters that showed Trump winning anywhere, had Trump winning in Florida by four, which is actually more than he won by. And as I mentioned the highlighting, the reason that we showed Trump ahead was because we were able to make sure we had the right number of registered Republicans in our poll, not rocket science, but because as you just saw, you knew my party registration.

We know how many Republicans, independents, and Democrats there are in every state, therefore, we can make sure our polls have the right number of each, and that happens to be pretty good for making sure you have a representative sample in terms of partisanship. I should also note that, although this is pretty good in 2016, it looked even better in our post-election analysis. This is the first time we did polls like this, and so we learned a lot. And when we did it after 2016, we were like, “Wow, we can be pretty good at this, actually. We might’ve nailed this thing with the benefit of hindsight.” And so unlike most polling organizations, we went big and we did 100 polls in 2018, which were great, earned that A+ rating from FiveThirtyEight, and then in 2020, unfortunately, we were just as bad as everyone else.

This shows that we, on average, were off by 5.5 percentage points. That’s not good, that’s about as well as you could do with no polling at all honestly, just given how correlated states are from one election to the next. And honestly, I think this sells short how badly we did because it isn’t simply that our polls were off, they were also biased, which means that in the aggregate, they were skewed towards one side on average in 2020. This isn’t just our polls, this is all polls in the era of the modern poll average. On average, polls underestimated Donald Trump by 4.7% in 2020. That is a huge amount. That’s the difference between whether the race seems close or not, and we were just as bad as this, even though I’m showing you all pollsters, not just ourselves. And as if that’s bad enough, no one knows why.

This is a quote from the post-election report of AAPOR, which is the big polling consortium out there, and I’ll read what it says in highlighting, “Identifying why polls overstated the democratic margin appears impossible.” So that’s pretty bad, off by five points, and figuring out why is impossible in their view.

There are many hypotheses for why the polls were wrong, of course, and the core hypothesis is something called non-response bias. And that’s the idea that the people who took the poll were just more likely to support Trump than people who didn’t take the poll, controlling for all the things you used in the design of your survey. So yes, you had the right number of Democrats who were college educated, who looked just like me on my spreadsheet over there, but the people who took your poll were just more for Biden than people who looked exactly like the people who didn’t take the poll.

The convenient thing about that explanation, of course, is that it’s entirely non-falsifiable by … and were to flip it around to be a little bit more positive about it, it’s a diagnosis of exclusion, to use a medical term. Once you have ruled out every other possible diagnosis, it has to be that one. That’s not true, strictly speaking. We don’t understand polling as well as I hope doctors understand medicine, but the logic is the same.

And there were a lot of different debates about the mechanism for why there was non-response bias, why Biden voters were likelier to support … sorry, why Democrats were likely to participate in surveys than demographically similar people who did not. One of them was that social trust was lower among Trump supporters. Another was COVID. All the liberals stayed home. And so they were there to take political polls while Republicans were out living their lives.

Another possibility was the horrible coverage of Donald Trump that just like, if you’re a Trump voter, but he’s in the hospital with COVID, and he’s doing whatever he did in that debate, and so you’re just not taking polls anymore. Another possibility is that we have this all flipped around the other way, and it’s actually the liberals who are super excited to take polls because they hate Trump, they’re energized by the Black Lives Matter movement, they’re part of the resistance, they’re flying to DC for their marches, and consequently, they’re surging to take the polls, not the other way around.

What’s relevant is that there’s no evidence brought by the polling community to distinguish between any of those theories, at least in public. There are partisan outlets … not outlets, partisan firms and organizations who have done research in this area behind closed doors, but they’re not available to you and me. And depending on which of those theories you accept, the prospect for polling going forward is very different. If it’s COVID, we’re fine, COVID’s gone, or at least gone enough for this non-response bias to be significantly attenuated. If it’s liberal exuberance in 2020, I think that’s faded a lot, too. I think Democrats have fallen back into a somewhat somber mood about the state of the world. They’re not contributing to candidates the way they used to be and so on. Coverage of Trump is also gone … not gone, but it’s not 24/7 Trump, Trump, Trump, Trump, Trump like it was in 2020. To the extent that’s a factor, it has faded.

If it is specific to the characteristics of the people who backed Trump, like they’re low in social trust and they simply won’t talk to telephone pollsters anymore, that one we couldn’t do very much about, and we would expect it to continue in the future. So depending on your theory for why the polls are wrong in 2020, you have a very different take on what the prospects are for polling going forward. What I will say is that I do think that there is evidence that at least something in this non-response field is probably the right answer. This is a chart that we published after the election, and it showed that democratic primary voters became more likely to respond to political surveys. In 2020, it stayed high. That doesn’t mean that non-response bias is a factor, but we at least know that Democrats did become likelier to respond to political surveys.

And so again, it doesn’t prove it, but it’s the pattern we would expect if the theory is right, even if the theory is not proven by it. Here at the Times, we’ve tried to do what we can to try and understand who doesn’t respond to political surveys. In 2022, we fielded an experiment in Wisconsin, where we sent mail to thousands of registered voters. And the mail contained a return envelope, it contained a paper questionnaire, it contained $5, which you can see partly depicted, and it contained a letter telling the respondent that, “If you reply to this survey, we’ll pay you 20 bucks.” At the same time, we feel that a normal phone survey and a couple of other experimental survey designs that we have never used before, and I don’t know that we’re likely to use since, but they were also in play.

In the end, this survey received a 27% response rate, while our normal survey in Wisconsin got 1.5% response rate. So we’re clearly tapping into a different population by paying people, and we did find some important differences between the people who took our survey when we paid them vs. those who took it on the phone.

And it’s worth cautioning that not all of these differences can be attributed to the higher response rate, it’s possible they’re attributed to some people taking it on the phone vs. people taking it online, and so on. It’s not a perfect experiment when we’re using different modes and methods, but I think that it’s nonetheless instructive, and some of them correspond with preexisting theories of why non-response bias exists. So I know that you can’t all read, but the left-hand side is how the voters are most different between the two kinds of surveys.

And the biggest difference is, “Almost never answers calls on a personal phone from non-recognized numbers.” That’s boring. The second-biggest difference, though, is interesting. It’s, “Identifies as a political moderate.” Ipsos mail, 42% moderate, the Siena/Times phone poll, 30% moderate, a 12 percentage point difference. And that really lines up to me with what we would expect. The sort of people who aren’t interested enough to take a political survey on the phone for no incentive are likelier to be political junkies, hobbyists, or fans, whatever you’d like to use, while the people who can be paid to take a survey, they’re not political junkies, they’re more likely to be moderate and not invested in American politics. The third one is also interesting, “Uses signs outside the home to deter strangers, such as, No Trespassing.” 31% of people in the mail survey said that they either had a no trespassing sign or considered it, compared to 20% on the phone. That also makes sense to me. These are people who aren’t, let’s call them socially engaged.

They may not have different answers on questions like social trust, as we’ll see in a second, but they don’t like talking to people. And it actually comes up a lot. There are questions on here like, “Prefers a job behind a computer,” as opposed to a job talking with other people. And maybe it goes without saying, but these questions were designed to test the various theories for why maybe people wouldn’t respond to political surveys.

The areas where they were similar though are also pretty striking. Contrary to what you might think, both groups were equally likely to trust the media. So I’ve always thought it was a distinct possibility, people don’t want to take our polls because they hear about polls all the time in the media and that didn’t seem to be a major difference between the two. No difference on, “Thinks most people can be trusted.” In fact, when people were more trusting on the mail, they’ll all note that maybe they were feeling trusting after they got five bucks from a stranger.

“Volunteered to work for an organization,” also about the same. And although I don’t have it on here, the results on the political ballot questions, like the Senate race and the governor’s race in Wisconsin, were virtually identical in both of the two tests. They were not necessarily the same on support for Trump. The mail poll did get a higher level of support for Trump, even though the level of support for Biden was about the same on both.

So it is possible that this group of less engaged, we’ll call them antisocial, although I personally would not be inclined to take a call from an unknown number, I wouldn’t have a no trespassing sign, so I don’t know where we want to draw these lines, but we’ll call them antisocial voters who don’t want to participate in political surveys, they’re less politically engaged, and they like Trump more than people who look a lot like them otherwise, the person next door demographically identical without the no trespassing sign. That I think is a plausible theory here.

Now, it doesn’t necessarily give us very much actionable information. In the basic model of polling, what you hope to do is if you see that there’s some dimension on what your survey could be biased, you want to know the truth and then you want to adjust your survey to match the truth. And we don’t know the truth for a lot of this stuff, we don’t know what proportion of people have a no trespassing sign nationwide to adjust our surveys on this subsequently. I don’t know how we could do that, unless there’s anyone here who wants to get on the Google Street View API and figure that out for us. It’s something I’ve been curious about whether we might try, and that’s true for a lot of the questions that we ask. This is more diagnostic to again return to our medical analogy, than it offers any clear prescription for what we can do about polling going forward.

One thing that makes this harder is that despite these differences that we saw in 2022 … sorry, despite the differences that we saw in this experiment, our polls were pretty similar … I’m sorry, our polls were very good. In 2022, our polls were dead on, best of class, and it was associated with a decline in apparent democratic response rates with respect to Republicans.

So this is the ratio of Democratic response rate to Republican response rate. So a number over one indicates that Democrats are responding at higher rates than Republicans, and this is only among white voters because Black and Latino voters, and Asian voters as well, tend to respond at lower rates and tend to be more Democratic. So when you look at the whole population, the aggregate, the response rate among Democrats and Republicans is pretty equal. But that’s mostly because non-white voters tend to be less likely to respond … Well, white democrats are more likely to respond than white Republicans can cancel it out.

I’m basically creating an artificial control for you by filtering down to white people. And what it shows here is that in 2020, white Democrats were 27% more likely to respond than white Republicans. And in 2019 and 2022, that number was a more reasonable 6% and 8%. And it’s possible that 6% and 8% can be attributed to socioeconomic factors, like higher educational attainment, the things that we would deal with in due course, as opposed to something that would systematically bias our polls going forward.

Going forward, I don’t think there’s anything that we can do … Sorry, I don’t think there’s anything that I know how to do that guarantees there won’t be another polling misfire in 2024, like there was in 2020. I do think that some of the data that I introduced here, for some reason, I think the risks of that have declined. But there’s no guarantee. I am continuing to monitor the bar chart you’re looking at here. I don’t know that an increase in democratic response rates among white people would be some siren that proves that we’re about to have another polling misfire. But I think it’s a plausible indicator that we can look at. And at the moment, I’m not even sure it’s the biggest problem that we face in polling right now. At the very least, I don’t think it was the biggest problem that we faced in 2022.

In 2022, I think I might posit the biggest problem that we had was with understanding why voters would do what they would do, not what they would do. As you could see in this prior one, our polls were awfully accurate, but I’m not sure we told the story of the election quite right.

Here’s what I wrote after the election. I wrote that, “The Democrats seemed to do too well because of two unusual issues, democracy and abortion. And we looked district by district and found that MAGA Republicans fared systematically worse than Republicans who were not from MAGA districts.” And here’s what I wrote before the election. I said, “The spotlight on those matters,” those matters being democracy and abortion, “is fading. Voters are less frequently citing them as top concerns while expressing worries about the economy, crime, and immigration, issues that tend to favor Republicans.” And these are all the poll questions that we tried to ask that we hoped would lend insight into why voters were doing what we were doing.

One is we asked them what they thought the most important issue facing in the country was. Issues like the economy, and inflation, Trump, abortion. We asked them what kinds of issues mattered more to their vote. Here again, they said the economy was more important than those social issues that we believe was a really important factor in determining the outcome of the election. And we also asked them what kind of candidate they would rather vote for. And most voters said they just didn’t really care that much, whether a Republican candidate thought Trump won the election, or they actually preferred a candidate who thought Trump won the election. So the poll questions that we were asking before the election, I don’t think did a very good job of telling the story of what would happen in the election even though our results were pretty good.

To me, that’s a pretty serious problem, and I think that if we could get this on the same scale as poll error in 2020 or 2016, I’d put this as an error just as big as that, if we could scale them the same way. And this is another area where I think that we continue to have a lot of work. Heading into 2024, the stakes for all of these issues feel as high as ever. This is today’s poll average as I calculated it this morning. Not 100% sure that I would take this to the bank, but it’s in the ballpark. And it’s a close race nationwide, Trump actually leads at the moment. In the battleground states, everything’s extremely close. Suppose these numbers would be the exact election result, and if you’ve gotten something out of this presentation, I hope it’s that you should not expect that.

But if you did suppose that Donald Trump would narrowly win by the margin of Pennsylvania, where he has a one point lead. And our polling suggested the reason for this is because Biden is struggling among less engaged, young, Black and Hispanic voters. And this is again, probably beyond the ability of many of you to see, but here, I’m breaking down the results of our surveys over the last year by whether respondents voted in 2022 or not. And it shows that Biden leads among people who voted in 2022, 47, 43 in our national polling. He trails among people who didn’t vote in 2022, 41, 39. And that’s true even though the people who didn’t vote in 2022 are disproportionately young, Black, and Latino. It just turns out, according to our polling, that the Black, Latino and young voters who didn’t vote in 2022, are very dissatisfied with Joe Biden at the moment.

You can see that among Black voters, Biden has an 81-8 lead among those who voted in 2022. While among those who didn’t vote, Biden’s only up 62-14, which is, for a Democrat among Black voters, a very weak showing. Same pattern exists among Hispanic and young voters. It’s also true among Republican constituencies as well. The white working class voters who stayed home in 2022, according to our data, are very conservative. And as I mentioned in the context of immigration reform, these findings will wind up playing a role in the democratic process right now.

Just last month I wrote that there are consistent signs of erosion in Black and Hispanic support for Biden, and it was watched by the White House, and they will be acting on this kind of information. I’m not the only person who’s telling them this. They have their own polling that shows them, of course, that Biden is faring worse among Black, Hispanic, and young voters than prior Democrats.

But it matters that this stuff is in the public as well. You can imagine that the democratic coalition consists of all kinds of stakeholders who believe all kinds of things, and they’ll want the president to talk about their thing in his reelection campaign. And the availability of this data may help explain to them why maybe they’ll be talking about something else, that’s not what they want. That’s especially true on something like appealing to minority voting groups. You can imagine that all kinds of people on the internet might be like, “Oh, why are you talking about Black Lives Matter? Isn’t it true that you’re just going to alienate more white people that way?” The availability of this data makes it easier for the rest of the world to understand the kind of political calculus that Joe Biden will probably be undertaking. And all of that, of course, is separate from the really important question, which is whether that calculation is being informed by accurate information.

If it is not true that Biden is faring worse among non-white voters, as I’ve just suggested here, imagine the consequences. Now, Biden might be saying exactly the wrong thing in his pursuit of reelection. And so in the end, whether voters are given the messages to lead them to support a politician, will come down to the kind of information that politicians, and political activists, and campaigners have to make decisions, and ultimately, the health of the democracy will rest on whether those calculations make sense or not. I don’t think I have a timer up here, so I’m hoping that I’m in the ballpark. Yeah. OK. I got a thumbs up, so that’s that. That’s it. Sorry, I almost lost my voice there a couple of times. Thank you for the water, whoever provided this in advance. Yeah, OK. Oh, I missed what that was all about. I saw a person come and I was like, “Oh, you’re closer than I would’ve guessed you would be.”

Audience 1: What would you say to people that believe that polling has essentially become the opposite of a self-fulfilling prophecy? I remember in the 2016 election, because there was a lot of polls coming out saying that Hillary Clinton was almost guaranteed to win, that democratic voters became complacent, and therefore didn’t come out.

Nate Cohn: Embedded in that question, there are several assumptions, and some of them I’m sympathetic to, and some, I’m not. So in 2016, I don’t think that turnout among Democrats was low, I don’t think that’s the reason she lost that election. I’m also not convinced it’s because the polls didn’t show the race being close. I just showed you a poll that we had that showed Trump up, but other polls still showed the race pretty close, even if they showed Clinton ahead. I think the main reason that people didn’t think Trump could win was not just because talking heads and pundits said he couldn’t win, because many people didn’t believe he could win because he was not someone who many people could imagine as winning a presidential election. .

This was a reality TV star who said crazy things. He was constantly being attacked, and didn’t fit their archetype for what they thought they could win. Now that I’ve talked about those specific things, in terms of the actual thrust of your question, I think that if there are people who make political decisions about whether to vote based on a narrow difference in the polls, I’m not sure that that person is the person who can be assured to be voting otherwise. I’m just not sure I’m convinced by the counterfactual that in the world without polling, that that person really is staying home. And also know that I think the research on this is pretty mixed and that intuitively makes sense. One thing that I was noting to someone I was speaking to earlier, is before the election, Donald Trump was complaining that the polls showing him behind were intending to suppress the enthusiasm of his voters.

And then after the election, Hillary Clinton argued that the fact that she was in the lead was the reason that people stayed home. I have no idea which of those theories aren’t right, I’ll be honest, I do think it’s entirely possible that people’s belief about whether a candidate is likely to win affects, at the margins, a small number of quite informed people’s decisions about whether they’ll turn out. I’m inclined to think that most of those highly informed people are regular voters, and I’m also inclined to think that a lot of those people would still think mostly the same thing. I don’t know that I agree that there are many races that people think would be competitive, but because of the polls, they’re like, “Oh, I just saw the Times, CBS poll showed Clinton up four, not a race, going to stay home.” I am not saying there’s nothing to it, I just think it’s unlikely.

Audience 2: Inflation, it’s a novelty to young people. It’s a big shock …

Nate Cohn: It’s also a novelty to a lot of the political data that’s been collected since 1980.

Audience 2: So that issue is killing Biden, but isn’t the fact that rents have gone up around the country, isn’t that hurting Biden among young people? Are your surveys starting to see the cost of housing being mentioned by people at all?

Nate Cohn: We don’t distinguish in the way that our surveys are being coded between the cost of living and housing. So I can’t say it’s one or the other, or the relative contribution of the rising costs of gas or food vs. rental costs vs. interest rates, or anything else. But I think it’s definitely fair to say that in the aggregate, the rising costs of living is hurting Biden, and that housing prices are undoubtedly a major part of it.

Audience 3: Yeah. Would you care to comment on the interaction between this electoral college and polling? Because normally, when you hear about polling, it’s about the general electorate, and obviously, what matters is the electoral college.

Nate Cohn: The most serious public polling firms almost exclusively focus on national surveys, and that does partly go to what I said at the beginning, that they think that their highest calling is to give voice to the nation and the people in the democratic process. To be blunt, why does it matter whether people in Pennsylvania think X about an issue? You can see why in their framework for the way polling matters that a single state poll isn’t especially important. For us, state polling has been our overwhelming priority. We’ve done far more state and district polling. There’s far more state and district polls than national polls, it’s partly because it’s like our value add and our opportunity. There are tons of national polls and very few state polls, as you just mentioned, so it’s a place where we can make an additional contribution. But it’s also because, as you’ve alluded to, many of our political outcomes are decided by states and the electoral college critical battleground states and the Senate battleground house districts.

Most of the time, the nation and the states and the battleground states look fundamentally similar. Even in 2020, when the battleground states were like Biden plus one and the nation was Biden plus four, there are a lot of different purposes for which that Biden plus one vs. Biden plus five difference isn’t that much, it’s a few points. Obviously, that’s everything when it comes down to the actual outcome of a presidential election. So for our purposes, we focus overwhelmingly on those states, but there are a lot of other purposes for which focusing on the battlegrounds as opposed to the nation isn’t necessary.

Audience 4: Will the rise of artificial intelligence change how you go about polling?

Nate Cohn: So before I answer your question, I do think that AI might already be affecting the presence … the present of polling, not the presence of polling, I’m sorry, by creating challenges for online pollsters who rely on these huge online panels that may now be mostly bots. It depends on the vendor, every online panel has different quality control checks, each pollster has different quality control checks. So it’s very hard to generalize about this problem. But I talk to people who say that they’re getting rid of 30 to 40% of their online respondents now on data quality checks. And by data quality checks, I mean, eliminating people or non-people who are not engaged, attentive, and cognizable respondents.

How much of a problem that is today? Hard to judge, and again, hard to generalize. Going forward, I’m not really sure what the role of AI is. One thing I’ll say is that there are some cases where machine learning and AI can be helpful to pollsters, but we have really granular and big data, as you already saw, and one consequence of having really granular and big data is that a lot of our problems are not necessarily ones that require the most sophisticated modeling techniques. The modeling helps, but it’s the acquisition of the data itself in the first place, that’s the real challenge.

And I don’t know that AI helps very much in the data acquisition part of our task, which I think is the hard one. One fun thing that someone’s posited to me is maybe we don’t even need to poll if AI gets such a great model for itself of the sentiment of the public. I don’t know how that would work, but maybe that’s conceivable. With all of the sentiment analysis on social media plus the polling, maybe it’ll eventually just learn to be the pool of the people, and we’ll all just ask, “What does the public want Biden to do?” And then we’ll all know. Maybe that’ll be the end state there.

But I don’t know whether in our processes as they currently exist, at least I am not sure that’ll be a big role. I should also be honest, I have not given a tremendous amount of thought to it. One thing about being the New York Times is that we get to be innovators, but I’m not sure we get to be radicals, and we can’t take too many risks. People will say the New York Times did blah, blah blah. If we were a different outlet, maybe this would be the thing that we could start experimenting in. But as it is, I think other people are going to play with this first, and if they figure out something then that works, then we can try and refine it and use it for our own purposes. But in the short term, I can’t imagine being the industry leader on this until something else has happened.

Audience 5: I have a question about the experiment, which is awesome, and I think it’s great that you wrote that. One question is, is the data publicly available in a way that … because I have a bunch of follow-up research questions, and the question two is, the slide that came next was not the one I expected. And so I’m curious if y’all looked at that and …

Nate Cohn: What was the slide next?

Audience 5: Well, I think the next slide was how well y’all did in 2022, which is awesome too. But my question would be, so you have this really rich data, I would imagine you could come up, in your mail poll, with a guess of the people who would’ve also responded on the phone poll. I would guess you could model that pretty well, having tried this exercise in other realms, and so you could construct a sample from your mail poll that would’ve looked like your phone pole, and if you limit yourself to that, or you could wait, or however you want to go about doing this, do the results differ? Because it’s interesting on the dimensions in which these people are different, but it only matters if they are also then different on these outcomes we care about. So I was curious if you could speak to that.

Nate Cohn: So first, to go in order, the data is not publicly available yet and it’s also not final. One interesting part of this process is that we are relying on Ipsos for the sake of preserving the anonymity of our respondents to do this match between their mail-based sample and the voter file. And as you may know, or as I know you know, and as many other people may not know, Wisconsin’s a state where there’re same day registration on election day. So we need Ipsos to do a match for the post-election voter file in Wisconsin, and it turns out that the person who they had, who knows how to do this, is no longer with Ipsos. And so we are waiting for them to provide this data to us.

So that’s why the analysis is not even final, strictly speaking, yet. We have also done exactly what you’ve said. It’s true it’s not the next slide, maybe it should be, but to step back for a second, the one important difference between the mail survey and our survey is that our survey is conducted off of a voter registration file, you saw that data, the mail survey that Ipsos commissioned was not, it was just a random sample of addresses. They then waited the survey to match normal demographic targets for the adult population of Wisconsin, but they didn’t use all that voter file stuff that I showed you that we know about myself.

I told you that the results of the two surveys were about the same in their top line results. But if you control for all of the information that we have on the voter file, or if you want to take the approach that Erin suggests, and you actually model what our telephone respondents would look like based on our mail data, the mail data becomes more conservative than the Times/Siena data, even though the reported top lines are essentially identical. And that’s true in the senate race, the governor’s race, and the presidential race, even though it’s only the presidential race in the top line figures. There are a lot of weird reasons that appear to be true here. One of the most interesting is the sourcing of the telephone numbers.

So as I mentioned earlier, there are some people who give their telephone number when they respond to a poll, and then there are other people where we get their numbers from an external source. The people who give their telephone numbers respond to surveys in much higher numbers. That’s partly because their numbers are just better, it’s not some match that an algorithm could get wrong. You said that when you registered to vote, maybe they’re someone who’s lying to the state of Wisconsin when they fill in their form, but it’s a pretty reliable number that these registrants give. And the second reason could be, I’ll just note, that they may also be the sort of people who are likelier to want to talk to people on the phone, or are willing to talk to people on the phone, or people who are willing to share their information more publicly.

And the people who we talked to … you can imagine three different groups of voters, people who give their number, people who don’t give their number who we call, and people who do not have a number. The people who do not have a number, and the people who did not give their number but we have anyway, are more conservative. Controlling for that covers about half of the difference between the Times survey and the Ipsos survey, adding in all of our base voter file traits that we would usually use.

Another series of things that you may have seen on this, but they do show up even in the top lines here, are things like born in Wisconsin. Registration date is a really interesting factor. The people who give their cell phone numbers when they register to vote, by definition, these are people who have registered to vote in the last 15 years. If you registered to vote 20 years ago, you didn’t have a cell phone to give the state of Wisconsin when you registered. So the kinds of people who are in that category I just mentioned, who gave their cell phones, they’re, by definition, a more mobile group of people who have registered to vote more recently, and they’re ever so slightly more democratic. There are a few other traits too, and I think some of them are in the engagement, like the vote choice … sorry, the vote history style questions, but if you do exactly what you proposed, and you model … Our telephone respondents is a function in the mail survey, the telephone is three points to the left.

If you control for all the things that are on the file and at our disposal, including the telephone sourcing, that eliminates almost all of it. The self-reported questions that I have here, like trespassing signs, that doesn’t seem to be relevant, to my surprise, even though I just talked to you all about them as this nice indication, I think it tells a colorful story about who these people are, it doesn’t really seem like if we do the thing that you proposed, that that is the thing that makes a big difference. And in a way, that’s actually helpful because that would actually be a much harder problem for us to crack. I should also note that then there’s one final question here that we don’t know the answer to, and that’s whether the mail respondents themselves are possibly too far to the left. That’s harder for us to gauge. Sorry, we got this guy running around back.

Audience 6: Hi. Thank you for the talk, it was really interesting. You talked about how prior to Trump, the GOP was anticipating the changing demographics and start to support immigration reform, gun reform, and God knows we can’t understand what goes on in his head, but how did Trump predict that? Did he realize that it was really the white votes that mattered more or …

Nate Cohn: I talked a lot about calculating political actors looking at polls, I don’t think Trump was one of those people. And I think that was probably part of why he managed to get it right, given that the polls were wrong. And I think that the reason Trump got it right was probably a product of his lived experience. Donald Trump was from outer Queens, and he grew up in the ’60s and ’70s. This is a time of toxic racial politics. And it’s a rare part of the country, I think, and a rare place to have grown up in, where they’re not evangelical Christians, they’re not very conservative on the Paul Ryan, Mitt Romney kind of conservatism. They may be populist and hate those liberal elites in Manhattan in fact, but they are pretty well off. You could be working class potentially, but you can also have quite a bit of money in Long Island or Queens. These are not people who are looking for unions or for redistribution of wealth.

And I think that that experience encapsulates a certain thread of the Republican Party, not a thread that’s been dominant for the last 60 years, but maybe it’s the one we would’ve went down if Nixon hadn’t been impeached and Agnew had become president instead of Reagan. I think this is a thread that existed throughout Republican politics the whole time that Trump is organically part of, and that he running organically as this kind of person at this political moment happened to capitalize on an opportunity that he probably intuited from his own level of dissatisfaction with the major parties. That’s my personal theory of it and how he came to this. I definitely don’t believe that it was the people at Cambridge Analytica doing the five traits analysis that they say, “Let them magically see the truth.” I think that’s all nonsense.

Audience 7: Thanks for the talk. I wanted to ask about sponsor effects across different survey vendors. So there’s academic literature which says they’re not a thing where they compare across different universities. As someone who’s administered a survey that goes out on UC Berkeley letterhead and been called lots of nasty names, I have to think it is correlated with non-response, at least in my own lived experience. For a while, there was this thought that maybe Trafalgar had figured it out and it seems like maybe they didn’t, they weren’t actually doing exactly what they said. I guess where I’m going with this is, I think, given education polarization, given trust in institutions, given what you showed us there’s an 86% chance that you voted for Biden, are there things that some survey vendors could be doing to make themselves appear more friendly to conservatives?

Nate Cohn: I think it’s a really good question. One thing that I’ll just say at the top because I get asked this all the time, is our interviewers do not introduce themselves as calling from the New York Times, they are introduced as calling from the Siena College Research Institute. I don’t think Siena College is a brand name associated with the dreaded liberal elite, but it is still a college. So I’m open to the idea that it’s still something that perks the interest of college educated liberals and best does nothing to attract those who didn’t go to college, and at worst, could repel some. Another thing that people don’t know, though, is that most of our calls are not even placed by Siena College Research Institute anymore, they’re placed by call centers all over the country, the largest of which is a call center called ReconMR, and ReconMR is meaningless.

And we have done testing to see whether it matters at all, whether ReconMR or Siena College is a better name to lead with. And according to Siena, and I have not been part of this myself, they use Siena and Recon all the time, this is not like Times/Siena sponsored research, according to them, it doesn’t make a difference. Whether that’s true for others, Berkeley, I don’t know. All right, the mic is on the way.

Audience 8: Hi. Thanks for your talk today. Could you give us a prediction about third parties in the 2024 election? And also, one of my students asked in class about did Jill Stein cost Hillary Clinton 2016 election, and what are your views on that given the state of exit polling?

Nate Cohn: Well, if I recall correctly, Berkeley gave more votes to Jill Stein than Donald Trump, so that’s a salient question here. It’s one of a small number of places in the country where that’s true, there are parts of Western Massachusetts like North Hampton, Massachusetts, I believe, some other places that are in that category. So I assume that a fifth of the crowd at least voted for Jill Stein in that election. So if you go by the survey results and you reallocate Jill Stein supporters in proportion to how they say they voted, my recollection is that it is not enough for Hillary Clinton to win the election, but I haven’t looked at this in five years now. So you’re just going to have to put a pretty big asterisk on that recollection. Someone is welcome to tell me I’m wrong and I’ll just be like, they remember this better than I do where they Googled it.

In terms of the third party in 2024, I do not have a prediction about that. I do think that the conditions for a third party candidate to be relevant are certainly in place. There are a lot of voters who did not like either Donald Trump or Joe Biden. In 2020, I would guess that the preponderance of them voted for Joe Biden. If a third party candidate could appeal to those voters, I think it is reasonable to suppose that there is a more upside for them among Biden voters than Trump voters. That does not mean that they will run a campaign that lets them do that, but in terms of just what is the theoretical opportunity, if I could just know the truth of who could possibly vote for a third party candidate in the next election, my guess is that there are more of them who voted for Biden regardless of whether any third party candidate is poised to capitalize on that opportunity, or whether it would be decisive. And I’m absolutely not making a prediction on what that would involve.

Audience 9: Thanks very much for the talk, it’s just a lot of things to think about here. Let me ask some questions that come from a little different perspective. I’ve not been looking at the pre-election polls, I’ve been wallowing in the analysis of the American National Election Studies 2020, and it raised as many of the same types of questions. In 2016, the bread and butter of the most expensive survey there is in this area exaggerated Hillary Clinton’s margin of victory in the popular vote by a point, not much more. In 2020, of course they could not do face-to-face interviewing and they did almost no telephone interviewing, so the overwhelming proportion of those cases, they also didn’t send a printed questionnaire, they were afraid it would scare everybody else off because it would be this thick and no one would want-

Nate Cohn: Look, I’m afraid of the code book for the survey, which was like hundreds of different …

Audience 9: So it was different in a number of respects. Biden’s margin of victory in the three category rendering, Biden, Trump, and all the ways people abstained was 10 points. Whereas in truth it’s …

Nate Cohn: Which to put it again, I think that’s like Biden plus 14 is the result on it. Yeah.

Audience 9: It depends on which way you calculate the percentages. So I’m doing a little [inaudible] score here. Well, it’s not because telephone interviewers are somehow related to this, it’s not because they didn’t offer financial incentives, they paid a substantial financial incentive. There’s an old question, been around for a long time, whether discouraged voters are more likely to, A, not vote, and B, not do an interview. And so that’s the main prison that I’m thinking about here. The other one is the trust in institutions. The University of Michigan, Stanford University, those are the ugly that people don’t mind, I suppose there could be something to that …

Nate Cohn: Those are at least college football schools, though, I think …

Audience 9: OK. But I was wondering, what do you think of the general question that people who are on the discouraged side, both are less likely to vote and are less likely to answer questions.

Nate Cohn: So far, it’s entirely true. For those that aren’t aware, people who … and we have the vote history on our respondents, as we mentioned, to have a really clear idea of how turnout history and response are related. But the people who vote in primaries and regularly vote in general elections, they’re more than two and a half times likely to respond to a survey than someone who has not voted in a midterm or a general. And it’s entirely true, we do a lot of work to reach these people, which is to say we pay a lot of money, and the way we do that is that before we conduct the survey, and if you looked very carefully on my spreadsheet, you can see that in the bottom right here, we have a modeled response rate for the likelihood that I would respond to the survey on a cell phone or a landline. And we use these probabilities to determine the number of records that we call for each respondent.

You happen to have on staff, someone who is very familiar with this, over there, named Erin Hartman, who did this sort of thing for the Obama campaign. But the basic idea is that the less likely you are to respond, the more likely we are to call you. And that is very expensive, as you can imagine, because we’re calling many, many people who are extremely unlikely to respond to get a small number of interviews just to make sure that they represent their small share of the sample. This does not necessarily solve the question of whether the people that we get who are unlikely to respond are actually representative of that far larger group of people. We have all that voter file data to check whether they look right. So did we get the right number of low turnout Democrats? That’s a knowable thing for us. We know whether we do or don’t. And thanks to various statistical techniques, we do.

That still doesn’t answer the question of course of whether we’re getting the right low turnout Democrats even if we have the right number of them. And that’s the non-response problem that we can’t crack with the data at our disposal, then we’re at the diagnosis of exclusion stage. I will say that in our data, both in 2022 and in 2020, the low turnout voters are more for Trump. So I think that our expense pays off, at least in terms of helping Donald Trump and our polling. And it does not surprise me that pollsters who do not take similar steps could end up being more democratic. Now, since you asked me about the ANES, I do have to ask whether it’s a seven figure or eight figure budget for it in 2024.

Audience 9: Money was not their problem.

Audience 10: So Nate, great talk. Thank you. So I’ve been reading weekly what you have to say, or whenever it comes out, and I guess, also reading a book by Phil Tetlock about how difficult it is to forecast the future. And so maybe how should we think about the polls as they are today, including what you just showed us? I would say, we’d be more confident in predicting the outcome of the Republican nomination process …

Nate Cohn: Yeah, you should have asked me about that.

Audience 10: … than anything else, but really, as a citizen, someone who’s reading this stuff, how would you take today’s data and think about the election going forward, not as a pollster, not as a political scientist, but just an ordinary person?

Nate Cohn: My short answer on this is something like, I think that the poll should be taken seriously, but that doesn’t mean that I think they’re predictive, I don’t think that they necessarily indicate what the final result will be, but I take them seriously. And so far as I think they’re based on considered informed and well-developed views on the part of Americans.

In a different election, in contrast, I might say, “We shouldn’t even take them seriously. It’s a year to go. The candidates haven’t been nominated.” I don’t think that’s true in this case. We’ve literally had this election before, these candidates have been around for decades, so I do think we should take them seriously, and the attitudes that people express about them are real. Now, just because real doesn’t mean that they’re bound to last until the next November, and in fact, taking them seriously could potentially empower different people to change them.

The Biden campaign, as I mentioned, is going to make decisions about how it’s going to campaign based on where they stand today. And they will be doing that precisely because they take seriously what the numbers are now, not because they believe they’re predictive. They’re predictive … I mean, they could potentially give up under certain circumstances if the numbers got bad enough, and that certainly won’t happen. In general, I’m not a fan of predicting elections. A lot of people don’t realize this, but I have never had anything to do with predicting an election beforehand. Even when the Times has done it, my name’s not on it. And there are a lot of reasons for that. But one reason is that, in our era, none of the elections that we care about are clear enough for the prediction, to my mind, to be operationally useful. To the question about whether you would vote in 2016, there’s never been a presidential election, as long as I’ve been around doing this stuff, that you could make such a clear prediction about the outcome that you would be like, “Oh, they’ve predicted X would win, I don’t need to vote.”

That has been true in different points of history, where if we had made such predictions, that could have been a reasonable interpretation, but it hasn’t been in my time. Or to use a sports analogy, every election that I’ve been a part of has been an election in which the team has the ball in the fourth quarter and a chance to win.

Audience 11: Too close to call.

Nate Cohn: Yeah, if it really comes down to like can you call it, they’re all too close to call by that standard. So from the standpoint of an engaged citizenry, I would absolutely not say that any of the polls merit the word prediction to me. And I would be curious to look it up in the dictionary, but to me, a prediction is like a statement of what you believe will happen. It is not like some probabilistic claim that you think Biden is likely to win. I have an opinion on that, which I’m not going to count your question up prediction as asking, just so that we’re clear about the meaning of the term, no election that I’ve been a part of has been clear enough that a prediction could, I think, be safely made. You could do it for gambling or something, but not as a citizen.

Audience 12: On immigration, as you know, democratic governors are calling for tougher stance by Biden, what is the attitude in your polling of Latin voters toward, let’s say, more restrictive policies at the border?

Nate Cohn: In our pollings, voters of Hispanic or Latino origin are more conservative than people might guess on border issues. That doesn’t mean that they’re conservatives, but they are more supportive of Republican policies on the border and whether people should be allowed to seek asylum, than they are supportive of Republican candidates, which is to say that if you’re a Republican candidate and you’re talking about this issue, you may have the potential to make gains among Hispanic voters by talking about these issues. Now, this is all very sensitive to the question and there are questions that are relevant to the border where Hispanic voters have very different opinions. If we were talking a few years ago about comprehensive immigration reform, which is something that is now off the table, Hispanic voters backed comprehensive immigration reform by a huge margin in the polls.

So the fact that they’re sympathetic to conservative policies on the border, doesn’t necessarily make them conservative on immigration as a whole. But with our politics now focused pretty narrowly on the border, and to take us back a few years, not ridiculous things Trump said about Mexican judges, if you remember that stuff, or building a wall, now that all of that other stuff is gone and we’re mostly focused and just like, “Do you want more border security or not? Do you think that people should be allowed to enter the country on the basis of asylum and seek a legal status for some amount of time?” There’s not as much public support for that, if we’re counting that as conservative right now, even among Hispanic and Latino voters.

One thing I’ll note here, by the way, just to bring it back to one thing I said, is I think this is a really interesting case of where Donald Trump himself has at once been an important part of making the democratic system more responsive to issues that were previously not addressed, but also has himself gotten in the way of his own voters as causes here. There’s a version of Donald Trump who didn’t himself become the dominating story of the last decade, that wouldn’t have caused the same reaction among liberals and progressives, and even many moderates, and would’ve potentially created the political conditions for some of these issues to be addressed. But that didn’t happen. And I think now that Trump is gone, we actually are now having this strange bipartisan moment, where now, even Biden is willing to talk about building elements of the wall, and so on, and taking border security more seriously.

And that may actually prove to whatever you think of the policy on the merits from the standpoint of the health of the democracy, and if you believe that the will of the public should eventually make its way into the policies of government, there is something healthy to it.

David Broockman: All right, please join me in a round applause. So if you didn’t before, I’m sure you now agree that Nate is well deserving of the 2023 Citrin Award for Achievements in Public Opinion Research. So it’s my honor to present you this large obelisk.

Nate Cohn: It’s the Washington Monument, really.

David Broockman: Yes.

Nate Cohn: Thank you.

David Broockman: So thank you, Nate. We’ll take a few pictures and then we’ll have a reception here for the next hour. So please stay, I think we have some refreshments in the back. I see, I think, some tubs of wine back there. So please stay. If you have more questions for Nate, he’ll be here for the next little bit. And thank you all for coming.

[Music: “Silver Lanyard” by Blue Dot Sessions]

Outro: You’ve been listening to Berkeley Talks, a Berkeley News podcast from the Office of Communications and Public Affairs that features lectures and conversations at UC Berkeley. Follow us wherever you listen to your podcasts. You can find all of our podcast episodes, with transcripts and photos, on Berkeley News at news.berkeley.edu/podcasts.