Is it the fault of voters or pollsters that state poll results show a tie between Trump and Harris?

Recent polls in the seven key swing states show a stunningly close presidential race: 124 of the last 321 polls conducted in those states — nearly 39% — show margins of 1 percentage point or less.

In fact, the state polls show not only an astonishingly close race, but also an impossibly close race. Even in a truly undecided election, the inherent randomness of polling would produce more diverse and less grouped results—unless the state polls and polling averages are artificially close because of decisions pollsters make.

The results of a poll depend on the opinions of voters and the decisions of pollsters. Decisions about how to weight polls to reflect the expected composition of the electorate can shift the results of a poll by up to 8 points. This is true even when pollsters make perfectly reasonable decisions about weighting their survey data, as survey researchers have been forced to consider new methods and ideas for weighting and dealing with declining response rates following survey errors in 2016 and 2020.

But the fact that so many polls produce exactly the same ranges and results raises a troubling possibility: that some pollsters are making adjustments in such similar ways that these decisions tend to condense the results and create a potential illusion of certainty – or so Some pollsters even use the results of others to determine their own results (e.g. “herding”). If so, the artificial similarity of the polls could create a false impression that may not come true on election day. We could well be facing a very close election. But there is also a significant chance that one candidate or another could win every swing state and win the presidency somewhat comfortably, at least compared to the balanced picture in the polls.

Given randomness, what should we see in a perfect election world?

In a world perfect for polling – a researcher's paradise where every voter can be contacted and every contacted voter responds – we can use mathematics to calculate how much variation there should be since voters are randomly assigned to participate be selected in a survey.

If a race in this world were truly 50%-50% undecided, the polls would not produce all the results that 50%-50% would share. Imagine if pollsters in the world conducted 100 identical surveys of 863 randomly selected voters (that's the average sample size of this year's swing state polls). The results in 95 of these polls would show the candidates receiving support in the range of 46.7% to 53.3% – even though in this imaginary world we know that the race is actually tied at 50%. The other five polls would show the candidates earning something even larger or smaller outside that range.

This variation is known as the “margin of error” in a poll – that is, how much the random selection of voters who always respond can affect a poll's estimate of a candidate.

Because each candidate's support fluctuates randomly, these polls predict a lead in a tied race that is between -6.6 and +6.6 in 95 polls out of 100 (even larger in the other five).

It's important to highlight that the range of margins we can expect in a tied race (and in a perfect polling world) is much larger than the margins in the swing states in 2020. Even under ideal election circumstancesIt is difficult, if not impossible, for a poll to be very informative about who is leading in a close race. And this is arguably a lower bound for what we should observe in the messier real world, where polls vary in the way respondents are selected, contacted and weighted to match whatever election outcome pollsters believe that it will be canceled in 2024.

We can also calculate what proportion of the 863-person polls we should expect to see different margins in a truly undecided race. Rounded to the nearest percentage point, about 11% of polls should show a tie in a tied race.

This means that almost 9 out of 10 polls of a tied race should not produce a tie result due to randomness and margin of error.

About 32% of polls should have a lead of 1 point or less, 55% should have a lead of 2 points or less, and 69% should have a lead of 3 points or less. Even in a 50-50 race, about 10% of polls should have a lead of more than 5 points due to inherent randomness – almost the same percentage that results in a (rounded) tie!

Given enough polls, the predicted margin should also resemble a normal “bell curve” distribution – with both candidates ahead for a similar number of polls.

What do we see in swing state polls?

Actual swing state polls have far less variation than the benchmarks we would expect in a perfect polling world. Of the 321 polls in the seven swing states, only 9 polls (3%) report a lead of more than 5 points. Even if every race were tied – which it isn't – we would still expect about 32 of the 321 polls to have a lead of more than 5 points due to chance.

Considering how the reported poll results compare to what we would expect in a perfect election world strongly suggests that swing state poll results are being held close to the national poll averages. In these 321 state polls, 69 of them (21%) report an exact tie and 124 polls (39%) report a lead of 1 percentage point or less. Both numbers are about twice what we would expect in a perfect world of polls where the only source of variation is the random selection of responding voters.

Pennsylvania is perhaps the most troubling state. Fully 20 of 59 polls there (34%) show an exact tie and 26 (44%) show a lead of 1 point or less, even though the odds of a truly tied race result in a poll with more than 15% Leads is at 15% With a 5 point margin due to randomness, we see only 2 of 59 Pennsylvania polls (3.3%) with a margin greater than 5 points.

Even where the polling results are not as tightly clustered, such as in Arizona, Michigan, and Wisconsin, there are still far more polls than we would expect around the polling average, and too few polls with large margins.

What's up?

The concentrated margins we see in swing state polls likely reflect one of two possibilities.

One possibility is that pollsters sometimes adjust a poll result that seems “strange” to them by choosing a weighting scheme that produces results that are closer to the results of other polls. There appear to be strong incentives for risk-averse pollsters to do this. Unless a pollster conducts many surveys and can be confident that the effects of randomness are average, there may be reputational and financial costs if they receive an incorrect result due to randomness, since pollsters are evaluated on the accuracy of their polls.

A risk-averse pollster who gets a 5-point lead in a race he thinks is undecided might choose to adjust the results to something closer to what other polls show, so that his Outlier survey does not affect its reputation compared to its competitors.

Another, more likely possibility is that some of the tools pollsters are using in 2024 to address the polling problems of 2020, such as weighting by party affiliation, prior voting or other factors, will flatten the differences and reduce the variation in those reported survey results could reduce. The effect of such decisions is subtle but important because it means that the similarity of polls depends on the decisions of pollsters rather than voters.

And if these assumptions are wrong, which will only become apparent after the election, then the risk of potentially significant polling error increases as the differences in different polls decrease.

Why this is important

The fact that so many swing state polls are reporting similarly close margins is a problem because it raises the question of whether the polls in these races are indecisive because of voters or pollsters. Will 2024 be as close as 2020 because our politics are stable, or will the polls in 2024 look like 2020 results just because of the decisions government pollsters make? The fact that the polls appear to be more tightly clustered than we would expect in a perfect polling world raises serious questions about the second scenario.

The reported polls and polling averages create a consensus that the race will be very close and we will likely see a similar result to 2020. Maybe that's true. It would be wonderful if the polls successfully addressed the concerns of 2016 and 2020 in 2024.

But the fact that the surveys all report similar margins doesn't necessarily make it any more likely that those margins represent the bottom line. In fact, there is a possibility that the election results could be unexpectedly different than the razor-sharp presentation that the state poll clusters and poll averages suggest.