Correlated Error: When every poll is wrong the same way

In brief

Averaging polls cancels only the random part of their error, not a mistake that every poll shares. In 2024 the shared lean was real and industry-wide: all contest averages missed in the same direction, by a mean signed error of 2.16 points. Adding more polls shrank the interval but left the average just as far from the certified result. So a tight poll average tells you the polls agree, not that they are right.

How to use this

Separate precision from accuracy before you use a poll average. Ask how the polls were recruited, adjusted and modelled, and whether the lean runs across firms and methods, because a shared miss is an industry pattern, not one firm's habit. If every poll leans the same way, do not expect a narrower interval to correct it. Treat a shrinking margin of error as evidence that the polls agree with each other, not that they are right. When you need a winner call, remember that a shared bias can invalidate an interval while leaving most calls intact. If you are planning around the lean, do not assume you can predict its direction, do not treat expert forecasts as a safer substitute.

What the story is about

A single poll can be off in either direction, so the errors of several polls seem as though they should cancel each other out. That works for mistakes that pull in different directions. It does nothing for a mistake that every poll makes in the same direction, because there is nothing to cancel.

Two kinds of mistake sit inside a poll's error. One is random: a given poll lands a little high or a little low for reasons that differ from poll to poll. The other is shared: a pull in one direction that every poll feels. Average several such polls and the random part shrinks, because averaging divides it by the square root of the number of polls. The shared part does not shrink, because nothing in it changes as polls are added. The American Association for Public Opinion Research says the same of models that blend many surveys: "While such models dampen random error, shared biases can still skew results and inflate confidence" (American Association for Public Opinion Research, 2025).

The natural guess is that this shared part is a house lean, one firm's habit that a rival could avoid. A large study says otherwise. Selb et al. (2023), in a peer-reviewed journal, fitted a published model of poll error to 5,240 German national polls from 1994 to 2021 in a multiparty system. The study reports that it found little sign that the bias belonged to particular firms or to particular ways of asking people. Their reading of the cause: "Common biases indicate industry effects due to similar methodological problems." Average absolute election-day bias per party was about 1.5 percentage points, and the estimated variance was about twice what usual margins of error imply. Method choice offers no way out either: firms found respondents in different ways, adjusted the raw numbers in different ways and guessed in different ways who would actually vote, and the 2024 task force found that "No single methodological choice guaranteed more accurate results" (American Association for Public Opinion Research, 2025).

The shared miss is not new, and it has pointed the same way for three presidential elections in a row. The American Association for Public Opinion Research (2025) computed its accuracy figures "using all aggregated polls from the last two weeks of each election", so they describe averages, not single polls. Its evaluation states that "2024 marks the third consecutive presidential election in which Democratic support was systematically overestimated." The Democratic margin was overstated by 3.1 points in 2016, 4.6 in 2020 and 2.7 in 2024. Absolute state error fell over the same period, from 5.7 points in 2016 and 4.8 in 2020 to 3.0 in 2024, but the direction never varied.

The 2024 cycle gives the clearest picture, because the arithmetic can now be run over the whole public record. Working from a crawl of FiveThirtyEight's 2024 poll file (Internet Archive, 2024) and the certified results (Federal Election Commission, 2025), this computation counted 248 qualifying polls from 57 organisations across the national contest and the seven battlegrounds. Of those polls, 223, or 89.9%, overstated the Democratic margin. All eight contest averages erred the same way, with a mean signed error of +2.16 points. The lean was not confined to a handful of firms: 55 of the 57 organisations, 96%, leaned the same way, and all 21 organisations that published four or more polls did.

Adding polls does something, though not the thing a user wants. On the same data, the margin of error that the polls' own spread justifies shrinks as more polls are pooled: 2.01 points with two polls, 1.12 with ten, 0.92 with fifteen. Mean absolute error does not follow it down. It starts at 2.39 for a single poll, sits at 2.18 with three, and holds at 2.16 from five polls onward. So the average looks more precise while staying the same distance from the certified result. The interval reveals the cost: it contains the certified result 42% of the time with two polls, 25% with five, and 1% with fifteen. Pool every poll in a contest and the interval excludes the certified result in all eight. The polls' visible disagreement understates the problem too, since their spread within a contest, 1.30 to 2.65 points, is smaller than the historical total error on a state margin.

Winner calls mostly survived. Barnett and Sarfati (2023), writing in a peer-reviewed statistics journal, evaluated FiveThirtyEight's 2020 state forecasts with a method that allows for polls in similar states to be wrong together. They found the pollsters did well at predicting who would win individual states, including tipping-point states, while underestimating Trump's vote share by a modest and statistically significant amount. They also compared the sophisticated aggregate against a plain Real Clear Politics average and against no aggregation at all, and both fared surprisingly well. A shared bias can be large enough to invalidate an interval and still leave most winner calls intact.

Averaging cancels the part of the error that differs from poll to poll, and in 2024 there was plenty of that. It cannot cancel a mistake that every poll makes together, because cancellation needs disagreement. In 2024 the polls differed only in size. They leaned the same way, so their average leaned with them. All eight contest averages missed the same way, and adding more polls did not reduce the average miss.

So what

For anyone reading a poll average, two questions look like one. How precise is this number? And what would it take for the number to be aimed at the wrong place? Averaging answers the first and says nothing about the second. A tight interval is evidence that the polls agree with each other, not that they are right. When errors are shared, precision and accuracy come apart, and only one of them is visible in the number.

For political parties

A party that wants to plan around the lean has to know which way it will point, and that has not been predictable. The one test after 2022 compared a poll average with 4,494 expert forecasts across three German federal elections. Graefe (2024), in a peer-reviewed journal, reports that "domain knowledge was unable to improve upon the accuracy of polls", that expert error averaged 34% higher, and that "in more than half of the cases, experts’ forecasts pointed in the wrong direction". There is a second limit. This is a presidential-cycle pattern rather than a general law: the task force notes that "signed errors in 2018 and 2022 were small and varied in direction" (American Association for Public Opinion Research, 2025), and all three cycles in the run had the same Republican nominee on the ballot, so it is unclear whether the pattern persists without him.

For government

The shared component is real, and the argument is about how big it is. Tierney and Volfovsky (2024), in a peer-reviewed statistics journal, remodel poll error as a process observed only once, when the votes are counted, and conclude that earlier estimates of presidential polling bias were too extreme by roughly 10%, the Senate figure too extreme by roughly 25%, and estimates of excess variance too large. So the published magnitude is contested downward. The 2024 arithmetic has limits of its own. Nine defensible ways of selecting which polls to use land within 1.42 points of each other here, but the computation remains one country, one office and one cycle, and its interval is the interval the polls' own spread justifies, not the interval any published model reports. That is a description of a season, not an estimate of a standing parameter.

Case studies

That shared miss is visible across the 57 organisations and 248 polls already cited, through 89.9% and 96%, and in eight of eight averages. And the pattern is not only American or only recent: Selb et al. (2023) reached the same verdict, that the bias belongs to the industry rather than to any one firm within it.

References

American Association for Public Opinion Research (2025) Task Force on 2024 Pre-Election Polling: an evaluation of the 2024 general election polls. Chaired by J. Pasek. Alexandria, VA: AAPOR, 29 October. Available at: https://aapor.org/announcements/2024-pre-election-polling-report/ (Accessed: 10 September 2026).

Barnett, A. and Sarfati, A. (2023) 'The polls and the U.S. presidential election in 2020 …. and 2024', Statistics and Public Policy, 10(1), article 2199809. Available at: https://doi.org/10.1080/2330443x.2023.2199809 (Accessed: 10 September 2026).

Federal Election Commission (2025) Official 2024 presidential general election results. Washington, DC: Federal Election Commission. Available at: https://www.fec.gov/resources/cms-content/documents/2024presgeresults.xlsx (Accessed: 10 September 2026).

Graefe, A. (2024) 'Limits of domain knowledge in election forecasting: a comparison of poll averages and expert forecasts', International Journal of Public Opinion Research, 36(1). Available at: https://doi.org/10.1093/ijpor/edae002 (Accessed: 10 September 2026).

Internet Archive (2024) Snapshot of president_polls.csv, FiveThirtyEight polls page, crawled 29 November 2024 [dataset]. Available at: http://web.archive.org/web/20241129175903id_/https://projects.fivethirtyeight.com/polls-page/data/president_polls.csv (Accessed: 10 September 2026). Calculations from this data were made for this article and are available upon request.

Selb, P. et al. (2023) 'Bias and variance in multiparty election polls', Public Opinion Quarterly, 87(4), pp. 1025–1037. Available at: https://doi.org/10.1093/poq/nfad046 (Accessed: 10 September 2026).

Tierney, G. and Volfovsky, A. (2024) 'Bias and excess variance in election polling: a not-so-hidden Markov model', Journal of the Royal Statistical Society Series A: Statistics in Society, 188(2), pp. 566–582. Available at: https://doi.org/10.1093/jrsssa/qnae066 (Accessed: 10 September 2026).