We’ve all heard of the so-called ‘vaccine bounce’.
The story goes something like this: unlike its neighbours in the EU, the UK has rolled out coronavirus vaccinations at impressive pace, and as a result, Boris Johnson’s Conservative Party has enjoyed an uptick in support.
This is the latest in a long line of stories that we have used to make sense of changes in the polls. But though these stories are compelling – and might even be true – we should take them with a hefty pinch of salt.
Put simply, most short-term changes in the polls mean nothing at all. They are random and arise only due to minor differences in the make-up of each poll’s pool of respondents. This is one reason why all polls come with some ‘margin of error’.
To see why this matters, let’s imagine that we could rerun recent polls over and over again. We can think of each run of polls as comprising their own separate ‘polling-verse’. If we did this, we would expect each party’s average level of support to stay the same across polling-verses.
The Conservatives would be just ahead of Labour, and the Lib Dems would trail in third place. Yet though we would expect the average level of support to be the same, we would expect day-to-day changes between individual polls to vary from one polling-verse to the next.
While we can’t visit these new polling-verses, we can simulate them. To do this, we fit a statistical model that accounts for each poll’s margin of error, combines it with some reasonable assumptions, then estimates each party’s level of support. We then use this model to produce new, hypothetical polls based on the data from the last few months.
The graph above shows three separate polling-verses. Though each is equally plausible, each also tells a different story.
The first mirrors conventional wisdom: the Conservatives rise, and Labour falls. The second shows a small vaccine bounce, with support for the two parties, then more or less plateauing. And the third shows Labour falling behind the Conservatives, wobbling a little, then catching back up.
Although the difference between each polling-verse is random, we can’t help but learn different lessons from them.
Even without a statistical model like this to map out alternative trends, we easily fall into the trap of finding these stories in randomness.
This is because the stories that we tell are just another model of reality. Models are in our minds too, but these models are almost always ‘overfit’: they find it hard to distinguish signal from noise and end up attributing meaning to pure chance.
The way in which we consume polls serves to exacerbate this problem. Often, the media reports polling figures as a single, precise value. But these figures are not exact, they are probabilistic.
Consider this recent poll putting Conservative support at 44%. A four-point margin of error on this figure implies that the real level of support for the party might reasonably be as low as 40% or as high as 48%. The true signal is much more uncertain than we might first believe.
Crucially, we are told that these figures also represent an increase of two percentage points in the Conservative vote, while Labour has stagnated at 36%. This encourages us to overfit our mental model to look more like Polling-Verse 1 or 2 than 3.
Yet we know that this supposed ‘change’ since the last poll is well within the margin of error of the two polls. Indeed, accounting for the margin of error, it becomes clear that in an alternative polling-verse this could even look like a decrease in Conservative support.
The fact that the polls instead found an increase could be entirely down to chance. The story we tell does not recognise this happenstance. Our model is overfit to the data we just so happened to observe.
If we look at our three polling-verses, it is easy to see why people do this. In the first, the momentum is with the Conservatives.
In the second, it’s with neither party. And in the third, it’s with Labour. In all cases, these stories help us to extrapolate the trends forward to guess what is likely to happen next.
These expectations have the allure of scientific objectivity, but we may have just been ‘fooled by randomness’.
Recent comments from Keir Starmer provide an excellent example of this reasoning in real life. The Labour leader pinned the blame for the party’s potential underperformance at the upcoming local elections on the ‘very significant’ vaccine bounce. The Labour leader might be looking for a rational explanation for mere bad luck.
To some, these points might seem pedantic. But ignoring them can have troubling consequences. Some evidence suggests that reports emphasising a party’s momentum in the polls can lead voters to ‘jump on its bandwagon’.
Stories like the vaccine bounce have the potential to affect the way we vote. Yet these stories are often just one among many potential efforts to impose reason on randomness.
So how might we make sense of this chaos? We provide three simple rules of thumb:
- Consider the polls in unison, not in isolation. If the errors in polls are random, lots of them carried out at a similar time should, when viewed together, more closely approximate the truth.
- Consider the margin of error, not just the headline figure. The full range covered by the margin of error is very likely to include the true underlying support for a party.
- Question the meaning we assign to sudden changes in the polls. The stories polls are used to tell might sometimes be accurate, but they are frequently convenient rationalisations of random changes.
Polling is the most scientific means we have of understanding public opinion, but it is not an exact science. The numbers are uncertain, and the stories we tell about them even more so.
By Jack Bailey, Research Associate at the University of Manchester and Matthew Barnfield, Political Science PhD at Queen Mary University of London.
