General election campaigns provoke a lot of attention and criticism of opinion polls. Some of that is sensible and well-informed… and some of it is not. This is about the latter – a response to some of the more common criticisms that I see on social media. Polling methodology is not necessarily easy to understand and, given many people only take an interest in it at around election time, most people have no good reason to know much about it. This will hopefully address some of the more common misapprehensions (or in those cases where they aren’t entirely wrong, add some useful context).

This Twitter poll has 20,000 responses, TEN TIMES BIGGER than so-called professional polls!

Criticisms about sample size are the oldest and most persistent of polling criticism. This is unsurprising given that it is rather counter-intuitive that only 1000 interviews should be enough people to get a good steer on what 40,000,000 people think. The response that George Gallup, the founding father of modern polling, used to give is still a good one: “You don’t need to eat a whole bowl of soup to know if it’s too salty, providing it’s properly stirred a single spoonful will suffice.”

The thing that makes a poll meaningful isn’t so much the sample size, it is whether it is representative or not. That is, does it have the right proportions of men and women, old and young, rich and poor and so on. If it is representative of the wider population in all those ways, then one hopes it will also be representative in terms of opinion. If not, then it won’t be. If you took a sample of 100,000 middle-class homeowners in Surrey then it would be overwhelmingly Tory, regardless of the large sample size. If you took a sample of 100,000 working class people on Merseyside it would be overwhelmingly Labour, regardless of the large sample size. What counts is not the size, it’s whether it’s representative or not. The classic example of this is the 1936 Presidential Election where Gallup made his name – correctly predicting the election using a representative sample when the Literary Digest’s sample of 2.4 million(!) called it wrongly.

Professional polling companies will sample and weight polls to ensure they are representative. However well intended, Twitter polls will not (indeed, there is no way of doing so, and no way of measuring the demographics of those who have participated).

Who are these pollsters talking too? Everyone I know is voting for party X!

Political support is not evenly distributed across the country. If you live in Liverpool Walton, then the overwhelming majority of other people in your area will be Labour voters. If you live in Christchurch, then the overwhelming majority of your neighbours will likely be Tory. This is further entrenched by our tendency to be friends with people like us – most of your friends will probably be of a roughly similar age and background and, very likely, have similar outlooks and things in common with you, so they are probably more likely to share your political views (plus, unless you make pretty odd conversation with people, you probably don’t know how everyone you know will vote).

An opinion poll will have sought to include a representative sample of people from all parts of the country, with a demographic make-up that matches the country as a whole. Your friendship group probably doesn’t look like that. Besides, unless you think that literally *everyone* is voting for party X, you need to accept that there probably are voters of the other parties out there. You’re just not friends with them.

Polls are done on landlines so don’t include young people

I am not sure why this criticism has resurfaced, but I’ve seen it several times over recent weeks, often widely retweeted. These days the overwhelming majority of opinion polls in Britain are conducted online rather than by telephone. The only companies who regularly conduct GB voting intention polls by phone are Ipsos MORI and Survation. Both of them conduct a large proportion of their interviews using mobile phones.

Polls of single constituencies are still normally conducted by telephone but, again, will conduct a large proportion of their calls on mobile phones. I don’t think anyone has done a voting intention poll on landlines only for well over a decade.

Who takes part in these polls? No one has ever asked me

For the reason above, your chances of being invited to take part in a telephone poll that asks about voting intention are vanishingly small. You could be waiting many, many years for your phone number to be randomly dialled. If you are the sort of person who doesn’t pick up unknown numbers, they’ll never be able to reach you.

Most polls these days are conducted using internet panels (that is, panels of people who have given pollsters permission to email them and ask them to take part in opinion polls). Some companies like YouGov and Kantar have their own panels, other companies may buy in sample from providers like Dynata or Toluna. If you are a member of such panels you’ll inevitably be invited to take part in opinion polls. Though of course, remember that the vast majority of surveys tend to be stuff about consumer brands and so on… politics is only a tiny part of the market research world.

The polls only show a lead because pollsters are “Weighting” them, you should look at the raw figures

Weighting is a standard part of polling that everyone does. Standard weighting by demographics is unobjectionable – but is sometimes presented as something suspicious or dodgy. At this election, this has sometimes been because it has been confused with how pollsters account for turnout, which is a more controversial and complicated issue which I’ll return to below.

To deal with ordinary demographic weighting though, this is just to ensure that the sample is representative. So for example – we know that the adult British population is about 51% female, 49% male. If the raw sample a poll obtained was 48% female and 52% male then it would have too many men and too few women and weighting would be used to correct it. Every female respondent would be given a weight of 1.06 (that is 51/48) and would count as 1.06 of a person in the final results. Every male respondent would be given a weight of 0.94 (that is 49/52) and would count as 0.94 of a person in the final results. Once weighted, the sample would now be 51% female and 49% male.

Actual weighting is more complicated that this because samples are weighted by multiple factors – age, gender, region, social class, education, past vote and so on. The principle however is the same – it is just a way of correcting a sample that has the wrong amount of people compared to the known demographics of the British population.

Polls assume young people won’t vote

This is a far more understandable criticism, but one that is probably wrong.

It’s understandable because it is part of what went wrong with the polls in 2017. Many polling companies adopted new turnout models that did indeed make assumptions about whether people would vote or not based upon their age. While it wasn’t the case across the board, in 2017 companies like ComRes, ICM and MORI did assume that young people were less likely to vote and weighted them down. The way they did this contributed to those polls understating Labour support (I’ve written about it in more depth here)

Naturally people looking for explanations for the difference between polls this time round have jumped to this problem as a possible explanation. This is where it goes wrong. Almost all the companies who were using age-based turnout models dumped those models straight after the 2017 election and went back to basing their turnout models primarily on how likely respondents say they are to vote. Put simply, polls are not making assumptions about whether different age groups will vote or not – differences in likelihood to vote between age groups will be down to people in some age groups telling pollsters they are less likely to vote than people in other age groups.

The main exception to this is Kantar, who do still include age in their turnout model, so can fairly be said to be assuming that young people are less likely to vote than old people. They kept the method because, for them, it worked well (they were one of the more accurate companies at the 2017 election).

Some of the criticism of Kantar’s turnout model (and of the relative turnout levels in other companies’ polls) is based on comparing the implied turnout in their polls with turnout estimates published straight after the 2017 election, based on polls done during the 2017 campaign. Compared to those figures, the turnout for young people may look a bit low. However there are much better estimates of turnout in 2017 from the British Election Study, which has validated turnout data (that is, rather than just asking if people voted, they look their respondents up on the marked electoral register and see if they actually voted) – these figures are available here, and this is the data Kantar uses in their model. Compared to these figures the levels of turnout in Kantar and other companies’ polls look perfectly reasonable.

Pollster X is biased!

Another extremely common criticism. It is true that some pollsters show figures that are consistently better or worse for a party. These are know as “house effects” and can be explained by methodological differences (such as what weights they use, or how they deal with turnout), rather than some sort of bias. It is in the strong commercial interests of all polling companies to be as accurate as possible, so it would be self-defeating for them to be biased.

The frequency of this criticism has always baffled me, given to anyone in the industry it’s quite absurd. The leading market research companies are large, multi-million pound corporations. Ipsos, YouGov and WPP (Kantar’s parent company) are publicly listed companies – they are owned by largely institutional shareholders and the vast bulk of their profits are based upon non-political commercial research. They are not the personal playthings of the political whims of their CEOs, and the idea that people like Didier Truchot ring up their UK political team and ask them to shove a bit on the figure to make the party they support look better is tin-foil hat territory.

Market research companies sell themselves on their accuracy, not on telling people what they want to hear. Political polling is done as a shop window, a way of getting name recognition and (all being well) a reputation for solid, accurate research. They have extremely strong commercial and financial reasons to strive for accuracy, and pretty much nothing to be gained by being deliberately wrong.

Polls are always wrong

And yet there have been several instances of the polls being wrong of late, though this is somewhat overegged. The common perception is that the polls were wrong in 2015 (indeed, they nearly all were), at the 2016 referendum (some of them were wrong, some of them were correct – but the media paid more attention to the wrong ones), at Donald Trump’s election (the national polls were actually correct, but some key state polls were wrong, so Trump’s victory in the electoral college wasn’t predicted), and in the 2017 election (most were wrong, a few were right).

You should not take polls as gospel. It is obviously possible for them to be wrong – recent history demonstrates that all too well. However, they are probably the best way we have of measuring public opinion, so if you want a steer on how Britain is likely to vote it would be foolish to dismiss them totally.

What I would advise against is assuming that they are likely to be wrong in the same direction as last time, or in the direction you would like them to be. As discussed above – the methods that caused the understatement of Labour support in 2017 have largely been abandoned, so the specific error that happened in 2017 is extremely unlikely to reoccur. That does not mean polls couldn’t be wrong in different ways, but it is worth considered that the vast majority of previous errors have been in the opposite direction, and that polls in the UK have tended to over-state Labour support. Do not assume that polls being wrong automatically means under-stating Labour.


965 Responses to “How not to interpret opinion polls”

1 2 3 20
  1. I’ve heard all those.

  2. That makes for depressing reading as a Labour voter. Maybe the Tories really are streets ahead, going by the largest Tory leads.

  3. Easy blog post for Anthony. He recycles it before every election. Because he has to!

  4. Sadly, those most in need of Anthony’s lesson above will never see it – but it’s an excellent revision note for those of us who have forgotten some of what he taught us last time!

  5. “Polls are done on landlines so don’t include young people
    I am not sure why this criticism has resurfaced”

    It’s because this does still happen in the US, where political polling is dominated by telephone methodology, and people see criticism of it there and assume that it’s also the case here.

  6. Not the best read for Labour supporters!

  7. I don’t pick up my phone to witheld numbers anyway and I don’t know many who do.

  8. I’ve been called up by a polling company.

    Around 2015

  9. I’ve been called up by a polling company.

    Around 2015

  10. Weighting: it’s fine when it’s done with weights of e.g. 0.94 and 1.06. But weighting can be a source of error when done on poorly recruited samples such that the multiplier is too high. For instance, when ethnicity is used as a weight, or if interlocking weighting is used, occasionally you have polls where one individual is worth 5 or more respondents. This introduces a lot of variance that isn’t accounted for by the conventional “+-3%” rule.

    In normal, commercial market research, we try to avoid weights that are less than 0.5 or greater than 2. The frequency of political polling, and the intricacy of some of their weighting schemes, means that political polls seem to breach those barriers more often than would happen in normal, commercial market research.

  11. Great blog AW.
    @RAF
    Not totally recycled because of references to 2017 GE.
    —————————
    @John33
    “I don’t pick up my phone to witheld numbers anyway and I don’t know many who do.”

    Haven’t you just fallen into one of Anthony’s classic mistakes? :)

  12. @John33

    I don’t think the Tories are streets ahead.

    Depending on how the polls are interpreted they could be close to any of the following elections:

    1979, 1987, 1992 or 2017.

    My own guess is somewhere between 1992 and 1979 but who knows!

  13. Labour announcing train ticket reductions today out of the blue following the Waspi announcement out of the blue showed to me they are struggling to gain traction, expect more giveaways that aren’t in the manifesto between now and election day.

  14. @ jonesinbangor

    Was that in 2015 or at 2015? :)

  15. CHRIS GREEN
    “Polls are done on landlines so don’t include young people
    I am not sure why this criticism has resurfaced”

    It’s because this does still happen in the US, where political polling is dominated by telephone methodology, and people see criticism of it there and assume that it’s also the case here.

    does this contradict or support Trumps win?

  16. Pete B

    I was just musing that it must be hard to get hold of certain demographics on the phone.

  17. Bantams
    “…expect more giveaways that aren’t in the manifesto between now and election day.”

    Nobody I know believes a word of it, therefore no-one in the country does. :)

  18. This is a neat summary of the unjustified criticisms (fallacies really) of polls. Even if it is repeated, it is worth the repetition .

    [But it is not an excuse for not updating the demographics in terms of behavioural changes.]

  19. @Bantams

    Irrespective of the merits of this approach, these will be pre – planned as a part of a strategy to dominate the headlines and maintain momentum into the latter stages of the campaign. They shot themselves in the foot by making the initial manifesto such a bonanza and now people are either numb to the offers or find them even harder to take seriously. Not the best strategy in my view.

  20. John33
    “I was just musing that it must be hard to get hold of certain demographics on the phone.”

    I know, I was just pulling your leg. There’s problems with all methods of contacting people. For instance, only certain kinds of people will sign up for online panels. Probably a higher percentage on here than in the general population though.

  21. Some comment on the previous thread about the C4 focus group. Worth recalling that there were lots of focus group type things in 2017, and they were mostly highly misleading when looking for any pointers to the result.

    Saddened to see some of the respondents actually thinking that there will be a Brexit Boom shortly. These people have votes, but they don’t even understand the basics of the Brexit timetable, let alone what effects it wll have.

    On NATO; Again, those with little understanding think that the EU somehow threatens NATO and the US is what matters. Well the EU is legally bound by multiple treaty clauses that specifically state they are not legally permitted to do anything that undermines NATO, while the US has just announced it’s cutting it’s NATO spending, to a level on a par with Germany. As NATO chiefs gather, many are privately suggesting that a Trump re-election could spell the end of NATO.

    Yet some people think the EU is the threat and we need to get closer to the US?

  22. @ Pete B

    I overheard a group of primary school kids calling Boris ‘ a big poo poo head’ so there is no way Tories are winning this election!

  23. “This is a neat summary of the unjustified criticisms (fallacies really) of polls. Even if it is repeated, it is worth the repetition.”

    ——-

    Maybe if AW could do another post covering all the justified criticisms of polls, that would give some symmetry.

  24. John33
    That’s obviously going to be the deciding factor!

  25. This article is good. But it is worth adding that online polling does not automatically generate a random sample. Somehow people have to be selected and there is then sample selection bias.

  26. Interesting! Reuters have a scoop tonight suggesting the documents used by Corbyn when he accused the USA and UK of collaborating over privatising the NHS might be slightly dodgy.

    I’ve copied and pasted the article below:

    “The leak and distribution of classified British-U.S. trade documents online resembles a disinformation campaign uncovered this year that originated in Russia, according to experts who say it could signal foreign interference in Britain’s election.

    The opposition Labour Party said on Nov. 27 the classified documents, which first appeared online on Oct. 21, showed the ruling Conservatives were plotting to offer up the state-run National Health Service for sale in trade talks with Washington.

    The NHS is much loved by Britons and has become an important issue in the Dec. 12 election, in which Labour trails the Conservatives despite cutting its lead in some opinion polls.

    Researchers at Britain’s Oxford and Cardiff universities, the Atlantic Council thinktank and social media analytics firm Graphika said the way the documents were first shared online mirrored a campaign called Secondary Infektion.

    Secondary Infektion uncovered by the Atlantic Council in June, used fabricated or altered documents to try to spread false narratives across at least 30 online platforms, and stemmed from a network of social media accounts which Facebook said “originated in Russia.”

    “It’s on the same set of websites (as Secondary Infektion), it’s using the same types of accounts and making the same language errors. It’s either the Russian operation or someone trying hard to look like it,” said Ben Nimmo, head of investigations at Graphika.

    Reuters has been unable to verify whether the documents are genuine. Labour and the British government declined immediate comment. In Washington, the U.S. Trade Representative did not respond to requests for comment.

    It is not clear who was behind either operation and cyber experts say it is hard to attribute malicious actions online with certainty.

    Moscow has denied allegations of election meddling and the Kremlin did not immediately respond to a request for comment.”

  27. Journalists at Open Democracy have been working hard. First, Mary Fitzgerald has a piece about Lib Dem dishonesty. It seems a dishonest claim about a non-existent email was made in order to apply pressure on OD to withdraw or alter an article it had done.

    https://www.opendemocracy.net/en/opendemocracyuk/what-are-jo-swinsons-liberal-democrats-so-desperate-to-hide/

    Now Open Democracy is looking at all the crap being directed at voters in Bristol.

    https://www.opendemocracy.net/en/dark-money-investigations/online-falsehoods-and-dark-money-targeting-bristol-voters/

  28. @Bantams

    Surely if the documents were dodgy the Conservatives would have said so? Would certainly have knocked the wind out of Labour’s sails a bit. Unless they’re waiting until later on in the campaign?

  29. @Bantams

    Surely if the documents were dodgy the Conservatives would have said so? Would certainly have knocked the wind out of Labour’s sails a bit. Unless they’re waiting until later on in the campaign?

  30. @Fred – It’s neither, unconnected to Trump’s win. The national US polls were actually pretty accurate in 2016. They just didn’t account properly for the Electoral College. Analysts there have been warning for a while that there are too few state-level polls done, so this is an increasing problem.

    Trump’s win is relevant to the UK though, because people are more critical of a tiny polling error in a knife-edge race (e.g. 2016 US or 2017 UK – or 2016 UK referendum, for that matter) than of a huge error in a landslide (e.g. 1997 UK). If the polls had been out by a greater amount in 2016 and 2017 but had called the simplistic binary result “correctly”, then there wouldn’t be such criticism of polling today.

  31. @Alec
    if we’re going back to the previous I remember that there was some surprise that some of the focus group said that they thought Boris was a man of his word. I can see that for the left-leaning posters on here that could be unimaginable, but let me try to give some insight into why they might think that.

    1. He said he would get a deal. The world and his wife said that it was impossible (including Barnier). Whether or not you think it is any good, he did indeed come back with a revised deal.
    2. He warned Tory rebels that if they voted against one of the votes, he would sack them. 21 of them still rebelled and he did duly sack them, even though that made governing impossible. I certainly can’t remember such decisive action against his own MPs by a PM.

    Now I know that he has not always been entirely accurate on details and I’m sure people can point to statements that they think are l!es, but those two points above did resonate with a lot of people.

  32. @PeteB

    “For instance, only certain kinds of people will sign up for online panels. Probably a higher percentage on here than in the general population though.”

    I signed up to YG a while back. After a month of consumer research twaddle, and zero political stuff, I cancelled. I can’t be bothered with ads, spam, consumer stuff, and I avoid sales people in shops. I’m that sort of person.

    I was hoping to become 1 of 1,001 (or 1 of 130 if talking Scottish CBs), and it didn’t happen. Some might consider a month to be impatient, but with 20 or so polls in that month, it wasn’t crazy. Had YG been a little less spammy, I might have stuck around longer.

    Maybe polling companies can invite people to be part the ‘politics only’ aspect of research, and maybe gain a few samples. Not sure I like the sound of being called ‘a sample’ though.

  33. DRBASIL

    That was my thought when reading Bantams post. Besides, Labour already had a copy, just with redacted bits so they obviously compared the two. Seems highly implausible to me.

  34. @ DrBasil

    The same occured to me. The Government might not have seen the private notes of their representatives, only written up conclusions. I’ve really no idea.

  35. Let’s not get too hung up with Anthony’s comments. Pollsters do get it badly wrong. I well remember YouGov’s final poll on the eve of the 2017 GE showing a surge in the Tory lead from 3 points to 7 literally in one day.

    Only Survation nailed it last time around. All other pollsters.were forecasting a Tory majority.
    So let’s see what happens, We May now have an era of shy Labour voters who are uncomfortable saying they are voting for Corbyn’s party.

    So as a Labour voter nothing has changed here. All the weighting in the world does not equate to getting it right.

  36. @ Bantams

    If only the Russian interference report had been released we might all be in a better position to judge. A bit tricky for Cons to make hay with this

  37. Mike Pearce

    I am a shy Labour voter in Solihull. Mainly because the parents of the kids I tutor would pull them out if they knew what a rabid leftie I am!!

  38. @John33

    The information had been out in the public domain for over a month before Corbyn used it. As you said Labour already had a redacted version but someone could have just altered the original slightly and put a different slant on it.

  39. @Mike Pearce – You don’t know that Survation nailed it in 2017. For all we know it was just that the magnitudes of their errors happened to cancel each other out, and the other firms’ errors didn’t.

  40. “Let’s not get too hung up with Anthony’s comments. Pollsters do get it badly wrong.”

    —–

    Indeed Anthony is acknowledging they might be wrong, it’s just that he suggests they might be wrong in new ways, as opposed to being wrong in the same way as last time.

  41. Chris Green

    Baffled by your comment. Survation had the National share of the vote for Con and Lab either spot on or incredibly close. That’s good enough for me. If it isn’t for you then so be it.

  42. I would say that there is no strong a priori reason why using online polls is better in terms of sample selection bias then using phone polls. In each case there may be systematic differences in “unobservable types” of person that ends up participating.

    As for turnout Mr Wells states:

    “Almost all the companies who were using age-based turnout models dumped those models straight after the 2017 election and went back to basing their turnout models primarily on how likely respondents say they are to vote. ”

    But self reported turnout is not guaranteed to be accurate. For instance young people may be more likely to say they won’t vote, and then end up voting, etc etc.

  43. @JOHN33

    Agreed, does seem rather implausible.

    @BANTAMS

    Perhaps. Surely they would have checked once the story broke though? Can’t imagine they would have allowed this to fester if there was evidence to the contrary.

    @MIKE PEARCE

    Agreed, no-one really knows what’s going to happen. I suppose the hope for any Labour supporter is that some of the polls last time were so wildly inaccurate it seems reasonable to suggest that there may have been wider problems with the samples themselves, not just with the various weighting methods they use. I’ve seem some ponder as to whether there is a cohort of (Labour) voters who the polling companies cannot seem to reach. If that is the case anything could happen.

  44. @Bantams – the only problem with the Russia conspiracy theory is that the Kremlin wants the Conservatives to win.

    As others have pointed out, if it was a conspiracy and the documents are fake, that would have emerged in 5 minutes.

    Given that Russia desperately wants to weaken the EU and the UK and so fervently supports Brexit, the conspiracy here is actually the innuendo that the leaks are actually a fake?

    @Pete B – yes, I’ve noticed a few mentions of ‘Boris is a man of his word’. Doesn’t fit with wider polling, where his trust rating has fallen, but some still believe.

    There is a substantial revision going on though, in that few doubted Johnson could get a deal. It is Johnson himself who is creating that myth. The main point was that people said he wouldn’t get a deal without the backstop, which is what he promised, and on this, they were absolutely correct.

    With a compliant press, establishing these myths is pretty easy. The backstop is far more damaging to the UK’s integrity than May’s version, and will break the UK up, but few seem troubled by the fact that Johnson’s central promise on Brexit has been a complete failure.

    Such is politics.

  45. Carfrew

    Indeed, In fairness to YouGov the constituency polling in 2017 was a work of art.

  46. Chris Green

    Being the geek that I am, I looked below the surface at the pollsters’ methodologies during the campaign GE2017.

    It was clear to me BEFORE the election that Survation were the ones weighting correctly etc.

    When the results came in, it merely confirmed my worst fears.

  47. Chris Green

    Being the geek that I am, I looked below the surface at the pollsters’ methodologies during the campaign GE2017.

    It was clear to me BEFORE the election that Survation were the ones weighting correctly etc.

    When the results came in, it merely confirmed my worst fears.

  48. MartinW fpt
    I use a university gym and the changing rooms are already much quieter than a couple of weeks ago (when there were notices up reminding people to register to vote).

    I’m not sure when the term officially finishes, but I do know the university uses a semester system, under which – I think – most students comes back after Xmas to end of semester exams, so it’s possible they disappear extra-early because the last week or so of term is officially dedicated to revision in some courses.

  49. If it was a pdf file of the redacted veraion, it is surprisingly easy to remove the redaction (in spite of the claims contrary).

    But I agree that there are many surprising mysteries about the files (not last that nobody really had a go at evaluating them – or I i.missed it), but if they were fake, it would have been already stated.

  50. @MIKE PEARCE

    In support of Chris here, I think the point is that whilst Survation certainly predicted it correctly, you cannot be sure that the underlying reasons for that were correct. For example (and purely hypothetically), they may have simply overestimated the youth vote potentially making up for an underlying sample that could have been skewed in favour of the Tories.

1 2 3 20