Polling myths

Whenever a poll goes up that shows bad news for someone you get the same sort of comments on social media. As I write this piece in May 2017 comments like these generally come from Jeremy Corbyn supporters, but that’s just the political weather at this moment in time. When the polls show Labour ahead you get almost exactly the same comments from Conservative supporters, when UKIP are doing badly you get them from UKIP supporters, when the Lib Dems are trailing you get them from Lib Dem supporters.

There are elements of opinion polling that are counter-intuitive and many of these myths will sound perfectly convincing to people who aren’t versed in how polls work. This post isn’t aimed at the hardcore conspiracists who are beyond persuasion – if you are truly convinced that polls are all a malevolent plot of some sort there is nothing I’ll be able to do to convince you. Neither is it really aimed at those who already know such arguments are nonsense: this is aimed at those people who don’t really want to believe what the polls are saying, see lots of people on social media offering comforting sounding reasons why you can ignore them, but are thinking, “Is that really true, or is it rather too convenient an excuse for waving away an uncomfortable truth…”

1) They only asked 1000 people out of 40 million. That’s not enough

This question has been about for as long as polling has. George Gallup, the trailblazer of modern polling, used to answer it by saying that it wasn’t necessary to eat a whole bowl of soup to know whether or not it was too salty, providing it had been stirred, a single spoonful was enough. The mention of stirring wasn’t just Gallup being poetic, it’s vital. Taking a single spoonful from the top of a bowl of soup might not work (that could be the spot where someone just salted it), but stirring the soup means that spoonful is representative of the whole bowl.

What makes a poll representative is not the size of the sample, it is its representativeness. You could have a huge sample size that was completely meaningless. Imagine, for example, that you did a poll of 1,000,000 over 65s. It would indeed be a huge sample, but it would be very skewed toward the Tories and Brexit. What makes a poll meaningful or not is whether it is representative of the country. Does it have the correct proportions of men and women? Old and young? Middle class and working class? Graduates and non-graduates? If the sample reflects British society as a whole in all these ways, then it should reflect it in terms of political opinion too. A poll of 1000 people is quite enough to get a representative sample.

The classic example of this was at the very birth of modern polling – in the US 1936 Presidential election a magazine called the Literary Digest did a survey of over two million people, drawn from magazine subscribers, telephone directories and so forth. It showed Alf Landon would win the Presidential election. The then newcomer George Gallup did a far, far smaller poll properly sampled by state, age, gender and so on. He correctly showed a landslide for Roosevelt. A poll with a sample skewed towards people wealthy enough to have phones and magazines in depression era America was worthless, despite have two million respondents.

2) Who do they ask? I’ve never been asked to take part in a poll!

Sometimes this is worked up to “…and neither has anyone I’ve met”, which does raise the question of whether the first thing these people do upon being introduced to a new person is to ask if MORI have ever rung them. That aside, it’s a reasonable question. If you’ve never been polled and the polls seem to disagree with your experience, where do all these answers come from?

The simple answer is that pollsters obtain their samples either by dialling randomly generated telephone numbers or by contacting people who are members of internet panels. Back when polls were mostly conducted by telephone the reason you had never been polled was simple maths – there were about forty million adults in Britain, there were about fifty or so polls of voting intention of a thousand people conducted each year. Therefore in any given year you had about a 0.1% chance of being invited to take part in a poll.

These days most opinion polls are conducted using online panels, but even if you are a member of a panel, your chances of being invited to a political poll are still relatively low. Most panels have tens of thousands of people (or for the better known companies, hundreds of thousands of people) and 95% of surveys are about commercial stuff like brands, pensions, grocery shopping and so on. You could still be waiting some time to be invited to a political one.

3) But nobody I know is voting for X!

We tend to know and socialise with people who are quite like ourselves. Our social circles will tend to be people who live in the same sort of area as us, probably people who have a similar sort of social status, a similar age. You probably have a fair amount in common with your friends or they wouldn’t be your friends. Hence people we know are more likely than the average person to agree with us (and even when they don’t, they won’t necessarily tell us; not everyone relishes a political argument). On social media it’s even worse – a large number of studies have shown that we tend to follow more people we agree with, producing self-reinforcing bubbles of opinion.

During the Labour leadership contest almost every one of my friends who is a member of the Labour party was voting for Liz Kendall. Yet the reality was that they were all from a tiny minority of 4.5% – it’s just that the Labour party members I knew all happened to be Blairite professionals working in politics in central London. Luckily I had proper polling data that was genuinely reflective of the whole of the Labour party, so I knew that Jeremy Corbyn was in fact in the lead.

In contrast to the typical friendship group, opinion polls samples will be designed so that they reflect the whole population and don’t fall into those traps. They will have the correct balance of people from all across the country, will have the correct age range, will have the correct balance of social class and past vote and so on. Perhaps there are people out there who, by some freak co-incidence, have a circle of acquaintances who form a perfectly representative sample of the whole British public, but I doubt there are very many.

4) Pollsters deliberately don’t ask Labour/Conservative supporters

In so far as there is any rationale behind the belief, it’s normally based upon the perception that someone said they were going to vote for x in a poll, and weren’t asked again. As we’ve seen above, it’s a lot more likely that the reason for this is simply that it’s relatively rare to be invited to a political poll anyway. If you’ve been asked once, the chances are you’re not going to be asked again soon whatever answers you gave.

Under the British Polling Council rules polling companies are required to publish the details of their samples – who was interviewed, what the sample was weighted by and so on. These days almost every company uses some form of political sampling or weighting to ensure that the samples are politically representative. Hence in reality pollsters deliberately include a specific proportion of 2015 Labour supporters in their polls, generally the proportion who did actually vote Labour in 2015. Pollsters are required to report these figures in their tables, or to provide them on request. Hence, if you look at last weekend’s Opinium poll you’ll find that 31% of people in the poll who voted in 2015 voted Labour, the proportion that actually did, if you look at the ICM poll you’ll find that 31% of the people who voted at the last election say they voted Labour, the proportion that actually did, and so on with every other company.

5) Pollsters are biased, and fix their figures

Again, this an accusation that is as old as polling – if you don’t like the message, say the person making it is biased. It’s made easier by the fact that a lot of people working in political polling do have a background in politics, so if you want to look for someone to build a conspiracy theory upon, you don’t need to look far. Over the years I think we’ve been accused of being biased towards and against every party at one time or another – when Labour were usually ahead in the polls YouGov used to be accused of bias because Peter Kellner was President. When the Conservatives were ahead different people accused us of being biased because Stephen Shakespeare was the CEO. The reality is, of course, that polling companies are made up of lots of people with diverse political views (which is, in fact, a great benefit when writing questions – you can get the opinion of colleagues with different opinions to your own when making sure things are fair and balanced).

The idea that polling companies would bias their results to a particular party doesn’t really chime with the economics of the business or the self-interest of companies and those who run them. Because political polls are by far the most visible output of a market research company there is a common misapprehension that it brings in lots of money. It does not. It brings in very little money and is often done as a loss-leader by companies in order to advertise their wares to the commercial companies that spend serious money doing research on brand perceptions, buying decisions and other consumer surveys. Voting intention polls are one of the very few measures of opinion that get checked against reality – it is done almost entirely as a way of the company (a) getting their name known and (b) demonstrating that their samples can accurately measure public opinion and predict behaviour. Getting elections wrong, however, risks a huge financial cost to market research companies through reputational damage and, therefore, huge financial cost to those running them. It would be downright perverse to deliberately get those polls wrong.

6) Polls always get it wrong

If the idea that polling companies would ruin themselves by deliberately getting things wrong is absurd, the idea that polls can get it wrong by poor design is sadly true: polls obviously can get it wrong. Famously they did so at the 2015 general election. Some polls also got Brexit wrong, though the picture is more mixed that some seem to think (most of the campaign polls on Brexit actually showed Leave ahead). Polls tend to get it right a lot more often than not though – even in recent years, when their record is supposed to have been so bad, the polls were broadly accurate on the London mayoral election, the Scottish Parliamentary election, the Welsh Assembly election and both of the Labour party leadership elections.

Nevertheless, it is obviously true to say that polls can be wrong. So what’s the likelihood that this election will be one of those occasions? Following the errors of the 2015 general election the British Polling Council and Market Research Society set up an independent inquiry into the polling error and what caused it, under the leadership of Professor Pat Sturgis at Southampton University. The full report is here, and if you have some spare time and want to understand how polling works and what can go wrong with them it is worth putting aside some time to read it. The extremely short version is, however, that the polls in 2015 weren’t getting samples that were representative enough of the general public – people who agreed to take part in a phone poll, or join an internet panel weren’t quite normal, they were too interested in politics, too engaged, too likely to vote.

Since then polling companies have made changes to try and address that problem. Different companies have taken different approaches. The most significant though are a mix of adding new controls on samples by education and interest in politics and changes to turnout models. We obviously won’t know until the election has finished whether these have worked or not.

So in that context, how does one judge current polls? Well, there are two things worth noting. The first is that while polls have sometimes been wrong in the past, their error has not been evenly distributed. They have not been just as likely to underestimate Labour as they have been to overestimate Labour: polling error has almost always overstated Labour support. If the polls don’t get it right, then all previous experience suggests it will be because they have shown Labour support as too *high*. Theoretically polls could have tried too hard to correct the problems of 2015 and be overstating Conservative support, but given the scale of the error in 2015 and the fact that some companies have made fairly modest adjustments, that seems unlikely to be the case across the board.

Secondly is the degree of error. When polls are wrong they are only so wrong. Even those elections where the polls got it most wrong, like 1992 and 2015, their errors were nowhere near the size of the Conservative party’s current lead.

Short version is, yes, the polls could be wrong, but even the very worst polls have not been wrong enough to cancel out the size of lead that the Tories currently have and when the polls have been that wrong, it’s always been by putting Labour too high.

So, if you aren’t the sort to go in for conspiracy theories, what comfort can I offer if the polls aren’t currently showing the results you’d like them to? Well, first the polls are only ever a snapshot of current opinion. They do not predict what will happen next week or next month, so there is usually plenty of time for them to change. Secondly, for political parties polls generally contain the seeds of their salvation, dismissing them misses the chance to find out why people aren’t voting for you, what you need to change in order to win. And finally, if all else fails, remember that public opinion and polls will eventually change, they always do. Exactly twenty years ago the polls were showing an utterly dominant Labour party almost annihilating a moribund Tory party – the pendulum will likely swing given enough time, the wheel will turn, another party will be on the up, and you’ll see Conservative party supporters on social media trying to dismiss their awful polling figures using exactly the same myths.

117 Responses to “Polling myths”

1 2 3
  1. Rob

    “On France message is clear: it takes a centrist to defeat the hard right (who are often supported- when push comes to shove- by the hard left).”

    As it is in the French context – The CP is called the party of the executed there, while many centrists happily collaborated with the far right. So, to put it mildly, quite a problematic statement.

    As a matter of fact LePen has significant backing from big business – it is a different matter that radicalised people can vote against their own interest (for the far right) and that the left is occasionally captured by radicalised philistines (such as Mélenchon or Corbyn).

  2. Dave – if two people who are demographically identical to each other in every way (and in ways like political interest and past vote), vote differently depending on if they are online or not, then it’s an issue, albeit one that can only really be got round using mixed-mode, which brings it’s own complicated issues. Its gets to a point, however, where you have to accept that any method isn’t perfect, and have to accept those limitations.

    Age/education correlation is quite right. We interlock the two together (i.e. we have a quota for young people with a degree, middle-aged people with a degree, middle-aged people with equiv of A-Levels, etc, etc). It is, indeed, absolutely crucial as you can’t make up for having too few young non-graduates by over-representing elderly non-graduates.

  3. I am beginning to wonder if Juncker et al are actually involved in a double bluff? They know that posing EU-UK conflict will not undermine May but probably help to consolidate her support with the British electorate. Perhaps they wouldn’t mind her having an overwhelming mandate because then they could go on to prove just how powerless and irrelevant they regard the wishes of the British electorate. A bit like the way they have cowed the Greeks “you can vote as you like but as long as we can do what we like” to misquote Frederick the Great! Yanis Yuroufakis may have a point when he advises us not to negotiate at all.

  4. Trevor,

    1/ Is the GE a proxy for IndyRef2 – high/low factor

    2/ Does desired timing of IndyRef2 have a factor – SNP want it sooner, other parties later/never

    3/ Is Brexit even relevant?

    The answer is I just don’t know.

    Most but not all SNP supporters want a second referendum, want it soon and backed Remain.

    On the leave side, unsurprisingly most don’t want a second referendum and were more likely than average to vote leave.

    As to how much that shapes this vote or the balance between Yes/No and Leave/Remain will pan out and how it plays with the undecideds and less political I don’t know.

    I feel Brexit will be the main issue here too as apart from anything else, with the other main issues shaping up to be the NHS, Education and social care, they are all devolved anyway.


  5. @ PETER – thanks

    You’ve confirmed my suspicions and in terms of predicting seats it doesn’t make too much difference from an “aggregate shift” – if anything it reduces the marginal swing factor in some as obviously high YES/Remain areas are generally “safe” SNP seats.

    The “pockets” of LibDem support are very visible in Scotland. It looks like they might go up from 1 seat to 3 seats despite dropping in the polls!? Obvioulsy Tories will be the main winner of any drop in SNP support and “weaponisation” of IndyRef2 and Brexit.

    It will be sad if Angus Robertson uses his seat. With the predicted Tory landslide an MP of his calibre will be sorely missed in keeping a check on the Tories.

    Looks like new thread starting, but thanks again

  6. AW Thanks very much.
    “you have to accept that any method isn’t perfect, and have to accept those limitations.” The question being, ‘How big are those limitations?’
    “if two people who are demographically identical to each other in every way (and in ways like political interest and past vote), vote differently depending on if they are online or not, then it’s an issue,”
    But you can’t know if whether they go online or not is their only difference, can you? Being elderly and prepared to go online may correlate with several other traits which affect likelihood to vote in one direction or another.
    I guess, though, that the overall uncertainties mask any attempt to refine the sampling further, the whole thing being subject to MoEs based on 95% confidence limits?
    It reminds me of the time I struggled for 2-3 years to find and eliminate some sources of error in a method to measure thermal conductivity. I succeeded, only to find that other sources still limited the accuracy to about 5%. I had found where some errors came from, and how to eliminate them, but that didn’t improve overall accuracy, as my ‘fix’ retained others, and introduced one or two new ones. Getting the +/-5% results was quicker, though.

  7. S Thomas

    You may be right but you can’t say that exactly the same would or would not have happened if a UK election hadn’t been called. So to suggest interference is stretching matters for your partisan view too.

    I’d go so far as to say that the EU believe this may be the only way to communicate with the UK public without the UK Government acting as the megaphone and that it may continue in this manner ’til the two years are up.

  8. Colin
    “Her Industrial & Economic Policy is pure Tony Benn Corbyn could have uttered every word.”

    Hence why I referred to her as a left wing candidate yesterday in a post, when for some reason others refer to her as right wing. She is as left as Melenchon.

  9. In case anyone actually wants to read the article that has caused such electioneering furore, here’s the link to a translation:

    Some might say that both sides have something to answer for here.

  10. @Robert Newark,
    That is not even remotely true according to reputable French sources, e.g. the tool at http://www.lemonde.fr/les-decodeurs/article/2017/04/23/vous-n-avez-vote-ni-macron-ni-le-pen-de-qui-etes-vous-le-plus-proche_5116072_4355770.html which allows you to compare the different candidates’ positions. Le Pen is shown as economically (neo)liberal on economics, and on social protection as very much less interventionist than Mélenchon is.

  11. ROBERT


    There were some memorable put downs from both of them in the debate.-I remember Len Pen telling Macron he is “the candidate of the elite” who would sell off French companies to the highest foreign bidder. I think she talked of a Globalised wasteland.

    A French economic protectionist wall of regulation is what she proposed-and retirement at 60 too !!-Benn would have smiled I think.

  12. ToH/S Thomas.
    Forgive my asking for a quote of Merkel calling the PM delusional – I could have missed it?

  13. Anthony, re. the question of whether internet usage is a relevant demographic factor: would it be fair to say that if it were suspected that such a thing might be a factor, it ought to be possible to perform a one-off study to interrogate that issue in particular and to see whether it did have an effect, where it had an effect (maybe it turns out to be relevant for older voters but not younger ones, or whatever) and what the magnitude of any effect was?

    Even if it weren’t methodologically possible to control for that in every poll, wouldn’t that at least either give one a rough handle on how it would affect the reliability of regular polling, or else to reassure oneself that it was not a relevant factor?

  14. Popeye – there are ways of doing it. Early in the EU ref campaign when there was a big gulf between online and telephone polling there was quite a lot of people looking at it to try and understand the difference. One thing that came out of that was that there seemed to be an attitudinal difference – the people you got in online surveys seemed to be less socially liberal than the people you got in phone surveys. But that, of course, just lead onto the questions of which one was right (given the correlation with EU ref vote and the performance of online and phone polls there I’d suspect online was actually the more representative, but who knows).

    Ultimately that’s the issue. There are some things, like age and gender, where we have firm reliable data to compare samples to… but because we’ve firm data, it means we can use quotas and weights to put it right, so it’s not an issue. There are other things where there are no firm reliable targets, but if you find a difference there, you can’t tell which side of the difference is correct and which one is wrong.

  15. PatrickBrian,

    Thanks for that link. Very revealing, if not very hopeful.

    I think Theresa and the Brexiteers need to wake up to the fact that “no deal is better than a bad deal” is in fact a much more convenient viewpoint for the EU 27 than for the UK…. We do not hold the whip hand in these negotiations in any way, shape or form and a bit of humility on her part might do us all a lot of good….

    First time we have seen that May does NOT want to guarantee existing rights for EU nationals in Britain, but to put them on the same footing as any other nationality, for example Syrians, Somalians etc.. No wonder they are nervous!

  16. A serious question –

    Its well known that in 1991 there was a small but measurable percentage of “timid Tories” who would not tell pollsters that they intended to vote for Major.

    Might it be possible that there is currently a “timid Labour” effect?

  17. Voting intention Con/Lab lead average 17 per cent. Actual local election lead 11 per cent. Why should the low turnout of local elections fabour Labour?

1 2 3