On Tuesday the BPC/MRS’s inquiry into why the polls went wrong publishes its first findings. Here’s what you need to know in advance.

The main thing the inquiry is looking at is why the polls were wrong. There are, essentially, three broad categories of problems that could have happened. First, there could have been a late swing – the polls could actually have been perfectly accurate at the time, but people changed their minds. Secondly respondents could have given inaccurate answers – people could have said they’d vote and not done so, said they’d vote Labour, but actually voted Tory and so on. Thirdly the samples themselves could have been wrong – people responding to polls were honest and didn’t change their minds, but the pollsters were interviewing the wrong mix of people to begin with.

Some potential problems can straddle those groups. For example, polls could be wrong because of turnout, but that could be because pollsters incorrectly identified which people would vote or because polls interviewed people who are too likely to vote (or a combination of the two). You end up with the same result, but the root causes are different and the solutions would be different.

Last year the BPC held a meeting at which the pollsters gave their initial thoughts on what went wrong. I wrote about it here, and the actual presentations from the pollsters are online here. Since then YouGov have also published a report (writeup, report), the BES team have published their thoughts based on the BES data (write up, report) and last week John Curtice also published his thoughts.

The most common theme through all these reports so far is that sampling is to blame. Late swing has been dismissed as a major cause by most of those who’ve looked at the data. Respondents giving inaccurate answers doesn’t look like it will be major factor in terms of who people will vote for (it’s hard to prove anyway, unless people suddenly start be honest after the event, but what evidence there is doesn’t seem to back it up), but could potentially be a contributory factor in how well people reported if they would vote. The major factor though looks likely to be sampling – pollsters interviewing people who are too easy to reach, too interested in politics and engaged with the political process and – consequently – getting the differential turnout between young and old wrong.

Because of the very different approaches pollsters use I doubt the inquiry will be overly prescriptive in terms of recommended solutions. I doubt they’ll say pollsters should all use one method, and the solutions for online polls may not be the same as the solutions for telephone polls. Assuming the report comes down to something around the polls getting it wrong because they had samples made up of people who were too easily contactable and too politically engaged and likely to vote, I see two broad approaches to getting it right. One is to change the sampling and weighting in way that gets more unengaged people, perhaps ringing people back more in phone polls, or putting some measure of political attention or engagement in sampling and weighting schemes. The other is to use post-collection filters, weights or models to get to a more realistic pattern of turnout. We shall see what the inquiry comes up with as the cause, and how far they go in recommending specific solutions.

While the central plank of the inquiry will presumably what went wrong, there were other tasks within the inquiry’s terms of reference. They were also asked to look at the issue of “herding” – that is, pollsters artificially producing figures that are too close to one another. Too some degree a certain amount of convergence is natural in the run up to an election given that some of the causes of the difference between pollsters are different ways treating things like don’t knows. As the public make their minds up, these will cause less of a difference (e.g. if one difference between two pollsters is how they deal with don’t knows, it will make more of a difference when 20% of people say don’t know than when 10% do). I think there may also be a certain sort of ratchet effect – pollsters are only human, and perhaps we scrutinise our methods more if we’re showing something different from anyone else. The question for the inquiry is if there was anything more than that? Any deliberate fingers on the scales to make their polls match?

Finally the inquiry have been asked about how polls were communicated to the commentariat and the public, what sort of information is provided and guidance given to how they should be understood and reported. Depending on what the inquiry find and recommend, in terms of how polls are released and reported in the future this area could actually be quite important. Again, we shall see what they come up with.

71 Responses to “This week’s Polling Inquiry”

1 2
  1. ICM
    Con 40 Lab 35 LD 6 Ukip 10 Grn 3 SNP 4

  2. @Roger M

    “But of course the UK wasn’t the only places where the pollsters had problems in 2015. In the run-up to the Greek election on 20 September most of the polls showed Syriza only a point or two ahead or tied with ND. In the end they had a 7 point lead (as in the UK the exit polls were more reliable). They also got the referendum wrong in a similar manner.”

    What’s this now? Shy Syriza/left weing voters in Greece?

    Is this a good omen for Corbyn, do we think?


  3. @Graham

    Con 40 Lab 35 LD 6 Ukip 10 Grn 3 SNP 4”

    Is ICM the new Angus Reid of this Parliament?


  4. “Late swing has been dismissed as a major cause by most of those who’ve looked at the data. Respondents giving inaccurate answers doesn’t look like it will be major factor in terms of who people will vote for (it’s hard to prove anyway”


    Ever-present concern though, innit? Because late swing can happen, to catch you out, and not much you can do about it?

  5. CB
    Well who knows?perhaps time will tell! But ICM was formerly seen by many – particularly on the Political Betting site – as the Gold Standard and generally came up with the lower Labour scores to the benefit of the LibDems.

  6. Worth reading the ICM note on their methodology –

    This poll is another in a line of post-2015 phone polls from us which contains too many people saying they voted Labour at the last General Election. With unrepresentative samples long since in line of sight as a primary cause of error, it’s difficult to escape that very conclusion. Memory lapse might be forgivable and understandable if we were much more than a year from the last General Election, but we’re not, and it’s hard not to think that it’s simply easier to get hold of Labour folk, than it is Conservative.

    It’s also difficult to think what can be done about it. Telephone omnibus services seek a representative sample of the population, with the hope and expectation that a representative sample of voters naturally falls out. This not currently being the case, one might figure that some protocols to ensure a closer match between the election result and the recall of that result on any given poll might be the answer. However, omnibus services are there for multiple clients (often tracking their data) with consistency an absolute necessity. If protocols were changed in order to facilitate more accurate voting patterns, other clients on board that omnibus might be disadvantaged as a result.

    Translation –

    The VI results are rubbish, but we’re not going to screw up our important commercial clients for the sake of you geeks on UKPR!

  7. Cons will still be happy with 40, especially with the Corbynistas being so noisy and desperate to be polled and plenty of issues about to depress the government’s support. 40 seems to be the new 35.

  8. “The main thing the inquiry is looking at is why the polls were wrong”

    As mentioned above they were right in Scotland.

    My opinion, massively differential turnout among different segments of the C2/D/E population which heavily favours Lab in some seats and heavily disfavours them in others.

    For example compare demographics of
    – Heywood and Middleton (600 Lab Majority)
    – Oldham West (10,000 Lab majority)

  9. @OLDNAT

    ICM must also be aware that if these results are rubbish, their other conclusions using the same people may well also be rubbish. If you are excluding a major section of the population, you are unlikely to get a representative opinion.

  10. Graham
    “I agree about Blair but not Cameron. Macmillan was more left wing as was Heath post-the U turn.”

    MacMillan was never considered at all left wing from my memory and for much of his time he was up against Gaitskill, who wasn’t really left wing either. He was a ‘one nation tory’ of course, like Cameron and very much part of the establishment. Wasn’t he related to half his cabinet or something? I could never imagine homosexuality being legalized under MacMillan, or indeed any Tory at that time, whereas Cameron has done many things to attract the pink vote – I understand what you mean with Heath, not someone I ever liked, he lied to the people about the Common Market and U turned as soon as someone stood up to him.

  11. I am a big fan of ‘marginal incremental change’ as an explanation for much that we struggle to understand.


    I suspect that there was a ‘late swing’, but it was small so has not come up on the radar – perhaps half a point or so.

    There is a ‘shy Tory’ phenomenon – it was modest but significant.

    The samples were not representative – they never are. It is always something of a stab in the dark.

    The youngsters don’t vote. The elderly do. This was underestimated. Perhaps the weather played a part.

    And tactical voting plays a huge part. The important thing here is that the degree of tactical voting varies from one election to the next. Every tactical voter makes a new calculation every time he walks into the polling booth. So a tactical voter makes a new decision on a daily basis in the run-up to an election, depending upon the polling at that time.

    Lib Dem support was very soft.

    All these independent factors combined in small but significant ways to increase the Tory vote at the expense of Labour.

    I suspect the pollsters did not perform that badly – they simply experienced a rare case of several modest independent effects combining to push the result in the same direction.

  12. weren’t most the polls more or less within margin of error on the actual election?

  13. Robert Newark
    Macmillan was easily the most left wing PM since World War2 particularly with regard to economic policy. He was way to the left of Blair and I even recall him being critical of Ted Heath’s Government during its early Selsdon Man phase. Apparently at one point in the 1930s whilst MP for Stockton on Tees he came close to crossing the floor and joining Labour. He never concealed his contempt for Thatcherism and his comments on ‘selling off the family silver’ are well known.

  14. Correction
    ‘Macmillan was easily the most left wing Tory PM since World War 2 particularly with regard to economic policy.’

  15. Anthony,
    I see that Martin Boon is pouring some cold water on ICM’s findings by pointing out that too many respondents recall having voted Labour in May 2015. Do not pollsters deal with that problem by reweighting?

  16. @Flap Zappa

    weren’t most the polls more or less within margin of error on the actual election?

    Here are the last polls from the main pollsters before the GE (Con and Lab only):

    Pollster – Con – Lab

    Populus – 34 – 34
    Lord Ashcroft – 33 – 33
    IPSOS Mori – 36 – 35
    Yougov – 34 -34
    Comres – 35 – 34
    Survation – 31 -31

    If the only issue was margin of error, then you would expect some polls to be over the real vote share. If you have a poll mean for the Conservatives about 36%, you would need a range of values above and below it.

    Labour would have had a rough poll mean of about 30%, with polls above and below that.

    The fact that the polls overstated Labour support consistently and understated the Conservative support consistently shows there were systematic issues with the pollsters methodologies.

    Survation have claimed they got a late poll they pulled with Con 37, Lab 31, but I don’t accept this being accurate, rather it was lucky. The polling methodology that Survation used that had got a tie at 31/31 just before the 37/31. Any reasonable analysis at Survation’s polls over a period of time would show a 37/31 as being a suspicious poll, way out of kilter with other pollsters and their own previous polls too.

  17. Graham – “Do not pollsters deal with that problem by reweighting?”

    Yes, that’s what Martin is referring to in his last paragraph:

    Thus far then, our solutions have been to correct data outcomes at analysis stage – which, to my mind, are not succeeding to the extent they need to.

  18. My abiding memory if the election is that so much of the news & commentary output was based on “what’s trending on twitter”, creating what felt like unmitigated criticism mostly of Labour (and quite a bit of LibDem). Surely that self-fulfilling roll had an effect? To limit debate to “why is everyone critical of you”?

  19. I am surprised at the confusion over the inaccuracies of the general election polls. I predicted Con overall majority by looking at SEATS and not % of votes. Polling organisations nearly all base their findings on extrapolating numbers of votes across the country. this is extreme naïve and out-dated (and lazy and definitely hopeful!). The politics in this country have changed, the variety of new parties make counting votes in our system unreliable and the Scottish factor also had a massive impact that none of them understood properly.

    So, firstly, the lab party was set to lose 40-50 seats in Scotland to SNP. this is SEATS not votes, and it is SEATS that count. In fact nearly every Scottish labour vote was worth nothing (just one seat came to them). just repeat myself, ALL LABOUR VOTES IN SCOTLAND WERE VIRTUALLY WORTHLESS, and we new this was going to happen. So, even before you start counting SEATS in England and wales, this meant Labour had to gain 40plus seats in England and Wales just to stand still. There was never any way that labour would do this so how could polling organisations possibly work out that labour might be the largest party? … because they were counting NUMBERS of VOTES and not SEATS. Most are still confusing the two. Its a bit like the scoring in tennis where you can win a match by winning fewer points overall than your opponent (its the Games and the Sets that count, not how many points overall). Secondly, the Cons were very clever, realising Labour would lose significant Scottish seats to the SNP they concentrated all efforts on the marginal SEATS in Midlands and North, which they then mostly won. thirdly I believe polling orgs convinced themselves that the majority of Liberal votes would go to Labour, and maybe they “factored” that into their calculations. Again, far too simplistic, in todays politics with many Liberals either not voting at all or voting Tory or UKIP or Green.

    In future polling orgs will need to poll WITHIN each constituent if they want to persist in counting numbers or percentages of votes cast. One can only predict accurately if you investigate seat by seat. Come on guys, invest some money and pull your socks up, get it right. xx

  20. @Robert Newark/Graham

    Any judgement on how left wing Tory PMs in the 50s and 60s were (Churchill, Eden, MacMillan and Home) has to be made in the context that they were all operating within Attlee’s post war settlement. Heath too to some extent in the early 70s and, in defence of Blair and Brown, they were governing in the shadow of the Thatcherite settlement of the 80s.

    Accordingly, all those Tory PMs I’ve listed presided over nationalised railways, communications, steel, coal, shipbuilding and public utilities and ran the NHS much as Bevan had intended. They managed socialism for much of their time, merely tinkering around the edges of a vast welfare state. I expect their Chancellors were quite happy with exorbitant marginal rates of taxation too. They governed as Tories in a largely social democratic state, much as Blair did as a Labour PM, tinkering with a deregulated, low tax and largely privatised economy.

    Was Blair right wing because of that, and was MacMillan left wing, or were they both captives of an established political legacy bequeathed to them a generation before?

  21. @CB
    ‘Was Blair right wing because of that, and was MacMillan left wing, or were they both captives of an established political legacy bequeathed to them a generation before?’

    Blair was right wing because he had the power in Parliament to significantly redraw or even reverse much of the Thatcher settlement. He lacked the will to do so – despite the fact that the electorate had reached the point where it would have been more than happy to see him try. Applying the 1992 manifesto for example – rather than its supine 1997 successor – would have gone down pretty well and probably avoided the collapse in turnout seen in 2001.

1 2