Yesterday the British Polling Council had an event talking about how the polls had changed since 2015. This included collecting up data from all the companies on what they’ve done to correct the error and what they are now doing differently – all those summaries are collected here.

In looking at what’s changed it’s probably best to start with what actually went wrong and what problem the pollsters are trying to solve. As all readers will know, the polls in 2015 wrongly overstated Labour support and understated the Conservatives. The BPC/MRS inquiry under Pat Sturgis concluded this was down to unrepresentative samples.

Specially, it looked as if polls had too many younger people who were too engaged and too interested in politics. The effect of this was that while in reality there was a big difference between the high turnout among old people and the low turnout among young people, among the sort of people who took part in polls this gap was too small. In short, the sort of young people who took part in polls went out and voted Labour; the sort of young people who weren’t interested and stayed at home didn’t take part in polls either.

So, what have polling companies done to correct the problems? There is a summary for each individual company here.

There have been a wide variety of changes (including YouGov interlocking past vote & region, ICM changing how they reallocate don’t knows, ICM and ComRes now both doing only online polls during the campaign). However, the core changes seem to boil down to two approaches: some companies have focused on improving the sample itself, trying to include more people who aren’t interested in politics, who are less well educated and don’t usually vote. Other companies have focused on correcting the problems caused by less than representative samples, changing their turnout model so it is based more on demographics, and forcing it to more accurately reflect turnout patterns in the real world. Some companies have done a bit of both.

Changes to make samples less politically engaged…

  • ICM and YouGov have both added a weight by respondents level of interest or attention to politics, based upon the British Election Study probability survey. YouGov have also added weights by level of educational qualification.
  • Ipsos MORI haven’t added political interest weights directly, but have added education weights and newspaper readership weights, which correlate with political interest.
  • Kantar have added education weighting, and also weight down turnout to the level they project it to be as a way of reducing the overall level of political engagement in their sample.

Changes to base turnout on demographics…

  • ComRes have changed their turnout model, so it is based more on respondents’ demographics rather than how likely they claim they are to vote. The effect of this is essentially to downweight people who are younger and more working class on the assumption that the pattern of turnout that we’ve seen at past elections remains pretty steady. ICM have a method that seems very similar in its aim (I’m not sure of the technicalities) – weighting the data so that the pattern of turnout by age & social grade is the same as in 2015.
  • Kantar (TNS) have a turnout model that is partially based on respondents age (so again, assuming that younger people are less likely to vote) and partially on their self-reported likelihood.
  • ORB weight their data by education and age so that it matches not the electorate as a whole, but the profile of people who the 2015 British Election Study who actually voted (they also use the usual self-reported likelihood to vote weighting on top of this).
  • Opinium, MORI and YouGov still base their turnout models on people’s answers rather than their demographics, but they have all made changes. YouGov and MORI now weight down people who didn’t vote in the past, Opinium downweight people who say they will vote for a party but disapprove of its leader.
  • Panelbase and Survation haven’t currently made any radical changes since 2015, but Panelbase say they are considering using BES data to estimate likelihood to vote in their final poll (which sounds to me as if they are considering something along the lines of what ICM are doing with their turnout model)

In terms of actual outcomes, the pollsters who have adopted demographic turnout-models (ComRes, ICM and Kantar) tend to show larger Conservative leads than companies who have tried to address the problem only through sampling and weighting changes. We cannot really tell which is more likely to be right until June 8th. In short, for companies who have concentrated only on making samples more representative, the risk is that it hasn’t worked well enough, and that there are still too many of the sort of young engaged voters who are attracted to Jeremy Corbyn in their samples. For companies who have instead concentrated on demographic-based turnout models, the risk is that the pattern of turnout in 2017 differs from that in 2015, and that Jeremy Corbyn’s Labour really does manage to get more young people to come out to vote than Ed Miliband did. We will see what happens and, I expect, the industry will learn from whatever is seen to work this time round.


199 Responses to “How the polls have changed since 2015”

1 2 3 4
  1. But will they get it right this time?

  2. Social media is having a profound impact on VI as far as I can see. You can now effectively canvass via Facebook, and there’s certainly a big Jeremy Corbyn thing going on.

    Given the way the Tories have alienated their core vote with their ham-fisted manifesto, I’m really not sure where this election is going!

  3. Thank you for a very useful summary of where different companies are with polling.

  4. http://www.conservativehome.com/platform/2017/05/lord-ashcroft-my-election-models-probabilities-currently-suggest-a-potential-conservative-majority-of-142.html

    D13 Lord Ashcroft sample model

    N= 40,000

    22-26 May 2017

    Seat Projection:
    Con 396
    Lab 180
    SNP 47
    LD 6
    PC 2
    NI 18

    Con projected Majority 142

  5. Thanks, this is really interesting. Is it possible to tell what the results would have been if no changes had been made to methodology at all? Obviously not for the pollsters who have changed their method of sampling – but for the ones who have just made weighting changes.

  6. @JONESINBANGOR
    Given the way the Tories have alienated their core vote with their ham-fisted manifesto, I’m really not sure where this election is going

    But conversely Labour are still struggling in its appeal to elements of its traditional wc base. Amongst C2DE’s the Tories still have an 8% lead. Given that there is a strong possibility that many of these voters will prefer May’s stance concerning terrorism than Corbyn’s, and the Brexit issue, its unlikely that Corbyn will succeed in reversing the trend of this group moving away from Labour.

    Unfortunately for Labour these types of voters are numerous in the battleground seats, and as others have commented, I think there is a chance Laabour are piling up there vote in seats they will anyway but will go down to relatively slim defeats in many of swing seats.

  7. i think polling has becoming a lot harder as politics becomes increasingly febrile and with the impact of social media. I think there is a longer trend of people becoming less socially connected with each other – and this has provided fertile ground for the rise social media echo-chambers and “fake news”

    Given this I would not be at all surprised if the polling is getting stuff wrong – but what is hard to guess.
    It may well be that the tory vote is understated and labour over stated – this has been the pattern for decades but was particularly bad in 2015.
    However it may be the pollsters have now overcompensated for this and its now the other way around – esp if there is a bigger turnout from young people.
    Also vote shares for the smaller parties may be wrong – who knows?

    Be interesting post mortem come june 9th.

  8. Is there a possibility of shy labourites?

  9. Really interesting to see how the weightings have changed. My guess is the polls will be wrong again as anecdotally JC seems to be inspiring the 18-24 far more than EM or GB ever did. Now how much that increases the turnout in that demographic (it was 43% in 2015) will effect the error. if it’s only 2-3% it will be very small but if it is >5% it will make a bigger difference and could be the difference between small CON maj and Hung Parliament.

  10. Ar558:

    I think you might be overestimating the numbers of 18-24 and therefore the effect they could have. The BBC ran analysis a week or so ago asking what the difference would be if 18-24 year olds voted at the same level of turnout as 65+ did. It essentially amounted to around a 1% gain for Labour, not really significant at all.

  11. I noticed that in YouGov the increase in JC’s rating was overwhelmingly a switch from don’t knows.

    So this could be about bringing new voters into play. And if social media has an impact they in the turn out of young voters, it could be a massive reversal of fortune when votes get counted.

    The trouble with saying that Labour is being overstated is that there has been no change of methodology over the last weeks. So if is over estimated at 38%, does that not mean it was also being overestimated when knocking on 25%?

  12. @ redrich

    I agree with a lot of what you’re saying.

    I think incumbency will save a lot of Labour MPs here in Wales – it’s always a big factor.

    I think Scotland London will be the interesting places on the night.

    I’m sticking to my view that the Tories will not really improve on what they already have, particularly as they “blew it” on the manifesto. As well as dementia tax, I sense the fox hunting issue is also playing against them.

  13. @Joseph1832
    Possibly, but not necessarily. Perhaps the 25% figure was correct but lots of people who are unlikely to vote in the election have been encouraged to say they will vote by peer pressure (but in reality they still won’t).

  14. Survation claim their final pre-GE15 poll was highly accurate – but they did not publish at the time. However, perhaps this goes some way to explaining why they see little need to change their methodology. But if their final poll was only correct by chance error then perhaps they should be making changes… Who knows.

  15. RELICK

    Maybe nationally that is true but there will be certain constituencies where they will have a larger impact. (How many CON/LAB marg do we have with Unis?)

  16. None of this explains why Ashcroft is coming up with a very different picture based on about 12,000 respondents. Can someone explain? I never really see how a thousand people gives a decent picture. Seems to me Ashcroft, who doesn’t get a look in on a website like this, is asking a lot more people on a much wider social and geographic spectrum.

  17. joseph1832

    er…no.The 25% represented those who were going to vote labour and were likely to vote. The increase in labour support is from the dont knows and from those likely to vote.

    Maybe we will see what this younger generation is made of? Are they all outrage and comment or will they use the stubby pencil. They will come up against the Tory shield wall who are used to and battle hardened in the use of the said stubby.
    Snowflakes or veterans? I know where my money is going.Prediction 45/34

  18. It looks like Labour had done their preparation well for dealing with the Tory response to Corbyn’s foreign policy speech by briefing the media. It’s unusual to see Fallon who is normally silkily imperturbable so thrown:

    https://www.theguardian.com/politics/2017/may/27/fallon-left-red-faced-after-condemning-boris-johnson-extremism-comments

  19. Ar558:

    That’s a good question. I am in the Coventry South constituency – which has a large number of Warwick and Coventry University students. It was relatively safe Labour last election but a couple weeks ago when it looked like the UKIP vote was going 10:1 to the Tories it appeared that the Tories could easily snag this constituency.

    But if the youth here are enthused it might stay Labour after all. I don’t know if this anecdotal experience of mine would apply to particularly many constituencies, though.

  20. Turnout among the young, many of whom have been turned on to the importance of voting by the EU referendum result, will be higher this time. Possibly quite a lot higher.

  21. Difficulty for the pollsters is that Moe makes it hard to judge accuracy.
    It may be that a particular pollsters method overstates Lab at Cons expense by say 1.5% but they get the result right due to random variation.

    The BPC enquiry looking at aggregated results will have helped but still not an exact science.

    Also, as Anthony says factors skewing raw date numbers in one GE (E.G differential turnout rates) can be different next time for political reason.

    Trend is the best measure for me and if 5 out of 6 regular companies show a closing trend then that is probably occurring. Doesn’t help much in the last week though!

  22. Moderation

    AW can you personally review my awaiting moderation.Or am i being moderated automatically?

    Long live AW

    test

  23. redrich

    “Without that edge it comes down to who is the lesser of two evils – and I think May still has the edge here.”

    As someone else wrote a few days ago – stick that on a poster.

    “The lesser of two evils” would certainly make a bold change from “Strong and Stable”.

  24. CR

    “Is there a possibility of shy labourites?”

    In this everchanging world in which we live in Rachel, [as my mate Macca would say] all things are possible.

    It’s “probable” that you need to worry about.

    [Or hope for.]

  25. ROB SHEFFIELD

    But he actually said the most likely result was a majority of 80-90/

  26. RELICK

    You sound like your in the kind of seat I was thinking of. I’m guessing the UKIP vote is not splitting 10:1 to CON anymore either probably 5:1 plus they will be retaining a little more too. Combine that with any increased 18-24 Turnout should keep it LAB.

  27. I think the problem could even be more fundamental – last minute changes of mind, even down to the last few hours/minutes or in the polling booth. And, of course, people lying.

    In which case, all the polling in the world can’t tell you the precise result.

    But maybe the pollsters don’t want to admit this.

  28. Rob that is what I have been predicting approx 150 Maj.Hard to understand all this angst from the Conservatives if Lord Ashcroft is correct.

  29. With regard to Corbyn on Neill, I find it hard to understand why he didn’t make it clear that, for ISIL and similar groups, there really are no discussions or negotiations to be had, as clearly there is no compromise position that is even remotely possible, far less desirable.

    He really does seem to have a problem with reaching definite conclusions – even when they’re bleedin’ obvious.

  30. I think those variables are likely to cancel each other out.

  31. Soory, last to Robert c

  32. On shy voters – is there any firm reason to think these still exist on either side? Almost all polls are done online and the results obviously keep people anonymous. If shy voters in the sense of those too shy to admit the truth to an online poll really do exist, does someone have a reference for that?

  33. Ar558:

    It’s kind of hard to judge the mood. I have so far seen only 1 poster, for Labour, and it was in the neighbouring Cov NW constituency. My experience with fellow Warwick students is overwhelming Labour and Green support, but I felt it was that way in 2015 too.

    Also I did a bit more research and it seems that the student population of Warwick might actually be spread across about 5 constituencies (Cov NE/NW/S, North Warwickshire, and Warwick and Leamington), which could limit student vote power. Another thing worth mentioning is that June 8th for most universities is after exams. A lot of students may already have gone back home.

  34. I think the whole shy ( party ) thing is a sinecure for those who are distressed by a poll.

  35. The IFS judgement on the tory and labour manifesto ” undeliverable vs unworkable ” ” neither addresses long-term challenges we face” ” the PM’s blueprint for fresh cuts in the next parliament may not even be deliverable ” a tory pledge to cut net migration to tens of thousands would cause considerable damage by cutting tax revenues , whilst labour are criticized for pretending taxing business was victimless which in reality actually means the impact on families are just less transparent. For me this has been the most dispiriting of election campaigns , for the first time in my adult life I’m going to abstain Christ !!! I can’t even protest vote , where are the monster looney party when you need them?

  36. @Robert C,
    In my experience the pollsters are perfectly willing to acknowledge most of those caveats – the possibility of late swing, margin of error , etc. It is frankly the media which tend to overegg the findings and neglect the caveats.

  37. I am waiting for polls to comment extensively, but, regarding Lizh 93 year old mother etc, it’s actually a poor argument to say Wars in ME are responsible for terrorism.

    Read the ISIS statement claiming responsibility for Manchester.

    They did the bombing because it was a ‘shameful western concert’. That’s hatred of way of life, not middle eastern intervention. People need to wake up, you can’t appease these people.

  38. Redrich,
    ” there is a strong possibility that many of these voters will prefer May’s stance concerning terrorism than Corbyn’s”

    just posted on the last thread that I am listening to R4 any questions, where David Davis just tried to deny any link between foreign policy and the likelihood of terrorist action here. He was booed. Now, some of the labour party might agree with him, but at least part of the public very definitely does not. Booing when a minister is trying to state the party line on security does not seem encouraging.

  39. Rich:

    I’ve been trying to hold my tongue regarding the foreign policy/terrorism debate going on here but I’m getting quite irritated now that yourself and others continue to conflate the reasons that terrorists attack with the reasons that terrorists exist.

    The “foreign policy causes terrorism” view is simple – foreign policy at least since WWI but especially recently with the War on Terror and the way we tried to deal with the Arab Spring has been a significant factor leading towards the existence of death cults like ISIL. Whether it’s through creating power vacuums, providing Islamist rebels with funds and arms, buddying up with countries like Saudi Arabia that have shown some support for these groups, or dividing the ME by arbitrary boundaries after seizing it from the Ottoman Empire, it all contributes bit by bit. There are of course other factors and I don’t mean to diminish those, I don’t suggest for one second that foreign policy is the only cause.

    Just because you can’t appease murderous maniacs doesn’t mean we didn’t help create them. This is a view shared by many terrorism experts and politicians across political divides.

  40. RELICK

    Interesting. Obviously if there are multiple constituencies that will dilute it . You’re right some students may have gone home and I wonder how many are registered at home or at Uni? (The astute may have picked which ever was more marginal I guess).

  41. @Paul Croft
    In fairness he did make pretty clear he wouldn’t negotiate with ISIS. Perhaps he should have explained his reasons more clearly though.
    From text of the interview:-
    AN:Would you talk to them [terrorist leaders/ISIS]?
    JC: No, I wouldn’t.

  42. @DANNY

    I wouldnt pay any credence to booing from an R4 AQ audience. Its self selecting. Last week when Dianne Abbott was on the panel she was getting whoops of cheering. Yet we all know where opinion polling places the public perception of her.

  43. AW We will see what happens and, I expect, the industry will learn from whatever is seen to work this time round.

    Or, like the unfortunate generals, start off (by necessity) trying to fight the last war.

  44. Rich, no one is saying wars are the sole factor and certainly no one is saying they excuse the perpetrators. The debate is about the degree and kind of contributing factor they represent etc. in various respects.

    Danny, yes but the AQ audience is not necessarily representative. J Dimbleby said they don’t attempt to make it so. So one shouldn’t read too much into it, perhaps.

  45. Ar558:

    Yes, personally I am registered to vote both in Cov South and in my home Essex constituency, but I will be voting in Coventry since my home town is as blue as it gets, whatever I vote there will be a waste.

    Having said that, I have found it a complete pain to keep up with registration, since I am moving where I stay at university each year, so have to also deregister at my old residence each time I register at a new one. I can imagine that those who are less politically engaged than I am might be inclined to just give up – I know some people that aren’t even registered with a GP because of the carousel of student housing!

  46. “Opinium downweight people who say they will vote for a party but disapprove of its leader.”
    This is not a normal election so that method could be flawed. People who fundamentally disagree with JC will be voting Labour. Be it Anti Tory, Anti Brexit , Anti Austerity , threat of large majority, support of local MP

  47. @Paul Croft

    He [Corbyn] really does seem to have a problem with reaching definite conclusions – even when they’re bleedin’ obvious.

    As Steven Wheeler has pointed out this is unfair at least in relation to the part of the interview to which you refered. It is also at odds with the usual picture of Corbyn which is that he is dangerously and dogmatically of the left.

    I actually thought that your prediction of how Corbyn would do (e.g. that he would be well prepared and calm) was spot on). So I was surprised at what seems to be a sharp change of tone in the light of the actual event. I know that you have very strong views about religion and the harm it can do. Was it this particular part of the interview that changed your mind or were you reacting to the thing as a whole?

  48. I think the “shy Tory” is a misleading term for the behaviour, and it seems to me that it sometimes take the discussion of track.

    I would be very surprised if the youth vote went up significantly (we would see the Greens up, but it’s not the case). Also, although it’s more observational, the current student body feels more centrist than, let’s say 2012’s ones.

    Also anecdotally – of the three main Corbynites on my FB, two won’t vote – they will be on holiday, and didn’t arrange postal vote.

  49. Relick

    I dont think it’s quite that simple. Even Corbyn’s argument would appear to be that our inept foreign policy creates a vacuum in which organisations like ISIS thrive, not that foreign policy creates ISIS. It’s like saying we create wasps by leaving jam jars open. We don’t, but we create an environment for wasps to thrive and do us harm by leaving jam etc

  50. In Newcastle under Lyme we have a Labour MP sitting on a precarious 600 majority and the 2015 Tory vote plus a bit of the large UKIP vote would wipe him out . I want to vote LD but they don’t have a chance here and as the Labour guy isn’t a Corbyn supporter and is pro EU I will vote for him. NuL constituency also includes Keele Uni. so there may be a plus factor for Labour if the students get up and vote.

1 2 3 4