Next Friday is the public meeting of the British Polling Council inquiry into the failure of the polls at the 2015 election, at which point I expect we’ll get some insight into what the different polling companies are thinking, though probably not many firm conclusions yet.

In the meantime the British Election Study team have published some thoughts from Jon Mellon about what the BES data could tell us about why the polls were wrong. It doesn’t include any conclusions yet, but goes through a lot of the thought processes and ways of identifying what went wrong, which I suspect may reflect what many of the pollsters are doing behind closed doors.

As yet only the online BES data from during the campaign is available for download, but in time it will be joined by their online recontact survey after the election campaign, their face-to-face survey after the campaign and voter validation data for the people interviewed in the face-to-face survey. The article has some thoughts about what they can learn from the data that’s already available and what can be learnt from the bits that are still to come:

1) The BES campaign data appears to show some movement towards the Tories over the last couple of days, though not one that is beyond the margin of error. This is in contrast with YouGov’s daily polling data, despite them coming from the same panel. This is interesting, but as Jon says, the real proof will be when the BES publish their post-election data, showing if people actually did change their minds from their pre-election answers

2) If you only take people who said they were very likely to vote it would have been more Tory… but that’s very much a “Pope is Catholic” finding. The interesting bit here is what the BES team plan on doing in the future – they are once again going to validate their face-to-face data against the marked electoral register, to see if people who claim they voted genuinely did, and how well people’s stated intention to vote compares to whether they actually did. They are also going to match the online respondents to the electoral registers before and after the new electoral registers, to see if drop off from individual electoral registration was a factor.

3) Sampling and weighting. Jon hasn’t really said anything on the data so far – he’s waiting for the face-to-face probability sample, to compare that to the results from the online polling and see if it is significantly closer to the actual result.

4) Don’t knows. According to Jon the people who said don’t know before the election were a mixed bunch – their attitudes towards the leaders, issues and party id did not point to them being obviously likely to switch to Conservative or Labour. Again, the interesting bit will be to see how they said they ended up voting in the post-election wave.

5) “Shy tories”. Jon makes two interesting points. One is about question order. While the BES campaign data came from YouGov’s panel, its results seemed to show a movement towards the Tories that the main YouGov data didn’t show – in his article Jon presents Peter Kellner’s hypothesis that this may be because of question order. As regular readers will know, the published voting intention polls all religiously ask voting intention first, but the BES actually asks some questions about the most important issues facing the country and party leaders before asking VI. However, Jon also mentions what he judges to be “weak” evidence against “shy Tory” hypothesis – the BES included a grid of questions aimed at identifying people who tended to give socially desirable answers to questions, and Conservatives scored higher, not lower, amongst those people.


151 Responses to “The British Election Study on why the polls were wrong”

1 2 3 4
  1. So its taken a month or so to come up with something that I could have told you on day 1.

    1. Shy tories,
    2. Sampling errors.
    3. Sampling sizes.

    Tell me something I don’t know.

  2. “the BES included a grid of questions aimed at identifying people who tended to give socially desirable answers to questions, and Conservatives scored higher, not lower, amongst those people.”

    That is what I would have assumed (with no evidence at all!) anyway.

  3. Matt Singh of Number Cruncher Politics predicted a Tory near majority!

  4. Mention of question order is interesting. That Survation poll that wasn’t just about got the result bang on, and asked preliminary questions before the VI I recall.

  5. Alec – no, you’re thinking of the Labour party’s private polling by Greenberg, who used priming questions before VI

  6. Interesting stuff -i also think there were too many polls -commissioners ploughing their own political furrow.

    Quality ie accuracy is essential not quantity.You cant blame the polling companies -its business and a leader to other work but the newspapers and ashcroft need to think carefully their objectives.

    Also some other methodological changes are required.-as the uk drifts apart ,all polls should separate the constituent countries .And given the strategic importance of the midlands as labour tory marginal territory it shouldnt be lumped in with wales.

  7. There were some real elections last night. Labour won the Tower Hamlets Mayoral election, albeit by a relatively narrow margin over the ex-Tower Hamlets First candidate considering the scandal that triggered this election. Also in Tower Hamlets, Labour narrowly made a gain from THF (now running an Indy as THF have been wiped off the register) in Stepney Green ward.

    Meanwhile, in Wellington South on Sutton Council there was that rarest of birds that is a Lib Dem hold.

  8. 07022015

    I agree, they are only following the customers and the market.

    It is like many other things nowadays; people are unwilling or unable to pay to have things done properly, and then end up wondering why they end up with a half-a*sed job.

    So I have some sympathy for the pollsters too.

  9. Good morning all from a sunny Mount Florida.

    The Other Howard predicted a Tory majority. They should consult with him.

  10. Ashcroft’s poll of April 26th also pretty much nailed the result, perhaps they should examine that.

  11. Yes but toh predicted a tory win in 1997,2001 and 2005 as well ?

  12. The term ‘shy Tories’ is interesting, it could also be named ‘intimidated Tories’. I know people who work in the public sector and have expressed the thought that if their colleagues ever suspected they vote Tory, their working life would be made very difficult indeed, possibly to the conclusion that they may be ousted from their job. There appears to be a distinct left of centre bias in such spheres which is very vociferous, probably Union instilled, and almost an unwritten and unchallenged idea that only left of centre people can be caring and public spirited. In such circumstances it is not surprising those people keep quiet about their voting intentions.

  13. How do we know when the problem with the polls started?

    Labour are claiming that what turned things around for the Conservatives was the Labour-SNP possibility. They say that the constant use of this issue was what convinced the electorate to go Blue. But there’s no evidence for this; the polls remained level before and after this part of the campaign.

    For all we know, the polls have been wrong for the past five years, consistently overstating Labour support and understating Conservative support.

    I think that every election will have its own polling factors, and that polling companies will forever be playing catch-up.

  14. @Shy Tories

    I agree with what you that shy Toryness is a thing. However, I don’t know why that would convert into a phone poll conversation with a faceless operator, still less into answers inputted into an online form. If ‘shyness’ is a cause of error it may be more a psychological effect with people who end up voting Tory paying at least lip service to being undecided until later on.

  15. “Alec – no, you’re thinking of the Labour party’s private polling by Greenberg, who used priming questions before VI”

    I thought Survation had an election eve experimental poll that they didn’t publish as the results were out of line and so obviously ‘wrong’?

  16. @ITF Tory

    “For all we know, the polls have been wrong for the past five years, consistently overstating Labour support and understating Conservative support.”

    I suspect you may have something here – I remember being surprised by the strength of the Tory showing at the last European Election despite the UKIP overall lead. It marginally confounded the polls in that election, if my memory serves me well?

    I suspect that when “modern” polling is sampling opinion during a lead by the Left era, they get them about right. However, during a Right era they get them wrong. It is almost as if a swing or surge to the Right is somehow below the radar, whereas the same to the Left is monitorable.
    Quitea conundrum for the polling industry I feel?

    I cannot wait to read what they think went wrong.

  17. Further analysis and polling suggests that the swing in the ethnic minority vote was not as dramatic as had been suggested by Survation polling:

    https://yougov.co.uk/news/2015/06/12/are-conservatives-really-breaking-through-extend/

  18. @AnthonyWells,

    I wanted to go to the RSS meeting next week but cannot. Will you be doing a writeup of it?

  19. Creagh drops out of the race -jeremy still 17 short ,be interesting to see if the whips nominate.

  20. Labour list readers vote 47per cent for corbyn ,13 for burnham,11 kendall,9 cooper-looks like a cchq operation to me !!

  21. Tony Dean
    “Quitea conundrum…”

    I thought I was going to have to brush up my Latin until I realised it was a typo!

  22. @Sunreada

    The grassroots Labour party might like Mr Corbyn.

    However, seeing as Labour MPs don’t take any notice of grassroots members, it won’t matter one bit!

  23. No jeff we resolved that battle in the 1980s ,it will be interesting to see whether anyone lends jeremy one or two nominations.

    After david miliband lending abbott some nominations in 2010 to get her on the ballot paper giving red ed some cover I think we can say kendall wont be doing it.

    Starmer nominated burnham tonight -with dan jarvis two very key nominations.

  24. As an outsider, the candidates for Labour leadership look a pretty sorry bunch. This isn’t meant to be partisan, but just a comparison to the previous generation like Blair, Brown, Clark, Reid etc. They all had faults but surely all of them seem (to a non-Labour person) to have more stature than the present lot?

    Perhaps it’s because more of them were in government for a long time, so were more in the public eye?

  25. @Sunreada

    Members can only vote for Jeremy in a Leadership election if 35 MPs nominate,

    Therefore, the people who MPs nominate to get on the ballot paper is entirely up to the MPs themselves.

  26. Very true jeff -be interesting to see who nominates him from here ,reckon its very much in burnhams interest to get him on the ballot paper -you can withdraw your nomination I believe.

  27. @Pete B

    I think whoever wins the leadership is probably aware that they have a massive mountain to climb, and are likely to be a person who passes the torch to someone more likely to win a GE in the future.

    Therefore, the more interesting aspects isn’t who will hold the fort up to 2020, but who will proper beneath them, ready to push for summit post 2020.

    The 2010 intake included some bright people, untainted by the Blair/Brown Governments. I recall observing Labour under John Smith, and it was clear then that Blair and Brown were shaping up well for the future.

    Who will shine now?

  28. CMJ
    That almost sounds as though you think 2020 is lost for Labour already?

  29. Too pessimistic jeff ,loads of variables-but the new leader if halfway successful would be faced with the miliband dilemma- deal or no deal with the snp.

  30. @PeteB

    If Labour can overcome unpleasant boundaries changes, an economy naturally on a growing part of the cycle, and starting 112 seats behind the Tories in England, it will be an amazing achievement.

    Strange things can happen, but the path to Labour win in 2020 looks much less than 50/50 to me.

    Of course, I may be totally wrong ;-)

  31. CMJ
    It doesn’t seem long since the conventional wisdom seemed to be that the Tories would be forever unable to form a majority government, and yet they have, Although for the foreseeable future our governments will be dominated by Con or Lab, the new parties bring some unpredicability into the situation.

  32. Thats to get overall majority jeff ,largest party would be tough but more realistic objective.

  33. sunreada

    “Too pessimistic jeff ,loads of variables-but the new leader if halfway successful would be faced with the miliband dilemma- deal or no deal with the snp.”

    To quote Pete B “That almost sounds as though you think [Scotland] 2020 is lost for Labour already?

  34. Oldnat
    BTW my last post referred to ‘new parties’. I mentally included SNP in that though of course I know they have a long history. I was thinking of them being new in the sense of a significant UK-wide influence.

  35. I’d have thought Burnham will be very keen for Corbyn to get over the line so that he isn’t seen as the left candidate… AV means any loss of first preference votes of real lefties is essentially inconsequential.

    I think he can get 35. There are still some on the left of the party undeclared and quite a few of those on Mary Creagh’s list had said they were nominating her because they wanted more than three candidates (Stephen Kinnock, for example). He may, however, have hurt his chances by saying he doesn’t want Burnham supporters to lend their nominations as Miliband supporters did for Burnham and Abbott in 2010.

  36. I’d have thought Burnham will be very keen for Corbyn to get over the line so that he isn’t seen as the left candidate… AV means any loss of first preference votes is essentially inconsequential.

    I think he can get 35. There are still some on the left of the party undeclared and quite a few of those on Mary Creagh’s list had said they were nominating her because they wanted more than three candidates (Stephen Kinnock, for example). He may, however, have hurt his chances by saying he doesn’t want Burnham supporters to lend their nominations as Miliband supporters did for Burnham and Abbott in 2010.

  37. Surely it would be better for Burnham if Corbyn wasn’t in the ballot? Burnham could then get all the left-wing votes and probably win. It might not be so good in the general election, but in 5 years’ time only political geeks will remember that he was the lefty candidate.

  38. Labour’s trouble with the new leader is that it is too soon to pick someone to fight for 2020 when:

    (1) it is still not clear why they lost and how they can get the south and scotland back at the same time.
    (2) the Tories will change their leader half way through.

    In a way John Smith’s passing during the parliament was what made Blair so popular. The writing was on the wall for the Tories after some good opposition by Smith and Robertson to the Mastrict Treaty. Then Tony Blair came in to win trust in Middle England.

  39. Oh and btw I think Liz Kendle’s performance has been hopeless. She may have the political views of Blair but not the delivery, she is far to evasive. Burnham is not performing as well as I hoped but I think he is the best so far.

  40. CMJ
    I think you are being a tad pessimistic in that the Tories will need at least 310 seats post 2020 to be able to continue even as a minority Government. Also Labour’s performance in England in 2015 was better than 1992 and 1979 in terms of both seats and vote share deficit. The collapse in Scotland has seriously distorted perceptions but at the end of the day all bar one of Scotland’s MPs can be relied upon to vote against the Tories.

  41. Graham
    Aren’t seats going to be reduced to 600 before the next election? If so, Tories only need 301 for a majority, and perhaps a few less if Sinn Fein still abstain.

  42. Pete B

    Such a definition is fine by me.

    How the “new” parties in E&W perform in 2020 will help to determine whether the incoming government will have to “deal with the SNP” or not. Unless things change dramatically in Sco and/or NI, only a party that is the largest in HoC but can’t gain a UK majority from E&W seats will have to consider SNP/DUP/SDLP etc.

    It makes sense, therefore, to look at how Labour, under FPTP, can become the largest party in E&W – regardless of what happens in Sco/NI.

    UKIP, Greens, LDs (who should be considered similarly to the “new” parties) and the “regional” parties (other than Plaid, hardly registering on the radar) don’t have to win seats to help/damage Labour, just affect the Con/Lab balance in individual marginal constituencies.

    Regardless of who becomes the new/interim (delete as preferred) Lab leader, the following factors in the next Parliament may be decisive.

    1. EUref – Assuming a Yes vote on the basis of current polls, will the No vote be large (or passionate) enough to provoke the kind of response we saw in Scotland?

    2. Administrative devolution in England – will this defuse/intensify the current tensions between the North and London (both terms used loosely !) ?

    3. What are the demographics of Labour support? If older, then more likely to shift to a rejuvenated UKIP? Is there a difference between their support in London /the North / other marginals?

    I don’t know the answers to these, and similar, questions but Lab strategists should be gaming all these permutations – or else they aren’t doing their job.

  43. ON
    I agree with all that. Regarding your third point, I think that there is a difference in UKIP support in the North and South of England. There are large areas of the North where a vote for the Tories is unthinkable for most voters, yet they will be prepared to vote UKIP if they don’t like Labour.
    In the South, the UKIP vote is more likely to come from those who think Cameron is too left-wing.

  44. Pete B

    I don’t think it is entirely clear as to Cameron’s intentions re- boundary changes. I am using 310 on the basis that we continue with 650 MPs.

  45. @all

    I must confess myself saddened abd surprised: is *nobody* planning on going to the RSS meeting?

  46. Graham
    My understanding was that the boundary changes were all passed into law in the last Parliament, but the LibDems managed to delay the changes until 2017 (I think). If I’ve got it right, Cameron’s intentions don’t matter, unless he’s going to pass another law to undo the changes.

    Of course, I could be wrong.

  47. Pete B

    Only marginally wrong!

    The 4 Boundary Commissions will restart the Boundary Review early in 2016 – to produce their final reports in 2018 for Parliamentary approval.

    Unless the law is changed again, their briefs will remain the same as for the 2013 reviews.

  48. More on the Boundary Reviews

    The electoral registers to be used are those at 1 Dec 2015.

    If the proportions within the 4 nations remain the same as in May 2015, then the allocation of seats between them will change since the 2013 reviews, by my (late night, while watching the Women’s World Cup !) calculations..

    England – 500 seats (-2)
    Scotland – 55 seats (+3)
    Wales – 29 seats (-1)
    N Ire – 16 seats (nc)

    The effects of IER, devolved elections (on the basis of the new register) and possible referendum on that basis too, should prove interesting!

  49. Hello everyone, first post since the GE from me,

    ON
    ‘1. EUref – Assuming a Yes vote on the basis of current polls, will the No vote be large (or passionate) enough to provoke the kind of response we saw in Scotland?,

    I have been wondering about this too.
    No might achieve around 40-45%, many of these would have voted Con in May because of the promise of this Ref. The Tories won’t be able to make that offer again.
    The question is, how important is EU membership to this group? Will they support the only party that will still want out and vote Ukip in 2020?
    I think the Indyref has shown, referendums don’t necessarily put an issue to bed for a generation.

  50. Good Morning All on a cloudy start by the sea, here in our Premier League Town of AFCB.

    CLOUDSPOTTER.
    Good Morning to you. I think GE 2020, like 2015, will largely be determined by perceptions of leadership competence, the economy, and how far a Party is seen to identify with the larger part of the voters.

1 2 3 4