Next Friday is the public meeting of the British Polling Council inquiry into the failure of the polls at the 2015 election, at which point I expect we’ll get some insight into what the different polling companies are thinking, though probably not many firm conclusions yet.

In the meantime the British Election Study team have published some thoughts from Jon Mellon about what the BES data could tell us about why the polls were wrong. It doesn’t include any conclusions yet, but goes through a lot of the thought processes and ways of identifying what went wrong, which I suspect may reflect what many of the pollsters are doing behind closed doors.

As yet only the online BES data from during the campaign is available for download, but in time it will be joined by their online recontact survey after the election campaign, their face-to-face survey after the campaign and voter validation data for the people interviewed in the face-to-face survey. The article has some thoughts about what they can learn from the data that’s already available and what can be learnt from the bits that are still to come:

1) The BES campaign data appears to show some movement towards the Tories over the last couple of days, though not one that is beyond the margin of error. This is in contrast with YouGov’s daily polling data, despite them coming from the same panel. This is interesting, but as Jon says, the real proof will be when the BES publish their post-election data, showing if people actually did change their minds from their pre-election answers

2) If you only take people who said they were very likely to vote it would have been more Tory… but that’s very much a “Pope is Catholic” finding. The interesting bit here is what the BES team plan on doing in the future – they are once again going to validate their face-to-face data against the marked electoral register, to see if people who claim they voted genuinely did, and how well people’s stated intention to vote compares to whether they actually did. They are also going to match the online respondents to the electoral registers before and after the new electoral registers, to see if drop off from individual electoral registration was a factor.

3) Sampling and weighting. Jon hasn’t really said anything on the data so far – he’s waiting for the face-to-face probability sample, to compare that to the results from the online polling and see if it is significantly closer to the actual result.

4) Don’t knows. According to Jon the people who said don’t know before the election were a mixed bunch – their attitudes towards the leaders, issues and party id did not point to them being obviously likely to switch to Conservative or Labour. Again, the interesting bit will be to see how they said they ended up voting in the post-election wave.

5) “Shy tories”. Jon makes two interesting points. One is about question order. While the BES campaign data came from YouGov’s panel, its results seemed to show a movement towards the Tories that the main YouGov data didn’t show – in his article Jon presents Peter Kellner’s hypothesis that this may be because of question order. As regular readers will know, the published voting intention polls all religiously ask voting intention first, but the BES actually asks some questions about the most important issues facing the country and party leaders before asking VI. However, Jon also mentions what he judges to be “weak” evidence against “shy Tory” hypothesis – the BES included a grid of questions aimed at identifying people who tended to give socially desirable answers to questions, and Conservatives scored higher, not lower, amongst those people.


151 Responses to “The British Election Study on why the polls were wrong”

1 2 3 4
  1. http://www.telegraph.co.uk/news/politics/11669159/Austerity-at-home-profligacy-abroad.html

    An interesting take on the end result of the new found fad for passing laws to govern future spending policy. This kind of thing will only get worse, with both main parties happy to play the game, until someone wakes up to the fact that we should let future governments get on with what future governments need to get on with, and have future electors either back them or sack them. Daft.

    Meanwhile, S&P have downgraded UK’s credit rating outlook, based on the impacts of the EU vote, and the BoE warns that rates will start to rise soon.

    Interesting this – rates heading back to normal, as the OBR forecasts household debt to reach a new record high by 2019 (higher than 2008). In effect, national debt is being replaced by household debt.

  2. “I don’t know the answers to these, and similar, questions but Lab strategists should be gaming all these permutations – or else they aren’t doing their job.”

    ———–

    It’s reactive though. Trying to anticipate how people will respond to events like EU ref. etc.

    Whereas what Tories/SNP have been adept at doing is changing perceptions, and harnessing events to that end.

  3. GMB union chief Paul Kenny knighted.

    You have to laarf , don’t you :-) :-) :-)

  4. Good morning,

    Heather P

    Inclined to agree regarding Liz K. She’s in the right place politically, and has some of the solutions, but is being far too timid, and anxious not to offend.

    Labour politicians need to understand that having someone who will actually lead will help rather than hinder the holding together of the party.

    Whatever your political persuasion, you have to hand it to Nicola Sturgeon: clear messages, strongly, but calmly, espoused in a confident manner. It works.

    Pretty much all the Tory leaders understand this.

    Labour tend to chant mantras, and indulge in box-ticking party-speak. It is a massive turn-off to those who didn’t vote for them, and whose support they require.

  5. Allan Christie

    Just back from four days walking holiday on the South Down. Splendid weather for walking and excellent hotel.

    Although I forecast the election accurately I do not think my views very helpful to the pollsters. Remember I have always said I take very little notice of the actual poll headline numbers, it’s the answers to detailed questions on the economy and leadership which are important in forming my predictions.

  6. 07052015

    I did not predict any of those elections, although as it happens I thought Labour would win all three.

    However I did predict the last Euro elections and the Scottish referendum more accurately than any of the pollsters.

  7. @Millie

    A lot of the appeal of Blairism was Blair himself, up to 2002, after which people started to see through him. Labour’s campaign in 2005 had to be a Blair and Brown joint effort, fostering the impression that Blair would go shortly after the 2005 GE. (Brown was popular then, not least because he was seen as standing up to Blair.)

    If Blair had had the personal charisma of E Miliband or indeed L Kendall a lot of that early electoral appeal would have been lost.

    I think that (unlike Umunna) Liz Kendall will expose the fact that, shorn of a figurehead with personal appeal, Blairism itself is a busted flush with the wider public. There will no doubt be further polls over the next couple of months, and I think they won’t show a desire from the general public for her to win, just as initial polls haven’t (notwithstanding the high number of don’t knows).

  8. Pete B

    “In the South, the UKIP vote is more likely to come from those who think Cameron is too left-wing.ete”

    I think that’s correct. I thought long and hard about voting UKIP but in the end could not vote for a party that wants to re-nationalise the railways. I ended up voting Tory as I think Osborne is the nearest to my idea of the ideal COE.

  9. Colin ,why laugh ,he is retiring after years as a constructive negotiator.His work will have resolved conflict leading to better economic growth etc than otherwise.

    His union or labour will have nominated him and this batch are from the queen so he can say its nothing to do with the tories-not that his activists will believe him.

  10. SUNREADA

    I stand corrected-he has been such a staunch supporter of the unelected chamber, privilege & the Establishment.

    No wonder he is pleased. :-)

  11. ITF Tory & Tony Dean

    I think it does seem clear that pollsters were wrong for a significant period of at least the last parliament and possibly into the last one.

    Just been looking at the Welsh Assembly Yougov polling vs the actual result in 2011 and the polls almost all suggest a Lab lead over the Tories of around 24-27% dependent on FPTP or Regional Vote but instead they achieved 14-17% lead over the Tories.

    Either there was a swing of 5% in the final few weeks/days before the election or the polls were already significantly out in terms of reflecting public opinion.

  12. @Gary O

    In hindsight we should have picked up on the inaccuracies in by-election polling more… as NumberCrucher Politics showed in this terrifically bold blog the day before the GE there has been a consistent trend in recent (post-2014) by-elections for which there was polling to underestimate the Tories – http://www.ncpolitics.uk/2015/05/shy-tory-factor-2015.html/6/

    In most of these contests UKIP got most of the attention which probably diverted us from the scent…

    That this error is consistent with a similar error in Euro election polling during this period just adds to the evidence that polls were out well before the last week before the election.

  13. Colin didnt realise he would now be in the other place -but then you would know about that stuff I guess .Anyway reckon he has done a bit more for old blighty than the treasurer of the conservatives who also gets the call.

  14. I know toh we all regard you as a forecasting genius.

    What about 1992 or 1945 they were both rather tricky.?

  15. @AnthonyWells

    I hate to nag, but if you are going to the RSS meeting next Friday, I would be genuinely grateful if you (or anybody reading this) could do a writeup and retrieve any handouts.

  16. 07052015

    I did not follow the 1992 election very closely as I was ill but I thought the Tories would probably win. In 1945 I was only interested in the model Spitfire my Uncle had made for me.

    :-)

  17. “Idid not follow the 1992 election very closely as I was ill but I thought the Tories would probably win.”

    —–

    As it happens, the Tories did win, just so you know…

  18. @ Carfrew

    “As it happens, the Tories did win, just so you know…”

    I thought it was a draw, and there was a five-year long extra time (it was before the penalty shoot out was introduced). Well, with age memory often fades.

  19. Interesting information from AW. It just shows you how wrong the British voters got it. I think the electorate have let the Labour party and the BBC down shamefully.

  20. “I thought it was a draw, and there was a five-year long extra time (it was before the penalty shoot out was introduced). Well, with age memory often fades.”

    ————

    Well, I didn’t wanna get into what happened next, but I s’pose if it helps fill any gaps for Howard…

  21. @roly
    Has it ever crossed your mind to change the record?

  22. GUY MONDE
    No.

  23. @ ToH I don’t agree with the hypothesis that your views are of not relevance to the pollsters. I think they are very relevant.

    As I understand it you have two basic ideas:

    a) peoples views are in the end shaped by issues and
    b) what one might call the ‘hovering pencil’ hypothesis – when it comes to the vote people faced with their image of Ed Milliband or an out of control spending spree will vote for the devil they know in the hope of getting on with their lives.

    A) would get support from what Anthony says about the effect of the order of questions with pollsters who put questions about the issues before questions about voting intention getting closer to the result than those who didn’t
    b) Is I suspect also right, and is said to draw support from psychological evidence that when people actually come to choose they opt for the less risky option rather than the one that offers the greatest gains. (Hence perhaps a slight shift in the Scottish referendum back towards the No camp right at the end (And I remember somebody saying that the same happened in the Quebec one)

    Personally I think that other reasons for the failure of the polls to get things exactly right have to do with likelihood to vote and weighting (Either there was too little weight given to elderly voters or those that were included in the sample were not representative of old people as a group). However, I do not think these factors even if true played much of a role in your predictive success to which along with others I pay tribute.

  24. Agreed that the biggest problem the “Blairites” have is their label. If only they could call themselves something less controversial, that would help their cause immensely.
    There is nearly always strong demand for a viable and acceptable alternative to current incumbent.

    As for Cameron and the EU question, it does look as if he is being pulled in half. Ouch.

  25. The Telegraph is reporting that the new Eurosceptic group (Conservatives for Britain) has signed up 110 MPs including some (unnamed Cabinet ministers).

    The same piece also mentions an amendment calling for a 16 week campaign, and a possible sizeable rebellion over scrapping the usual purdah rules.

    The purdah rules issues strikes me as a difficult call for the (pro EU membership) opposition given that the government position is not guaranteed to be for staying in the EU.

    http://www.telegraph.co.uk/news/politics/conservative/11673356/Cabinet-ministers-join-Tory-rebels-in-latest-blow-to-David-Cameron-on-EU.html

  26. @Phil Haines

    I agree that Liz K is unlikely to win: I have said all along that Andy B is much the likelier. I note that he is shortening with the bookmakers.

    Matthew Parris ( who got the election result right incidentally ) wrote an interesting piece in yesterday’s Times about the ( perhaps temporary ) triumph of capitalism over socialism, and its relevance to the Labour situation.

    He argued that Labour should embrace capitalism, as the general population has done, largely. And to abandon socialism entirely. How many of the general population now describe themselves as ‘socialists’? Very few I would suggest.

    It seems to me that Parris properly summed up the core question that Labour needs to resolve: is it an unashamedly pro-capitalist party, or is it a reluctant participant seeking a socialist ‘control of the commanding heights’?

    If they choose the latter route then I would not say they are doomed, but they may well have a very long wait for power.

  27. @ Millie

    With respect to the contests to lead the losing parties (Lab/LD/UKIP) at the GE, the results don’t matter. NF has “unresigned” and forfeited credibility by his actions. The next Lab leader won’t be PM; the next LiS and LD leaders are an irrelevance as these parties were annihilated. BTW, whose mad idea is it to suggest that the other Milliband would be any less toxic than his brother?

  28. @ ExileinYorks

    There is no way that the UK will leave the EU. The referendum is a charade, which DC has only been forced to carry out as the price for enticing enough UKIP voters to switch to the Cons at the GE to give them an absolute majority. The entire establishment plus the SNP are in favour of staying in, so there won’t be a no vote. Peter Hitchens in his DM blog has some interesting comments about these issues.

  29. @Millie

    I would suggest that what proportion of people are ‘socialists’ depends on a subjective definition of what socialism is.

    While only a tiny minority of people would want to totally dismantled all aspects of capitalism, very many more worry that pure free market market economics are not a panacea, and some control by Government is required to control market excesses. Free markets are rubbish at delivering social services and emergency health care. Free markets reward winners, but punish losers. How acceptable is it to allow ‘losers’ ie those without the skills, or capability to compete, to fall away at the the bottom?

    This is where the heart of the debate lies.

    This debate is as crucial today as it has ever been.

  30. @Millie

    May I also add that in fact Labour has embraced capitalism for a very long time.

    I see no evidence to the say otherwise.

  31. Dunham111

    DC made himself a hostage to an EU Ref a long, long, time before the 2015 GE. Pandering to the Eurosceptics in the Tory party was part of his price for getting elected as party leader.

    While the negotiation exercise is a bit of a charade, the vote itself is real, and far from a foregone conclusion. Voters will not necessarily meekly go along with the establishment’s wishes.

    Who will be motivated to turn out and vote?

    If, as is quite likely, Greece is forced out of the Euro, what turmoil will follow, and how will this affect the EU Ref?

    Will the electorate treat the vote purely as a question on membership of the EU, or will they to some extent treat it as a referendum on the government itself?

    Which parts of the media will support Yes and which No?

    Will the Eurosceptics be successful in muzzling the BBC on the issue?

    Will the government openly campaigning for Yes help or hinder?

    I do not share your optimism that there won’t be a No vote, there are far too many unknowns at this stage.

  32. @ ExileinYorks

    You are implying that I am optimistic that there will be a YES vote. I am merely realistic re prediction of the result. I don’t think that EU membership is necessarily good from an English perspective, given German dominance of the EU, whereas the minnows of the Celtic fringe benefit from the EU’s largesse.

  33. Dunham

    My apologies for appearing to attribute you with an allegiance to YES.

  34. Interesting analysis in todays Sunday Times by Liam Byrne on why Labour lost.

    The Progressive consensus is a myth.

    Labour lost Blue Collar , and Older voters to Cons.

    CMJ-no one in UK is advocating “pure free market market economics “-if by that you mean unregulated markets.

    The regulation of markets is necessary-thats why we have so much of it. It is why Governor Carney stood up and explained at Mansion House what a time bomb the poor regulation of Financial Markets in UK was before the Coalition gave him the right tools.

    If you continue to believe in all that Private Bad /Public Good stuff you will become more & more detached from the reality of modern society , in which the State Funds but no longer Provides all the services its citizens require.

  35. The EU referendum will come down to ‘the economy stupid’.

    Coming up with a list of EU issues people aren’t happy with is easy.

    Persuading them that the economic uncertainty of withdrawing from the EU is worth it to assuage these grievances is quite different.

    I think it may mirror the Scottish referendum. The deck is stacked in favour of the less risky status quo.

  36. @Colin

    I don’t believe that all private is good, and all public is bad.

    It’s far more nuanced, issue by issue.

  37. I hear, both in my own personal experience and in media vix-pops and so on, lots of people saying that they don’t know much about why we should be in the EU and would like to hear the argument before making up their mind. Once the argument has been heard I expect around two thirds of the electorate will back staying in (already, before the argument has been heard, Yes has a fairly clear polling lead). Not because the public necessarily do what the establishment tells them but because people don’t like risk and there are a lot more holes that will be exposed in the out case than the in one.

    Inevitably Eurospectics with the assistance of media noise are turning up the heat. Committee stage of the referendum bill this week will certainly be interesting. Conservatives for Britain seems to me to be itself divided – some members pretty much sharing Cameron’s position, others throwing out completely ridiculous red lines like a total parliamentary veto on EU legislation. They could defeat the government on purdah though I expect a compromise amendment will be put down by the govt.

  38. @Colin

    I meant to say:

    I don’t believe that all private is bad and all public is good.

    The rest stands :-)

  39. CMJ

    “I think it may mirror the Scottish referendum. The deck is stacked in favour of the less risky status quo.”

    While I think that is true, there are other comparisons with the indyref that might be borne in mind for politics in the longer term –

    It’s not just winning the referendum battle, but securing the “peace” that matters.

    Did the indyref result produce a more united Kingdom – or a more divided one?

    While the most partisan losers will always scream “we wus cheated!”, how will those who were less definite about their choice respond? Trust in the “institutions of the Union” (whether that be UK or EU) can be enhanced or destroyed by how they are seen in the “normal (largely apolitical) population.

    Scare stories can be effective in a campaign, but can backfire in the longer term. Some of the scares that the winning side predicted would happen if the UK leaves/stays will happen anyway!

    Will either side produce a “vision for the future” that most can buy into (to some extent at least) or will both sides use Fear as their dominant tactic?

    The politics of campaigning can be as important as the result of a single vote on a single day.

  40. Charles

    Thanks for your comments. The point i was making was that I disregard the headline figures and of course this is what the pollsters are measured on.

  41. @ToH

    I understood that you distrust the headline figures. I thought you also had your own way of predicting the outcome which relied on opinions about the issues, particularly leadership and the economy. It seems to me that this way of going about things is relevant to the pollsters and they should take you seriously!

  42. @OldNat

    The EU referendum will not affect the underlying disenchantment in the UK (or at least England) with the big two UK parties.

    These underlying problems are rooted in the destruction and/ or collapse of UK mining, manufacturing and heavy industry and in particular the communities villages and towns that this industries sustained. Leaving the EU will not restore the quality of life these communities previously enjoyed and so the resentment will remain.

  43. @Oldnat

    I do agree with you.

    The way the different nations within the UK relate to each other, and the UK relates to the world around is out of it’s historical straight jacket.

    Constitutionally, Scotland is unsettled and will remain so in my opinion. The current settlement cannot hold.

    I think our EU position will follow this same path. A yes vote won’t put the driving forces behind the no vote back into the box.

    I would love a positive campaign, but I very much doubt we will get it. I expect mud-slinging, insults and negativity.

    Scotland and the EU position are sourced from the same place – the UK settlement is breaking up. I think we can only build something better once we have hit the bottom, a place we are not at just yet.

    I also think what parties will do well and which will wither in this new settlement is uncertain too.

  44. Charles

    You are certainly correct about how i make my predictions but I suspect the pollsters are paid on the basis of the correctness of those headline figures.

  45. CMJ

    Clearly no-one advocates unfettered capitalism, and regulation, and redistribution of profit is required. Matthew Parris’s point was that the debate between capitalism and socialism is effectively over: capitalism, albeit it not in its purest form, has triumphed.

    I think where you are wrong, if you will forgive me, is with the suggestion that Labour has ’embraced capitalism’: the Blairites probably did, and sometimes in a rather undignified way, but a large proportion of the Party clearly resent business, and still long for a socialist agenda to prevail.

    A modern Labour Party, which seriously wants to win, has to do exactly what you say it has done, but I respectfully suggest it hasn’t, which is ’embrace capitalism’.

    Parris went on to say that the Tories are so in love with business that they fail to properly regulate it, and this is the role that Labour should pursue.

    In other words, a healthy and rigorous scepticism, not resentment, animosity and ‘chip on shoulder’ moaning.

    They can’t go on as they are, can they?.

  46. Polling continues – despite difficulties with measuring the responses of supporters of two parties in a bit of one EU member. :-)

    Buzzfeed/Ipsos MORI ran a poll in 23 countries on attitudes to unrestricted abortion.

    http://www.buzzfeed.com/lesterfeder/this-is-how-23-countries-around-the-world-feel-about-abortio#.civ0WZYd2

    66% in GB support abortion “whenever a woman decides she wants one”.

  47. @Millie
    “Parris went on to say that the Tories are so in love with business that they fail to properly regulate it, and this is the role that Labour should pursue. In other words, a healthy and rigorous scepticism, not resentment, animosity and ‘chip on shoulder’ moaning.”

    Miliband wanted to do no more than that (i.e. properly regulate business and address profiteering where underregulated markets were failing). Not only would Labour be selling its soul if it watered down even that mild approach and returned to the days of New Labour, there would also be absolutely no electoral mileage in it..

  48. @Millie

    Thanks for your response.

    I haven’t seen anything close to a socialist agenda from any mainstream party for a very, very long time.

    Most of the policies that I see called for by people on the left in my view are not not socialist, but social democratic. Read ‘The Road to Wigan Pier’. That is genuine socialism. Perhaps Attlee’s Government could be called socialist, but no government since has been even close to those ‘ideals’, IMHO.

    It does make me laugh that when the media comment on ‘socialist’ policies, when they are clearly not by any definition I recognise. More often than not it’s just lazy journalism and intellectual nonsense.

    Go back to the clause 4 debate. Clause 4 was symbolic and a sacred cow, but no more than that. I think when most Labour Party members go misty-eyed about the loss of clause 4, it’s like John Major’s back to basics world of warm beer, people cycling to church and cricket. The world never existed quite like that.

    I would place a large amount of money that if a Labour leadership candidate offered a real socialist agenda, the membership would mostly run a mile. Across the board people hang onto ideas of ideological purity within their party, but in practice support those who are see the world as shades of grey, not black and white.

  49. MILLIE.
    Good Evening to you from Bournemouth East seat-by-the-sea, with the same number of points as MUFC.

    I think the new leader should address the capitalist theme with a new, revised, again, Clause 4.

1 2 3 4