Jon Mellon and Chris Prosser from the British Election Study team have written a new post and paper on the emerging evidence from the BES data on what went wrong with the polls. Last month they wrote a piece, which I covered here, on some of the potential causes of error they could use the BES data to look at. Now the BES post-election data is out they’ve done so, and come back with some findings.

Firstly, late swing – the BES data finds virtually no net change at all between how people said they would vote pre-election and how they reported having voted after the election. The BES team conclude from this that late swing is unlikely. We’ve now got published re-contact data from the British Election Study, ICM, Opinium, Populus and Survation, only Survation found any obvious evidence of late swing in their re-contact survey.

Secondly, Shy Tories. This is essentially the most difficult potential cause to evidence – if people lie before the election, and lie after the election and we can’t check their actual ballot papers, how do you detect it? You need to look for circumstantial evidence. The BES team have compared levels of Tory support in their polling in different types of area, on the assumption that if people feeling embarrassed to admit voting Tory really was a problem it would be less of an issue in heavily Tory areas than in areas where no one else voted Tory. They did not find this pattern. They also have some experimental data about question order and priming, one suggested solution to the polling error. The first three waves of the BES, conducted back in 2014, randomised where it asked the voting intention question – at the start, or later in the survey. Asking it later in the survey only made a minimal, non-significant change to the Tory vote (while it doesn’t say so in the article, the full paper also makes clear it doesn’t change the Labour vote either!)

Thirdly the BES team looked a bit at sampling, specifically around age, looking at an issue Opinium have already commented on at the BPC inquiry meeting. All pollsters weight by age using various age bands such as 18-24, 25-40, etc, etc. But are people evenly distributed within those bands? Within the top age group, for example, the BES found that people over the age of 80 were underrepresented and people in their early 60s were overrepresented. Whether that makes any different to the results they can’t yet say, as it may be countered by other things like political weighting.

Finally, and most importantly, they wrote about turnout and suggested that people may have been overestimating their likelihood to vote, and that the people who were actually less likely to vote were increasingly skewed towards Labour. Currently this pretty circumstantial evidence – we don’t know if people lied about voting in the general election, but there’s evidence to suggest they might have. For example, a small proportion of people said they had already voted by post before most of the ballot papers had even been sent out, in areas where there were not any local elections this May there was still a chunk of people who reported having voted in their local elections. At the moment, these people who look as if they might be lying disproportionately break to Labour, so would explain some of the error. In their article Jon and Chris instead try modelling people’s likelihood to vote based on their demographics and characteristics of their seat, and that increases the Tory lead by 1.8%. They conclude that turnout, people saying they’ll vote when they won’t, is a major factor behind the error, thought they conclude that it’s one pollsters can probably address quite easily through a better turnout model.

Two caveats though, before you think things are solved. One – at the moment we’re going on indirect evidence of people overstating their likelihood to vote. In the fullness of time the BES are going to do a validation exercise of their data (that is, checking respondents names against the marked electoral register) so they will be able to conclusively prove whether or not there are a significant proportion of people who told pollsters they voted, but lied about it. Secondly, Jon and Chris estimate that getting turnout wrong probably explains about a quarter of the difference between the final polls and the election result, which would still leave us another three-quarters to explain…

Meanwhile there have been two new Scottish polls in the last week – both show the SNP still on course for another landslide at the Holyrood elections next year.

Survation for the Scottish Daily Mail have Holyrood voting intentions of
Constituency: CON 14%, LAB 20%, LDEM 7%, SNP 56%, Others 4%
Regional: CON 12%, LAB 19%, LDEM 8%, SNP 45%, GRN 11%, UKIP 5%
(Tabs here)

TNS have Holyrood voting intentions of:
Constituency: CON 14%, LAB 20%, LDEM 5%, SNP 60%, Others 2%
Regional: CON 13%, LAB 21%, LDEM 5%, SNP 51%, GRN 7%
(Tabs here)


301 Responses to “More BES findings on What Went Wrong”

1 2 3 7
  1. They are extraordinary polls for the SNP. The party appears to be on the crest of a wave both at Westminster and Holryrood.

    As far as I’m concerned the Holyrood election is already over.

  2. “At the moment, these people who look as if they might be lying disproportionately break to Labour, so would explain some of the error. In their article Jon and Chris instead try modelling people’s likelihood to vote based on their demographics and characteristics of their seat, and that increases the Tory lead by 1.8%”

    “They conclude that turnout, people saying they’ll vote when they won’t, is a major factor behind the error, thought they conclude that it’s one pollsters can probably address quite easily through a better turnout model”
    _______

    Some call it White Van Man syndrome or something like that.

    I don’t think it’s so much people are lying but rather those who say they will vote Labour really would vote Labour if they bothered to get their backsides round to the polling stations.

    Older people (ie Tory voters in England) are more likely to vote but also more likely to hold their cards to their chests when asked who they will vote for. They probably see it as no ones business and like to give pollsters the run around.

  3. So is the registration issue still on the table then….

  4. Anthony

    Early analysis showed that the polling problem was concentrated in England, outside London.

    Have the BES anything on that?

    If they concentrated on those areas might that increase the % of the difference explained by “lying Labour non-voters”?

  5. If they concentrated on those areas might that increase the % of the difference explained by “lying Labour non-voters”?

    ———-

    So what’s the euphemism for that, à la “shy voters”.

    “Teasers”?

  6. Carfrew

    “lying Labour non-voters”? WAS the euphemism! :-)

  7. Observant tweet from Jamie Ross at the Farron Press Conference –

    All of the young people in the room have been made to gather behind the podium where Farron is due to speak so I think we’re about to begin.

  8. @oldnat

    I’m not sure that really counts as a euphemism. But yeah, it’s prolly going easy on them by your standards. Am afeared as to what your description would be sans euphemism!!

  9. Carfrew

    That description is under tight security in storage – protected by a thorium powered electric fence.

  10. @oldnat

    Fair enough. At least someone sees the many uses of Thorium, and storage…

  11. oldnat

    carfrew

    “lying Labour non-voters” = brush-offers

    SNP MPs = brush-oners

  12. Think about this:

    Perhaps by constantly trying to reduce costs, pollsters have lost the random element which they surely need to produce accurate results?

    This thought came to me last week when we were approached by the British Social Attitudes survey run for the Government by NatCen.

    They explained how we were chosen.

    First, households were selected completely randomly from the postcode address file (not the same as the electoral register). One household in many thousands is chosen. Then a person is chosen from that household according to a formula, eg a one person household it is that person, a two personal household it is the second person (alphabetically), and there is a formula which carries on for larger households. As they say, ‘we randomly select addresses and individuals to take part. Unfortunately we cannot replace you with anybody else’.

    First a letter from them arrives, followed by a phone call to make an appointment, then an interviewer arrives to ask the questions which takes well over half an hour, there are a lot of questions and opinions canvassed. The interviewee gets a £15 voucher for their trouble, cashable at the post office.

    This is obviously an expensive process, but it must produce a truly representative sample.

    That, I am sure, is how opinion polls were also conducted in the early days.

    But what do we have now? Telephone polling, limited to those who have phones and who do answer calls from unknown numbers, and who are in fact willing to answer the questions truthfully.

    Or internet polling, directed at panels of people, but again limited to those who have an internet facility and want to participate, so they are almost self-selecting.

    Anyone can see the difference in accuracy between these two cheap methods and the thorough methods which such as NatCen use.

    We can conclude that the failure of the polls lay, not in late swing or shy Tories, or turnout, or anything else analysed by Mellon and Prosser above,

    It is caused by cutting corners and producing an unrepresentative sample in the first place!

    Opinion polls cannot be done on the cheap, and will continue to suffer loss of credibility until they are prepared to spend the money needed to get a truly representative sample.

  13. Everything emerging so far is consistent with the view that the failure of the new system of individual electoral registration contributed to the polling failure to a significant degree. Firstly, the evidence is pointing to differential turnout as the most significant culprit. Secondly, there is a further snippet within the BES summary which AW doesn’t report, that “the increased impact of differential turnout is not due to a change in the relative enthusiasm between Labour and Conservative supporters since 2010”. That is, the enthusiasm of Labour respondents (relative to Conservatives) didn’t decline, rather that such retained enthusiasm mysteriously didn’t translate into actually voting this time compared to last. And that’s what you would expect from a new system which made it harder for people without long term settled addresses to get onto the register.

    When the BES make comparisons with the marked register, they need to record not only whether people are recorded or not as having voted, but also whether their names appeared there in the first place.

  14. In their analyses, are the polling industry paying much attention to the registration thing? How often do they mention it?

  15. Phil Haines

    The much greater accuracy of polling in Scotland (where the referendum produced a much higher level of registration) would also support your thesis.

    However, what of Wales and London – where polling also seems to have been much more accurate? Are there special circumstances re registration there as well?

  16. @ AW – sorry this is completely off thread, and off subject, but are we going to get ANY proper polling of the Labour race apart from the Corbyn Campaign leaking their own polls saying their man is doing great.

    This would seem a good time to do it, if it is going to happen, as people are about to depart on holidays!

  17. Adrian B

    Has the Corbyn campaign been leaking private polls as well?

    The only report I’ve seen was of one of Corbyn’s opponents (Kendall? Cooper?) leaking their polling that Corbyn was “doing great”.

  18. Would be nice to have some more Labour Leadership polling. My home CLP (North West Leics. – 1997 Labour majority 13,000, 2015 Tory majority 11,000) has just nominated Burnham.

    If it doesn’t stray too far into the realms of spam, my students’ union is considering closing down half the student media hub (the half where the paper is done) and making 100 committee members across print, radio, TV and online work out of one room.

    Like all great causes, we have a petition: https://www.change.org/p/sheffield-students-union-don-t-close-down-a-prized-student-newspaper-office?recruiter=298171573

  19. Good Evening All, on a sultry Premier League Town day in Tobias Ellwood’s seat.
    I think the polling companies should consider avoiding any GE polling.
    Voters just vote, or not, the way they vote, and the mysteries are impossible to penetrate; IMO.

  20. Mr N

    As a matter of interest, how many people in North West Leics voted in the nomination process?

  21. Judging by the photo the CLP tweeted I’d say about 30, from a membership of about 300. This is why nominations should be taken with a pinch of salt.

  22. Just noticed how London heavy Liz Kendall’s nominations are. Eight of her 10 – the two outside London are Moray and Newton Abbott, both small CLPs.

    Now a good 40% of members are in London. But she can’t win on them alone.

  23. Thanks MrNameless you have just encouraged me that the CLP nominations mean diddly squat! Although, they obviously get reported and create a narrative.

  24. MrNameless

    Just noticed how London heavy Liz Kendall’s nominations are. Eight of her 10 – the two outside London are Moray and Newton Abbott, both small CLPs.

    Now a good 40% of members are in London. But she can’t win on them alone.

    I think this is a myth about the high percentage of members in London. Certainly the last figures we have from the 2010 leadership election showed only 20% of members in the 73 London constituencies. That’s more than pro-rata (73 is only 11% of 650) but it’s not overwhelming.

    The latest HoC report of Party membership:

    http://researchbriefings.files.parliament.uk/documents/SN05125/SN05125.pdf

    shows that Labour’s stayed fairly static at around 190,000 for the last four years. It’s possible that they have been losing members everywhere but London and gaining there in compensation, but it would be a big shift. Non-London constituencies would all have to lose an average 25% nett of members, while it doubled in London.

    Now there do seem to have been an increase in membership because new ones joined during or since the election, maybe 50,000 – 60,000[1]. But again there’s no reason why these should be concentrated in London – indeed the opposite might be true.

    Incidentally in 2010 Moray had the fourth smallest membership – and that’s counting Orkney and Shetland separately (at #1 and 2).

    [1] This isn’t unusual, many Parties saw a jump in 2010 though not as dramatic. Such boosts in membership tend to follow elections (or rather electoral events as we saw with the Scottish Referendum). There are various technical reasons for this, but interestingly it seems to even more marked in defeat than in victory (again see Scotland last year). The Lib Dems apparently gained 20,000 members – maybe 50% extra – though you do wonder if some may have been holding back waiting till Clegg went.

  25. Meanwhile, the Lib Dems have made two local by-election gains tonight… including in Wrexham where they won 52% of the vote, not having stood last time…

    Tim Farron obviously working his magic already :)

  26. Also the LDs managed to hold a seat in Kingston (previously uber-marginal) by almost 1000 votes – a massive margin in council by-election terms!

  27. Good late evening all from a very wet East Ren.

    I thought the photo of EM in Alex Salmond’s pocket was cruel but this takes it to the next level.

    https://www.facebook.com/AnimalRightsUK/photos/a.181645081854292.42393.181638208521646/1003085169710275/?type=1&theater

    Not sure if the link will work.

  28. Scottish polling provides a useful insight into the extent that people can be usefully labelled as “X Party voters” – in a way that just isn’t available in England’s one Parliament model.

    It seems unlikely that Scots are somehow strangely variant from the behaviours of those south of the border or elsewhere – so the different patterns may actually illuminate the extent to which people can be considered to “own” (or be owned by) parties.

    In the Survation poll, VI, for both Constituencies and Lists, is shown by recalled Westminster VI in May.

    W/M vote : Retention in HR Const : Vote transfer : : Retention in HR List : Vote transfer

    SNP : Const 98% : insignificant : List 83% : 8% SGP ; 4% Lab
    Lab : Const 85% : 8% SNP ; Con 7% : List 73% : 12% SNP; 6% SGP; 5% Con; 4% LD
    Con : Const 80% : 10% SNP; 6% Lab : List 79% : 8% LD; 6% Lab; 4% SGP; 2% UKIP

    There are the “obvious” vote transfers between the indy (SNP/SGP) and Unionist (Con/Lab/LD) parties on the constitutional issue and Left (SNP[1]/Grn/Lab) and Right (SNP[1]/Con/LD) on more traditional lines, but for different voters different issues can determine alternative courses of voting action.

    Voters aren’t clones that can be programmed by political leaders!

    [1] I’ve put the SNP into both sets, as from different perspectives they can exhibit both left & right characteristics!

  29. JACK SHELDON

    That’s a truly magnificent set of by-election results for the Lib/Dems. A truly remarkable turn around for the party in such a short space of time since its GE kicking but all credit must go to the new leader Tim Farron.

    It gets even better for them. In the most recent Scottish Holyrood polls the party looks set to gain a (list MSP) somewhere in the West of Scotland.

    Truly remarkable stuff and I only wish Richard in Norway was with us to join in the Lib/Dem bounty.

  30. Adrian B

    […]are we going to get ANY proper polling of the Labour race apart from the Corbyn Campaign leaking their own polls saying their man is doing great.

    That piece in the New Statesman which claimed that two ‘private polls’ showed Corbyn was actually ahead on the first round:

    http://www.newstatesman.com/politics/2015/07/jeremy-corbyn-course-come-top-labour-leadership-election

    doesn’t necessarily come from his camp, and you certainly wouldn’t expect two different ones. the ‘information’ may be more from other candidates’ teams trying to scare voters to back their candidate to keep out the Bogey Man. It sounds like the sort of thing SpAds get up to.

    One interesting thing I found in the piece was the reference to the 50,000 plus recent Labour members who it was claimed were heavily biased towards Corbyn. Then consider in the ‘supporters’ who have also registered to vote – perhaps 50,000 with 30,000 from UNITE alone. These also may be Corbyn backers – as UNITE has recommended. Both sets of these will not be much involved in such things as nominations and yet may make up a third of the electorate. So Corbyn may be doing even better than hinted.

    As to polling, I suspect we will see YouGov running a poll sometime later, but not before the end of August. The electoral roll doesn’t even close till 12 August and ballot papers go out on the Th. Voting is open till 10 September and there will be no point in doing a poll much before most people have received their form. In 2010 the poll was taken 7-9 September.

  31. So does this mean Chrislane is gonna start saying it’s a bit low for the LibDems?

  32. Above post refers to TNS data – not Survation! and on second thoughts, Lab should have been treated in the same way in my analysis above – ambivalent as whether they should be considered Left or Right.

    While the regional crossbreaks in TNS are too small to be reliable, the strength of the SGP vote in different areas doesn’t seem unlikely – Glasgow 23% (the power of Pat Harvie!) : Lothian 15% (the traditional Green heartland) : Highlands & Islands 12% (popular List MSPs defecting from SNP to SGP) : more surprisingly Central 7% (Pat Harvie again?)with little support showing elsewhere.

    That might suggest a more radical SGP group of MSPs pushing the SNP hard on issues like Land Reform, where they have been active, but not pushing a hard line.

  33. Damn! No idea where the “above post” on the TNS poll went.

  34. @Oldnat

    Prolly a “shy post”!!

  35. Meanwhile back in UKIP land (Clacton), after five defections and expulsions and one resignation in just about a month, UKIP still held one of their Council seats in a by-election yesterday. Seems like nothing is going to shake them round these parts!

  36. Polling is obviously important. At the very least party leaders’ standing and parties themselves are evaluated partly on how well they are doing ‘in the polls.’ Polling even cropped up at an evening class I went to earlier this week.

    Individual registration may well be an issue, though I thought I had heard that no-one had been deleted from the current registers so far. But differential levels of registration in future will certainly need to be taken into account.

    Like some others on here I have had a few doubts about the true randomness of the samples, but I guess the companies have to start somewhere.

    One less scientific thought on popularity. Some party leaders, Margaret Thatcher for example, seem to have attracted strong feelings for or against, some other politicians produce less reaction overall. I wonder if it is ever possible for some politicians to annoy their detractors intensely, while at the same time somehow failing to inspire their own supporters to go out and vote?

  37. Norbold,

    To be fair, they held it with a much reduced vote share and with big swings to both Labour and the Tories.

  38. Thanks AW an interesting insight into what has been established so far. I remember the polls showing that Labour voters were almost as likely to vote as Tory voters. I also remember thinking “I bet that’s wrong”.

    Looking at your caveats there is obviously still a long way to go, and personally I suspect there will never be a clear answer. I though Clive Elliot’s post above made some good points. Is any work being done on how truly random the sampling is?

  39. The total number of clp nominations ia apparently well below 2010-activists taking defeat harder than then when it was obvious ? Or underwhelmed by the choice.

    Here in the progressive capital of England we have all the candidates in town for hustings tomorrow.

  40. Clive Elliot’s post is very thoughtful-but:-

    Is the objective in a Political VI OP a “random sample” actually ? I thought the objective was a demographically representative sample-hence the use of weightings applied to the raw sample data.

    If the 2015 GE error was due to fundamental sampling errors , wouldn’t the results have produced zig zag graphs , with no discernable trend-rather than the steady polldrums which were produced?

  41. Mr N

    True. But I’m shocked they held it at all after all the goings on with them recently. I forgot to mention the four UKIP members who defected and then returned…..

  42. @AnthonyWells

    The complacency (not by you!) of the pollsters regarding differential turnout troubles me. All we know is that people with a certain set of characteristics had different turnout this time. But that doesn’t mean they had the same differential last time and it *especially* doesn’t mean they’ll have the same differential next time. A poll that uses a differential turnout model isn’t fixed, it’s broken.

    I’m not under any illusion that anybody will listen to me on this, but I am saddened. I quite like polling and will be disheartened by this.

  43. “The total number of clp nominations ia apparently well below 2010-activists taking defeat harder than then when it was obvious?”

    CLP nominations are far less important than in 2010 because prior to the Collins Report the point was often to influence constituencies’ MPs who held a third of votes.

  44. I would have a lot more confidence in a report on what went wrong with the polls if it came from an independent source. But this one comes from the very people, BES, who made one of the worst predictions of all (their final prediction actually showed Labour as the largest party!)

    So I am not at all surprised to see that their report makes the excuse ‘people lied to us’ rather than admit ‘we completely messed up’.

  45. @Adrian B

    There was some discussion about the practicality of polling party members when the “Corbyn is ahead” story was published. Apparently only YouGov identify party members, so realistically only they could actually conduct such a poll. The other companies simply couldn’t do it cost-effectively because there aren’t that many party members relative to general population.

    I think somebody from YouGov denied conducting the “Corbyn is ahead” poll, so it’s a bit of a mystery what that story was based on. Canvass returns?

  46. We’ve got our CLP nomination meeting on 27 July.

    The way things are going down here, we’ll probably nominate Carswell!!!

  47. Mr N very true -Works the other way round now as mps nominations are key and then embarrassing for them if their clp nominates another candidate .

    I see Frank Field the man who gets nothing wrong and who nominated Ed M in 2010(whatever he says now) nominated Corbyn .Be afraid be really afraid.

  48. My wife’s trade union sent a post in which she has to declare that she doesn’t support any other party than Labour (I guess it is for the leadership election). She could – the party that she would support doesn’t exist, and she would support Labour more than others. Qualitative and quantitative differences …

  49. Laszlo,

    I’m fairly sure by support they mean “campaign for or contribute money to”.

  50. @ Roger Mexico,

    Last time in 2010 You Gov did two polls – one in July (around now) and one at the beginning of September (check with the Wikipedia entry for the 2010 election). So I was hoping to see something around now – particularly as people (who knows who) are leaking their own “private polling”.

    It is setting a narrative and may be totally fictitious and some independent properly weighted polling would help everyone have a more sensible (or at least more informed) debate.

1 2 3 7