The day after the election the British Polling Council announced it was going to have an inquiry into what went wrong with the polls, we’ve now got some more information about how the inquiry is going to proceed. Over on the National Centre for Research Methods website they have announced the membership of the inquiry team, timings and terms of reference.

The Chairman of the inquiry, Pat Sturgis, was announced earlier this month. The rest of the team include several names who regular readers will be familiar with: Steve Fisher from Oxford University who ran the ElectionEtc model and worked on the exit poll, as did Jouni Kuha of the LSE, Will Jennings of Southampton University who is part of the Polling Observatory team, Jane Green of Manchester University who is the current Director of the British Election Study and Ben Lauderdale of the LSE who did the ElectionForecast model that was on Newsnight and 538. The rest of the inquiry team are Nick Baker of Quadrangle Research, Mario Callegaro of Google and Patten Smith of Ipsos MORI.

The terms of reference for the inquiry are to assess the accuracy of the 2015 polls and investigate the cause of any inaccuracy, whether it’s connected to inaccuracy at previous elections, to look into the possibility of herding, to see if enough information was provided and communicated to people about how polls were done and what they meant and make recommendations on how polls are conducted and published in the future and on the rules and obligations of the BPC.

The inquiry are inviting written submissions via their website, and there will be a public meeting on the 19th June – it’s due to report to the BPC and MRS by the 1st March next year.


571 Responses to “Details of the polling Inquiry”

1 2 3 12
  1. 83% of the public polled think the polling inquiry will help pollsters be more accurate in future. 68% disagree.

  2. What if they find that landline phone polling is to blame? the Polling industry goes the way of ostrich feathers in hats…

  3. 1st March next year!

    An 11 month wait . I imagined they would be reporting much sooner than that.

  4. Colin – in 1992 it took two years to report.

    Antiochian – telephone polling is unlikely to be the cause of online polls getting it wrong too…

  5. Three words: Social Desirability Bias.

  6. At what cost is this inquiry going to come at?

    I just tried to look for the youtube montage of Keith Vaz calling for an inquiry over about 40-50 different things with culumative cost of all the inquiries in the corner… Sadly I couldn’t find it… Very ammusing, barring the cost to the country!

  7. Colin and Phil Haines FPT

    Looks like this inquiry will tell us if supposed Labour voters (that is people who polled Labour VI) were inaccurately polled in the Midlands. I presume we can agree that the election was chiefly lost by Labour in this area (disregarding the given of Scotland, which was correctly polled, and LD making its contribution by losing its SW seats to Con, also correctly polled by one pollster),

    The problem is that the polling was not all carried out on a regional basis, except Scotland and the SW LD seats, so one is relying on cross break data. How will the inquiry look into the Midlands?

  8. What’s the margin of error on the Inquiry?
    ‘s nice to have a Polling Inquiry, but when’s the Posting Inquiry? We used to have a lot more posting, but since the election, it’s tailed right off.

    There seems to be insufficient concern about the devastating impact an election can have on posting.

    And the cricket has started, and there’s

  9. …and there’s scarcely a mention. No Crossbat etc. to chat about it with. Country’s going to the dogs…

  10. Never mind the margin of error, I think the more pertinent point is how do we know that we will have a representative sample of inquirers!

  11. @Popeye

    Exactly, for example I’m not sure the good members of this board are sufficiently represented on the Panel.

    and it seems a rather small sample: shouldn’t we be using a t-test or summat?

    In any event, chances are, they’ll prolly conclude it woz the voters fault, like last time.

  12. @ Statgeek

    As I commented earlier Yorkshire and Humber has been done (and it’s all correct), I may find time for West Midlands this evening (just in case).

  13. Thanks AW for keeping us up to date on this. I presume that the members of the BPC will be effectively required to take part in the review – however, have Lord Ashcroft or the Conservative and Labour private pollsters indicated whether they are willing to take part?

    @BristolianHoward

    I agree that a key component will be examining what happens in the Midlands, as most of the expected Labour gains from the Conservatives would have been there. I haven’t gone through Ashcroft’s polls in the Midlands to see if his constituency polls (either SVI or CVI) gave any good guide as to what would happen.

    @Everyone

    I’m sure that everyone here is aware of the marriage equality referendum in Ireland today. Unsurprisingly, there has been a very large buzz on social media during the day, and it appears (although it’s very difficult to know accurately) that turnout is high by Irish referendum standards. As well as being a very significant piece of social legislation in its own right, it’s also (putting my Paddy Ashdown psephology hat on) an interesting test of opinion polling for referendums generally. Since turnout is expected to be reasonably high, one would expect that the opinion polls would give a reasonable idea of the result.

    One reason why I bring this up is that, in the event of an EU referendum, a big challenge for the pollsters will be to try to estimate the overenthusiastic voters on either side. In the Scottish independence referendum, this probably didn’t matter so much because of the very high turnout (but still, the yes side were overestimated a bit in the final calls). I think following the BPC review into the general election palaver, this will be a major test of any new methods/approaches that come out of the recommendations.

  14. IPSOS Mori have published a summary of voting by demographic group.

    Of some interest to the polling inquiry might be the fact that turnout amongst private renters (where Lab was 11 points ahead the Conservatives) fell by 7% relative to that by owners and those with mortgages (where the Conservatives were 24% and 8% ahead of Labour respectively). That is, turnout amongst private renters was down from 55% to 51%, whilst amongst owner occupiers it was up by 3%.

    I think that relative (if not absolute) turnout is here defined relative to demographics rather than just those registered.

    The only plausible reason that comes to mind for the relative fall in turnout amongst private renters is that as a group under the new system of individual registration they found it far harder to get onto the register in the first place, given that they are more likely as a group to move home. So this snippet of evidence suggests that Individual Electoral Registration did indeed explain some of the inaccuracy in the polls as well as affecting the result itself.

    The polling inquiry has to look very hard at the impact IER might have had on the result.

  15. Interesting data out from Ipsos MORI (estimates weighted to final results) – https://www.ipsos-mori.com/researchpublications/researcharchive/3575/How-Britain-voted-in-2015.aspx?view=wide

    They reckon:

    -Biggest CON to LAB swing (7.5%) among 18-24s, biggest LAB to CON (5.5%) among 65+
    -Particularly big LAB to CON swing (5.5%) among AB women
    -Home-owners and mortgagers swinging LAB to CON, but big CON to LAB among renters of both types (8.5% among private rented…)
    -Small LAB to CON swing among BMEs (though LAB still have a big lead in this group)

  16. @ Carfrew

    the voters fault

    Naturally. I stated it before the election (Colin perhaps thought I meant it seriously), that it is very irresponsible to allow anyone to cast a vote just because they are on a register, which is not carefully weighed by demographic characteristics long proven to be the gold standard, and the selecting the votes that would show the result within the MoE.

    That was my only right prediction in this election campaign….

  17. Anthony-good lord-that long !

  18. @Carfrew

    Cricket is a bit touchy in my house right now.

    As a Yorkshireman, England have not only taken our best players, they now want our Coach.

    I feel ‘pillaged’ by the ECB….

  19. @Phil Haines

    I’m not convinced that IER was a major factor or even a minor one. The main group it affected was young people and turnout among that group was virtually the same as last time (1% down). There was a high profile campaign on social media/ street adverts/ in universities to get people to register. Therefore I imagine the vast majority of those unregistered people who had any intention of voting registered.

    In fact the biggest impact of IER may have been to confuse some people. Though I fall into the young category who the campaigns seemed to imply would all need to register I, and many others who still live with parents (at least outside term-time), are in fact registered under the old system and did not need to re-register. But some may have thought they were unregistered come polling day when in fact they were.

    Having said that people who didn’t notice/ respond to the registration campaign were probably always unlikely to vote anyway. So I don’t think it will have made much difference.

  20. @ Jack Sheldon

    While this is a large sample, the figures are still crossbreaks (or appear to be to me) as the research question wasn’t how did 27 years old male marketing consultants, who voted LibDems in 2010 voted in 2015.

    Considering voting, turnout, population changes, thei results on the LD vote would suggest a huge movement of voting intention.

    Also not a word (if I read it carefully enough) on differences by regions/constituencies.

  21. @Laszlo

    Yep… definitely treat with caution

  22. @Laszlo

    “That was my only right prediction in this election campaign….”

    ———-

    Ah, well, then clearly you need to make more predictions, on the basis that the more predictions you make, the more chance you’ll make some that might be correct.

  23. You might call it the Kellner strategy…

  24. One leak is cleared up (ahem …) only three more to go (ahem …)

    On leaks ..

    @ Anthony Wells

    Anthony, do polling companies share information with each other (like releasing polls, changing methodology etc.). I don’t think it is a problem, but the equivalent of it (and more) is a standard practice in German chemical industry with some interesting results (mainly positive).

  25. @Catman

    It might be a bit easier to take if, having whisked them away, they actually played them, eg Rashid…

  26. @ Carfrew

    In one of the leading IB journals an article identified 105 variables, and guess what? Five were in the p<0.05 range …

  27. “In one of the leading IB journals an article identified 105 variables, and guess what? Five were in the p<0.05 range …"

    ———-

    I suppose that makes it a bit easier to understand why they didn't play Rashid, but then there's the whole Peterson thing. Clearly an opportunity for an Inquiry…

  28. Pietersen

  29. @ Carfrew

    Sorry, even after 22 years, I still can’t understand cricket references. But enquiries are good. If there are enough of them, two thirds of the population could get involved, like in the general elections.

  30. @Jack Sheldon

    Turnout amongst 45+ year olds was up on average 3%, whereas amongst the younger half of the adult population it was down 1-2%. Those relative trends seem to me to be again broadly consistent with the impact that IER could have.

    That said, young 18-24 adults fall into two groups – those still settled in the family home (or at uni but registered there) and those who have left home and are in probably transient private accommodation. The former are probably the more numerous, so there is not going to be that good an association between youth and population turnover.

    I consider private rented tenure, rather than age, to be a much better predictor of high population turnover.

    Note also that I and others in this marginal parliamentary seat worked quite closely with the ERO here particularly in the latter half of 2014 when the inadequacy of results from the IER data matching exercise were becoming apparent. It was clear that the areas with problems were those with low cost private housing, rather than the better private housing or better social housing areas. Those same areas of cheap private housing are those where a lot of the homes are rented out privately.

    I remain of the view that the results from the MORI survey are pretty well in line with what should be expected if IER did affect the result.

  31. @Laszlo

    Many thanks, and no rush. I’ll tackle London and South East in the next couple of hours.

  32. Good Afternoon All, on a day when another Upper Sixth goes off to exams, university and employment; tired but satisfied.
    Two questions about the review, and thank you, Anthony for the information.

    1. Could it be the case that no one really knows who and why anyone votes or does not vote and why?

    2. Will we find out why the GE 2015 numbers for the Liberal Democrats were so high?

  33. @Laszlo

    That’s OK, no one understand’s England cricket, especially the selection policy. I don’t think one is supposed to understand English cricket, it’s more that one suffers it, perplexed…

    Inquiries have a built in flaw, if peeps who got it wrong get paid to sit on them, since it creates an incentive to get things wrong so you can have an Inquiry.

    But yeah, if more peeps were involved, like a referendum… We could start with the populace picking the England team. Then we could pick more Yorkies…

  34. @Phil Haines

    Thanks for elaborating. Interesting.

  35. @Phil Haines

    I’m not sure that IER hobbling the private renters is “the only plausible reason”. It’s certainly *a* plausible reason.

    If that is simply a bulk measure, uncorrected for other factors, what about changes in the make-up of that group between 2010 and 2015? The proportion of private renters is increasing overall, but the trend is also that the private rental vs. homeownership is becoming more a marker of a generational divide rather than a socio-economic one. That could also very plausibly be behind the statistic, as the turnout differential by age is more stark than the differential by socio-economic group.

  36. Cabinet Office has concluded its investigation into the French Ambassador/Sturgeon story in the Telegraph.

    As assumed by myself and others at the time, the memo was leaked by Carmichael’s SPadD Euan Roddin, with Carmichael’s agreement.

    Afficionados of election time leaking will know that the then SoS for Scotland made public statements which were lacking in even the vaguest approximation to the truth

    Neither Mr Carmichael nor Mr Roddin will take their severance pay. (£16,876 in Carmichael’s case).

    https://www.gov.uk/government/news/scotland-office-memorandum-leak-cabinet-office-inquiry-statement

    Perhaps, another 818 voters in Orkney and Shetland would have changed their vote away from Carmichael had they known that he was more Spencer than Frank.

  37. Sorry. Didn’t close the italics. The quote from the Cabinet Office report was “Neither Mr Carmichael nor Mr Roddin will take their severance pay”

    Nicola Sturgeon has accepted Carmichael’s apology.

  38. @Phil Haines
    “The only plausible reason that comes to mind for the relative fall in turnout amongst private renters is that as a group under the new system of individual registration they found it far harder to get onto the register in the first place, given that they are more likely as a group to move home.”

    Why would it be harder for private renters to be registered individually rather than by household? Did the landlord used to do it for them or something?

    Isn’t it more likely that the growth of the private rented sector makes any type of registration more difficult for more people?

  39. @Pete B

    It’s not private renting per se, rather population turnover which is much higher in privately rented properties particularly the least desirable such as HMOs and bedsits. High turnover made it harder to obtain data matches with official records in the initial IER exercise and put an onus on people each to remember that they needed to register and to then register individually, whereas the names of people who had stayed put over the last 18 months were carried forward automatically even if they did nothing.

    @Popeye

    Might be a factor, but I wouldn’t have thought it were that much of one.

  40. Phil
    I understand that, but still don’t understand why IER is harder for these folks than was household registration, especially when you think that a high proportion of single people live in such accommodation, so there would be no difference at all for them.

  41. There is another factor. Up to and including this election councils will leave people on the register until they have information to the contrary. Since private renters move often, there is a high proportion of “dead” names obn the register. This has always been accepted as a major reason for the lower apparent turnout in areas with high proportions of private renters. In my younger days I was sometimes registered in 3 or 4 different places, and sometiimes not at all.

    These days there is an attempt to remove people from their old address when they register for a new one. However where people only register when there’s an election they care about, and then forget about for n years until there’s another election by which time they’ve moved more than once, then this doesn’t apply.

    You have to conduct a survey of private renters to see whether and where they were registered (many of them won’t know to add to the problem), and whether and where they voted. And you ahve the problem of getting a representative sample of this hard to contact group, which takes us back where we started.

    I’m not convinced that private renters who are on the register where they currently live had a lower turnout than others from the same demographic.

  42. One fewer LibDem MP soon?

  43. Lazlo

    If not, then possibly two fewer LD MSPs in a year’s time?

  44. Can’t we set up our own enquiry? I suggest Unicorn, Stageek, Laszlo, Number Cruncher, Robin Hood and TOH.

    That should do it. But any other suggestions are welcome.

  45. Someone wrote the other day that Orkney had voted SNP but the Shetlanders had voted LD by more, thus assuring the LDs of the seat. Anyway, it would be an historic moment were the Northern Isles to vote SNP.

    Wouldn’t it be more likely that they vote for a Norwegian party and apply to transfer?

    It seems to me to be very unlikely Carmichael would resign, and equally improbable that Shetland would ever vote SNP. But stranger things have happened of late…..

  46. I may a giddy Bristolions high on life after Bristol Rovers were promoted out of the conference league but surely the answer is…

    ‘They didn’t vote the way we thought they would….’

  47. Morally there is probably a reasonable case for Carmichael triggering a by-election… If this was public the result may well have been different…. There is, of course, the new recall procedure though without double checking the rules he probably gets off because he didn’t commit a criminal offence and could not have breached parliamentary standards because he was not an MP

  48. @Norbold (RE: Enquiry – Some random thoughts and questions, some of which are rhetorical)

    What the point of polls if they predict the right answer the day before you get the answer, but not for the days and weeks prior?

    Are polls there to inform, or to influence?

    Are we not crossing from measuring the mood of the people to attempting to set the narrative with polls that are of the moment?

    Is it a reflection on 24-hour news and social media, that polls are too attempting to inform us today of the mood of yesterday?

    Is it not scandalous that politicians might react more to a blip (rather than a trend) in a poll, than to a real need in the community? Examples being Cameron / Miliband in September ’14, or Miliband / Brown in this election.

    My experience is that they only come out from under their rocks when their jobs (pensions/investments/monies) are under threat, rather than when they are needed. As such my disdain for politicians has grown a little this last month, and has yet to be seen if the new batch from the North will be any different.

    Carmichael in the news as well. Popped up after the vote etc. Nuff said, partisan stuff.

    No point in enquiries into polling if polling companies are going to attempt to become the media in their practices. They should stick to measuring and recording the mood of the people, and perhaps it would be a good thing to put a window of release on all BPC polls, so that the data is old by the time we get it. That way it can’t be used to set agendas (I appreciate that agenda is the plural, but its usage is thus nowadays).

    Perhaps a ban on polling in election week too. The more blind politicians are they more they would be forced to campaign blindly, rather than use polling companies as tactical measuring tools.

    All my humble opinion etc.

  49. This is probably not the place for discussion on the subject, but it does seem to me that Carmichael was not alone in seriously misreading the mood. LiS will take ten years to recover from the catastrophic choice of JM, and even longer if they choose Dugdale as new leader. IMO

    As for whether he ought to resign, I would expect the Northern Isles to vote for an independent rather than for the SNP.

  50. @John

    All personal inclinations aside, an Orkney and Shetland by-election in the current mood wouldn’t half be interesting from the perspective of the tactical SNPout types.

1 2 3 12