Today the polling inquiry under Pat Sturgis’ presented its initial findings on what caused the polling error. Pat himself, Jouni Kuha and Ben Lauderdale all went through their findings at a meeting at the Royal Statistical Society – the full presentation is up here. As we saw in the overnight press release the main finding was that unrepresentative samples were to blame, but today’s meeting put some meat on those bones. Just to be clear, when the team said unrepresentative samples they didn’t just mean the sampling part of the process, they meant the samples pollsters end up with as a result of their sampling AND their weighting: it’s all interconnected. With that out the way, here’s what they said.

Things that did NOT go wrong

The team started by quickly going through some areas that they have ruled out as significant contributors to the error. Any of these could, of course, have had some minor impact, but if they did it was only minor. The team investigated and dismissed postal votes, falling voter registration, overseas voters and question wording/ordering as causes of the error.

They also dismissed some issues that had been more seriously suggested – the first was differential turnout reporting (i.e, Labour people overestimating their likelihood to vote more than Conservative people), in vote validation studies the inquiry team did not found evidence to support this, suggesting if it was an issue it was too small to be important. The second was the mode effect – ultimately whether a survey was done online or by telephone made no difference to its final accuracy. This finding met with some surprise from the audience, given there were more phone polls showing Tory leads than online ones. Ben Lauderdale of the inquiry team suggested that was probably because phone polls had smaller sample sizes and hence more volatility, hence spat out more unusual results… but that the average lead in online polls and average lead in telephone polls were not that different, especially in the final polls.

On late swing the inquiry said the evidence was contradictory. Six companies had conducted re-contact survey, going back to people who had completed pre-election surveys to see how they actually voted. Some showed movement, some did not, but on average they showed a movement of only 0.6% to the Tories between the final polls and the result, so can only have made a minor contribution at most. People deliberately misreporting their voting intention to pollsters was also dismissed – as Pat Sturgis put it, if those people had told the truth after the election it would have shown up as late swing (but did not), if they had kept on lying it should have affected the exit poll, BES and BSA as well (it did not).

Unrepresentative Samples

With all those things ruled out as major contributors to the poll error the team were left with unrepresentative samples as the most viable explanation for the error. In terms of positive evidence for this they looked at the differences between the BES and BSA samples (done by probability sampling) and the pre-election polls (done by variations on quota sampling). This wasn’t a recommendation to use probability sampling (while they didn’t do recommendations, Pat did rule out any recommendation that polling switch to probability sampling wholesale, recognising that the cost and timing was wholly impractical, and that the BES & BSA had been wrong in their own way, rather than being perfect solutions).

The two probability based surveys were, however, useful as comparisons to pick up possible shortcomings in the sample – so, for example, the pre-election polls that provided precise age data for respondents all had age skews within age bands, specifically within the oldest age band there were too many people in their 60s, not enough in their 70s and 80s. The team agreed with the suggestions that samples were too politically engaged – in their investigation they looked at likelihood to vote, finding most polls had samples that were too likely to vote, and didn’t have the correct contrast between young and old turnout. They also found samples didn’t have the correct proportions of postal voters for young and old respondents. They didn’t suggest all of these errors were necessarily related to why the figures were wrong, but that they were illustrations of the samples not being properly representative – and that ultimately led to getting the election wrong.

Herding

Finally the team spent a long time going through the data on herding – that is, polls producing figures that were closer to each other than random variation suggests they should be. On the face of it the narrowing looks striking – the penultimate polls had a spread of about seven points between the poll with the biggest Tory lead and the poll with the biggest Labour lead. In the final polls the spread was just three points, from a one point Tory lead to a two point Labour lead.

Analysing the polls earlier in the campaign the spread between different was almost exactly what you would expect from a stratified sample (what the inquiry team considered the closest approximation to the politically weighted samples used by the polls). In the last fortnight the spread narrowed though, with the final polls all close together. The reason for this seems to be because of methodological change – several of the polling companies made adjustments to their methods during the campaign or for their final polls (something that has been typical at past elections, companies often add extra adjustments to their final polls). Without those changes them the polls would have been more variable….and less accurate. In other words, some pollsters did make changes in their methodology at the end of the campaign which meant the figures were clustered together, but they were open about the methods they were using and it made the figures LESS Labour, not more Labour. Pollsters may or may not, consciously or subconsciously, have been influenced in the methodological decisions they made by what other polls were showing. However, from the inquiry’s analysis we can be confident that any herding did not contribute to the polling error, quite the opposite – all those pollsters who changed methodology during the campaign were more accurate using their new methods.

For completeness, the inquiry also took everyone’s final data and weighted it using the same methods – they found a normal level of variation. They also took everyone’s raw data and applied the weighting and filtering the pollsters said they had used to see if they could recreate the same figures – the figures came out the same, suggesting there was no sharp practice going on.

So what next?

Today’s report wasn’t a huge surprise – as I wrote at the weekend, most of the analysis so far has pointed to unrepresentative samples as the root cause, and the official verdict is in line with that. In terms of the information released today there were no recommendations, it was just about the diagnosis – the inquiry will be submitting their full written report in March. It will have some recommendations on methodology – but no silver bullet – but with the diagnosis confirmed the pollsters can start working on their own solutions. Many of the companies released statements today welcoming the findings and agreeing with the cause of the error, we shall see what different ways they come up with to solve it.


224 Responses to “What the Polling Inquiry said”

1 3 4 5
  1. @COLIN
    Do you think it might be Crossbatt11?
    @CROSSBATT11
    Do you think it might be Colin?

  2. @Dave Colby

    Don’t listen to Colin. He’s just trying to distract from Neil A’s last post bigging up Crosby, which is a dead giveaway…

  3. Good evening all from Westminster North.

    I will let the cat out of the bag and tell you all that our very own The Other Howard is Lynton Crosby.

    TOH, sorry Lynton predicted the Tory majority when everyone else from Miss Piggy to ole ET said “nae chance”

    More proof…”After graduating from the University of Adelaide, Crosby first became involved in politics with the Liberal Party of Australia, eventually being appointed federal director of the party in 1997. He oversaw the party’s successful campaigns at the 1996, 1998, 2001, and 2004 federal elections, which made the Howard Government Australia’s second-longest serving federal government”
    ____

    “Howard Government” “Liberal” The Other Howard is a bit of a Tory’ish Liberal is he not?

    I rest my case…

  4. If polling is unreliable, focus groups may be much less reliable than them.

    However, Labour’s focus group has some familiar analyses of why they lost.

    http://www.itv.com/news/2016-01-25/revealed-secret-labour-report-published-in-full/

    However, are the views expressed by participants the product of commentators’ ideas – or the sources for the commentators?

    Classic chicken and egg question, I’d suggest.

  5. @Candy

    The PLP ‘Establishment’ has been firmly controlled by the Blairite faction, to the point that Corbyn not giving them the majority of front bench positions is a ‘sign’ that he’s hopelessly left wing. Let’s not forget how generous Milliband was to giving the Blairite faction front bench positions, in-spite of being constantly undermined by press briefings by them.

  6. If polling is unreliable, focus groups may be much less reliable than them.

    However, Labour’s focus group has some recognisable features in the analyses of why they lost.

    http://www.itv.com/news/2016-01-25/revealed-secret-labour-report-published-in-full/

    However, are the views expressed by participants the product of commentators’ ideas – or the sources for the commentators?

    Classic chicken and egg question, I’d suggest.

  7. @NeilJ

    “My understanding is that Google will target your search results to match things you have previously shown an interest in. I think it is called a filter bubble.”

    It goes a bit further than that. Everything you search for is logged in their servers, alongside the IP address and Google account if you have one (includes e-mails) which helps them to tailor their search results effectively.

    And lower their tax rates, if course.

    @Candy

    Not really. Labour never had a plan to target Clegg. They used him as a symbol, which maybe drew more activists there, but it wasn’t paet of a strategy.

    The real problem was with Cleggs strategy. In the election he was signalling very strongly that he would support the Consevatives and continue the coalition (his red lines were all rather easy for the Tories to accommodate, but harder for Labour, e. g. no energy price freeze).

    This, of course, had the effect of turning away the Lab voters in the SW he needed to support the party in order to keep his MPs in. He was effectively asking Conservative voters to switch to voting Lib Dem in order to get a watered down Conservative government over a real one. It had the predictable result.

    Fundamentally Clegg never understood the nature and make up of his pary’s support, coupled with a delusion that the Lib Dems would be rewarded for their responsibility. It was also a similar misunderstanding that Labour ran head first into.

  8. @ Neil A

    I was interested by your assessment ‘In my constituency the young and slightly rebellious Tory candidate apparently eschewed CCHQ assistance, but he was a towering figure on Facebook. ‘

    Reading Conservative Home comment threads under articles about the 40/40 strategy suggests that some local Conservative associations reacted very negatively to Lynton Crosby’s strategy. I remember several saying that 2015 was a one off because they were not happy to be ordered to ignore their own constituencies, the centralised control was heavy-handed (and impolite) and that the political landscape was fortuitous for them, with the rise of the SNP/Ukip/Greens and the fall of the LDs. The real concern was the lack of local organisation on the ground… and latterly, the Young Conservatives Battlebus and RoadTrip 2015.

    Instructive that Facebook seems to have worked for your candidate because rather than eschewing CCHQ assistance his was quite definitely listed as a non-target seat:

    ‘Individual seats aside, there’s a more general issue here, too. We’ve written before about the growing resentment among non-40/40 candidates about the way they are treated by the centre (and our anonymous Candidate’s Diary gives a personal account of the experience). Being publicly labelled due to a cock-up like this won’t do anything to improve relations.’

    http://www.conservativehome.com/thetorydiary/2015/02/a-cchq-cock-up-leaks-the-names-of-101-non-target-seats.html

  9. @AU

    I also think Clegg somehow fell for the idea that you can just u-turn on things like tuition fees and austerity and simply blame it on Greece and peeps will just go “oh, ok then, fair enough, as much as you like!! When Greece sneezes, everyone catches a cold and peeps in the UK have to pay tuition fees, bedroom taxes and much more besides, everyone knows that!”

    It was amazing. Even if the economic powerhouse that is Greece had indeed somehow hammered our economy, that doesn’t suddenly make cuts a necessity. If you were against the deflationary effects of cuts in hard times, that doesn’t suddenly change if you get hard times via Greece.

    Cable looked like he was chewing on several wasps as he delivered this master stroke…

  10. @AU
    ‘The real problem was with Cleggs strategy. In the election he was signalling very strongly that he would support the Consevatives and continue the coalition (his red lines were all rather easy for the Tories to accommodate, but harder for Labour, e. g. no energy price freeze).
    This, of course, had the effect of turning away the Lab voters in the SW he needed to support the party in order to keep his MPs in. He was effectively asking Conservative voters to switch to voting Lib Dem in order to get a watered down Conservative government over a real one. It had the predictable result.’

    @CARFREW
    ‘ also think Clegg somehow fell for the idea that you can just u-turn on things like tuition fees and austerity and simply blame it on Greece and peeps will just go “oh, ok then, fair enough, as much as you like!! When Greece sneezes, everyone catches a cold and peeps in the UK have to pay tuition fees, bedroom taxes and much more besides, everyone knows that!”

    Agree with both of the above, the reality is that in England the conservatives were second in most of the Libdem seats. It did not need many voters to turn away from the Lib dems to let the conservatives in. Many Lib dem voters from 2010 and before had in effect lent the Libdems their vote to keep a conservative out. Enough of these felt betrayed and either stayed at home or voted Labour. Some blue libdems would also have voted conservative, seeing a real chance of getting a conservative in.
    Scotland was different only in as much as the SNP were rampant and disaffected Lib Dem voters found a natural home there.

    I do think it was these global considerations and disaffections rather than any local campaign by any particular party which swung the election.

  11. @NeilJ,

    Probably true in much of the country, but in my constituency we don’t really have any Liberal Democrats, it’s a safe Labour seat, it had a decent (if unexciting) long-serving MP, there were no significant demographic changes to make it more Tory, and as SYZYGY has uncovered it wasn’t a Tory target (I would add that the Tory candidate made this into a sort of a plus, saying that he wouldn’t take party money because he wouldn’t toe the party line – which he’s made good on so far to some extent).

    Yet the Tory triumphed, and in my opinion this was partly because lots of young ladies enthusiastically shared videos of him wearing jeans and an unbuttoned shirt making pledges about (relatively) non-political local issues, and his friends and family manned the pumps of a very active social media campaign.

    My wife started following him on FB (to my surprise, she’s not that political – or wasn’t). Now she’s a party member.

  12. ALLAN Christie

    “The Other Howard is Lynton Crosby.”

    While I admire Crosby’s ability, I’m afraid I am too old to be him, unless he uses monkey glands and is actually 75.

    :-)

  13. @Carfew

    True, I think that tied in with the whole responsibility thesis. That ‘we made a bunch of promises but we didn’t realize how bad things were, now we’ve seen the books we know we can’t do x because it would be irresponsible’. They tried to co-opt the ‘blame Labour’ strategy, but it didn’t really work – especially as it took them about three years to realize that they were always in the news delivering the bad news, and the Tories were always on air delivering the good news.

    Latterly I think it was also a case that Clegg, as with tuition fees, didn’t really want/believe in the policy he’d signed up to and thought he spied a convenient chance to junk it and pretend it was something he didn’t want to do. That Cable went along with it was somewhat surprising, but then power (or a flash of its leg) does funny things to people’s minds.

    @NeilJ

    They really needed to convince Labour supporters to grit their teeth and plump for them in order to survive. I really think that the Lib Dems, for whatever reason, honestly believed that people would view them as a party in their own right, and not the rather large protest party for the disgruntled that they were. In other times it might have worked, but their problem was that there were lots of new protest parties (UKIP, Greens, SNP to an extent) that would eat into their support from all directions.

    @Neil A

    So he employed the Berlusconi strategy of electoral politics? ;)

  14. @TOH

    It’s all right, Howard, we all know that you’re really the Oracle of Delphi ;)

  15. Hi folks, haven’t been about recently, because I needed a bit of time off after masterminding the Conservative’s 2015 election victory – #iamlynton ;)

    Anyway… I was avoiding political forums like the plague recently because of some of the pretty nasty stuff that passes for discourse. Between Corbynistas and CyberNats and Red Tories and Trumpites and Colonel Sanders, it’s been a massive turnoff for someone like me. I have some opinions on things, and some of them are contradictory, but this intense shouting down of people (should I say peeps?) who dare to express an honestly held feeling is infuriating. I don’t know if it’s a view generally held by the population (perhaps we should take a poll?), but I want politicians and pundits to discuss policy and issues, not prattle on about favourability ratings and back-stabbing!

    I wanted to talk briefly about two countries that are definitely not part of the UK, but are a major part of my life, namely Ireland and Germany.

    In the next week or so (perhaps even tomorrow), the elections to Dail Eireann will be called. Polling in Ireland generally does a pretty good job, but because of the large number of parties and independents taking part, it will be particularly difficult for the pollsters. The most recent polls have Fine Gael (Christian Democrats) on around 30, with their Labour coalition partners on around 7-8%, which would probably not be enough to return a majority. On the opposition, Fianna Fail (roughly Liberal) are at around 18-20% with Sinn Fein (Left) perhaps a point or two behind. Then – the rest. When you load up the Greens, Anti-Austerity Alliance, Left Unity, People before Profit, Renua and a bunch of sundry independents, they amount to about 20-25% of the vote. A nightmare scenario (at least in my view) would be a government propped up by say 8-10 independents, which would hardly be stable. However, unless a lot changes during the campaign, it’s fairly possible that around 20 populist-type candidates will be returned.

    In Germany, on 13th March, there will be elections to three state parliaments. Most of the focus will be on the success of the AfD – the anti-euro party who are sitting at 10%+ in the polls. Although Angela Merkel has been under a fair amount from her own party and from the CSU, it’s actually the SPD (junior coalition partners) who may be under severest pressure in these elections. In BW and S-A, they could come in 4th place behind the AfD, which would be disastrous, while in Rhineland-Pfalz they are likely to lose control of the state government. I know it’s unwise to draw conclusions across a whole continent, but there may be a lesson here for centre-left politicians – with the European (and US) economies under some pressure, it’s easy to leak votes to harder-left factions. On the other hand, with concern over immigration being a key issue, socially right-wing parties can benefit. The AfD themselves are a contradictory confection of economic conservatives, social libertarians and social conservatives with a slightly sinister turn in recent months to fairly open xenophobia (particularly since founder Bernd Lucke left). An interesting issue is that the SPD refuse to debate with the AfD (on political talkshows etc), but have had to agree to a election debate on SWR (public broadcaster for RP and BW), to which the AfD have been invited. The SPD have, however, refused to send Minister-President Dreyer, instead sending the leader of the parliamnentary grouping.

    Coming back to the issue of groupthink and Internet echo chambers, it seems to me that this issue (and Trump refusing to attend the Fox Iowa debate) is a small example of it in practice. Rather than engage with the opposition “mano-y-mano”, and argue the issues like adults, it’s now becoming easier to hide behind armchair warriors and blame the MSM or neoliberalism or communism! I think as voters, we deserve a bit more respect from politicians and pundits.

    I know it’s sometimes hard to get excited by elections outside the UK (particularly given the name of this website…..), but for a politigeek like me, it’s always intriguing to see how the issues of the day play out under different electoral circumstances and under differing election systems.

  16. Off topic: the voting intention graph here seems to have been replaced by nonsense: http://ukpollingreport.co.uk/voting-intention-2

    Is the correct graph coming back?

  17. @Neil A
    Your new Tory MP though did rather misbehave on Polling Day and was forced to cough up for the cost of removing his posters illegally displayed on public property. A complaint was made to the police – I am not sure that the courts have become involved.

  18. @Mico

    “Off topic: the voting intention graph here seems to have been replaced by nonsense:”

    I fear political opinion polling generally has been replaced by nonsense. In that sense, the graph you refer to may in fact be entirely representative of the current state of the polls!

    :-)

  19. @David Colby

    “@COLIN
    Do you think it might be Crossbatt11?
    @CROSSBATT11
    Do you think it might be Colin?”

    You are a tease!

    :-)

  20. @AU

    True, indeed you can also tie it in with the “didn’t understand their supporters thesis”. I still think it’s worth posting as a reminder of how sometimes the reasoning of our political peeps beggars belief.

  21. Ipsos-MORI poll showing big Tory lead

    https://www.ipsos-mori.com/researchpublications/researcharchive/3690/Conservatives-take-their-biggest-lead-over-Labour-as-party-with-best-team-of-leaders-and-most-clear-and-united.aspx

    However, it seems worth questioning whether they have over-corrected for the difficulty of finding Tories, and too many Labour.

    In the Scottish crossbreak, they found a nonsensical 30% with a Tory VI (16% Lab, 44% SNP).

    While their GB numbers may (or may not) be approximately accurate, that would seem to indicate that they still can’t find enough English Tories, and are compensating by methodologically creating Scots ones! :-)

  22. Sorry. That was 34% Tories in Scots crossbreak!

  23. @Graham,

    I don’t think it went anywhere in terms of electoral law, as there’s no certainty over who actually put them up.

    In a way it perhaps illustrates the point, in that against expectations the Conservatives in a relatively poor, working class Labour stronghold had enough enthusiasts to go out flyposting…

    Whether that is because of the social media presence of the candidate is of course hard to say.

  24. O/T Slowing London property market thanks to overseas buyers drying up (collapsing rouble, collapsing oil, collapsing far east currencies) and brexit fears:

    http://www.bloomberg.com/news/articles/2016-01-28/london-home-presales-slump-19-as-demand-damped-by-sales-taxes

    Will the Conservatives benefit from this if it makes it easier for Brits to buy? Or will homeowners feel cross at their paper profits evaporating and blame Osborne? Or will people think it’s about global events that no-one controls?

1 3 4 5