Last year the election polls got it wrong. Since then most pollsters have made only minor interim changes – ComRes, BMG and YouGov have conducted the biggest overhauls, many others have made only tweaks, and all the companies have said they are continuing to look at further potential changes in the light of the polling review. In light of that I’ve seen many people assume that until changes are complete many polls probably still overestimate Labour support. While on the face of it that makes sense, I’m not sure it’s true.

The reason the polls were wrong in 2015 seems to be the samples were wrong. That’s sometimes crudely described as samples including too many Labour voters and too few Conservative voters. This is correct in one sense, but is perhaps describing the symptom rather than the cause. The truth is, as ever, rather more complicated. Since the polls got it wrong back in 1992 almost all the pollsters have weighted their samples politically (using how people voted at the last election) to try and ensure they don’t contain too many Labour people or too few Conservative people. Up until 2015 this broadly worked.

The pre-election polls were weighted to contain the correct number of people who voted Labour in 2010 and voted Conservative in 2010. The 2015 polls accurately reflected the political make up of Britain in terms how people voted at the previous election, what it got wrong it how they voted at the forthcoming election. Logically, therefore, what the polls got wrong was not the people who stuck with the same party, but the proportions of people who changed their vote between the 2010 and 2015 elections. There were too many people who said they’d vote Labour in 2015 but didn’t in 2010, too many people who voted Tory in 2010 but said they wouldn’t in 2015, and so on.

The reason for this is up for debate. My view is that it’s due to poll samples containing people who are too interested in politics, other evidence has suggested it is people who are too easy to reach (these two explanations could easily be the same thing!). The point of this post isn’t to have that debate, it’s to ask what it tells us about how accurate the polls are now.

The day after an election how you voted at the previous election is an extremely strong predictor of how you’d vote in an election the next day. If you voted Conservative on Thursday, you’d probably do so again on Friday given the chance. Over time events happen and people change their minds and their voting intention; how you voted last time becomes a weaker and weaker predictor. You also get five years of deaths and five years of new voters entering the electorate, who may or may not vote.

Political weighting is the reason why the polls in Summer 2015 all suddenly showed solid Conservative leads when the same polls had shown the parties neck-and-neck a few months earlier, it was just the switch to weighting to May 2015 recalled vote**. In the last Parliament, polls were probably also pretty much right early in the Parliament when people’s 2010 vote correlated well with their current support, but as the Lib Dems collapsed and UKIP rose, scattering and taking support from different parties and in different proportions polls must have gradually become less accurate, ending with the faulty polls of May 2015.

What does it tell us about the polls now? Well, it means while many polling companies haven’t made huge changes since the election yet, current polls are probably pretty accurate in terms of party support, simply because it is early in the Parliament and party support does not appear to have changed vastly since the election. At this point in time, weighting samples by how people voted in 2015 will probably be enough to produce samples that are pretty representative of the British public.

Equally, it doesn’t automatically follow that we will see the Conservative party surge into a bigger lead as polling companies do make changes, though it does largely depend on the approach different pollsters take (methodology changes to sampling may not make much difference until there are changes in party support, methodology changes to turnout filters or weighting may make a more immediate change).

Hopefully it means that polls will be broadly accurate for the party political elections in May, the Scottish Parliament, Welsh Assembly and London Mayoral elections (people obviously can and do vote differently in those elections to Westminster elections, but there will be a strong correlation to how they voted just a year before). The EU referendum is more of a challenge given it doesn’t correlate so closely to general election voting and will rely upon how well pollsters’ samples represent the British electorate. As the Parliament rolls on, we will obviously have to hope that the changes the pollsters do end up making keep polls accurate all the way through.

(**The only company that doesn’t weight politically is Ipsos MORI. Quite how MORI’s polls shifted from neck-and-neck in May 2015 to Tory leads afterwards I do not know. They have made only a relatively minor methodological change in their turnout filter. Looking at the data tables, it appears to be something to do with the sampling – ICM, ComRes and MORI all sample by dialing random telephone numbers, but the raw data they get before weighting it is strikingly different. Looking at the average across the last six surveys the raw samples that ComRes and ICM get before they weight their data has an equal number of people saying they voted Labour in 2015 and saying they voted Tory in 2015. MORI’s raw data has four percent more people saying they’d voted Conservative than saying they’d voted Labour, so a much less skewed raw sample. Perhaps MORI have done something clever with their quotas or their script, but it’s clearly working.)


There are two new polls on the EU referendum out tonight – YouGov for the Times and ComRes for the Mail. YouGov have topline figures of REMAIN 37%, LEAVE 38%, DK/WNV 25%; ComRes have topline figures of REMAIN 51%, LEAVE 39%, DK 10%. ComRes was asked Friday to Monday (so started before Cameron’s deal was finalised), YouGov’s poll was asked between Sunday and Tuesday, so was after Cameron’s renegotiation, but straddled Boris Johnson’s endorsement of the Leave campaign.

As we’ve come to expect there’s a sharp difference between the online YouGov poll and the telephone ComRes poll. Online polls on the referendum have tended to show a neck-and-neck race, telephone ones have tended to show a lead for Remain. The level of support for leaving is actually pretty much the same regardless of mode – the difference all seems to be in the proportion who say stay and the proportion who say don’t know (I speculated about that a little last month, here)

Anyway, while the different modes produce different shares, just as interesting is the direction of travel. YouGov’s previous poll was conducted just after the draft renegotiation had been published and showed a significant shift towards leave, giving them a nine point lead. My suspicion then was that it could just be a short-term reflection of the extremely bad press that the deal received in the papers, and that does appear to be the case – the race has tightened right back up again. A fortnight ago YouGov found 22% thought the draft renegotiation was a good deal, 46% a bad deal. That’s now closed to 26% good, and 35% bad. After a blip from the initial bad publicity over the draft deal, the effect according to YouGov seems broadly neutral.

ComRes’s last poll found a similar trend to YouGov – it was conducted after the draft deal had been published, and found a sharp shift towards Leave, with the remain lead dropping by ten points, from eighteen to eight. Today’s poll finds that negative reaction to the draft deal fading a bit now the final deal is done, with the remain lead creeping back up to twelve points. The net effect is still negative, but not by nearly as much as the early polls suggested. ComRes’s specific question on the renegotiation provides a more positive verdict than YouGov’s – among the three-quarters of the sample asked after the deal was struck 46% say it was a success, 39% a failure.

Note that this poll also represents the first outing for some methodology changes from YouGov. Most significantly, they’ve started sampling and weighting by the attention respondents say they pay to politics, have added educational qualifications as a sampling/weighting variable and have shifted up the top age bracket from 60 and over to 65 and over. Also, at the risk of getting very technical, past vote and grouped region are now interlocked (to explain – in the past YouGov weighted everyone’s past vote to match the overall shares of the vote in Great Britain, now they are weighting respondents in London’s past vote to match the shares of the vote in London, respondents in the Midlands’ past vote to match the shares in the Midlands and so on). There isn’t actually much impact on today’s results; the old sampling and weighting would also have shown the race tightening to neck-and-neck. The main difference is that a lot of questions have a higher number of don’t knows, reflecting the higher proportion of respondents who don’t follow politics closely.

Full tables for ComRes are here, for YouGov here.


-->

A quick update on some polling figures from the last few days.

ComRes released a new telephone poll for the Daily Mail on Friday. Topline voting intention figures were CON 37%, LAB 32%, LDEM 6%, UKIP 12%, GRN 4% (tabs are here.) On the EU referendum ComRes had voting intentions of REMAIN 54%, LEAVE 36%, DK 10%.

YouGov also released new figures on voting intention and the EU referendum on their website. Their lastest topline VI figures are CON 39%, LAB 30%, LDEM 6%, UKIP 17%, GRN 3% (tabs are here). On the EU referendum they have Leave slightly ahead – REMAIN 38%, LEAVE 42%, DK/WNV 20%.

Finally Ipsos MORI also released EU referendum figures (part of the monthly Political Monitor survey I wrote about earlier in the week). Their latest figures are REMAIN 50%, LEAVE 38%, DK 12%.

There continues to be a big contrast between EU referendum figures in polls conducted by telephone, and conducted online. The telephone polls from ComRes and Ipsos MORI both have very solid leads for remain, the online polls from ICM, YouGov, Survation and others all tend to have the race very close. In one sense the contrast seems to be in line with the contrast we saw in pre-election polls – while there was little consistent difference between online and telephone polls in terms of the position of Labour and the Conservatives (particularly in the final polls), there was a great big gulf in terms of the levels of UKIP support they recorded – in the early part of 2015 there was a spread of about ten points between those (telephone) pollsters showing the lowest levels of UKIP support and those (online) pollsters showing the highest levels of UKIP support. It doesn’t seem particularly surprising that this online/telephone gap in terms of UKIP support also translates into an online/telephone gap in terms of support for leaving the EU. In terms of which is the better predictor it doesn’t give us much in the way of clues though – the 13% UKIP ended up getting was bang in the middle of that range.

The other interesting thing about the telephone/online contrast in EU referendum polling is the don’t knows. Telephone polls are producing polls that have far fewer people saying they don’t know how they’ll vote (you can see it clearly in the polls in this post – the two telephone polls have don’t knows of 10% and 12%, the online poll has 20% don’t knows, the last couple of weekly ICM online polls have had don’t knows of 17-18%). This could have something to do with the respective levels of people who are interested in politics and the EU that the different sampling approaches are picking up, or perhaps something to do with people’s willingness to give their EU voting intention to a human interviewer. The surprising thing is that this is not a typical difference – in polls on how people would vote in a general election the difference is, if anything, in the other direction – telephone polls find more don’t knows and refusals than online polls do. Why it’s the other way round on the EU referendum is an (intriguing) mystery.


ICM released their final monthly voting intention poll of 2015 yesterday, with topline figures of CON 39%, LAB 34%, LDEM 7%, UKIP 10%, GRN 3%. I assume it’s the last voting intention poll we will see before Christmas. The full tables are here, where ICM also make an intriguing comment on methodology. They write,

For our part, it is clear that phone polls steadfastly continue to collect too many Labour voters in the raw sample, and the challenge for phone polling is to find a way to overcome the systematic reasons for doing so. The methodological tweaks that we have introduced since the election in part help mitigate this phenomenon by proxy, but have not overcome the core challenge. In our view, attempting to fully solve sampling bias via post-survey adjustment methods is a step too far and lures the unsuspecting pollster into (further) blase confidence. We will have more to say on our methods in the coming months.


One of the key bits of evidence on why the polls got it wrong has today popped into the public domain – the British Election Study face to face survey. The data itself is downloadable here if you have SPSS or Stata, and the BES team have written about it here and here. The BES has two elements – an online panel study, going back to the same people before, during and after the election campaign, and a post-election random face-to-face study, allowing comparison with similar samples going back to the 1964 BES. This is the latter part.

The f2f BES poll went into the field just after the election and fieldwork was conducted up until September (proper random face-to-face polls take a very long time). On the question of how people voted in the 2015 election the topline figures were CON 41%, LAB 33%, LDEM 7%, UKIP 11%, GRN 3%. These figures are, of course, still far from perfect – the Conservatives and Labour are both too high, UKIP too low, but the gap between Labour and Conservative – the problem that bedevilled all the pre-election polls, is much closer to reality.

This is a heavy pointer towards the make-up of samples having been a cause of the polling error. If the problems had been caused by people incorrectly reporting their voting intentions (“shy Tories”) or people saying they would when they did not then it is likely that exactly the same problems would have shown up in the British Election Study (indeed, given the interviewer effect those problems could have been worse). The difference between the BES f2f results and the pre-election polls suggests that the error is associated with the thing that makes the BES f2f so different from the pre-election polls – the way it is sampled.

As regular readers will know, most published opinion polls are not actually random. Most online polls are conducted using panels of volunteers, with respondents selected using demographic quotas to model the British public as closely as possible. Telephone polls are quasi-random, since they do at least select randomised numbers to call, but the fact that not everyone has a landline and that the overwhelming majority of people do not answer the call or agree to take part means the end results is not really close to a random sample. The British Election Study was a proper randomised study – it randomly picked consistencies, then addresses within in them, then a person at that address. The interviewer then repeatedly attempted to contact that specific person to take part (in a couple of cases up to 16 times!). The response rate was 56%.

Looking at Jon Mellon’s write up, this ties in well with the idea that polls were not including enough of the sort of people who don’t vote. One of the things that pollsters have flagged up in the investigations of what went wrong is that they found less of a gap in people’s reported likelihood of voting between young and old people than in the past, suggesting polls might no longer be correctly picking up the differential turnout between different social groups. The f2f BES poll did this far better. Another clue is in the comparison between whether people voted, and how difficult it was to get them to participate in the survey – amongst people who the BES managed to contact on their first attempt 77% said they had voted in the election, among those who took six or more goes only 74% voted. A small difference in the bigger scheme of things, but perhaps indicative.

This helps us diagnose the problem at the election – but it still leaves the question of how to solve it. I should pre-empt a couple of wrong conclusions that people will jump to. One is the idea polls should go back to face-to-face – this mixes up mode (whether a poll is done by phone, in person, or online) with sampling (how the people who take part in the poll are selected). The British Election Study poll appears to have got it right because of its sampling (because it was random), not because of its mode (because it was face-to-face). The two do not necessarily go hand-in-hand: when face-to-face polling used to be the norm in the 1980s it wasn’t done using random sampling, it was done using quota sampling. Rather than asking interviewers to contact a specific randomly selected person and to attempt contact time and again, interviewers were given a quota of, say, five middle-aged men, and any old middle-aged men would do.

That, of course, leads to the next obvious question of why don’t pollsters move to genuine random samples? The simple answers there are cost and time. I think most people in market research would agree a proper random sample like the BES is the ideal, but the cost is exponentially higher. This isn’t more expensive in the sense of “well, they should pay a bit if they want better results” type way – it’s more expensive as in a completely difference scale of expense, the difference between a couple of thousand and a couple of hundred thousand. No media outlet could ever justify the cost of a full scale random poll, it’s just not ever going to happen. It’s a shame, I for one would obviously be delighted were I to live in a world where people were willing to pay hundreds of thousands of pounds for polls, but such is life. Things like the BES only exist because of big funding grants from the ESRC (and at some elections that has need to be matched by grants from other charitable trusts).

The public opinion poll industry has always been about a finding a way of measuring public opinion that can combine accuracy with being affordable enough for people to actually buy and speedy enough to react to events, and whatever the solutions that emerge from the 2015 experience will have those same aims. Changing sampling techniques to make them resemble random sampling more could, of course, be one of the routes that companies look at. Or controlling their sampling and weighting in ways to better address shortcomings of the sampling. Or different ways of modelling turnout, like ComRes are looking at. Or something else yet unspeculated. Time will tell.

The other important bit of evidence we are still waiting for is the BES’s voter validation exercise (the large scale comparison of whether poll respondents’ claims on whether they voted or not actually match up against their individual records on the marked electoral register). That will help us understand a lot more about how well or badly the polls measured turnout, and how to predict individual respondents’ likelihood of voting.

Beyond that, the polling inquiry team have a meeting in January to announce their initial findings – we shall see what they come up with.