Survation have a poll in today’s Mail on Sunday. Topline figures are CON 37%(-1), LAB 45%(+1), LDEM 6%(-1). Fieldwork was Thursday and Friday and changes are since early October.

The eight point Labour lead is the largest any poll has shown since the election, so has obviously attracted some attention. As regular readers will know, Survation carry out both telephone and online polls. Their telephone method is unique to them, so could easily explain getting different results (Ipsos MORI still use phone polling, but they phone randomly generated numbers (random digit dialling), as opposed to Survation who phone actual numbers randomly selected from telephone databases). However, this was an online poll, and online there is nothing particularly unusual about Survation’s online method that might explain the difference. Survation use an online panel like all the other online polls, weight by very similar factors like age, gender, past vote, referendum vote and education, use self-reported likelihood to vote and exclude don’t knows. There are good reasons why their results are better for Labour than those from pollsters showing the most Tory results like Kantar and ICM (Kantar still use demographics in their turnout model, ICM reallocate don’t knows) but the gap compared to results from MORI and YouGov don’t have such an easy explanation.

Looking at the nuts and bolts of the survey, there’s nothing unusual about the turnout or age distribution. The most striking thing that explains the strong Labour position of the poll is that Survation found very few people who voted Labour in 2017 saying they don’t know how they would vote now. Normally even parties who are doing well see a chunk of their vote from the last election now saying they aren’t sure what they would do, but only 3% of Labour’s 2017 vote told Survation they weren’t sure how they would vote in an election, compared to about 10% in other polls. Essentially, Survation are finding a more robust Labour vote.

Two other interesting findings worth highlighting. One is a question on a second referendum – 50% said they would support holding a referendum asking if people supported the terms of a Brexit deal, 34% said they would be opposed. This is one of those questions that get very different answers depending on how you ask it – there are plenty of other questions that find opposition, and I’m conscious this question does not make it clear whether it would be a referendum on “accept deal or stay in EU”, “accept deal or continue negotiations” or “accept deal or no deal Brexit”. Some of these would be less popular than others. Nevertheless, the direction of travel is clear – Survation asked the same question back in April when there was only a five point lead for supporting a referendum on the deal, now that has grown to sixteen points (50% support, 34% opposed).

Finally there was a question on whether Donald Trump’s visit to the UK should go ahead. 37% think it should, 50% think it should not. This echoes a YouGov poll yesterday which found 31% think it should go ahead, 55% think it should not. I mention this largely as an antidote to people being mislead by twitter polls suggesting people want the visit to go ahead – all recent polls with representative samples suggest the public are opposed to a visit.

Tabs for the Survation poll are here.


Kantar have published a new voting intention poll ahead of the budget, the first I’ve seen from them since the general election. Topline figures are CON 42%, LAB 38%, LDEM 9%, UKIP 5%. Fieldwork was between last Tuesday and this Monday.

This is the first poll to show a Conservative lead since September and the largest Tory lead in any poll since the election. As ever, it’s best to look carefully at any poll that shows an unusual result before getting too excited/dismayed. The reason for the unusual result appears to be methodological, rather than from some sudden Tory recovery, and down to the way Kantar treat turnout. As regular readers will know, many polls came horribly unstuck at the 2017 election because instead of basing turnout on how likely respondents said they were to vote, they predicted respondents likelihood to vote based on factors like their age and class. These methods assumed young people would be much less likely to vote, and produced large Conservative leads that ended up being wrong. Generally speaking, these socio-economic models have been dropped.

At the election Kantar took a sort of halfway position – they based their turnout model on both respondents’ self-assessed likelihood to vote, whether they voted last time and their age, assuming that older people were more likely to vote than younger people. This actually performed far better than most other companies did; Kantar’s final poll showed a five point Conservative lead, compared to the 2.5 they actually got. As such, Kantar appear to have kept using their old turnout model that partly predicts likelihood to vote based on age. The impact of this is clear – before turnout weighting Labour would have had a one point lead, very similar to other companies’ polls. After turnout weighting the Conservatives are four points ahead (the full tabs and methodology details are here).

(Another noticable difference between Kantar’s method and other companies is that they use the leaders’ names in their voting intention question, though given there is not nearly as much of a gap between Theresa May and Jeremy Corbyn’s ratings as there used to be I’m not sure that would still have an impact.)


-->

Yesterday the British Polling Council had an event talking about how the polls had changed since 2015. This included collecting up data from all the companies on what they’ve done to correct the error and what they are now doing differently – all those summaries are collected here.

In looking at what’s changed it’s probably best to start with what actually went wrong and what problem the pollsters are trying to solve. As all readers will know, the polls in 2015 wrongly overstated Labour support and understated the Conservatives. The BPC/MRS inquiry under Pat Sturgis concluded this was down to unrepresentative samples.

Specially, it looked as if polls had too many younger people who were too engaged and too interested in politics. The effect of this was that while in reality there was a big difference between the high turnout among old people and the low turnout among young people, among the sort of people who took part in polls this gap was too small. In short, the sort of young people who took part in polls went out and voted Labour; the sort of young people who weren’t interested and stayed at home didn’t take part in polls either.

So, what have polling companies done to correct the problems? There is a summary for each individual company here.

There have been a wide variety of changes (including YouGov interlocking past vote & region, ICM changing how they reallocate don’t knows, ICM and ComRes now both doing only online polls during the campaign). However, the core changes seem to boil down to two approaches: some companies have focused on improving the sample itself, trying to include more people who aren’t interested in politics, who are less well educated and don’t usually vote. Other companies have focused on correcting the problems caused by less than representative samples, changing their turnout model so it is based more on demographics, and forcing it to more accurately reflect turnout patterns in the real world. Some companies have done a bit of both.

Changes to make samples less politically engaged…

  • ICM and YouGov have both added a weight by respondents level of interest or attention to politics, based upon the British Election Study probability survey. YouGov have also added weights by level of educational qualification.
  • Ipsos MORI haven’t added political interest weights directly, but have added education weights and newspaper readership weights, which correlate with political interest.
  • Kantar have added education weighting, and also weight down turnout to the level they project it to be as a way of reducing the overall level of political engagement in their sample.

Changes to base turnout on demographics…

  • ComRes have changed their turnout model, so it is based more on respondents’ demographics rather than how likely they claim they are to vote. The effect of this is essentially to downweight people who are younger and more working class on the assumption that the pattern of turnout that we’ve seen at past elections remains pretty steady. ICM have a method that seems very similar in its aim (I’m not sure of the technicalities) – weighting the data so that the pattern of turnout by age & social grade is the same as in 2015.
  • Kantar (TNS) have a turnout model that is partially based on respondents age (so again, assuming that younger people are less likely to vote) and partially on their self-reported likelihood.
  • ORB weight their data by education and age so that it matches not the electorate as a whole, but the profile of people who the 2015 British Election Study who actually voted (they also use the usual self-reported likelihood to vote weighting on top of this).
  • Opinium, MORI and YouGov still base their turnout models on people’s answers rather than their demographics, but they have all made changes. YouGov and MORI now weight down people who didn’t vote in the past, Opinium downweight people who say they will vote for a party but disapprove of its leader.
  • Panelbase and Survation haven’t currently made any radical changes since 2015, but Panelbase say they are considering using BES data to estimate likelihood to vote in their final poll (which sounds to me as if they are considering something along the lines of what ICM are doing with their turnout model)

In terms of actual outcomes, the pollsters who have adopted demographic turnout-models (ComRes, ICM and Kantar) tend to show larger Conservative leads than companies who have tried to address the problem only through sampling and weighting changes. We cannot really tell which is more likely to be right until June 8th. In short, for companies who have concentrated only on making samples more representative, the risk is that it hasn’t worked well enough, and that there are still too many of the sort of young engaged voters who are attracted to Jeremy Corbyn in their samples. For companies who have instead concentrated on demographic-based turnout models, the risk is that the pattern of turnout in 2017 differs from that in 2015, and that Jeremy Corbyn’s Labour really does manage to get more young people to come out to vote than Ed Miliband did. We will see what happens and, I expect, the industry will learn from whatever is seen to work this time round.


Two new voting intention polls today. The first by Survation for Good Morning Britain had topline figures of CON 48%(+1), LAB 30%(nc), LDEM 8%(+1), UKIP 4%(nc). Clearly there is no substantial change since their poll a week ago. Fieldwork was conducted on Friday and Saturday, after the leak of the Labour manifesto, and doesn’t show any sign of any impact.

The second was the weekly ICM poll for the Guardian. Topline figures there are CON 48%(-1), LAB 28%(+1), LDEM 10%(+1), UKIP 6%(nc). As many have noted, ICM are now are, along with TNS, one of only two pollsters still showing Labour support below thirty points (MORI last poll did the same, but that was several weeks ago when everyone showed Labour that low). It’s not that ICM haven’t shown Labour support rising a little. ICM have been showing Labour recovering slightly, it’s just they’ve been doing so at a slightly lower figures: at the start of the campaign ICM had Labour at 25-26% and they now have them at 27%-28%.

This seems to be a consistent methodological difference. The methodological differences between pollsters are complicated and various, and some of them work in opposite directions (ICM, for example, also reallocate don’t knows in a way that helps Labour) but the most obvious one at the moment is probably the approach to turnout. Traditionally British pollsters have accounted for people’s likelihood to vote by getting respondents to estimate their own likelihood to vote – put crudely, they ask people to say how likely they are to vote on a scale of 0 to 10, and then either weight them accordingly (someone who says they are 8/10 likely to vote is only counted as 8/10ths of someone who says 10/10), or apply a cut off, ignoring people who rate their chances below 5/10 or 9/10 or 10/10. Since 2015 several companies, including YouGov and Ipsos MORI, have also factored in whether people say they have voted in the past, weighting down past non-voters.

ICM and ComRes have adopted new approaches. Rather than basing their turnout model on people’s self-reported likelihood to vote, they base it on their demographics – estimating respondent’s likelihood to vote based on their age and social grade – the assumption being that younger people and working class people will remain less likely than older, more middle class people to vote. This tends to have the effect of making the results substantially more Conservative, less Labour, meaning that ICM and ComRes tend to produce some of the biggest Tory leads.

Full tabs for the ICM poll are here and the Survation poll here.


Polling myths

Whenever a poll goes up that shows bad news for someone you get the same sort of comments on social media. As I write this piece in May 2017 comments like these generally come from Jeremy Corbyn supporters, but that’s just the political weather at this moment in time. When the polls show Labour ahead you get almost exactly the same comments from Conservative supporters, when UKIP are doing badly you get them from UKIP supporters, when the Lib Dems are trailing you get them from Lib Dem supporters.

There are elements of opinion polling that are counter-intuitive and many of these myths will sound perfectly convincing to people who aren’t versed in how polls work. This post isn’t aimed at the hardcore conspiracists who are beyond persuasion – if you are truly convinced that polls are all a malevolent plot of some sort there is nothing I’ll be able to do to convince you. Neither is it really aimed at those who already know such arguments are nonsense: this is aimed at those people who don’t really want to believe what the polls are saying, see lots of people on social media offering comforting sounding reasons why you can ignore them, but are thinking, “Is that really true, or is it rather too convenient an excuse for waving away an uncomfortable truth…”

1) They only asked 1000 people out of 40 million. That’s not enough

This question has been about for as long as polling has. George Gallup, the trailblazer of modern polling, used to answer it by saying that it wasn’t necessary to eat a whole bowl of soup to know whether or not it was too salty, providing it had been stirred, a single spoonful was enough. The mention of stirring wasn’t just Gallup being poetic, it’s vital. Taking a single spoonful from the top of a bowl of soup might not work (that could be the spot where someone just salted it), but stirring the soup means that spoonful is representative of the whole bowl.

What makes a poll representative is not the size of the sample, it is its representativeness. You could have a huge sample size that was completely meaningless. Imagine, for example, that you did a poll of 1,000,000 over 65s. It would indeed be a huge sample, but it would be very skewed toward the Tories and Brexit. What makes a poll meaningful or not is whether it is representative of the country. Does it have the correct proportions of men and women? Old and young? Middle class and working class? Graduates and non-graduates? If the sample reflects British society as a whole in all these ways, then it should reflect it in terms of political opinion too. A poll of 1000 people is quite enough to get a representative sample.

The classic example of this was at the very birth of modern polling – in the US 1936 Presidential election a magazine called the Literary Digest did a survey of over two million people, drawn from magazine subscribers, telephone directories and so forth. It showed Alf Landon would win the Presidential election. The then newcomer George Gallup did a far, far smaller poll properly sampled by state, age, gender and so on. He correctly showed a landslide for Roosevelt. A poll with a sample skewed towards people wealthy enough to have phones and magazines in depression era America was worthless, despite have two million respondents.

2) Who do they ask? I’ve never been asked to take part in a poll!

Sometimes this is worked up to “…and neither has anyone I’ve met”, which does raise the question of whether the first thing these people do upon being introduced to a new person is to ask if MORI have ever rung them. That aside, it’s a reasonable question. If you’ve never been polled and the polls seem to disagree with your experience, where do all these answers come from?

The simple answer is that pollsters obtain their samples either by dialling randomly generated telephone numbers or by contacting people who are members of internet panels. Back when polls were mostly conducted by telephone the reason you had never been polled was simple maths – there were about forty million adults in Britain, there were about fifty or so polls of voting intention of a thousand people conducted each year. Therefore in any given year you had about a 0.1% chance of being invited to take part in a poll.

These days most opinion polls are conducted using online panels, but even if you are a member of a panel, your chances of being invited to a political poll are still relatively low. Most panels have tens of thousands of people (or for the better known companies, hundreds of thousands of people) and 95% of surveys are about commercial stuff like brands, pensions, grocery shopping and so on. You could still be waiting some time to be invited to a political one.

3) But nobody I know is voting for X!

We tend to know and socialise with people who are quite like ourselves. Our social circles will tend to be people who live in the same sort of area as us, probably people who have a similar sort of social status, a similar age. You probably have a fair amount in common with your friends or they wouldn’t be your friends. Hence people we know are more likely than the average person to agree with us (and even when they don’t, they won’t necessarily tell us; not everyone relishes a political argument). On social media it’s even worse – a large number of studies have shown that we tend to follow more people we agree with, producing self-reinforcing bubbles of opinion.

During the Labour leadership contest almost every one of my friends who is a member of the Labour party was voting for Liz Kendall. Yet the reality was that they were all from a tiny minority of 4.5% – it’s just that the Labour party members I knew all happened to be Blairite professionals working in politics in central London. Luckily I had proper polling data that was genuinely reflective of the whole of the Labour party, so I knew that Jeremy Corbyn was in fact in the lead.

In contrast to the typical friendship group, opinion polls samples will be designed so that they reflect the whole population and don’t fall into those traps. They will have the correct balance of people from all across the country, will have the correct age range, will have the correct balance of social class and past vote and so on. Perhaps there are people out there who, by some freak co-incidence, have a circle of acquaintances who form a perfectly representative sample of the whole British public, but I doubt there are very many.

4) Pollsters deliberately don’t ask Labour/Conservative supporters

In so far as there is any rationale behind the belief, it’s normally based upon the perception that someone said they were going to vote for x in a poll, and weren’t asked again. As we’ve seen above, it’s a lot more likely that the reason for this is simply that it’s relatively rare to be invited to a political poll anyway. If you’ve been asked once, the chances are you’re not going to be asked again soon whatever answers you gave.

Under the British Polling Council rules polling companies are required to publish the details of their samples – who was interviewed, what the sample was weighted by and so on. These days almost every company uses some form of political sampling or weighting to ensure that the samples are politically representative. Hence in reality pollsters deliberately include a specific proportion of 2015 Labour supporters in their polls, generally the proportion who did actually vote Labour in 2015. Pollsters are required to report these figures in their tables, or to provide them on request. Hence, if you look at last weekend’s Opinium poll you’ll find that 31% of people in the poll who voted in 2015 voted Labour, the proportion that actually did, if you look at the ICM poll you’ll find that 31% of the people who voted at the last election say they voted Labour, the proportion that actually did, and so on with every other company.

5) Pollsters are biased, and fix their figures

Again, this an accusation that is as old as polling – if you don’t like the message, say the person making it is biased. It’s made easier by the fact that a lot of people working in political polling do have a background in politics, so if you want to look for someone to build a conspiracy theory upon, you don’t need to look far. Over the years I think we’ve been accused of being biased towards and against every party at one time or another – when Labour were usually ahead in the polls YouGov used to be accused of bias because Peter Kellner was President. When the Conservatives were ahead different people accused us of being biased because Stephen Shakespeare was the CEO. The reality is, of course, that polling companies are made up of lots of people with diverse political views (which is, in fact, a great benefit when writing questions – you can get the opinion of colleagues with different opinions to your own when making sure things are fair and balanced).

The idea that polling companies would bias their results to a particular party doesn’t really chime with the economics of the business or the self-interest of companies and those who run them. Because political polls are by far the most visible output of a market research company there is a common misapprehension that it brings in lots of money. It does not. It brings in very little money and is often done as a loss-leader by companies in order to advertise their wares to the commercial companies that spend serious money doing research on brand perceptions, buying decisions and other consumer surveys. Voting intention polls are one of the very few measures of opinion that get checked against reality – it is done almost entirely as a way of the company (a) getting their name known and (b) demonstrating that their samples can accurately measure public opinion and predict behaviour. Getting elections wrong, however, risks a huge financial cost to market research companies through reputational damage and, therefore, huge financial cost to those running them. It would be downright perverse to deliberately get those polls wrong.

6) Polls always get it wrong

If the idea that polling companies would ruin themselves by deliberately getting things wrong is absurd, the idea that polls can get it wrong by poor design is sadly true: polls obviously can get it wrong. Famously they did so at the 2015 general election. Some polls also got Brexit wrong, though the picture is more mixed that some seem to think (most of the campaign polls on Brexit actually showed Leave ahead). Polls tend to get it right a lot more often than not though – even in recent years, when their record is supposed to have been so bad, the polls were broadly accurate on the London mayoral election, the Scottish Parliamentary election, the Welsh Assembly election and both of the Labour party leadership elections.

Nevertheless, it is obviously true to say that polls can be wrong. So what’s the likelihood that this election will be one of those occasions? Following the errors of the 2015 general election the British Polling Council and Market Research Society set up an independent inquiry into the polling error and what caused it, under the leadership of Professor Pat Sturgis at Southampton University. The full report is here, and if you have some spare time and want to understand how polling works and what can go wrong with them it is worth putting aside some time to read it. The extremely short version is, however, that the polls in 2015 weren’t getting samples that were representative enough of the general public – people who agreed to take part in a phone poll, or join an internet panel weren’t quite normal, they were too interested in politics, too engaged, too likely to vote.

Since then polling companies have made changes to try and address that problem. Different companies have taken different approaches. The most significant though are a mix of adding new controls on samples by education and interest in politics and changes to turnout models. We obviously won’t know until the election has finished whether these have worked or not.

So in that context, how does one judge current polls? Well, there are two things worth noting. The first is that while polls have sometimes been wrong in the past, their error has not been evenly distributed. They have not been just as likely to underestimate Labour as they have been to overestimate Labour: polling error has almost always overstated Labour support. If the polls don’t get it right, then all previous experience suggests it will be because they have shown Labour support as too *high*. Theoretically polls could have tried too hard to correct the problems of 2015 and be overstating Conservative support, but given the scale of the error in 2015 and the fact that some companies have made fairly modest adjustments, that seems unlikely to be the case across the board.

Secondly is the degree of error. When polls are wrong they are only so wrong. Even those elections where the polls got it most wrong, like 1992 and 2015, their errors were nowhere near the size of the Conservative party’s current lead.

Short version is, yes, the polls could be wrong, but even the very worst polls have not been wrong enough to cancel out the size of lead that the Tories currently have and when the polls have been that wrong, it’s always been by putting Labour too high.

So, if you aren’t the sort to go in for conspiracy theories, what comfort can I offer if the polls aren’t currently showing the results you’d like them to? Well, first the polls are only ever a snapshot of current opinion. They do not predict what will happen next week or next month, so there is usually plenty of time for them to change. Secondly, for political parties polls generally contain the seeds of their salvation, dismissing them misses the chance to find out why people aren’t voting for you, what you need to change in order to win. And finally, if all else fails, remember that public opinion and polls will eventually change, they always do. Exactly twenty years ago the polls were showing an utterly dominant Labour party almost annihilating a moribund Tory party – the pendulum will likely swing given enough time, the wheel will turn, another party will be on the up, and you’ll see Conservative party supporters on social media trying to dismiss their awful polling figures using exactly the same myths.