There have been three polls over the last week – in the Sunday papers there were polls from ComRes and Opinium, the regular YouGov poll for the Times last week. Voting intention figures were:

Opinium – CON 37%, LAB 25%, LDEM 16%, BREX 13%, GRN 2% (tabs)
ComRes – CON 28%, LAB 27%, LDEM 20%, BREX 13%, GRN 5% (tabs)
YouGov – CON 32%, LAB 23%, LDEM 19%, BREX 14%, GRN 7% (tabs)

There isn’t really a consistent trend to report here – YouGov and ComRes have the Conservatives declining a little from the peak of the Johnson honeymoon, but Opinium show them continuing to increase in support. My view remains that voting intention probably isn’t a particularly useful measure to look at when we know political events are looming that are likely to have a huge impact. Whatever the position is now, it is likely to be transformed by whether or not we end up leaving the European Union next month, on what terms and under what circumstances.

What did receive some comment was the sheer contrast between the reported leads, particularly because the ComRes (1 point Tory lead) and Opinium (12 point Tory lead) were published on the same day.

Mark Pickup, Will Jennings and Rob Ford wrote a good article earlier this month looking at the house effects of different pollsters. As you may expect if you’ve been watching recent polls, ComRes tend to show some of the largest Labour leads, YouGov some of the biggest Tory leads. Compared to the industry average Opinium actually tend to be slightly better for Labour and slightly worse for the Tories, though I suspect that may be changing: “House effects” for pollsters are not set in stone and can change over time, partly because pollsters change methods, partly because the impact of methodological differences change over time.

What that doesn’t tell us why there is a difference. I saw various people pointing at the issue of turnout, and how pollsters model likelihood to vote. I would urge some caution there – in the 2017 election, most of the difference between polls was indeed down to how polling companies predicted likelihood to vote, and this was the biggest cause of polling error. However when those new turnout models backfired and went wrong, polling companies dropped them. There are no longer any companies using demographic based turnout models that have a huge impact on voting intention figures and weight down young people. These days almost everyone has gone back to basing their turnout models primarily on how likely respondents themselves say they are to vote, a filter that typically only has a modest impact. It may be one factor, but it certainly wasn’t the cause of the difference between ComRes and Opinium.

While polling companies don’t have radically different turnout models, it is true to say (as Harry does here) that ComRes tends to imply a higher level of turnout among young people that Opinium. One thing that is contributing to that in the latest poll is that Opinium ask respondents if they are registered to vote, and only include those people who are, reducing the proportion of young people in their final figures. I expect, however, that some of it is also down to the respondents themselves, and how representative they are – in other words, because of the sample and weights ComRes may simply have young people who say they are more likely to vote than the young people Opinium have.

As regular readers will know, one important difference between polling companies at the moment appears to be the treatment of past vote weighting, and how polling companies account for false recall. Every polling company except for Ipsos MORI and NCPolitics use past vote in their weighting scheme. We know how Britain actually voted at the last election (CON 43%, LAB 41%, LDEM 8%), so a properly representative sample should have, among those people who voted, 43% people who voted Tory, 41% people who voted Labour, 8% who voted Lib Dem. If a polling company finds their sample has, for example, too many people who voted Tory at the previous election, they can weight those people down to make it representative. This is simple enough, apart from the fact that people are not necessarily very good at accurately reporting how they voted. Over time their answers diverge from reality – people who didn’t vote claim they did, people forget, people say they voted for the party they wish they’d voted for, and so on. We know this for certain because of panel studies – experiments where pollsters ask people how they voted after an election, record it, then go back and ask the same people a few years later and see if their answers have changed.

Currently it appears that people are becoming less likely to remember (or report) having voted Labour in 2017. There’s an example that YouGov ran recently here. YouGov took a sample of people whose votes they had recorded in 2017 and asked them again how they had voted. In 2017 41% of those people told YouGov’s they’d voted Labour, when re-asked in 2019 only 33% of them said they had voted Labour. This causes a big problem for past vote weighting, how can you weight by it, if people don’t report it accurately? If a fifth of your Labour voters do not accurately report that they voted Labour and the pollster weights the remaining Labour voters up to the “correct” level they would end up with too many past Labour voters, as they’d have 41% past Labour voters who admitted it, plus an unknown amount of past Labour voters who did not.

There are several ways of addressing this issue. One is for polling companies to collect the data on how their panellists voted as soon as possible after the election, while it is fresh in their minds, and then use that contemporaneous data to weight future polls by. This is the approach YouGov and Opinium use. The other approach is to try and estimate the level of false recall and adjust for it – this is what Kantar have done, instead of weighting to the actual vote shares in 2017, they assume a level of false recall and weight to a larger Conservative lead than actually happened. A third approach is to assume there is no false recall and weight to the actual figures – one that I think currently risks overstating Labour support. Finally, there is the approach that Ipsos MORI have always taken – assuming that false recall is such an intractable problem that it cannot be solved, and not weighting by past vote at all.

Dealing with false recall is probably one reason for the present difference between pollsters. Polling companies who are accounting for false recall or using methods that get round the problem are showing bigger Tory leads than those who do not. It is, however, probably not enough to explain all the difference. Neither, should we assume that the variation between pollsters is all down to those differences that are easy to see and compare in the published tables. Much of it is probably also down to the interaction of different weighting variables, or to the very samples themselves. As Pat Sturgis, the chair of the 2015 enquiry into polling error, observed at the weekend there’s also the issue of the quality of the online panels the pollsters use – something that is almost impossible to objectively measure. While we are wondering about the impact of weights and turnout filters, the difference may just be down to some pollsters having better quality, more representative panels than others.


On Saturday YouGov released a new poll of Tory party members for the Times, timed to coincide with ballot papers going out and members actually starting to cast their votes. If the race was to be in any way close it would really need to have shown a substantial drop in Boris Johnson’s lead. It did not show any drop at all – Boris Johnson continued to have a 48 point lead over Jeremy Hunt, 74% to 26%.

Boris Johnson’s private life was seen as irrelevant, members would be happier with him as leader, trusted him more, thought he would be a better Prime Minister. In terms of the race itself, the poll was very much cut and dried. With that in mind, perhaps the more interesting thing to look at is members’ expectations. Despite Boris Johnson’s stated aim, only 45% of party members think he will actually be able to negotiate a better deal. His attraction seems more because 90% of members think he would be prepared to leave without one. Even then, only 54% of party members think a Johnson led party would actually end up leaving without a deal by Oct 31st (26% think he will leave with a deal, 13% that we won’t have left by then). Even so, most party members don’t seem to be in the mood to set red lines – only 34% think that it would be a resigning offence if the new leader failed to deliver Brexit by October 31st.

Full tables are here.

Since I’ve been asked about it by a lot of journalists over the last week or so, I should probably also explain a bit more about how polling party members works. First up, it is hard. If you think about the traditional approaches towards polling, they simply aren’t plausible for polling members of political parties. The Conservative party themselves are not likely to provide polling companies with a list of party members’s contact details to randomly select people from. Given far less than 1% of the population are Conservative party members it is certainly not feasible to randomly ring people up in the hope of finding Conservative party members, neither do members live in geographically concentrated areas that would make the sort of clustered face-to-face sampling that is sometimes used for BME polling feasible. Apart from an academic study in the 1990s that had the co-operation of the party itself, polling of party members was simply impossible before the advent of internet polling.

The only way that it is possible these days is to use an internet panel, either a small, specially recruited one like ConHome’s mailing list, or the way YouGov do it – by having a panel of the general public that is so large that you can draw very niche samples like party members from within it. YouGov identify Conservative members as part of the general process of collecting demographic information about respondents – as well as age, gender, occupation and so on panellists are asked if they are a member of organisations such as the National Trust, WI, RSPB, English Heritage, Conservative party, Labour party and so on. The parties are asked alongside other organisations, at neutral times (and the occasional clever bugger who claims to be a member of every party to get into all the surveys is excluded from them all). Party membership is asked again during the survey to check answers are consistent.

It remains tricky however because of a lack of demographic targets. For normal polling of the British public quotas and weights will be set based on known demographics of the target population. For example, we know from the census and ONS population estimates that around 49% of the adult population in Britain are male, 51% female, so polling companies will ensure samples reflect that. The Conservative party does not publish any such targets, so polling companies are flying a little blind. YouGov estimate targets based on the demographics of party members on our wider panel and known skews within it, but it poses an additional difficulty.

So polls of party members pose particular challenges, but in this case Boris Johnson’s lead is so large and, more importantly, so consistent across groups that he is likely to win regardless. He leads among different age groups, men and women, working class and middle class, and every region – so in the event that the balance of those groups were a bit off, it wouldn’t change the victor. The only group Jeremy Hunt leads amongst is those party members who voted to Remain.

For whats worth, YouGov’s record of polling party leadership contests has been extremely good in the past. If anything, the problems that have bedevilled polls in recent decades and companies have spent so much time and money addressing – getting respondents who are too interested in politics – have been a positive in recruiting respondents to polls of party members.


-->

To start with, here’s an update of all the pre-election polls (Ipsos MORI, Survation and NCPolitics all published theirs on the morning of election day, after my last post).

Note that ComRes and Hanbury also produced polls during the campaign, but not with fieldwork conducted on or after the final weekend of the campaign. For what it’s worth, they tended to show high Labour support, though we’ll never know what their polls would have shown in the final week.

Needless to say, the pre-election polls varied wildly from one another for all the main parties. Labour had a twelve point spread (13% to 25%), the Conservatives eight points (7% to 15%), the Liberal Democrats (12% to 20%), the Brexit party eleven points (27% to 38%). In the event, the polls that had low Labour scores and high Liberal Democrat scores were closest to reality. Compared to the final results, Ipsos MORI took the laurels, getting close to the correct result for all parties. YouGov were next, getting within a point or two of most parties but overstating the Brextit party. Other companies recorded significant errors, with a few double-digit overstatements of Labour support.

It is difficult to point at a single obvious cause for the wide variation. When there were huge differences between polls at the 2017 election the reasons the were clear: pollsters had adopted demographic turnout models and other post-fieldwork adjustments which backfired and overstated Tory support. There is no such easy explanation for the 2019 polls – pollsters have mainly reversed the missteps of 2017 and, while there are some variations in approaches to turnout, the elaborate turnout models that made such a difference in 2017 have disappeared. Different approaches to turnout perhaps explain differences of a point or two, they don’t explain differences of 10 points. The differences here look as if they are more likely to be down to pollsters’ different approaches to sampling or weighting, and the representativeness of their samples.

From the beginning these European elections were going to be a challenge. They are a low turnout election, when at recent elections polls have struggled to correctly reflect the demographic pattern of turnout. In recent decades most British pollsters have also relied upon past-vote weighting to ensure their polls are politically representative, and this was an election when past vote was a particularly poor predictor of current voting intention.

In terms of what this means for wider polling, errors here don’t necessarily transfer directly across to Westminster polls. The challenges posed by high-turnout elections can be very different to those posed by low-turnout elections and just because some polls overstated Labour support in the European elections does not necessarily mean they are overstating Labour support for general elections. On the other hand, given the recent history of errors, it probably isn’t something we in the polling industry should be complacent about.


Opinion polling on Brexit has not necessarily been the best. Highly politically contentious issues do tend to attract polling that is sub-optimal, and Brexit has followed that trend. I’ve seen several Brexit polls coming up with surprising findings based on agree/disagree statements – that is, questions asked in the form:

Do you agree with the following statement? I think Brexit is great
Agree
Disagree
Don’t know

This is a very common way of asking questions, but one that has a lot of problems. One of the basic rules in writing fair and balanced survey questions is that you should try to given equal prominence to both sides of the argument. Rather than ask “Do you support X?”, a survey should ask “Do you support or oppose X?”. In practice agree-disagree statements break that basic rule – they ask people whether they agree/disagree with one side of the argument, without mentioning the other side of the argument.

In some cases the opposite side of the argument is implicit. If the statement is “Theresa May is doing a good job”, then it is obvious to most respondents that the alternative view is that May is doing a bad job (or perhaps an average job). Even when it’s as obvious as this it still sometimes to make a difference – for whatever reason, decades of academic research into questionnaire design suggest people are more likely to agree with statements than to disagree with them, regardless of what the statement is (generally referred to as “acquiescence bias”).

There is a substantial body of academic evidence exploring this phenomenon (see, for example Schuman & Presser in the 1980s, or the recent work of Jon Krosnick) it tends to find around 10%-20% of people will agree with both a statement and its opposite, if it is asked in both directions. Various explanations have been put forward for this in academic studies – that it’s a result of personality type, or that it is satisficing (people just trying to get through a survey with minimal effort). The point is that it exists.

This is not just a theoretical issue that turns up in artificial academic experiments – they are plenty of real life examples in published polls. My favourite remains this ComRes poll for UKIP back in 2009. It asked if people agreed or disagreed with a number of statements including “Britain should remain a full member of the EU” and “Britain should leave the European Union but maintain close trading links”. 55% of people agreed that Britain should remain a full member of the EU. 55% of people also agreed that Britain should leave the EU. In other words, at least 10% of the same respondents agreed both that Britain should remain AND leave.

There is another good real life example in this poll. 42% agreed with a statement saying that “divorce should not be made too easy, so as to encourage couples to stay together”. However, 69% of the same sample also agreed that divorce should be “as quick and easy as possible”. At least 11% of the sample agreed both that divorce should be as easy as possible AND that it should not be too easy.

Examples like this of polls that asked both sides of the argument and produced contradictory findings are interesting quirks – but since they asked the statement in both directions they don’t mislead. However, it is easy to imagine how they would risk being misleading if they had asked the statement in only one direction. If that poll had only asked the pro-Brexit statement, then it would have looked as if a majority supported leaving. If the poll had only asked the anti-Leave statement, then it would have looked as if a majority supported staying. With agree-disagree statements, if you don’t ask both sides, you risk getting a very skewed picture.

In practice, I fear the problem is often far more serious in published political polls. The academic studies tend to use quite neutrally worded, simple, straightforward statements. In the sort of political polling for pressure groups and campaigning groups that you see in real life the statements are often far more forcefully worded, and are often statements that justify or promote an opinion – below are some examples I’ve seen asked as agree-disagree statements in polls:

“The Brexit process has gone on long enough so MPs should back the Prime Minister’s deal and get it done”
“The result of the 2016 Referendum should be respected and there should be no second referendum”
“The government must enforce the minimum wage so we have a level playing field and employers can’t squeeze out British workers by employing immigrants on the cheap”

I don’t pick these because they are particularly bad (I’ve seen much worse), only to illustrate the difference. These are statements that are making an active argument in favour of an opinion, where the argument in the opposite direction is not being made. They do not give a reason why MPs may not want to back the Prime Minister’s deal, why a second referendum might be a good idea, why enforcing the minimum wage might be bad. It is easy to imagine that respondents might find these statements convincing… but that they might have found the opposite opinion just as convincing if they’d been presented with that. I would expect questions like this to produce a much larger bias in the direction of the statement if asked as an agree-disagree statement.

With a few exceptions I normally try to avoid running agree-disagree statements, but we ran some specially to illustrate the problems, splitting the sample so that one group of respondents were asked if they agreed or disagreed with a statement, and a second group where asked if they agreed-disagreed with a contrasting statement. As expected, it produces varied results.

For simple questions, like whether Theresa May is doing a good job, the difference is small (people disagreed with the statement that “Theresa May is doing a good job by 57% to 15% and agreed with the statement that “Theresa May is doing a bad job” by 52% to 18%. Almost a mirror image. On some of the other questions, the differences were stark:

  • If you asked if people agree that “The NHS needs reform more than it needs extra money” then people agree by 43% to 23%. However, if you ask if people agree with the opposite statement, that “The NHS needs extra money more than it needs reform”, then people also agree, by 53% to 20%.
  • If you ask if people agree or disagree that “NHS services should be tailored to the needs of populations in local areas, even if this means that there are differences across the country as a whole” than people agree by 43% to 18%. However, if you ask if they agree or disagree with a statement putting the opposite opinion – “NHS services should be the same across the country” – then people agree by 88% to 2%!
  • By 67% to 12% people agree with the statement that “Brexit is the most important issue facing the government and should be its top priority”. However, by 44% to 26% they also agree with the statement “There are more important issues that the government should be dealing with than Brexit”

I could go on – there are more results here (summary, full tabs) – but I hope the point is made. Agree/disagree statements appear to produce a consistent bias in favour of the statement, and while this can be minor in questions asking simple statements of opinion, if the statements amount to political arguments the scale of the bias can be huge.

A common suggested solution to this issue is to make sure that the statements in a survey are balanced, with an equal amount of statements in each direction. So, for example, if you were doing a survey about attitudes towards higher taxes, rather than asking people if they agreed or disagreed with ten statements in favour of high taxes, you’d ask if people agreed or disagreed with five statements in favour of higher taxes and five statements in favour of lower taxes.

This is certainly an improvement, but is still less than ideal. First it can produce contradictory results like the examples above. Secondly, in practice it can often result in some rather artificial and clunky sounding questions and double-negatives. Finally, in practice it is often difficult to make sure statements really are balanced (too often I have seen surveys that attempt a balanced statement grid, but where the statements in one direction are hard-hitting and compelling, and in the other direction are deliberately soft-balled or unappetising).

The better solution is not to ask them as agree-disagree statements at all. Change them into questions with specific answers – instead of asking if people agree that “Theresa May is going a good job”, ask if May is doing a good or bad job. Instead of asking if people agree that “The NHS needs reform more than it needs more money”, ask what people think the NHS needs more – reform or more money? Questions like the examples I gave above can easily be made better by pairing the contrasting statements, and asking which better reflects respondents views:

  • Asked to pick between the two statements on NHS reform or funding, 41% of people think it needs reform more, 43% think it needs extra money more.
  • Asked to pick between the two statements on NHS services, 36% think they should be tailored to local areas, 52% would prefer them to be the same across the whole country.
  • Asked to pick between the two statements on the importance of Brexit, 58% think it is the most important issue facing the government, 27% think there are more important issues the government should be dealing with instead.

So what does this mean when it comes to interpreting real polls?

The sad truth is that, despite the known problems with agree-disagree statements, they are far from uncommon. They are quick to ask, require almost no effort at all to script and are very easy for clients after a quick headline to interpret. And I fear there are some clients to whom the problems with bias are an advantage, not a obstacle; you often see them in polls commissioned by campaigning groups and pressure groups with a clear interest in getting a particular result.

Whenever judging a poll (and this goes to observers reading them, and journalists choosing whether to report them) my advice has always been to go to polling companies websites and look at the data tables – look at the actual numbers and the actual question wording. If the questions behind the headlines have been asked using agree-disagree statements, you should be sceptical. It’s a structure that does have an inherent bias, and does result in more people agreeing than if the question had been asked a different way.

Consider how the results may have been very different if the statement had been asked in the opposite direction. If it’s a good poll, you shouldn’t have to imagine that – the company should have made the effort to balance the poll by asking some of the statements in the opposite direction. If they haven’t made that effort, well, to me that rings some alarm bells.

If you get a poll that’s largely made up of agree-disagree statements, that are all worded in the direction that the client wants the respondent to answer rather than some in each direction, that use emotive and persuasive phrasing rather than bland and neutral wording? You would be right to be cautious.


There have been several new polls with voting intention figures since the weekend, though all so far have been conducted before the government’s defeat on their Brexit plan.

ComRes/Express (14th-15th) – CON 37%(nc), LAB 39%(nc), LDEM 8%(-1), UKIP 7%(+1)
YouGov/Times (13th-14th)- CON 39%(-2), LAB 34%(-1), LDEM 11%(nc), UKIP 6%(+2)
Kantar (10th-14th) – CON 35%(-3), LAB 38%(nc), LDEM 9%(nc), UKIP 6%(+2)

Looking across the polls as a whole Conservative support appears to be dropping a little, though polls are still ultimately showing Labour and Conservative very close together in terms of voting intention. As ever there are some differences between companies – YouGov are still showing a small but consistent Tory lead, the most recent polls from BMG, Opinium and MORI had a tie (though Opinium and MORI haven’t released any 2019 polls yet), Kantar, ComRes and Suration all showed a small Labour lead in their most last polls.

Several people have asked me about the reasons for the difference between polling companies figures. There isn’t an easy answer – there rarely is. The reality is that all polling companies want to be right and want to be accurate, so if there were easy explanations for the differences and it was easy to know what the right choices were, they would all rapidly come into line!

There are two real elements that are responsible for house effects between pollsters. The first is the things they do to the voting intention data after it is collected and weighted – primarily that is how do they account for turnout (to what extent do they weight down or filter out people who are unlikely to vote), and what to do they with people who say they don’t know how they’ll vote (do they ignore them, or use squeeze questions or inference to try and estimate how they might end up voting). The good thing about these sort of differences is that they are easily quantifiable – you can look up the polling tables, compare the figures with turnout weighting and without, and see exactly the impact they have.

At the time of the 2017 election these adjustments were responsible for a lot of the difference between polling companies. Some polls were using turnout models that really transformed their topline figures. However, those sort of models also largely turned out to be wrong in 2017, so polling companies are now using much lighter touch turnout models, and little in the way of reallocating don’t knows. There are a few unusual cases (for example, I think ComRes still reallocate don’t knows, which helps Labour at present, but most companies do not. BMG no longer do any weighting or filtering by likelihood to vote, an adjustment which for other companies tends to reduce Labour support by a point or two). These small differences are not, by themselves, enough to explain the differences between polls.

The other big differences between polls are their samples and the weights and quotas they use to make them representative. It is far, far more difficult to quantify the impact of these differences (indeed, without access to raw samples it’s pretty much impossible). Under BPC rules polling companies are supposed to be transparent about what they weight their samples by and to what targets, so we can tell what the differences are, but we can’t with any confidence tell what the impact is.

I believe all the polling companies weight by age, gender and region. Every company except for Ipsos MORI also votes by how people voted at the last election. After that polling companies differ – most vote by EU Ref vote, some companies weight by education (YouGov, Kantar, Survation), some by social class (YouGov, ComRes), income (BMG, Survation), working status (Kantar), level of interest in politics (YouGov), newspaper readership (Ipsos MORI) and so on.

Even if polling companies weight by the same variables, there can be differences. For example, while almost everyone weights by how people voted at the last election, there are differences in the proportion of non-voters they weight to. It makes a difference whether targets are interlocked or not. Companies may use different bands for things like age, education or income weighting. On top of all this, there are questions about when the weighting data is collected, for things like past general election vote and past referendum vote there is a well-known phenomenon of “false recall”, where people do not accurately report how they voted in an election a few years back. Hence weighting by past vote data collected at the time of the election when it was fresh in people’s minds can be very different to weighting by past vote data collected now, at the time of the survey when people may be less accurate.

Given there isn’t presently a huge impact from different approaches to turnout or don’t knows, the difference between polling companies is likely to be down some of these factors which are – fairly evidently – extremely difficult to quantify. All you can really conclude is that the difference is probably down to the different sampling and weighting of the different companies, and that, short of a general election, there is no easy way for either observers (nor pollsters themselves!) to be sure what the right answer is. All I would advise is to avoid the temptation of (a) assuming that the polls you want to be true are correct… that’s just wishful thinking, or (b) assuming that the majority are right. There are plenty of instances (ICM in 1997, or Survation and the YouGov MRP model in 2017), when the odd one out turned out to be the one that was right.