On Saturday YouGov released a new poll of Tory party members for the Times, timed to coincide with ballot papers going out and members actually starting to cast their votes. If the race was to be in any way close it would really need to have shown a substantial drop in Boris Johnson’s lead. It did not show any drop at all – Boris Johnson continued to have a 48 point lead over Jeremy Hunt, 74% to 26%.

Boris Johnson’s private life was seen as irrelevant, members would be happier with him as leader, trusted him more, thought he would be a better Prime Minister. In terms of the race itself, the poll was very much cut and dried. With that in mind, perhaps the more interesting thing to look at is members’ expectations. Despite Boris Johnson’s stated aim, only 45% of party members think he will actually be able to negotiate a better deal. His attraction seems more because 90% of members think he would be prepared to leave without one. Even then, only 54% of party members think a Johnson led party would actually end up leaving without a deal by Oct 31st (26% think he will leave with a deal, 13% that we won’t have left by then). Even so, most party members don’t seem to be in the mood to set red lines – only 34% think that it would be a resigning offence if the new leader failed to deliver Brexit by October 31st.

Full tables are here.

Since I’ve been asked about it by a lot of journalists over the last week or so, I should probably also explain a bit more about how polling party members works. First up, it is hard. If you think about the traditional approaches towards polling, they simply aren’t plausible for polling members of political parties. The Conservative party themselves are not likely to provide polling companies with a list of party members’s contact details to randomly select people from. Given far less than 1% of the population are Conservative party members it is certainly not feasible to randomly ring people up in the hope of finding Conservative party members, neither do members live in geographically concentrated areas that would make the sort of clustered face-to-face sampling that is sometimes used for BME polling feasible. Apart from an academic study in the 1990s that had the co-operation of the party itself, polling of party members was simply impossible before the advent of internet polling.

The only way that it is possible these days is to use an internet panel, either a small, specially recruited one like ConHome’s mailing list, or the way YouGov do it – by having a panel of the general public that is so large that you can draw very niche samples like party members from within it. YouGov identify Conservative members as part of the general process of collecting demographic information about respondents – as well as age, gender, occupation and so on panellists are asked if they are a member of organisations such as the National Trust, WI, RSPB, English Heritage, Conservative party, Labour party and so on. The parties are asked alongside other organisations, at neutral times (and the occasional clever bugger who claims to be a member of every party to get into all the surveys is excluded from them all). Party membership is asked again during the survey to check answers are consistent.

It remains tricky however because of a lack of demographic targets. For normal polling of the British public quotas and weights will be set based on known demographics of the target population. For example, we know from the census and ONS population estimates that around 49% of the adult population in Britain are male, 51% female, so polling companies will ensure samples reflect that. The Conservative party does not publish any such targets, so polling companies are flying a little blind. YouGov estimate targets based on the demographics of party members on our wider panel and known skews within it, but it poses an additional difficulty.

So polls of party members pose particular challenges, but in this case Boris Johnson’s lead is so large and, more importantly, so consistent across groups that he is likely to win regardless. He leads among different age groups, men and women, working class and middle class, and every region – so in the event that the balance of those groups were a bit off, it wouldn’t change the victor. The only group Jeremy Hunt leads amongst is those party members who voted to Remain.

For whats worth, YouGov’s record of polling party leadership contests has been extremely good in the past. If anything, the problems that have bedevilled polls in recent decades and companies have spent so much time and money addressing – getting respondents who are too interested in politics – have been a positive in recruiting respondents to polls of party members.


To start with, here’s an update of all the pre-election polls (Ipsos MORI, Survation and NCPolitics all published theirs on the morning of election day, after my last post).

Note that ComRes and Hanbury also produced polls during the campaign, but not with fieldwork conducted on or after the final weekend of the campaign. For what it’s worth, they tended to show high Labour support, though we’ll never know what their polls would have shown in the final week.

Needless to say, the pre-election polls varied wildly from one another for all the main parties. Labour had a twelve point spread (13% to 25%), the Conservatives eight points (7% to 15%), the Liberal Democrats (12% to 20%), the Brexit party eleven points (27% to 38%). In the event, the polls that had low Labour scores and high Liberal Democrat scores were closest to reality. Compared to the final results, Ipsos MORI took the laurels, getting close to the correct result for all parties. YouGov were next, getting within a point or two of most parties but overstating the Brextit party. Other companies recorded significant errors, with a few double-digit overstatements of Labour support.

It is difficult to point at a single obvious cause for the wide variation. When there were huge differences between polls at the 2017 election the reasons the were clear: pollsters had adopted demographic turnout models and other post-fieldwork adjustments which backfired and overstated Tory support. There is no such easy explanation for the 2019 polls – pollsters have mainly reversed the missteps of 2017 and, while there are some variations in approaches to turnout, the elaborate turnout models that made such a difference in 2017 have disappeared. Different approaches to turnout perhaps explain differences of a point or two, they don’t explain differences of 10 points. The differences here look as if they are more likely to be down to pollsters’ different approaches to sampling or weighting, and the representativeness of their samples.

From the beginning these European elections were going to be a challenge. They are a low turnout election, when at recent elections polls have struggled to correctly reflect the demographic pattern of turnout. In recent decades most British pollsters have also relied upon past-vote weighting to ensure their polls are politically representative, and this was an election when past vote was a particularly poor predictor of current voting intention.

In terms of what this means for wider polling, errors here don’t necessarily transfer directly across to Westminster polls. The challenges posed by high-turnout elections can be very different to those posed by low-turnout elections and just because some polls overstated Labour support in the European elections does not necessarily mean they are overstating Labour support for general elections. On the other hand, given the recent history of errors, it probably isn’t something we in the polling industry should be complacent about.


-->

Opinion polling on Brexit has not necessarily been the best. Highly politically contentious issues do tend to attract polling that is sub-optimal, and Brexit has followed that trend. I’ve seen several Brexit polls coming up with surprising findings based on agree/disagree statements – that is, questions asked in the form:

Do you agree with the following statement? I think Brexit is great
Agree
Disagree
Don’t know

This is a very common way of asking questions, but one that has a lot of problems. One of the basic rules in writing fair and balanced survey questions is that you should try to given equal prominence to both sides of the argument. Rather than ask “Do you support X?”, a survey should ask “Do you support or oppose X?”. In practice agree-disagree statements break that basic rule – they ask people whether they agree/disagree with one side of the argument, without mentioning the other side of the argument.

In some cases the opposite side of the argument is implicit. If the statement is “Theresa May is doing a good job”, then it is obvious to most respondents that the alternative view is that May is doing a bad job (or perhaps an average job). Even when it’s as obvious as this it still sometimes to make a difference – for whatever reason, decades of academic research into questionnaire design suggest people are more likely to agree with statements than to disagree with them, regardless of what the statement is (generally referred to as “acquiescence bias”).

There is a substantial body of academic evidence exploring this phenomenon (see, for example Schuman & Presser in the 1980s, or the recent work of Jon Krosnick) it tends to find around 10%-20% of people will agree with both a statement and its opposite, if it is asked in both directions. Various explanations have been put forward for this in academic studies – that it’s a result of personality type, or that it is satisficing (people just trying to get through a survey with minimal effort). The point is that it exists.

This is not just a theoretical issue that turns up in artificial academic experiments – they are plenty of real life examples in published polls. My favourite remains this ComRes poll for UKIP back in 2009. It asked if people agreed or disagreed with a number of statements including “Britain should remain a full member of the EU” and “Britain should leave the European Union but maintain close trading links”. 55% of people agreed that Britain should remain a full member of the EU. 55% of people also agreed that Britain should leave the EU. In other words, at least 10% of the same respondents agreed both that Britain should remain AND leave.

There is another good real life example in this poll. 42% agreed with a statement saying that “divorce should not be made too easy, so as to encourage couples to stay together”. However, 69% of the same sample also agreed that divorce should be “as quick and easy as possible”. At least 11% of the sample agreed both that divorce should be as easy as possible AND that it should not be too easy.

Examples like this of polls that asked both sides of the argument and produced contradictory findings are interesting quirks – but since they asked the statement in both directions they don’t mislead. However, it is easy to imagine how they would risk being misleading if they had asked the statement in only one direction. If that poll had only asked the pro-Brexit statement, then it would have looked as if a majority supported leaving. If the poll had only asked the anti-Leave statement, then it would have looked as if a majority supported staying. With agree-disagree statements, if you don’t ask both sides, you risk getting a very skewed picture.

In practice, I fear the problem is often far more serious in published political polls. The academic studies tend to use quite neutrally worded, simple, straightforward statements. In the sort of political polling for pressure groups and campaigning groups that you see in real life the statements are often far more forcefully worded, and are often statements that justify or promote an opinion – below are some examples I’ve seen asked as agree-disagree statements in polls:

“The Brexit process has gone on long enough so MPs should back the Prime Minister’s deal and get it done”
“The result of the 2016 Referendum should be respected and there should be no second referendum”
“The government must enforce the minimum wage so we have a level playing field and employers can’t squeeze out British workers by employing immigrants on the cheap”

I don’t pick these because they are particularly bad (I’ve seen much worse), only to illustrate the difference. These are statements that are making an active argument in favour of an opinion, where the argument in the opposite direction is not being made. They do not give a reason why MPs may not want to back the Prime Minister’s deal, why a second referendum might be a good idea, why enforcing the minimum wage might be bad. It is easy to imagine that respondents might find these statements convincing… but that they might have found the opposite opinion just as convincing if they’d been presented with that. I would expect questions like this to produce a much larger bias in the direction of the statement if asked as an agree-disagree statement.

With a few exceptions I normally try to avoid running agree-disagree statements, but we ran some specially to illustrate the problems, splitting the sample so that one group of respondents were asked if they agreed or disagreed with a statement, and a second group where asked if they agreed-disagreed with a contrasting statement. As expected, it produces varied results.

For simple questions, like whether Theresa May is doing a good job, the difference is small (people disagreed with the statement that “Theresa May is doing a good job by 57% to 15% and agreed with the statement that “Theresa May is doing a bad job” by 52% to 18%. Almost a mirror image. On some of the other questions, the differences were stark:

  • If you asked if people agree that “The NHS needs reform more than it needs extra money” then people agree by 43% to 23%. However, if you ask if people agree with the opposite statement, that “The NHS needs extra money more than it needs reform”, then people also agree, by 53% to 20%.
  • If you ask if people agree or disagree that “NHS services should be tailored to the needs of populations in local areas, even if this means that there are differences across the country as a whole” than people agree by 43% to 18%. However, if you ask if they agree or disagree with a statement putting the opposite opinion – “NHS services should be the same across the country” – then people agree by 88% to 2%!
  • By 67% to 12% people agree with the statement that “Brexit is the most important issue facing the government and should be its top priority”. However, by 44% to 26% they also agree with the statement “There are more important issues that the government should be dealing with than Brexit”

I could go on – there are more results here (summary, full tabs) – but I hope the point is made. Agree/disagree statements appear to produce a consistent bias in favour of the statement, and while this can be minor in questions asking simple statements of opinion, if the statements amount to political arguments the scale of the bias can be huge.

A common suggested solution to this issue is to make sure that the statements in a survey are balanced, with an equal amount of statements in each direction. So, for example, if you were doing a survey about attitudes towards higher taxes, rather than asking people if they agreed or disagreed with ten statements in favour of high taxes, you’d ask if people agreed or disagreed with five statements in favour of higher taxes and five statements in favour of lower taxes.

This is certainly an improvement, but is still less than ideal. First it can produce contradictory results like the examples above. Secondly, in practice it can often result in some rather artificial and clunky sounding questions and double-negatives. Finally, in practice it is often difficult to make sure statements really are balanced (too often I have seen surveys that attempt a balanced statement grid, but where the statements in one direction are hard-hitting and compelling, and in the other direction are deliberately soft-balled or unappetising).

The better solution is not to ask them as agree-disagree statements at all. Change them into questions with specific answers – instead of asking if people agree that “Theresa May is going a good job”, ask if May is doing a good or bad job. Instead of asking if people agree that “The NHS needs reform more than it needs more money”, ask what people think the NHS needs more – reform or more money? Questions like the examples I gave above can easily be made better by pairing the contrasting statements, and asking which better reflects respondents views:

  • Asked to pick between the two statements on NHS reform or funding, 41% of people think it needs reform more, 43% think it needs extra money more.
  • Asked to pick between the two statements on NHS services, 36% think they should be tailored to local areas, 52% would prefer them to be the same across the whole country.
  • Asked to pick between the two statements on the importance of Brexit, 58% think it is the most important issue facing the government, 27% think there are more important issues the government should be dealing with instead.

So what does this mean when it comes to interpreting real polls?

The sad truth is that, despite the known problems with agree-disagree statements, they are far from uncommon. They are quick to ask, require almost no effort at all to script and are very easy for clients after a quick headline to interpret. And I fear there are some clients to whom the problems with bias are an advantage, not a obstacle; you often see them in polls commissioned by campaigning groups and pressure groups with a clear interest in getting a particular result.

Whenever judging a poll (and this goes to observers reading them, and journalists choosing whether to report them) my advice has always been to go to polling companies websites and look at the data tables – look at the actual numbers and the actual question wording. If the questions behind the headlines have been asked using agree-disagree statements, you should be sceptical. It’s a structure that does have an inherent bias, and does result in more people agreeing than if the question had been asked a different way.

Consider how the results may have been very different if the statement had been asked in the opposite direction. If it’s a good poll, you shouldn’t have to imagine that – the company should have made the effort to balance the poll by asking some of the statements in the opposite direction. If they haven’t made that effort, well, to me that rings some alarm bells.

If you get a poll that’s largely made up of agree-disagree statements, that are all worded in the direction that the client wants the respondent to answer rather than some in each direction, that use emotive and persuasive phrasing rather than bland and neutral wording? You would be right to be cautious.


There have been several new polls with voting intention figures since the weekend, though all so far have been conducted before the government’s defeat on their Brexit plan.

ComRes/Express (14th-15th) – CON 37%(nc), LAB 39%(nc), LDEM 8%(-1), UKIP 7%(+1)
YouGov/Times (13th-14th)- CON 39%(-2), LAB 34%(-1), LDEM 11%(nc), UKIP 6%(+2)
Kantar (10th-14th) – CON 35%(-3), LAB 38%(nc), LDEM 9%(nc), UKIP 6%(+2)

Looking across the polls as a whole Conservative support appears to be dropping a little, though polls are still ultimately showing Labour and Conservative very close together in terms of voting intention. As ever there are some differences between companies – YouGov are still showing a small but consistent Tory lead, the most recent polls from BMG, Opinium and MORI had a tie (though Opinium and MORI haven’t released any 2019 polls yet), Kantar, ComRes and Suration all showed a small Labour lead in their most last polls.

Several people have asked me about the reasons for the difference between polling companies figures. There isn’t an easy answer – there rarely is. The reality is that all polling companies want to be right and want to be accurate, so if there were easy explanations for the differences and it was easy to know what the right choices were, they would all rapidly come into line!

There are two real elements that are responsible for house effects between pollsters. The first is the things they do to the voting intention data after it is collected and weighted – primarily that is how do they account for turnout (to what extent do they weight down or filter out people who are unlikely to vote), and what to do they with people who say they don’t know how they’ll vote (do they ignore them, or use squeeze questions or inference to try and estimate how they might end up voting). The good thing about these sort of differences is that they are easily quantifiable – you can look up the polling tables, compare the figures with turnout weighting and without, and see exactly the impact they have.

At the time of the 2017 election these adjustments were responsible for a lot of the difference between polling companies. Some polls were using turnout models that really transformed their topline figures. However, those sort of models also largely turned out to be wrong in 2017, so polling companies are now using much lighter touch turnout models, and little in the way of reallocating don’t knows. There are a few unusual cases (for example, I think ComRes still reallocate don’t knows, which helps Labour at present, but most companies do not. BMG no longer do any weighting or filtering by likelihood to vote, an adjustment which for other companies tends to reduce Labour support by a point or two). These small differences are not, by themselves, enough to explain the differences between polls.

The other big differences between polls are their samples and the weights and quotas they use to make them representative. It is far, far more difficult to quantify the impact of these differences (indeed, without access to raw samples it’s pretty much impossible). Under BPC rules polling companies are supposed to be transparent about what they weight their samples by and to what targets, so we can tell what the differences are, but we can’t with any confidence tell what the impact is.

I believe all the polling companies weight by age, gender and region. Every company except for Ipsos MORI also votes by how people voted at the last election. After that polling companies differ – most vote by EU Ref vote, some companies weight by education (YouGov, Kantar, Survation), some by social class (YouGov, ComRes), income (BMG, Survation), working status (Kantar), level of interest in politics (YouGov), newspaper readership (Ipsos MORI) and so on.

Even if polling companies weight by the same variables, there can be differences. For example, while almost everyone weights by how people voted at the last election, there are differences in the proportion of non-voters they weight to. It makes a difference whether targets are interlocked or not. Companies may use different bands for things like age, education or income weighting. On top of all this, there are questions about when the weighting data is collected, for things like past general election vote and past referendum vote there is a well-known phenomenon of “false recall”, where people do not accurately report how they voted in an election a few years back. Hence weighting by past vote data collected at the time of the election when it was fresh in people’s minds can be very different to weighting by past vote data collected now, at the time of the survey when people may be less accurate.

Given there isn’t presently a huge impact from different approaches to turnout or don’t knows, the difference between polling companies is likely to be down some of these factors which are – fairly evidently – extremely difficult to quantify. All you can really conclude is that the difference is probably down to the different sampling and weighting of the different companies, and that, short of a general election, there is no easy way for either observers (nor pollsters themselves!) to be sure what the right answer is. All I would advise is to avoid the temptation of (a) assuming that the polls you want to be true are correct… that’s just wishful thinking, or (b) assuming that the majority are right. There are plenty of instances (ICM in 1997, or Survation and the YouGov MRP model in 2017), when the odd one out turned out to be the one that was right.


There is plenty of new polling in today’s papers, including two polls proporting to show that large numbers of people would vote for new political parties. One by BMG for the Huffington Post, claiming 58% of people would consider backing a new party at the next election, and a ComRes poll for BrexitExpress, claiming 53% of people in a selection of Tory constituencies would consider voting for a single issue party campaigning to “conclude Brexit as quickly and as fully as possible”. There have been various other polls in recent weeks asking similar questions about how popular new parties would be.

These sound like large figures, but you should take them all with a huge pinch of salt – the reality is that quantifying the prospects of a new political party before it exists is an almost impossible task. Certainly it is not something that can be done with a single question.

First let’s look at the question itself. Polls tend to take two approaches to this question, both of which have flaws. The first is to say “Imagine there was a new party that stood for x, y and z – how likely would you be to consider voting for it?”. The problem with that as a question is that “consider” is a pretty low bar. Does thinking about something for a fleeting second before dismissing it count as “considering”?

An alternative approach is to say “Imagine there was a new party that stood for x, y and z. How would you vote if they stood at the next election?” and then prompt them alongside the usual political parties. This does at least force a choice, and sets the new hypothetical party alongside the alternative established parties, prompting to people to consider whether they would actually vote for their usual party after all.

There are, however, rather deeper problems with the whole concept. The first is the lack of information about the party – it asks people whether they would vote for a rather generic new party (a new anti-Brexit party, a new pro-Brexit party, a new pro-NHS party, or whatnot). That misses out an awful lot of the things that determine people’s vote. Who is the leader of the party? Are they any good? Do the party appear competent and capable? Do they share my values on other important issues? Can I see other people around me supporting them? Are they backed by voices I trust?

Perhaps most of all, it misses out the whole element of whether the party is seen as a serious, proper contender, or a wasted vote. It ignores the fact that for most new parties, a major hurdle is whether voters are even aware of you, have ever heard of you, or think you are a viable challenger. That is the almost insoluble problem with questions like this: by asking a question that highlights the existance of the new party and implies to respondents that it is a party that is worthy of serious consideration a pollster has ignored the biggest and most serious problem most new parties face.

That’s the theory of why they should be treated with some caution. What about their actual record? What about when people polled about hypothetical parties that later became real parties that stood in real elections? Well, there aren’t that many cases of large nationwide parties launching, though there are more instances of constituency level polls asking similar questions. Here are the examples I can find:

  • At the 1999 European elections two former Conservative MEPs set up a “Pro-Euro Conservative party”. Before that a hypothetical MORI poll asked how people would vote in the European elections “if breakaway Conservatives formed their own political party supporting entry to the single European currency”. 14% of those certain or very likely to vote said they would vote for the new breakaway pro-Euro Conservatives. In reality, the pro-Euro Conservative party won 1.3%.
  • Back in 2012 when the National Health Action party was launched Lord Ashcroft did a GB poll asking how people would vote if “Some doctors opposed to the coalition government’s policies on the NHS […] put up candidates at the next election on a non-party, independent ticket of defending the NHS”. It found 18% of people saying they’d vote for them. In reality they only stood 12 candidates at the 2015 election, getting 0.1% of the national vote and an average of 3% in the seats they contested.
  • Just before the 2017 election Survation did a poll in Kensington for the Stop Brexit Alliance – asked how they might vote if there was a new “Stop Brexit Alliance” candidate in the seat, 28% of those giving a vote said they’d back them. In the event there were two independent stop Brexit candidates in Kensington – Peter Marshall and James Torrance. They got 1.3% between them (my understanding, by the way, is that the potential pro-Europe candidates who did the poll are not the same ones who actually stood).
  • Survation did a similar poll in Battersea, asking how people would vote if a hypothetical “Independent Stop Brexit” candidate stood. That suggested he would get 17%. In reality that independent stop Brexit candidate, Chris Coghlan, got only 2%.
  • Advance Together were a new political party that stood in the local elections in Kensington and Chelsea earlier this year. In an ICM poll of Kensington and Chelsea conducted in late 2017 64% of people said they would consider voting for such a new party. In reality Advance Together got 5% of the boroughwide vote in Kensington and Chelsea, an average of 7% in the wards where they stood.

In all of these examples the new party has ended up getting far, far, far less support than hypothetical polls suggested they might. It doesn’t follow that this would always be the case, and that a new party can’t succeed. I suspect a new party that was backed by a substantial number of existing MPs and had a well-enough known leader to be taken seriously as a political force could do rather well. My point is more that hypothetical polls really aren’t a particularly good way of judging it.