While the gap between online and telephone polls on the EU referendum has narrowed of late, it is still there, and Populus have put out an interesting paper looking at possible explanations and written by James Kanagasooriam of Populus and Matt Singh of Number Cruncher Politics. The full paper is here.

Matt and James essentially suggest three broad reasons. The first thing is don’t knows. Most telephone polls don’t prompt people with the option of saying don’t know, but respondents are free to volunteer it. In contrast in online polls people can only pick from the options that are presented on the screen, so don’t know has to be presented up front as an option (Personally, I have a suspicion that there’s a mode effect as well as a prompting effect on don’t knows. When there is a human interviewer people may feel a certain social pressure to give an answer – saying don’t know feels somehow unhelpful).

Populus tested this in two parallel surveys, one online, one phone, both split. The phone survey was split between prompting people just with the options of Remain or Leave, or explicitly including don’t know as an option in the prompt. The online survey had a split offering don’t know as an option, and a split with the don’t know option hidden away in smaller font at the bottom of the page (a neat idea to try and simulate not explicitly prompting for an option in an online survey).

  • The phone test had a Remain lead of 11 points without a don’t know option (the way phone polls normally ask), but with an explicit don’t know it would have shown only a 3 point Remain lead. Prompting for don’t knows made a difference of eight points in the lead.
  • The online survey had a Leave lead of six points with a don’t know prompt (the way they normally ask), but with the don’t know option hidden down the page it had only a one point Leave lead. Making the don’t know prompt less prominent made a difference of six points in the lead.

The impact here is actually quite chunky, accounting for a fair amount of the difference. Comparing recent phone and online polls the gap is about seven or so points, so if you looked just at the phone experiment here the difference in don’t knows could in theory account for the whole lot! I don’t think that is the case though: things are rarely so simple, earlier this year there was a much bigger gap and I suspect there are probably also some issues to do with sampling make up and interviewer effect in the actual answers. In the Populus paper they assume it makes up about a third of a gap of fifteen points between phone and online, obviously that total gap is smaller now.

The second thing Populus looked at was attitudinal differences between online and phone samples. The examples looked at here are attitudes towards gender equality, racial equality and national identity. Essentially, people give answers that are more socially liberal in telephone polls than they did in online polls. This is not a new finding – plenty of papers in the past have found these sort of differences between telephone and online polling, but because attitudinal questions are not directly tested in general elections these are never compared against reality and it is impossible to be certain which are “right”. Neither can we really be confident how much of the difference is down to different types of people being reached by the two approaches, and interviewer effects (are people more comfortable admitting views that may be seen as racist or sexist to a computer screen than to a human interviewer?). It’s probably a mixture of both. What’s important is that how socially liberal people were on these scales correlated with how pro-or-anti EU they were, so to whatever extent there is a difference in sample make-up rather than interviewer effect, it explains another couple of points difference between EU referendum voting intention in telephone and online polls. The questions that Populus asked had also been used in the face-to-face BES survey: the answers there were in the middle – more socially liberal than online polls, less socially liberal that phone polls. Of course, if there are interviewer effects at play here, face-to-face polling also has a human interviewer.

Populus think these two factors explain most of the difference, but are left with a gap of about 3 points that they can’t readily explain. They float the idea that this could be because online samples have more partisan people who vote down the line (so, for example, online samples have fewer of those odd “UKIP for Remain” voters), when in reality people are more often rather contradictory and random. It’s a interesting possibility, and chimes with my own views about polls containing people who are too politically aware, too partisan. The impact of YouGov adopting sampling and weighting by attention paid to politics last month was mostly to increase don’t knows on questions, but when we were doing testing it before rollout it did increase the position of remain relative to leave on the EU question, normally by two or three points, so that would chime with Populus’s theory.

According to Populus, therefore, the gap comes down partially to don’t know, partially towards the different attitudinal make-up and a final chunk because they think online samples are more partisan. Their estimate is that the reality will be somewhere inbetween the results being shown by online and telephone, a little closer towards telephone. We shall see.

(A footnote for the just the really geeky among you who have paid close attention to the BPC inquiry and the BES team’s posts on the polling error, but is probably too technical for most readers. When comparing the questions on race and gender Populus also broke down the answers in the BES face-to-face survey by how many contacts it took to interview them. This is something the BES team and the BPC inquiry team also did when investigating the polling error last May. The inquiries looking at the election polls found that if you took just those people the BES managed to interview on their first or second go the make up of the sample was similar to that from phone polls, and was too Labour, but people who were trickier to reach were more Conservative. Hence they took “easy for face-to-face interviewers to reach” as a sort of proxy for “people likely to be included in a poll”. In this study Populus did the same for the social liberal questions and it didn’t work the same way: phone polls were much more liberal than the BES f2f poll, but the easy to reach people in the BES f2f poll were the most conservative and the hard to reach the most liberal, so “easy to reach f2f” didn’t resemble the telephone sample at all. Populus theorise that this is a mobile sampling issue, but I think it raises some deeper questions about the assumptions we’ve made about what difficulty of contacting in the BES f2f sample can teach us about other samples. I’ve never seen any logical justification as to why people who it takes multiple attempts to reach face-to-face will necessarily be the same group that it’s hard to reach online – they could easily be two completely different groups. Perhaps “takes multiple tries to reach face-to-face” is not a suitable proxy for the sort of people phone polls can’t reach either…)


69 Responses to “Populus and Matt Singh on the EU polls”

1 2
  1. Yes, but Allan, if you have to check your IP to know where you are, and weren’t aware you were in Baghdad, you might have more pressing concerns.

    (Especially if, like me, you are also behind the curve on the golf club thing….)

  2. “Say what you like about the Scot Indy manifesto (actually don’t!) but at least there was some basis for deciding how to vote.”

    I don’t really agree with this. There was a mountain of scary fiction on one side and an equally unimpressive mountain of happy fiction on the other, with one or two nuggets of compelling analysis from more independent sources giving some support to either option. The EU referendum is panning out in a very similar vein so far I think, although I would agree that the change option is less well defined than in Scotland. However, in the Indy ref, the quality of this definition was (in my opinion) so poor as to be worthless, as the current fiscal situation of iScotland would attest to, had there been a yes vote.

    In terms of the campaigns, my view is that Leave have been far more effective, although I won’t say they have been impressive. Two particular things have shone through for me. Firstly, they have assembled a reasonably impressive group of business leaders to back Brexit. This is nowhere near the scale of the remain campaign, but given the weight of expectation that business would almost universally back remain, having former heads one of the world’s largest banks and other high profile business leaders back Brexit is a major coup.

    The second area where I think leave has had success is in partially neutralising the risk message. In the first period of the campaign, it has been notable how rapidly leave has mounted a counter operation when there is a major speech from remain about the risks of Brexit. As has been discussed above, this is all conjecture, on both sides, but the chief impression from the Indyref was that the Yes campaign weren’t willing or able to shift the concept of risk onto the No option. This time, possibly after having studied the Indyref dynamics, Leave appear determined to challenge the risk concept quickly and loudly, whenever it appears.

    This means we have had David Owen warning that staying in the EU is a risk to UK defence and security because the EU is incompetent and incapable of organising security matters, IDS and others using the recent terrorist attacks to state that being in the EU means a risk that we cannot secure our borders, business leaders stating that remaining in the EU risks excess red tape and loss of competitiveness, etc etc.

    I’m not here to argue whether or not these attack lines are sensible or plausible, but I do get the sense that at the very least, they have clouded the simple interpretation that Remain would like to project, namely that Brexit is a major gamble with high risk levels.

    On the above basis, I would caution against the somewhat simplistic analysis that both campaigns have been substandard – to date, the Leave campaign has exceeded my expectation and I think they have aquitted themselves rather well, most of the time, and I think it can be argued that this is reflected in the polling situation, confused though that is.

  3. I agree entirely with Alec’s above comments.

  4. @crossbat11
    If the Remainers really do have a 5-10% lead at the moment, then the differential turnout would have to favour Leave significantly, but there seems to be some signs that the Remain lead is slowly reducing and if things get very tight come June, the passion and determination to vote of the Leavers may just swing it.

    I was thinking along similar lines, and that if the same demographic trend as voted in the general election was replicated in the referendum (with the over 65’s more likely to vote) this would favour Leave. If the same GE/Ref turnout ratio as 1974/75 is replicated then turnout in June should be approx. 60%, and as I am assuming the Leave voters are more motivated this would tilt the result in their favour. I think the Ipsos poll last week did an adjustment along the lines of the demographic split for last year’s GE and it increased the remain lead by 6/7%.

    I also had a chat in the kitchen at work with a colleague this morning (neither of us have allotments btw), and came to conclusion that the closer the polls the assumed more ‘soft’ Remain voters are likely to vote.

    Either way I think it will be very close – perhaps football fans who want to see more home grown talent will swing it for leave.

  5. @Alun009

    If you read in context I was saying that the impact of 250,000 EU citizens, filtered by their overall benefit to the UK, leaving our shores would have a pretty small impact, not that leaving the EU in itself would have. After all, more than that number arrive every single year currently, and Remainers are pretty sanguine that This Doesn’t Matter.

    Don’t get me wrong, I’m not a Brexiteer. I am fairly convinced by the argument that exit would be a net detriment to the economic wellbeing of the UK. John was talking about whether any thought had gone in to the situation of EU citizens currently residing in the UK and the impact on them (i.e. the Leave campaigns don’t seem to have thought that issue through) and I was expressing the view that it’s probably not as complex and difficult an issue as all that.

    I agree about the quid pro quo, but I think it would be in more literal terms than you suggest. I don’t think EU countries would act punitively, or quantitatively (i.e. you b****ds have deported 10,000 of the 1m Poles in the UK, so we Poles are going to deport all of the 4,500 Britons living in Poland). I think it would a qualitative quid pro quo. Whatever rules the UK set for allowing residency would probably be approximated by the EU countries. So UK citizens living abroad who hadn’t been there long, who were on benefits or in unspecialized / easy to recruit jobs, or who had proved themselves undesirable through criminality, anti-social behavior or extremism, might be required to leave.

    There would probably be quite a few Britons turfed out of Spain, but as a police officer I actually think that would be a good thing. Globalization is a thing in the criminal world too, and UK criminals in the Costa del Crime still impact on the UK, we just find it difficult to get to them. Better that they return to live in Chigwell where we can target them without Europol’s help.

    In general, I think EU countries are unlikely to turf out anyone whose presence represents an economic benefit to them. So UK pensioners, for example, drawing their income from UK pension funds and spending it in the local economy might not be targeted. Particularly in places where there probably isn’t a demand for the housing those pensioners currently occupy. I expect they’d lose out on current reciprocal health and welfare arrangements (although not necessarily – its perfectly possible new agreements could be come to) and have to fund their own health insurance. But most of them will have lived abroad long enough to qualify for residency or citizenship in one form or another so they’d have that option.

    Certainly it’s an area that needs some thought, but it’s yet another item on the “We Can’t Leave Because” list that I think is being exaggerated for effect.

  6. “I also had a chat in the kitchen at work with a colleague this morning (neither of us have allotments btw…”

    ———

    Allotments are no longer sufficient in the new, more bourgeois UKPR. You gotta have a golf club too now…

  7. Allan Christie
    “The EU at present is aesthetically pleasing as watching Diane Abbott v Eric Pickles in a mud bath competition.”

    Love it! What an image!

  8. @Carfrew

    Allotments are no longer sufficient in the new, more bourgeois UKPR. You gotta have a golf club too now…

    And an Italian wife so it seems…

  9. Not as horrible as Ken Clarke vs Nicholas Soames.

  10. ALLAN “MUDWRESTLING FAN” CHRISTIE
    “The EU is nothing more than the political wing of America”

    This is one of those curious conspiracy theories that does the rounds in the more lunatic fringe of the Brexiteers. And to date I’ve never quite been able to get to the bottom of why on earth anyone really thinks this.

    I would ask for more information on this, but in all honesty I don’t think you actually believe what you are saying. In any case, my previous attempts to seek an explanation for this odd claim have met with silence. Is it an in joke from the Leave lot, or are you actually serious?

  11. The Tata story looks like exerting an influence on both the EU referendum and Tory party fortunes.

    On the EU issue, there are attempts to seize upon this as an example of the EU being incapable of protecting vital industries, with the states aid rules heavily in the firing line. This is an area I know well, and there are many elements to this that I suspect many UK voters would find unacceptable, were they to understand how this works.

    Basically, any government aid up to a fixed ‘de minimis’ level is illegal, if the receipt of that aid is likely to affect trade between member states. There are various standard exemptions to enable things like health service contracts etc, and some industrial sectors have their own rules, but in a nutshell, any organisation, including charities, that engage in any activity that could be delivered by the market are affected.

    The European Commission (unelected) has complete authority in determining such issues, backed up by the ECJ. Their interpretations are extremely conservative and very restrictive. For example, a charity running a local youth club that receives grants for doing so will need to assume that the grant needs to be counted as states aid, meaning there is a restriction of €200,000 in any three year period. There are cases of charities closing down as they cannot secure further funding due to the rules, whereas in reality, the notion that they are affecting trade between member states is, frankly, risible. However, the fact that a youth club could theoretically be supplied by the free market, even if it isn’t, is sufficient to determine grants as states aid.

    So on the face of it the Brexiteers have a strong case, but the details of the Tata case are making extremely uncomfortable reading for Tories, and George Osborne in particular. Whereas the US have applied penalty tarrifs on Chinese steel imports of 240%, the EU has only managed around 10%. Again, an apparent godsend for Brexit, until Brussels angrily revealed that it was fierce UK lobbying that prevented stiffer penalties, presumably in order to maintain good Sino/British relationships and keep Hinkley C on track.

    I think this issue could get very serious for Tories in general, but is another serious blow to Osborne’s leadership ambitions, especially if EDF pull out or delay Hinkley.

    In terms of the EU referendum, this is one of many classic examples of people blaming the collective institution while ignoring the failings at home. We had this aplenty in the Indyref, and are having it again now, but the critical issue is who gets their message across most effectively. My guess currently is that Remain will suffer, and the government will be embarassed.

  12. Alec

    There is a further problem for Cameron if he clings on after the referendum: now two of his allies in cabinet (Hunt and Javid) would be on the waiting list to get canned.

  13. “Again, an apparent godsend for Brexit, until Brussels angrily revealed that it was fierce UK lobbying that prevented stiffer penalties, presumably in order to maintain good Sino/British relationships and keep Hinkley C on track.”

    ———

    Yep, generally the attempt to keep certain ideologies in play tends to wind up as a game of whackamole…

  14. CARFREW
    Yes, but Allan, if you have to check your IP to know where you are, and weren’t aware you were in Baghdad, you might have more pressing concerns.
    (Especially if, like me, you are also behind the curve on the golf club thing….)
    ________

    According to your IP you’re in TOH’s allotment pinching his vegetables. ;-)
    __

    PETE B

    :-)

  15. ALUN009

    It’s just the way I see the situation in Europe. America has its hands in everything. They interfered with the Scottish indy ref, soon the EU vote and are overseeing NATO expansionism towards Russia.

    Its no conspiracy theory that the US would see a BREXIT as a weakening of their influence over the political direction of Europe.

  16. AC:
    “Its no conspiracy theory that the US would see a BREXIT as a weakening of their influence over the political direction of Europe.”

    Sure, but it’s a mighty leap to infer the EU is a “political wing” of the USA. It’s also widely held (rightly or wrongly) in the UK that the election of Donald Trump would be a Bad Thing, but that doesn’t mean Hillary is a UK agent.

  17. @AC

    “According to your IP you’re in TOH’s allotment pinching his vegetables. ;-)”

    ———-

    Oi!! Don’t be blaming your shady practices on me!!

    Actually, it can’t be you, you’re in Baghdad. So who is it then?

    (Unless…

    …ToH’s allotment is in Baghdad…)

  18. @Alec
    “For example, a charity running a local youth club that receives grants for doing so will need to assume that the grant needs to be counted as states aid, meaning there is a restriction of €200,000 in any three year period. There are cases of charities closing down as they cannot secure further funding due to the rules, whereas in reality, the notion that they are affecting trade between member states is, frankly, risible. However, the fact that a youth club could theoretically be supplied by the free market, even if it isn’t, is sufficient to determine grants as states aid.”

    Sorry Alec but my experience as the Director of a major charitable foundation, an advisor to local community groups and the architect of two national major community-based grant programmes tells me that your statement is utterly wrong. State aid rules may affect some social enterprises but even then very rarely.

  19. I am wondering how accurate the figures are about the numbers of EU citizens living here . Posts here have referred to 2 million or so. I have seen an alleged home office statistic as saying 3.7 million. But do they really know? How are the statistics collated? Are they in any way reliable, or just an estimate? My wife (who is Dutch) has been here so long I think it is quite possible the Home Office don’t know about her (and many others). As with almost everything to do with this referendum I’d be very grateful to hear from anybody who knows any facts…..

1 2