While the gap between online and telephone polls on the EU referendum has narrowed of late, it is still there, and Populus have put out an interesting paper looking at possible explanations and written by James Kanagasooriam of Populus and Matt Singh of Number Cruncher Politics. The full paper is here.

Matt and James essentially suggest three broad reasons. The first thing is don’t knows. Most telephone polls don’t prompt people with the option of saying don’t know, but respondents are free to volunteer it. In contrast in online polls people can only pick from the options that are presented on the screen, so don’t know has to be presented up front as an option (Personally, I have a suspicion that there’s a mode effect as well as a prompting effect on don’t knows. When there is a human interviewer people may feel a certain social pressure to give an answer – saying don’t know feels somehow unhelpful).

Populus tested this in two parallel surveys, one online, one phone, both split. The phone survey was split between prompting people just with the options of Remain or Leave, or explicitly including don’t know as an option in the prompt. The online survey had a split offering don’t know as an option, and a split with the don’t know option hidden away in smaller font at the bottom of the page (a neat idea to try and simulate not explicitly prompting for an option in an online survey).

  • The phone test had a Remain lead of 11 points without a don’t know option (the way phone polls normally ask), but with an explicit don’t know it would have shown only a 3 point Remain lead. Prompting for don’t knows made a difference of eight points in the lead.
  • The online survey had a Leave lead of six points with a don’t know prompt (the way they normally ask), but with the don’t know option hidden down the page it had only a one point Leave lead. Making the don’t know prompt less prominent made a difference of six points in the lead.

The impact here is actually quite chunky, accounting for a fair amount of the difference. Comparing recent phone and online polls the gap is about seven or so points, so if you looked just at the phone experiment here the difference in don’t knows could in theory account for the whole lot! I don’t think that is the case though: things are rarely so simple, earlier this year there was a much bigger gap and I suspect there are probably also some issues to do with sampling make up and interviewer effect in the actual answers. In the Populus paper they assume it makes up about a third of a gap of fifteen points between phone and online, obviously that total gap is smaller now.

The second thing Populus looked at was attitudinal differences between online and phone samples. The examples looked at here are attitudes towards gender equality, racial equality and national identity. Essentially, people give answers that are more socially liberal in telephone polls than they did in online polls. This is not a new finding – plenty of papers in the past have found these sort of differences between telephone and online polling, but because attitudinal questions are not directly tested in general elections these are never compared against reality and it is impossible to be certain which are “right”. Neither can we really be confident how much of the difference is down to different types of people being reached by the two approaches, and interviewer effects (are people more comfortable admitting views that may be seen as racist or sexist to a computer screen than to a human interviewer?). It’s probably a mixture of both. What’s important is that how socially liberal people were on these scales correlated with how pro-or-anti EU they were, so to whatever extent there is a difference in sample make-up rather than interviewer effect, it explains another couple of points difference between EU referendum voting intention in telephone and online polls. The questions that Populus asked had also been used in the face-to-face BES survey: the answers there were in the middle – more socially liberal than online polls, less socially liberal that phone polls. Of course, if there are interviewer effects at play here, face-to-face polling also has a human interviewer.

Populus think these two factors explain most of the difference, but are left with a gap of about 3 points that they can’t readily explain. They float the idea that this could be because online samples have more partisan people who vote down the line (so, for example, online samples have fewer of those odd “UKIP for Remain” voters), when in reality people are more often rather contradictory and random. It’s a interesting possibility, and chimes with my own views about polls containing people who are too politically aware, too partisan. The impact of YouGov adopting sampling and weighting by attention paid to politics last month was mostly to increase don’t knows on questions, but when we were doing testing it before rollout it did increase the position of remain relative to leave on the EU question, normally by two or three points, so that would chime with Populus’s theory.

According to Populus, therefore, the gap comes down partially to don’t know, partially towards the different attitudinal make-up and a final chunk because they think online samples are more partisan. Their estimate is that the reality will be somewhere inbetween the results being shown by online and telephone, a little closer towards telephone. We shall see.

(A footnote for the just the really geeky among you who have paid close attention to the BPC inquiry and the BES team’s posts on the polling error, but is probably too technical for most readers. When comparing the questions on race and gender Populus also broke down the answers in the BES face-to-face survey by how many contacts it took to interview them. This is something the BES team and the BPC inquiry team also did when investigating the polling error last May. The inquiries looking at the election polls found that if you took just those people the BES managed to interview on their first or second go the make up of the sample was similar to that from phone polls, and was too Labour, but people who were trickier to reach were more Conservative. Hence they took “easy for face-to-face interviewers to reach” as a sort of proxy for “people likely to be included in a poll”. In this study Populus did the same for the social liberal questions and it didn’t work the same way: phone polls were much more liberal than the BES f2f poll, but the easy to reach people in the BES f2f poll were the most conservative and the hard to reach the most liberal, so “easy to reach f2f” didn’t resemble the telephone sample at all. Populus theorise that this is a mobile sampling issue, but I think it raises some deeper questions about the assumptions we’ve made about what difficulty of contacting in the BES f2f sample can teach us about other samples. I’ve never seen any logical justification as to why people who it takes multiple attempts to reach face-to-face will necessarily be the same group that it’s hard to reach online – they could easily be two completely different groups. Perhaps “takes multiple tries to reach face-to-face” is not a suitable proxy for the sort of people phone polls can’t reach either…)


Two days to go. The huge rush in final polls won’t be until tomorrow, but there are still a fair number of polls out today. I don’t think any of them are proper final calls yet – most companies will produce their eve-of-election numbers tomorrow or on election day itself (it’s illegal to publish an exit poll before polls close, but it’s fine to publish a poll conducted on the eve of election on the morning of polling day). All of today’s look as if they are penultimate polls…

  • Populus today had topline figures of CON 34%, LAB 34%, LDEM 10%, UKIP 13%, GRN 5%
    (tabs). According to the FT we still have another Populus poll to come before the election.
  • Lord Ashcroft’s weekly poll had topline figures of CON 32%, LAB 30%, LEM 11%, UKIP 12%, GRN 7%, coming into a much closer race than the rather incongruous six point Tory lead last week. Tabs are here). Ashcroft will have a final call poll on Thursday morning, so one more to come from him.
  • Survation for the Mirror have topline figures of CON 33%, LAB 34%, LDEM 9%, UKIP 16%, GRN 4% (tabs). Survation have said they’ve got new figures everyday before the election, so we’ll be getting some new figures from them tomorrow too.

UPDATE: We now have three more polls out:

  • A ComRes telephone poll for the Mail and ITV has topline figures of CON 35%, LAB 32%, LDEM 9%, UKIP 14%, GRN 4%. Again, this is their penultimate poll, with one more to come (presumably tomorrow). Tabs are here.
  • There is also a second BMG poll for May 2015 (which in their case DOES appear to be their final call poll) topline figures are CON 34%, LAB 34%, LDEM 10%, UKIP 12%, GRN 4%. Full details here.
  • Finally YouGov’s penultimate poll for the Sun has topline figures of CON 34%, LAB 34%, LD 9%, UKIP 12%, GRN 5% – still neck and neck. Their final call will follow tomorrow night.

-->

Ten days to go to the election and we’ve had interesting day of polls – four new GB polls, some new constituency polling and a new Scottish poll. The four GB polls today are the weekly Ashcroft and ICM telephone polls, the twice weekly Populus poll and, to come later on tonight, the daily YouGov poll for the Sun:

  • Populus had figures of CON 33%, LAB 36%, LDEM 8%, UKIP 14%, GRN 5% (tabs). They continue to produce figures that are more favourable to Labour than many of the other pollsters – you have to go all the way back to August to find a Populus poll with a Conservative lead.
  • In contrast ICM have tended to produce some of the better polls for the Conservatives – their last four polls showed Conservative leads and today’s has topline figures of CON 35%, LAB 32%, LDEM 9%, UKIP 13%, GRN 5% (tabs)
  • Lord Ashcroft’s weekly poll had topline figures of CON 36%, LAB 30%, LDEM 9%, KIP 11%, GRN 7% (tabs) – this is obviously a particularly good poll for the Conservative party, but all the usual caveats apply. No other poll is showing such positive figures for them.

Lord Ashcroft also released four new constituency polls, this time covering four UKIP target seats (or at least, four where he had previously found them doing well, I’m not sure whether Cannock Chase was ever a seat they were targetting – certainly Ashcroft’s poll found respondents reporting a lower level of UKIP activity there). When Ashcroft previously polled these seats he found UKIP in an extremely close second place, this time he found them falling back and seemingly out of serious contention in three of them:

  • Cannock Chase is a seat the Conservatives won on a vast swing last time, but where the new MP has stood down after various gaffes. In October 2014 Ashcroft found UKIP two points behind Labour, 30% to Labour’s 32%. The latest poll still shows Labour ahead, but UKIP now trail in third place on 21%.
  • Great Grimsby is widely regarded as the best opportunity for a UKIP gain from Labour at this election – a Lincolnshire fishing port where the veteran MP Austin Mitchell is standing down. In December Ashcroft found UKIP just a point behind Labour, but they’ve fallen back considerably since then and today’s poll has them 17 points behind Labour
  • Great Yarmouth fits the pattern for a typical UKIP target seat, a seaside town and marginal seat out on England’s east coast. Last July Ashcroft found a tight three way fight – Con 33%, UKIP 31%, Lab 28%. Today’s poll has UKIP falling back to 24%, but Conservative and Labour still in a close battle – Con 36%, Lab 34%
  • Castle Point is the only one of the three where UKIP still seems to be in the race. It’s an unusual seat – the former Conservative MP Bob Spink sort of defected to UKIP in 2008 and contested the seat as an Independent in 2010, coming second with 27%. In February Ashcroft found UKIP just one point behind the Tories, in today’s poll the Conservatives have widened their lead to 5 points.

Finally a new TNS poll of Scotland shows the SNP moving into an even stronger lead. Their topline figures with changes from their last Scottish poll are CON 13%(nc), LAB 22%(-2), LDEM 6%(nc), SNP 54%(+2), UKIP 2%(+1), GRN 2%(-1). Tabs are here.


It’s Monday, so as usual we are spoilt for polls, with new figures from Populus, Ashcroft and ICM (who are now on a weekly rota until the general election), with YouGov to come later on tonight.

Populus have voting intentions of CON 32%, LAB 34%, LDEM 9%, UKIP 15%, GRN 4%. Tabs are here).

ICM have topline figures of CON 34%, LAB 32%, LDEM 10%, UKIP 11%, GRN 5% (tabs). The Conservatives are down 5 points since last week, UKIP up four points – something that is almost certainly a reversion to the mean after the incongruous six point Tory lead last week.

Lord Ashcroft has topline figures of CON 34%, LAB 30%, LDEM 10%, UKIP 13%, GRN 4% (tabs). Ashcroft has also updated his Scottish constituency polls from last week with two new polls in Scotland, both in Edinburgh. Edinburgh North and Leith and Edinburgh South were both Labour -v- Lib Dem marginals at the last election, with the SNP in a poor fourth place. Ashcroft’s latest polls finds the SNP ahead in both, with a 13 point lead in Edinburgh North and Leith and a narrow three point lead in Edinburgh South (tabs, tabs).

YouGov are still to come later on tonight…

UPDATE: The last GB poll of the day – YouGov for the Sun – has topline figures of CON 34%, LAB 35%, LD 7%, UKIP 13%, GRN 5%. So we have two polls giving Labour leads, two giving Conservative leads, and the polls apparently still fluctuating around an underlying picture of Labour and Conservative neck-and-neck.


The worst thing you can do in analysing polls of voting intention is to get excited at polls that show something exciting and different and ignore those that show the same old pattern. Occassionally the unusual poll will herald a genuine movement in public opinion – after all, whenever there is a change, one poll has to pick it up first. More often than not, the unusual poll will turn out to be a freak result, the product of unusual sampling or methods. If there is genuinely a change in public opinion, other polls will pick it up sooner or later, so it’s always wise to withhold your judgement.

Today we have one of those unusual polls, and we have the overexcitement you’d expect. ICM’s monthly poll in the Guardian has topline figures of CON 39%(+3), LAB 33%(-2), LDEM 8%(nc), UKIP 7%(-2), GRN 7%(+3) (tabs). This is pretty odd all round – a storming six point lead for the Tories, up on thirty-nine percent; the Greens and UKIP equal on seven percent.

In the Guardian’s write up they are rightly dubious, and include a welcome caveat from ICM’s Martin Boon about the inevitability of random variation and the sample perhaps being a touch too Tory. I’ll just leave it with the usual caveats – it’s one poll, and an odd looking one at that. Sure, it could be the start of some Tory surge, but if it is we will see it echoed in other polls today…and luckily enough we have at least three of them.

Populus this morning had topline figures of CON 33%, LAB 33%, LDEM 8%, UKIP 15%, GRN 5% (tabs). The Conservatives are up two points (possibly helped by an update in weighting targets), but no big tory lead.

Still to come are the weekly Ashcroft poll and the daily YouGov poll. Come the end of the day, the way to judge where we are is too look at all them as a whole – not fixate on the unusual one.

UPDATE: Lord Ashcroft’s weekly poll has topline figures of CON 33%(-3), LAB 33%(-1), LDEM 9%(+3), UKIP 13%(+3), GRN 6%(-1). Changes are from a fortnight ago – Ashcroft took a week off to avoid bank holiday fieldwork. As with today’s Populus poll, there is nothing here to support the big Tory lead in the ICM poll. Full details are on Lord Ashcroft’s website here.

UPDATE2: Finally the daily YouGov poll for the Sun has topline figures of CON 33%, LAB 34%, LD 8%, UKIP 13%, GRN 6% – a one point Labour lead. Putting all four polls together that ICM poll looks very much like an outlier. Such things are an unavoidable part of polling – and well done to Guardian for reporting it in a heavily caveated way within the context of other polls showing no movement, rather than getting all excited about it.