ComRes have an interesting post over on their site about differences between online and telephone polling so far this year (as well as making some extremely sensible points about the polls not being all over the place). As they correctly say, telephone polls this year have been showing a tiny Conservative lead, online polls a tiny Labour one. It’s only a small difference, but it’s there and it is not new – at the start of the year I produced a chart showing house differences between the different polling companies over 2014, and even then an online vs telephone tendency was observable: the two most “Toryish” polls were Ipsos MORI and ICM, both done by telephone. The most “Laboury” polls were TNS and Opinium, both done online.

Look a little closer though, and things are not quite that cut and dried. There are many causes of variation between polls, telephone or online fieldwork is just one of them. There is variation between different online companies and between different phone companies. Last year ComRes’s telephone polls actually produced some of the more Laboury figures, the online Populus polls tended be on the Tory side of average. Below is the average for each company so far this year (given the polls have been pretty static in 2015 I haven’t worried too much about timings of different companies polls, it’s just a straight average).

phoneonlineleads

So all three companies who have been showing a Tory lead are done by phone, all the online polls have been showing an average Labour lead. But note the variation – MORI use the telephone, but they are showing a Labour lead on average. Two online polls (YouGov and Opinium) show barely any Labour lead at all, Survation, TNS and Panelbase average around a 2 point Labour lead. This is because there are plenty of other reasons for variation between pollsters too, different approaches to weighting, turnout, don’t knows and so on – I summarised lots of them here. Just looking at one can sometimes be misleading, for example, ICM and Ashcroft also reallocate don’t knows by past vote, which normally bumps up the Tory position by a point or so, so that will also be a major part of the difference between them and companies showing worse results for the Conservatives (one should also bear in mind that the monthly polling companies have only produced 3 or 4 polls this year – so a single odd poll like ICM’s this month has a large impact on the average).

I’ve no doubt that telephone vs online is one of the reasons for differences though, especially when it comes to UKIP. The graph below has even starker differences. With Labour vs Conservatives the difference between phone and online polls is a matter of a few points. With UKIP there is a vast gulf between the figures from different pollsters…

phoneonlineukip

The companies showing lower UKIP scores are all telephone. The companies showing higher UKIP scores are all online. While there is little difference between the phone company showing the highest UKIP support (Ashcroft) and the online company showing the lowest (YouGov), there is a gulf of 9 points between the highest and lowest ends of the scale. Why there should be such a difference between online and telephone polling of UKIP we cannot tell – some of it may be an interviewer effect (people being more willing to tell a computer screen they are voting for a non-mainstream party than a human interviewer), some of it may be sampling (some online samples getting too many of the sort of people who vote UKIP, or some phone samples getting too few, or both). Until the results are in we won’t really know.


566 Responses to “Phone and online differences”

1 2 3 4 12
  1. @Mikey

    I’ve worked it out now!

  2. That was a post Lab manifesto poll so we should see a bounce back tomorrow, although as we know a two point difference is neither here nor there.

  3. If this is the response from the Labour manifesto launch, one point movement, not very aspiring. Or have I got something wrong ?

  4. Ta

  5. Ta, Mikey

  6. AW.. Please comment on sample size, as 1,000 plus or minus a few is throwing up results all over the place. plus 6 to evens, Paddy Power still have DC on a minority govt but very close…?

  7. BristolianHoward

    I agree with you, just 1 point gain.
    Lets see what happens tomorrow.

  8. A safe launch?

  9. Labour lead by two tonight.

    What on earth was that ICM poll yesterday all about? A bit Toryish I would say!

    :-)

  10. 1 point is more than the Tories got from the Budget!

  11. Bill Whitehurst

    Is the way that opinion polling is undertaken hostage to the commercial interests of both the polling companies and the newspapers that pay their wages?

    Not really. Political opinion poling makes up a pretty small percentage of the work that most polling companies get and they don’t make much money on them. Most polling work is market research for commercial companies not for newspapers (who are notoriously stingy).

    The reason that they do political polls it is as a form of advertising – it gets the pollster’s name in the media and businesses become more aware of their services[1]. This is why some pollsters even conduct and release polling that nobody seems to be paying for – particularly at high profile times such as before an election. At the moment TNS, Panelbase and Populus are conducting some polls without any sponsorship[2].

    Would it actually be better (i.e. provide more accurate data) if there was, say, one poll per week (two in the last few weeks before an election) with about 20,000 respondents based upon a methodology that has the best track record.

    Not really either. Partly because no one has any idea what the ‘best’ methodology is – otherwise they would all do it. But a bigger sample doesn’t really always give you that much more accuracy to justify the extra effort. A sample of 20,000 isn’t 20 times ‘better’ than one of a 1000 – it’s maybe four times better at most and anyway the extra accuracy may not tell you much. There’s also the danger of putting all your eggs in one basket.

    The optimum figure for most polls is probably 1000 to 2000, though larger ones can be useful if you want look at sub-samples – for example that 8000 poll I mentioned above meant that there were around 500 each of LA and HA tenants in the sample. Smaller number wouldn’t be reliable.

    [1] In some case this may include doing private polling for political Parties or pressure groups of course.

    [2] Usually they are doing these polls by tacking questions onto other commercial polling they would be carrying out anyway. So the main cost is the extra analysis and set up. It’s also possible that they feel the people in their sample would expect to be asked how they will vote at this time so they are including the questions to show they are ‘proper’ pollsters.

  12. Labour’s launch was meant to calm nerves I think rather than be inspiring so I’m not surprised there is little change in the polls. Think they made a mistake myself but we’ll see.

    I think part of the telephone polling effect may be to be with the fact that the respondant is not always alone and may be overheard by other members of the family.

  13. Interestingly the Yougov Centre has

    Lab 36
    Con 34
    UKIP 13
    LD 8
    Green 5
    Other 4

  14. @assiduosity

    “But does it represent a benefit to too few to speak more widely of a party ‘for working people’? Time and polls will tell I guess.”

    I’m thinking now it was more about invoking the ghost of Mrs T to win back some BlueKippers.

  15. This is excellent – the comparison is finally partially up to date with modern times and methods of polling compared with American polling standards…. however, we do need a comparison between landline telephone and mobile telephone polling to really adjudicate data to the current norm.

  16. […]

    The reality is nearly every pollster in the last 48 hours has shown a swing back to Con and only YG putting Labour ahead.

    A simple average of 10 seat forecasts as of this afternoon puts Con on 276 and Lab on 275. This is as close as Lab will get to overtaking Con on seats now – the UKIP vote will continue to ebb away and LDs will climb a little – leaving Lab struggling to make a handful of net gains.

  17. We cant say that there has been a 1% Lab boost from the manifesto launch.
    the before and after polls are all subject too moe as will the ones tomorrow which will not tell us if the Cons got a boost.

    A flat poll may, for example, be a boost as the earlier one may have been a tad low within moe but we will never know.

    BH- time for a 4 in a row warning I think.

  18. Will both sides be happy with that poll?

    For Labour any lead is a good lead, and for the Tories to be only two points behind after a day of Labour news and manifesto launch.

  19. @SWebb

    Maybe. Or maybe not. At this stage the most likely result is something approaching a dead heat in terms of votes beteeen Lab and Con.

  20. @MartinW

    I agree entirely and said as much in a rather lengthy post in a previous post.

    Invoking Mrs Thatcher and trying to rekindle the magic electoral flame that was right to buy the first time round – a political obsession of all political parties ever since that policy proved such a game changer.

  21. Polling like politics is an art not a science. It’s impossible to disprove any theory. Anything can be explained by anybody at any time. Any result can be spun any way. A bit like accountancy but less subject to regulation, politics and polling rely on statistical analysis plus the subjective input of analysts and their adherence to the analytic modelling they’ve adopted.

  22. Ashman
    “If this is the response from the Labour manifesto launch, one point movement, not very aspiring. Or have I got something wrong ?”

    I suspect you’ve got something wrong.

    Most folks’ minds were made up on the Lab-Con question a long time ago.

    The manifestos are about not dropping the ball – there’s nowt earth-shattering in any of them.

    This poll doesn’t reflect any movement to Labour.

    The next few days’ polls are unlikely to reflect any movement to the Tories.

  23. Roger Mexico

    Agree a poll of sample size 20000 is not effective to do on a regular basis, however doing one pre election is probably worth it to avoid being the unlucky pollster who got screwed by sampling error

    A poll of 20k would give a sampling error of 0.7% for major parties (so would also expose any significant methodology errors that might get hidden otherwise)

    The difference between 1k and 2k is noticeable but you do get diminishing returns so I can’t see more than that happening regularly (unless the cost per response falls)

  24. Wonder why the early reports said 35-33 when now it seems to have become 36-34.

  25. Somewhat belatedly, a summary of the accuracy of the four different projection models (as benchmarked against the latest batch of Ashcroft constituency polls).

    May2015 only offers a projection of the margin for the frontrunning party over the runner-up. Using this measure all four models were equally accurate (or inaccurate). Across the ten constituencies polled the mean absolute difference between the model’s margin and the polled margin was: Electionforecast – 5.4%; YouGov Nowcast – 4.9%; Electoral Calculus – 4.9% and May2015 – 5.8%. \none of these discrepancy measures differ reliably from one another. So, by this criterion all four models are equally accurate in predicting the polls.

    Turning to the models’ accuracy in predicting VIs for challenger parties, I use the Euclidean Distance (ED) metric. (A high score means a bad miss and a score of zero would imply that projected VIs were exactly right for every single party.) On this measure EC (mean ED = 6.2) and EF (mean ED = 7.2) were both reliably more accurate than YouGov were (mean ED = 10.3), but the former two did not differ in accuracy from one another.

    This pattern is exactly the same as emerged from last week’s batch of Ashcroft’s polls. (For that sample the corresponding mean ED scores were: EF Model: 6.76; EC model: 7.41 and YouGov Nowcast: 9.50). As on this occasion, the performance of the first two did not differ significantly but both were did a better job than YouGov.

    This strengthens the evidence that the YouGov model is out of line with the Ashcroft polls. In 24 days we should be able to tell whether or not this is a good place to be.

  26. DM going in very hard on their front page against the chief executives of housing associations, describing them as ‘fat cat hypocrites’ over their opposition to RTB for tenants when they earn ‘six figure sums’.

    One can see why they would do this, but I can’t imagine that Conservative HQ, who appeared to be looking to drop negativity and move to ‘The Good Life’ message, will be overly delighted with their keynote upbeat policy descending into an ugly spat so soon.

    Meanwhile ‘The Times’ leads on ‘Brussels Vows to Block Cameron on EU Treaty’, again hardly helpful if the UKIP flank is to be fended off.

    A cautionary note to those who believe that the right of centre press always act in concert to support the Conservative agenda.

  27. Labour’s manifesto does seem a bit lame compared to the Tories but is difficult to attack whereas the Tory manifesto and its funding of pledges will now be the subject of considerable scrutiny. The next few days may well decide who comes out on top.

  28. Time once more for

    On This Day in 2010 on UKPR

    Today, Wednesday 14th April, 2010, saw a relatively quiet day for polls, with just three. YG had Cons up to 41% and 9% ahead, suggesting a possible manifesto bounce, while the earlier ComRes showed both the big two down on 35/29 while Harris showed Cons easing slightly on 36/27.

    @John Fletcher opined – “Which ever way you cut it GB is not getting back into no. 10.”

    while @Richard O was a vaguely concerned Conservative supporter, happy in general with the state of the polls, but saying –
    “No idea how the debates will impact. My worry as a Conservative supportor, is that Cameron is going in to the debates expected to win, which is not always the best way to enter…”.

    How very prescient! With just 24 hours to the first debate, the swagger of Cameron was about to dodge Brown’s clunking fist, only to break upon the novelty of Clegg. But that was for tomorrow.

    @Charliew gave us a good insight into the science of UKPR election result forecasting with the following; –

    “I think Harris is closest to the real position tonight – maybe under by 1% for each of the 3 parties.
    However, when unknowns actually get to the booth, I think a majority will go blue because “it’s time for change”.
    So, come polling day, I think a slim (single digit) majority for DC”

    For any newbies out there, just in case you missed it, this is how it’s done;
    1) Decide what you would like the result to be
    2) Assume that sufficient voters will make their minds up in the polling booth
    3) Take the most recent polls, and then discard the ones you don’t like
    4) Add 2) and 3) to equal 1)
    5) If they don’t equate precisely, make up something else to fill the gap.

    It’s easy, when you get the hang of it.

    @John Fletcher informed the board that he had – “..been telephone canvassing myself.”

    John – it’s great to see real political activism, but here’s a hint – it’s a little more effective if you canvass other people.

    @Peterbell reported – “Have just heard on Sky News that the Torygraph is reprting that a number of economists have written that Tory economic proposals will probably result in a double dip recession.”

    Economists 1 Politicians 0

    Meanwhile, @Rosie P complained – “I too felt an unnecessary ‘swipe’ was taken at my (IMHO) objective comments yesterday evening”

    See, that’s the trouble with objectivity – it’s always IMHO, IMHO.

    @Polly Ticks (sweet girl, but at heart, a joker) said –

    “…there are only 2 possible outcomes I believe.
    Labour will not win a majority or be the biggest party. So that only leaves the question how many short or by how many will the Tories win by…?”

    Which led @Eoin Clarke to respond –

    “I very politely disagree…if the trend I have pointted out continues, i have a theory that GB will remain as PM….”

    Well I think Eoin I can, on behalf of all at UKPR, say (very politely) you were wrong.

    More later.

  29. To be fair to Eoin Clarke he was a lot closer than many people expected.

  30. Labour will be quite satisfied with 36 pts…and 2pt lead…and confirmation that yesterdays Tory 6 pt lead appears an outlier…nothing more…no crossover as yet.

    Anyone know why the Greens would launch their manifesto on the same day as the Tories?

  31. I think that we will really need to wait to the weekend for the effects (if any) of the manifestos and the debate to become clear.

  32. @Alan

    I’ll tabulate the MOE of difference vs sum of party from 60 to 80% in a bit

    I am impressed!

  33. So I’ve skim read both manifestos. In the spirit of this website I’ll leave my own political persuasion to one side. Labour were low on details but set out some principles. The Tories were the opposite; lots of details (mostly continuing what has been set out already) but light on the guiding principles.

    How this has been reflected in the media is Labour saying “honestly, we won’t be reckless with the economy” whereas the Tories is all about the Right to Buy for Housing Association tenants.

    What this means for public opinion is, I suspect, negligible. I doubt that there will be many floating voters in marginals who will think of labour, “oh, that’s OK then; now you’ve talked about a triple lock, I’ll vote for you”. Equally, the only voters in marginal constituencies to switch to Tory today are HA tenants able and willing to buy their homes.

    So I don’t see the last two days as marking an end to the polldrums. Nor do I see them igniting the campaign and engaging more than those engaged already.

  34. The tory internal rhetoric is that they will gain VI so late no poll will pick it up.!!!

    All down to voters who set out for the polling station still making up their mind but who by the time they arrive stick with the devil they know.

    Labour clearly believes these same voters are deeply suspicious of all politicians promises and will vote for the party who tells the fewest porkies.

    If I were advising the pair of them I would tell them get their prayer mat out.

  35. @SWEBB: “A simple average of 10 seat forecasts as of this afternoon puts Con on 276 and Lab on 275.”

    The Tories will likely need at least 300 seats to stay in government while Labour could probably cobble together a majority with no more than 280.

  36. @swebb wishful thinking? What evidence there is points more likely to a dead heat. But we shall see.

  37. Thanks for the replies to my question – it was just a question, i have no knowledge at all of this subject; a little more now though :-).

    Indeed, what a lovely site this is. It is the LV-CC of online chat sites – genteel, polite and full of people wanting to help and chat !! Been at a LV-CC match today, in fact, watching DPRY win.

  38. Alan

    The trouble with very large samples is the ‘eggs in one basket’ situation. If there is a systematic problem with your sampling, unrelated to sample size, then your results may be more precise but just as inaccurate. In some cases there may even be factors that increase with the sample size (panel exhaustion for example) that may actually play against the benefits of a larger sample.

  39. According to Allegra on Newsnight, the Conservatives have changed tack. The approach taken last week,has been dropped, and this week’s initiatives are more positive.

  40. @Daibach

    You’ve set out clearly in a few words what I took a lot more and a rather extended metaphor concerning the cast of The Good Life to say on the previous thread.

    Labour’s ‘light’ manifesto seems largely defensive, giving their opponents little with which to find fault or attack them and not much that can be turned into gaffes.

    This seems consistent with the alleged ‘35% strategy’.

    The Conservatives’ much more traditional ‘stocked shop window’ of promises, literally pledging to do everything from commission nuclear submarines to repair church roofs, is a consumer offer.

    Both have risks, Labour could be seen as mean, the Conservatives may have offered hostages to fortune.

    Who would have thought that a week ago?

    With turnarounds like that I’d say it was to soon to say that there aren’t more surprises ahead in this campaign.

  41. @Billy W
    “Indeed, what a lovely site this is. It is the LV-CC of online chat sites – genteel, polite and full of people wanting to help and chat !! Been at a LV-CC match today, in fact, watching DPRY win.”

    I agree with your first sentence, but possibly because of my age I struggle with modern acronyms. The only LV I know is Luncheon Vouchers. Can you explain LV-CC and DPRY for an old fogey? Thanks.

  42. @BillyWhitehurst

    A cricketing Yorkshireman?

  43. Well to be fair we already had an election in 2014, the European elections, so we can see from that who got UKIP right. UKIP got 27.49% of the vote at a GB level on 22 May.

    From the right end of Anthony’s chart

    Survation – 19-20 May poll said UKIP would get 32, so 4.5% over

    TNS 15-19 May said UKIP would get 31, so 3.5% over.

    Comres online – 16-18 May said UKIP would get 33, so 5.5% over

    Panelbase – no poll

    Opinium – 19-21 May said UKIP would get 32, so 4.5% over

    Populus – no poll

    Yougov – 20-21 May said UKIP would get 27, so 0.5 under

    Ashcroft – no poll

    Comres phone – no poll

    MORI – no poll

    ICM – 14-15 May said UKIP would get 25, so 2.5% under.

    So the answer is in – Yougov is the best pollster in terms of getting UKIP support right!

    Phone polls understate. Non Yougov online panels overstate.

  44. @ Pete B

    Liverpool Victoria (cricket) County Championship / Democratic Peoples Republic (of) Yorkshire

    @ Omnishambles re: 2012 – Four more beers, etc. It was a good night.

  45. Learned today that Michael Fallon is a Scot and defeated Alex Salmond in a Student Union election in St Andrews.

  46. Roger Mexico

    If there’s problems with methodology, it’ll only be exposed by doing a higher accuracy poll before the election, until then it’s too easy to put any weirdness down to sampling error. It’s the best change to identify methodology errors you might have.

    I agree if other things start to go wrong like panel exhaustion, then there are other issues involved, sometimes you just aren’t set up to make such a poll, fortunately the phone book doesn’t run out of numbers quite so easily.

    In a way the pollsters have all their eggs in one basket the day before the election anyway, if you have methodological errors with a large poll, you’ll likely have then with a small poll (if anything the larger poll will likely be more representative) as that’s normally the measure taken to see who got it right and who got it wrong. In that situation I’d want as big a basket as reasonable.

  47. Does anyone know whether YouGov were using online or telephone polls in 2010?

  48. @PeteB

    The interweb suggests the cricket county championship for LV-CC.

    For DPRY, it suggests the Democratic People’s Republic of Yemen.

    This may not be what was intended by BillyW :)

  49. I think the Conservatives might get a small push up to equal Labour on 34% each from their manifesto launch.

    Beyond that, back to wobbling around 34 for each I think, depending on the vagaries of individual polls.
    So it could easily be called polldrums. Although it looks to me that the Conservatives have a bit more variation than Labour, ie 32-36 (excluding “weird” polls) vs 33-35. Could that possibly be UKIPers wavering?

  50. Matt
    Thanks for that – a bit bl—y obscure.

1 2 3 4 12