ComRes have an interesting post over on their site about differences between online and telephone polling so far this year (as well as making some extremely sensible points about the polls not being all over the place). As they correctly say, telephone polls this year have been showing a tiny Conservative lead, online polls a tiny Labour one. It’s only a small difference, but it’s there and it is not new – at the start of the year I produced a chart showing house differences between the different polling companies over 2014, and even then an online vs telephone tendency was observable: the two most “Toryish” polls were Ipsos MORI and ICM, both done by telephone. The most “Laboury” polls were TNS and Opinium, both done online.

Look a little closer though, and things are not quite that cut and dried. There are many causes of variation between polls, telephone or online fieldwork is just one of them. There is variation between different online companies and between different phone companies. Last year ComRes’s telephone polls actually produced some of the more Laboury figures, the online Populus polls tended be on the Tory side of average. Below is the average for each company so far this year (given the polls have been pretty static in 2015 I haven’t worried too much about timings of different companies polls, it’s just a straight average).

phoneonlineleads

So all three companies who have been showing a Tory lead are done by phone, all the online polls have been showing an average Labour lead. But note the variation – MORI use the telephone, but they are showing a Labour lead on average. Two online polls (YouGov and Opinium) show barely any Labour lead at all, Survation, TNS and Panelbase average around a 2 point Labour lead. This is because there are plenty of other reasons for variation between pollsters too, different approaches to weighting, turnout, don’t knows and so on – I summarised lots of them here. Just looking at one can sometimes be misleading, for example, ICM and Ashcroft also reallocate don’t knows by past vote, which normally bumps up the Tory position by a point or so, so that will also be a major part of the difference between them and companies showing worse results for the Conservatives (one should also bear in mind that the monthly polling companies have only produced 3 or 4 polls this year – so a single odd poll like ICM’s this month has a large impact on the average).

I’ve no doubt that telephone vs online is one of the reasons for differences though, especially when it comes to UKIP. The graph below has even starker differences. With Labour vs Conservatives the difference between phone and online polls is a matter of a few points. With UKIP there is a vast gulf between the figures from different pollsters…

phoneonlineukip

The companies showing lower UKIP scores are all telephone. The companies showing higher UKIP scores are all online. While there is little difference between the phone company showing the highest UKIP support (Ashcroft) and the online company showing the lowest (YouGov), there is a gulf of 9 points between the highest and lowest ends of the scale. Why there should be such a difference between online and telephone polling of UKIP we cannot tell – some of it may be an interviewer effect (people being more willing to tell a computer screen they are voting for a non-mainstream party than a human interviewer), some of it may be sampling (some online samples getting too many of the sort of people who vote UKIP, or some phone samples getting too few, or both). Until the results are in we won’t really know.


566 Responses to “Phone and online differences”

1 10 11 12
  1. Saffer

    “Pointless nostalgia anecdote”

    Nostalgia anecdote? Yes

    Pointless? Certainly not! I read that with great interest (and admiration).

  2. New thread

  3. Safer
    Great way of putting a sobering perspective
    This site needs posts like yours!

  4. Pointless nostalgia anecdote.

    Reading here the reports of UKPR posters’ experiences on the doorsteps, brings back memories of my own endeavours in S Africa, in the dark days of apartheid,

    My first ever political experience was in the 1970 GE. Fresh out of the army after compulsory military service, one of the first things I did was to sign up, aged 18, as a member of the Progressive Party (whose only MP was Helen Suzman) and began canvassing, putting up posters, leaffeting and general dogsbody in a no hope constituency in which I could not even vote – my own district had no PP candidate, just like 146 more of the 166 total. Our most optimistic hope was to gain a 2nd MP, to give some support to Helen. I vividly remember my first election night party, listening to the results, and crying bitterly as one after another result showed that we had made good progress in many areas, but not enough to get that 2nd seat.

    In many more GE’s and by elections thereafter I repeated my efforts, but never ever in a seat with a realistic chance of winning. Initially, that was just because there were not any within (geographic) reach. Later, it was a point of principle – it was more important to me to take the message of liberal values and non- racialism to the unconverted, than to shore up support in what were becoming safe seats. By the 1987 and 1989 elections, I was campaign manager for the DP (successor to Suzman’s Progs) in “Helderkruin”, a large seat on the outskirts of Johannesburg, which we did not even contest just 10 years earlier – and made enough noise against a cabinet minister that the press began to tout it as possible surprise upset.

    We didn’t win – but by then, we had become the Official Opposition in the (White) parliament, and ust a few years later, we won the bigger prize – SA’s first democratic election.

    Along the way, in Helderkruin and earlier, I had to cope with a challenge that won’t have bothered too many here (except possibly AS, in Canada) – attempting to conduct a doorstep or telephone canvass in a second language. This was a Nationalist stronghold, with mostly Afrikaans speaking voters, and my home language was English.

    Years earlier, I’d done the same thing in a genuinely no-hope seat: a by-election in Johannesburg West, where we had never previously had a candidate, way back in 1972. I vividly recall my very poor attempts to conduct a canvass in Afrikaans, with students on the campus of the “Randse Afrikaanse Universiteit”. The winning Nat candidate, one Carel de Wet, was later SA Ambassador to London. In the circumstances, we thought our vote share of just 4% was a reasonable achievement. On the same day, we also contested for the first time a seat in the industial town of Vereeniging, where we achieved only 2%, against one FW de Klerk – later to be state president, and the man who finally faced up to reality, and prepared for a transition to multi – party democracy.

    My interest in psephology began way back with that first election in 1970. To this day, I can name the 19 constituencies in which the PP had candidates, and recall the approximate share of the vote we scored in each (in some cases, the majority to the exact number – Colin Eglin in Sea Point fell short by just 231). For subsequent elections, I can recall fairly accurately the sequence in which one seat after another fell to the Progressive Party, and it’s successors, the Progressive Federal Party and the Democratic Party. Now, I continue to obsess over the polls, projections, and results for elections here in the UK, the USA, and back home in SA.

    But I’m deeply grateful that I no longer have to take the results quite as seriously as I did in the past – when some years (1974 for instance) had me whooping out loud, and walking on air for days, but others (1970, and 1987) had me literally crying myself to sleep.

    And with the greatest respect to all those here, the issues here just don’t seem quite as important as the fight against apartheid which dominated my election experience for 20 years.

  5. If anyone thought that the negative personal attacks on Ed Miliband and Labour were a feature of the campaign that had been consigned to the past after the ‘backstabber’ misfire of last week, a glance at tomorrow’s front pages should set them straight.

    Though my personal view is the press have only a fraction of the influence they had in the past for a whole host of reasons, I do wonder whether the public associate certain papers so closely with the Conservatives that a continuation of these stories might actually come to be seen as part of the official campaign and therefore reflect on the Conservatives to their detriment.

    Also, someone needs to brief George Osborne if there’s been a change of tone to ‘The Good Life’, his response to the LibDem and UKIP manifestos – an attack on Ed Miliband and repetition of ‘competence vs chaos’ – looked and sounded ‘so last week’.

    The Conservative campaign seemed to steady around the manifesto launch, the next few days will tell if this discipline will be maintained, especially if the polls don’t move.

  6. @Prof Howard I think the dynamics of the challengers debate are more fluid than that; UKIP could use tomorrow to take votes from others but those others are most likely to be the absent Tories as he’ll be the sole right wing voice. of the others it’s difficult to see how NS can win as she did so well last time….. unless she wants to go much further with clarity on where they stand vis a vis Labour. Miliband on the other hand has potentially a lot to gain and something to lose; he has improved perceptions of his leadership significantly during the campaign but still has a way to go & this could be another step in a contest where he’ll be stuck in the middle of the spectrum on display.

    The other one to watch is Leanne Wood; partly because this will again raise her profile which was far lower than the others at the start and partly because I suspect she’ll plan to go for Farage at the slightest opportunity which will at least raise the entertainment stakes

  7. Thanks for the replies to the Cameron question.
    MOG,
    Personally I hope it isn’t EM who gets the knives stuck in him.I can’t remember
    whether the Ides of the month were the 13th or 17th but I am fairly sure they
    weren’t the 16th.

  8. @Saffer

    Thank you. A great post.

  9. @paulbristol

    Re: Bristol West. My son is currently campaigning there. He seemed to suggest it was neck and neck until recently as a three way marginal: Green 28%, Lib Dem and Labour likewise. However his feelign is that a lot of Green support comes from the student population and they are much less likely to vote/or be registered for that area.

  10. @paulbristol

    I was surprised to see Labour on 22% in Weston. Obviously higher than many Somerset areas. They’re about 10% in our bit of the county.

  11. @saffer

    Certainly puts things in perspective. Thanks.

  12. Anyone else think Clegg is now quite definitely putting all his eggs in the Conservative basket…. aiming for a second coalition with them if at all possible. His tone certainly seems to have hardened. I think this may well be because he is now concerned he is losing voters who are worried he may support a Labour government. His efforts to woo back tactical or left leaning Lib Dems with policy shifts towards the end of government and many apologies have failed. I think he is now heading firmly to reassure voters who are anti-Labour rather than fully pro-Conservative that he is a viable option.

  13. I am surprised to hear people here cheering for Labour because they are still up one percent. First of all, its just one point, the lead went down and every poll has a margin of error of 2 to 3 percent for Conservatives and Labour each. Therefore with a moe of 2.5 percent its possible that Labour is up six points or Conservatives being up 4. Thats the range, one poll doenst tell you anything.
    I still believe Labour will perform worse than expected due to the fact that most people cant stand Ed.

  14. McClane

    In case you are interested, we’ve been discussing the margin of error of a lead recently, for typical Con vs Lab leads the MOE is 5% (ish) for polls of size 1000 and about 4% (ish) for polls of size 2000.

    Fundamentally what you say is true, a lead of 1 is nothing in isolation, it’s only when multiple polls come together can you hope to learn anything useful.

  15. @Mcclane – thanks for that valuable insight on behalf of the ‘people’

  16. Just out of interest… how difficult/annoying would it be to do a one-off aggregate of just phone polls and just online polls?

    Obviously until the election we won’t know whether one is more accurate than the other. But AFTER the election, it will be interesting to see whether the result from one is more accurate than the other, and by how much.

    I’ve often suspected that online polls are prone to bias without really knowing why. Then I looked into how online pollsters get their sample and… it seems like it’s entirely self-selecting. Which… is obviously a major problem, surely?

    I mean, not to generalise, but in my experience, people who support UKIP *RRRRREALLY* support UKIP; but their enthusiasm doesn’t change how many votes they get. For me, that would be the most obvious explanation for online polls being skewed towards them.

1 10 11 12