ComRes have an interesting post over on their site about differences between online and telephone polling so far this year (as well as making some extremely sensible points about the polls not being all over the place). As they correctly say, telephone polls this year have been showing a tiny Conservative lead, online polls a tiny Labour one. It’s only a small difference, but it’s there and it is not new – at the start of the year I produced a chart showing house differences between the different polling companies over 2014, and even then an online vs telephone tendency was observable: the two most “Toryish” polls were Ipsos MORI and ICM, both done by telephone. The most “Laboury” polls were TNS and Opinium, both done online.

Look a little closer though, and things are not quite that cut and dried. There are many causes of variation between polls, telephone or online fieldwork is just one of them. There is variation between different online companies and between different phone companies. Last year ComRes’s telephone polls actually produced some of the more Laboury figures, the online Populus polls tended be on the Tory side of average. Below is the average for each company so far this year (given the polls have been pretty static in 2015 I haven’t worried too much about timings of different companies polls, it’s just a straight average).

phoneonlineleads

So all three companies who have been showing a Tory lead are done by phone, all the online polls have been showing an average Labour lead. But note the variation – MORI use the telephone, but they are showing a Labour lead on average. Two online polls (YouGov and Opinium) show barely any Labour lead at all, Survation, TNS and Panelbase average around a 2 point Labour lead. This is because there are plenty of other reasons for variation between pollsters too, different approaches to weighting, turnout, don’t knows and so on – I summarised lots of them here. Just looking at one can sometimes be misleading, for example, ICM and Ashcroft also reallocate don’t knows by past vote, which normally bumps up the Tory position by a point or so, so that will also be a major part of the difference between them and companies showing worse results for the Conservatives (one should also bear in mind that the monthly polling companies have only produced 3 or 4 polls this year – so a single odd poll like ICM’s this month has a large impact on the average).

I’ve no doubt that telephone vs online is one of the reasons for differences though, especially when it comes to UKIP. The graph below has even starker differences. With Labour vs Conservatives the difference between phone and online polls is a matter of a few points. With UKIP there is a vast gulf between the figures from different pollsters…

phoneonlineukip

The companies showing lower UKIP scores are all telephone. The companies showing higher UKIP scores are all online. While there is little difference between the phone company showing the highest UKIP support (Ashcroft) and the online company showing the lowest (YouGov), there is a gulf of 9 points between the highest and lowest ends of the scale. Why there should be such a difference between online and telephone polling of UKIP we cannot tell – some of it may be an interviewer effect (people being more willing to tell a computer screen they are voting for a non-mainstream party than a human interviewer), some of it may be sampling (some online samples getting too many of the sort of people who vote UKIP, or some phone samples getting too few, or both). Until the results are in we won’t really know.


566 Responses to “Phone and online differences”

1 2 3 12
  1. Am I the only person not out Canvassing?

    Peter!

  2. Great work as always @AW. Fascinating stuff.

  3. Interesting analysis, as always

  4. It seems like there was only a small window of time where you could get a good sample of the voting population by calling random telephone lines. A couple of decades where everyone had a telephone, and reliably answered polling firms if they called.

  5. I thought it was more or less established that the internet reflected a hugely disproportionate pro-UKIP voice? (Forums etc..)

    If so wouldnt that give a better rating for UKIP from online polls? After all if GOM (Grumpy Old Men) – who make up the largest proportion of UKIP support – are those most likely to be online and rattling around things like opinion poll sites (oops) and forums, they are also likely to be slightly over-represented in polling groups.

  6. Thank you very much, Anthony, as always extremely helpful.

    MORI are interesting this year aren’t they? Telephone-based, but with Labour leads and a low UKIP average.

  7. “Toryish” is a really awful word

    Otherwise, good analysis

  8. Unicorn/Charles

    Replied to previous thread. In summary the formula is kosher, the numbers people were feeding were not.

    I set up a simulation as the numbers people were coming up with were barmy and got the correct result. As part of doing that, I saw what was going wrong with the formula.

  9. @AW

    Interesting additional material. I had a shrewd suspicion you might be drawn into this one.

    Do you have any comment to make on the claim that telephone polls not only produce different patterns of results but are more accurate in predicting real voting behaviour?

    If the evidence for that is solid, then it’s not good news for a vast amount of YouGov output is it? Can you point to a thorough analysis of the accuracy question?

    [If you’re lucky enough to have access to academic journals, this is good – http://www.tandfonline.com/doi/abs/10.1080/17457289.2011.563309#.VS2P2PnF8Wg – AW]

  10. @ CMJ and Unicorn

    A brief response in the previous topic.

  11. @Anthony

    Is there any evidence available to us that can show which has proved to be the most accurate method of polling? In other words, can we look at actual election results and then determine which polls turned out to be the most accurate and what methodology they used?

    Do pollsters opt for online or telephone because of cost and speed or do they do so because they feel one or the other gives more accurate results?

    This may be more helpful research than determining which method tends to be more “Laboury” or “Toryish”.

  12. Okay, here’s a theory based on no real evidence whatsoever: UKIP-leaning ex-Tory voters don’t like admitting they are voting UKIP, due to the social stigma attached with support for an allegedly ‘racist’ party.

    Therefore, given the privacy of online, they opt for UKIP. But when asked over the phone, they opt for Tory.

    This would seem to resolve the mystery.

  13. Election night will be a nervous time for pollsters as well as politicians then!
    Someone on here a couple of days ago pointed out that parties seen as on the up often are over estimated on polls. Will the SNP really get around 45-50% in May?
    We’ll just have to wait…

  14. @drmibbles

    +1

    Election night is going to be fun!

  15. Dr Mibbles
    Who’s to say which they will put there cross by though.

    Jayblanc makes an interesting point. Who still answers the phone if the number isn’t known. 90% of the time it’s a cold call.

  16. DRMIBBLES

    Could be true – or could be the reverse – could be that when clickling an online poll people choosed UKIP but maybe the verbal interaction over the phone makes them reasess their voting intention more seriously and choose Tory (with a heavy heart presumably).

    Shy Tories or Shy Kippers? Hopefully we might know more when we get the results.

  17. @sinbad

    Election night is NOT going to be fun

  18. @omnishambles

    Why? You’d rather know who is going to win weeks earlier? ZZZZZZZZZZ.

  19. I am a outsider on this site / subject (although not totally green – I am former student of Lord Norton, no less (!!) and I am a cheerful anorak on other subjects), so forgive me if this is naive:

    Is the way that opinion polling is undertaken hostage to the commercial interests of both the polling companies and the newpapers that pay their wages? Would it actually be better (i.e. provide more accurate data) if there was, say, one poll per week (two in the last few weeks before an election) with about 20,000 respondents based upon a methodology that has the best track record. With, for example, parliament commissioning such a poll. Or something similar in spirit.

    Just curious.

  20. @Omni

    Genuinely intrigued – why not?

  21. Is it possible that opinion polls are massively under-estimating UKIP support, especially in northern Labour strongholds? I recall that in the Heywood and Middleton by-election in October 2014, two separate opinion polls by Survation and Ashcroft, published only 10 days and 5 days before the vote, respectively each gave Lab a 19 percent lead over UKIP. In the event, Labour only just pipped UKIP by 2 percent. Is there any other by-election evidence to draw on?

  22. @CMJ, Laszlo, Charles and Alan

    Okay – just to sum up on what I think we now all agree…

    For a poll with N = 1000, approx the Margin of Error that needs to be used for any VI difference between the main two parties is about 5.2%.

    For N of about 2000 it reduces to about 3.7%.

    (According to the formula, the actual MoE varies depending on the values of the two VIs. However, in settling upon a rule-of-thumb figure this detail can be ignored because (a) the squared term makes an extremely small contribution to the eventual figure and (b) between now and May 7 we can safely assume that the Tory and Labour VIs will add to roughly 70% (which is the figure I used in the calculations above)

    The MoE would be smaller if you wanted to decide whether (say) a LibDem margin over the Greens was an outlier. In the case the Franklin paper would provide an exact method of calculating the MoE you would need to use.

    On a final point, the 5.2% MoE should preumably apply to each of the Ashcroft constituency polls (N roughly 1000) which means that a very substantial proportion of them have the two leading parties well within the MoE of each other.

  23. An interesting discussion on the problems with RTB for Housing Association tenants, to which I’d just add a few minor points. Firstly it’s a bit naive to assume that more expensive properties won’t be bought by impoverished HA tenants. Relatives, friends or those who want to make a profit can simply supply the cash and benefit from the discount when the original tenant dies or moves. Think of Alan Duncan for example who (to quote from his Wiki):

    <i. […] had used the right-to-buy programme to make profits on property deals. It emerged that he had lent his elderly next door neighbour money to buy his home under the right-to-buy legislation. The neighbour bought the 18th century council house at a significant discount and sold it to Duncan just over three years later

    There were much worse cases reported where confused or illiterate tenants were encouraged to buy and when they (inevitably) failed to keep up with the mortgage payments, were thrown out, leaving the lender with a discounted property as had been intended all along.

    I also seem to remember previous attempts at RTB for HA tenants being very strictly circumscribed b y the House of Lords, and Wiki seems to indicate that it does exist though so limited as to be insignificant.

    However we don’t need to speculate about what the public thinks about the proposals because they have already been asked:

    https://d25d2506sfb94s.cloudfront.net/cumulus_uploads/document/rii8qf86w7/NHF_Results_150408_Housing_Associations_website.pdf

    YouGov asked Generally speaking, do you think that all those living in housing associations should or should not have a right to buy their home at a government funded discount of 30%? and 35% said they should as against 45% who said they should not.

    The sample was 8077, collected over a week which meant that sub-samples were larger than normal and there was an opportunity to get decent size ones for different types of housing tenure as well (I suspect the sample was also weighted by this unlike normally with YouGov). There was surprisingly little difference in political response to the idea (Labour voters were most for, Lib Dems most against) or geographically (Scots were most against, but not by much). Most other demographic differences seem mainly related to tenure.

    Own outright: 29 – 54

    Own with mortgage: 30 -51

    Rent privately: 38 -41

    Local authority: 50 – 34

    Housing association: 61 -24

    Live with family/friends: 36 – 33

    Other: 35 – 30

    It’s not a surprise I suppose that those who would benefit should be most enthusiastic. “Would you like free money?” is always a popular question. Those will never benefit (existing owners) are most opposed while those who might benefit in future (private renters etc) are more split. LA tenants presumably back it because they already get it and possibly because they are uncertain as to the exact status of their landlord.

    The thinking behind such offers is always that those who are against are unlikely to change their vote based on this one issue, while those who approve and might benefit might well be swayed. Whether this is true in this case is another matter.

  24. Good grief Nick Robinson on BBC News sounds terrifying after his surgery.

  25. Personally I trust ICM and Yougov more than TNS and Survation. On the UKIP share and on the overall result. That would seem to suggest a better story for the Conservatives.

  26. If Online vs Telephone does create a bias, it still should be possible through other methodology to get back to relatively unbiased polls.

    Looking at the table above, yougov are looking a bit more “telephone” like than other online polls, if they need to correct further they can though more methodology fixes.

    The reasons for the methodology is just getting to the right answers consistently. Online polling is presumably a lot cheaper leading to larger sample sizes and more information. If you can remove bias (if any, there’s no reason you gov can’t be right)

    It might be prompting UKIP on an online poll pushes them too high, but on a phone poll gets it about right. We’re seeing a bit of methodology tweaks and I’d be surprised if the likes of Survation and TNS didn’t head for the middle of the pack before the election.

    On a slightly related point, it might well mess with forecasting sites that take account of house effects, only to be undone at the last minute by methodology changes. If TNS suddenly shifted a way UKIP -> Con then a neutral pre polling day poll would be seen as a pro Tory one once correcting for a non existent house effect as they headed for the herd.

    As far as I’m aware on EF, yougov 2 was from when yougov started prompting UKIP, we haven’t seen a yougov 3 when they moved to weighting by likelihood to vote. If they miss these changes things might go awry.

  27. There seems to be an interesting detail missing here: namely the proportion of phone respondents who use mobile phones. Isn’t that rather important?

    Another point as I emerge from the lurker’s shadows: are any pollsters currently asking respondents whether they are actually registered to vote?

    At least part of the 1992 polling company car-crash was down to extensive de-registration.

  28. @mitz

    Because the result matters. Following the polls is fun at the moment but once we get into the final week it will rapidly become not fun. At least in an election with an obvious result you can steel yourself for disappointment – but whatever outcome you’re hoping for, the situation at the moment gives you hope. Then you will discover the new government’s general approach to some very important issues suddenly, in a matter of hours after the polls close.

    Certainly not fun!

  29. billywhitehurst

    Increasing the sample size is subject to the law of diminishing returns (the statisticians here can explain why, in detail).

    It is theoretically possible that pollsters might decide to “influence” their results to tell a client what they think he wants to hear – but when their results are clearly rubbish, they’ll have to find a new job, because which company would ever hire them again?

    Political polling is a sideline – not the bread and butter work of polling companies.

  30. @BillyWhitehurst

    based upon a methodology that has the best track record.

    Aye, but that’s the rub. It is largely the innovation and competition between pollsters that refines the methods and helps to exploit newly emerging technologies. It’s not something that is fixed for all time.

    Also with many pollsters jostling for attention (and business) it should be quite easy to detect biases linked to commercial interest.

    So, in my view the present situation is (far) better than what you propose.

  31. Personally, I don’t “trust” any one company more than the others. Logically, an average poll of polls seems to be the likeliest to be closest to the truth.

  32. @Billywhitehurst

    Welcome. Just to add to ON’s concise summary – if there was an agreed best method they’d all use it, so it seems there isn’t one… yet.

  33. Sun front page tomorrow doesnt mention RTB ,goes with childcare,no tax on min wage,party of the grafters.

  34. YouGov/Sun poll tonight – Labour lead by two: CON 33%, LAB 35%, LD 8%, UKIP 13%, GRN 5%.

  35. YouGov/Sun poll tonight – Labour lead by two: CON 33%, LAB 35%, LD 8%, UKIP 13%, GRN 5%.

  36. Comment on YouGov poll (two point labour lead):

    This poll was conducted after people will have seen the news of Labour’s Manifesto, but largely before they had seen the Conservative Manifesto.

    Nothing here suggests that we are out of polldrums.

  37. Relief for Labour after the recent moves in other polls towards the Tories.

  38. the yougov poll, up to what time is it conducted

  39. @Omni

    Ah, don’t be so miserable. Get your friends round, set your drinking game rules (Dimbleby’s changed his tie – DRINK!) and enjoy the ride. If it’s anything like 5 years ago there will be plenty of time for the creeping horror to set in during the following days…

  40. The idea that there is a “perfect” method is in itself a bit dangerous and liable to produce groupthink. Better to have a crowded field, all looking at polling in subtly different ways and testing various theories.

  41. thanks Profhoward :-)

  42. I for one have booked annual leave for Friday 8th May and will be seeing it through to the end….

  43. Unicorn

    I’ll tabulate the MOE of difference vs sum of party from 60 to 80% in a bit.

    I think if you try measuring Green vs LD you might get non symmetric results, it might be just about OK, but you get into issues when close to 0 and 100% levels.

    I suspect the formula holds, just the distribution won’t be symmetric.
    With my bootstrapping I saw a very small positive skew (no, I didn’t run a bootstrapping on the skewness of the distributions attained from bootstrapping to see if it was significant), with very small VI this skewness should increase.

  44. @mitz

    Had great fun watching the US 2012 elections with friends but this ain’t the same thing. There will probably be drinking but no games.

  45. @Neil A
    “I for one have booked annual leave for Friday 8th May and will be seeing it through to the end….”

    In line with your reputation of never knowingly shirking a challenge! Mind you May 8th might just be the end of the beginning.

  46. Up all night on the 7th and then in to the Count at 9:00 a.m. on the 8th for the Council election.

  47. @Mikey

    Who are the “muff”?

  48. @RogerMexico

    On RTB in housing associations and your comment:

    “The thinking behind such offers is always that those who are against are unlikely to change their vote based on this one issue, while those who approve and might benefit might well be swayed. Whether this is true in this case is another matter.”

    Just so. The only additional factor to consider in this case is that for the various reasons discussed in the previous thread the actual number of people in a position to even think about exercising their right to buy is substantially lower than the 1.3 million that’s in the headlines today.

    I suspect this is a very marginal policy in terms of outcome for ‘working families’, its prominence is all down to underlying message it seeks to communicate.

    But does it represent a benefit to too few to speak more widely of a party ‘for working people’? Time and polls will tell I guess.

  49. Neil A

    I agree, but when your livelihood depends on being at least comparable to the other pollsters I can understand the desire to fix their methods.

    No point having pollsters who no one uses for political polling if they stuff the GE up. Where are Angus Reid now?

  50. Martin W

    Bournemouth.

1 2 3 12