In this post back in January I wrote about the partisan effects of the different methodologies the different polling companies used, of how some companies tend to show consistently higher or lower scores for different parties. Since then I’ve been meaning to do a reference post explaining those different methods between pollsters. This is that – an attempt to do a summary of different companies methods in one place so you can check whether company A prompts for UKIP or what company B does with their don’t knows. As ever, this is from the published methodology details of each company and my own understanding of it – any mistakes are mine and corrections are welcome!

Phone polls

There are four regular telephone polls – Ipsos MORI, ICM, Ashcroft and ComRes/Daily Mail (ComRes do both telephone and online polls). All phone polls are conducted using Random Digit Dialing (RDD) – essentially taking phone numbers from the BT directory and then randomising the digits at the end to ensure the sample includes some ex-directory numbers, all polls will now also include some mobile phone numbers, though the pollsters have all said this makes little actual difference to results and is being done as a precaution. All telephone polls are weighted by some common demographics, like age, gender, social class, region, housing tenure, holidays taken and car ownership.

Ipsos MORI

Now the most venerable of the regular pollsters, Ipsos MORI are also the most traditional in their methods. They currently do a monthly political poll for the Evening Standard. Alone among GB pollsters they use no form of political weighting, viewing the problem of false recall as unsurmountable, their samples are weighted using standard demographics, but also by public and private sector employment.

MORI do not (as of March 2015) include UKIP in their main prompt for voting intention. For people who say don’t know, MORI ask who people who are most likely to vote for and count that equally as a voting intention. People who still say don’t know or won’t say are ignored. In terms of likelihood to vote, MORI have the tightest filter of any company, including only those respondents who say they are absolutely 10/10 certain to vote.

ICM

ICM are the second oldest of the current regular pollsters, and were the pioneer of most of the methods that became commonplace after the polling industry changed methods following the 1992 debacle. They currently do a monthly poll for the Guardian. They poll by standard demographics and by people’s past vote, adjusted for false recall.

ICM don’t currently include UKIP in their main voting intention prompt. People who say they don’t know how they will vote are reallocated based on how they say they voted at the previous election, but weighted down to 50% of the value of people who actually give a voting intention. In terms of likelihood to vote, ICM weight by likelihood so that people who say they are 10/10 certain to vote are fully counted, people who say they are 9/10 likely to vote count as 0.9 of a vote and so on. Additionally ICM weight people who did not vote at the previous election down by 50%, the only pollster to use this additional weighting.

Ashcroft

Lord Ashcroft commissions a regular weekly poll, carried out by other polling companies but on a “white label” basis. The methods are essentially those Populus used to use for their telephone polls, rather than the online methods Populus now use for their own regular polling. Ashcroft polls are weighted by standard demographics and by past vote, adjusted for false recall.

Ashcroft’s voting intention question has included UKIP in the main prompt since 2015. People who say they don’t know how they will vote are reallocated based on how they say they voted at the previous election, but at a different ratio to ICM (Ashcroft weights Conservatives and Labour down to 50%, Lib Dems down to 30%, others I think are ignored). In terms of likelihood to vote, Ashcroft weights people according to how likely they say they are to vote in similar way to ICM.

ComRes

ComRes do a monthly telephone poll, previously for the Independent but since 2015 for the Daily Mail. This is separate to their monthly online poll for the Independent on Sunday and there are some consistent differences between their results, meaning I treat them as two separate data series. ComRes’s polls are weighted using standard demographics and past vote, adjusted for false recall – in much the same way as ICM and Ashcroft.

ComRes have included UKIP in their main voting intention prompt since late 2014. People who say they don’t know how they will vote or won’t say are asked a squeeze question on how they would vote if it was a legal requirement, and included in the main figures. People who still say don’t know are re-allocated based on the party they say they most closely identify with, though unlike the ICM and Ashcroft reallocation this rarely seems to make an impact. In terms of likelihood to vote ComRes both filter AND weight by likelihood to vote – people who say they are less than 5/10 likely to vote are excluded completely, people who say they are 5/10 to 10/10 are weighted according to this likelihood.

Online Polls

Online poll sampling can be somewhat more opaque than telephone sampling. In most cases they are conducted through existing panels of online volunteers (either their own panels, like the YouGov panel or PopulusLive, or panels from third party providers like Toluna and Research Now). Surveys are conducted by inviting panellists with the required demographics to complete the poll – this means that while panels are self-selecting, surveys themselves aren’t (that is, you can choose to join a company’s online panel, but you can’t choose to fill in their March voting intention survey, you may or may not get randomly invited to it). Because panellists demographics are known in advance, pollsters can set quotas and invite people with the demographics to reflect the British public. Some pollsters also use random online sampling – using pop-ups on websites to randomly invite respondents. As with telephone polling, all online pollsters use some common demographic weighting, with all companies weighting by things like age, gender, region and social class.

YouGov

YouGov are the longest standing online pollster, currently doing daily voting intention polls for the Sun and Sunday Times. The length of time they have been around means they have data on their panellists from the 2010 election (and, indeed, in some cases from the 2005 election) so their weighting scheme largely relies on the data collected from panellists in May 2010, updated periodically to take account of people who have joined the panel since then. As well as standard demographics, YouGov also weight by newspaper readership and party identification in 2010 (that is, people are weighted by which party they told YouGov they identified with most in May 2010, using targets based on May 2010).

YouGov have included UKIP in their main prompt since January 2015. They do not use any weighting or filtering by likelihood to vote at all outside of the immediate run up to elections (in the weeks leading up to the 2010 election they weighting by likelihood to vote in a similar way to Ashcroft, Populus and ICM). People who say don’t know are excluded from final figures, there is no squeeze question or reallocation.

Populus

Populus used to conduct telephone polling for the Times, but since ceasing to work for the Times have switched to carrying out online polling, done using their PopulusLive panel. Currently they publish two polls a week, on Mondays and Fridays. As well as normal demographic weightings they weight using party identification, weighting current party ID to estimated national targets.

Populus have included UKIP in their main prompt since February 2015. They weight respondents according to their likelihood to vote in a similar way to ICM and Ashcroft. People who say don’t know are excluded from final figures, there is no squeeze question or reallocation.

ComRes

Not to be confused with their telephone polls for the Daily Mail, ComRes also conduct a series of monthly online polls for the Independent on Sunday and Sunday Mirror. It is conducted partially from a panel, partially from random online sampling (pop-ups on websites directing people to surveys). In addition to normal demographic weightings they weight using people’s recalled vote from the 2010 election.

ComRes have included UKIP in their main prompt since December 2014. Their weighting by likelihood to vote is slightly different to their telephone polls – for the Conservatives, Labour and Liberal Democrats it’s the same (include people who say 5+/10, and weight those people according to their likelihood) but for UKIP and Green I believe respondents are only included if they are 10/10 certain to vote. Their treatment of don’t knows is the same as in their phone polls: people who say they don’t know how they will vote or won’t say are asked a squeeze question and included in the main figures, people who still say don’t know are re-allocated based on the party they say they most closely identify with.

Survation

Survation do a regular poll for the Daily Mirror and occasional polls for the Mail on Sunday. Data is weighted by the usual demographics, but uses income and education rather than social class. Recalled 2010 vote is used for political weighting. Survation have included UKIP in their main prompt for several years. They weight by likelihood to vote in the same way as ICM, Populus and Ashcroft. People who say don’t know are reallocated to the party they voted for in 2010, but weighted down to 30% of the value of people who actually give a voting intention.

Note that Survation’s constituency polls are done using a completely different method to their national polls, using telephone sampling rather than online sampling and different weighting variables.

Opinium

Opinium do regular polling for the Observer, currently every week for the duration of the election campaign. Respondents are taken from their own panel and is weighted by standard demographics. Historically Opinium have not used political weighting, but from February 2015 they switched to weighting by “party propensity” for the duration of the election campaign. This is a variable based on which parties people would and wouldn’t consider – for practical purposes, it seems to be similar to party identification.

Opinium do not include UKIP in their main prompt (meaning they only appear as an option if a respondent selects “other”). They filter people by likelihood to vote, including only respondents who say they will definitely or probably vote. People who say don’t know are excluded from the final figures.

TNS

TNS are a huge global company with a long history in market research. In terms of public opinion polling in this country they are actually the successors to System Three – who used to be a well known Scottish polling company and ended up part of the same company through a complicated series of mergers and buy-outs by BMRB, NFO, Kantar and WPP, currently their ultimate parent company. At the last election TNS were the final company doing face-to-face polling, since then they have switched over to online. The sample is taken from their Lightspeed panel and is weighted using standard demographics and recalled 2010 vote. TNS do include UKIP in their main prompt, and also prompt for the BNP and Green. TNS filter and weight people according to likelihood to vote and exclude don’t knows and won’t says from their final figures.

Putting all those together, here’s a summary of the methods.

methods2

As to the impact of the different methods, it not always easy to say. Some are easy to quantify from published tables (for example, ICM and Ashcroft publish their figures before and after don’t knows are reallocated, so one can comfortably say “that adjustment added 2 points to the Lib Dems this week”), others are very difficult to quantify (the difference the choice of weighting regimes makes is very difficult to judge, the differences between online and telephone polling even more so), many methods interact with one another and the impacts of different approaches changes over time (a methodology that helps the Tories one year may help the Lib Dems another year as public opinions change). Rather than guess whether each pollsters methods are likely to produce this effect or that effect, probably best to judge them from actual observed results.

UPDATE:
TNS have confirmed they do prompt for UKIP, and also prompt for the BNP and Green – I’ll update the table later on tonight.


491 Responses to “Different pollsters methods – a summary”

1 7 8 9 10
  1. Pete B
    Its been the headlines in The Times for the last two days! “Salmond to hold Milliband to Ransom”
    No headlines tomorrow, because someting else tragically happened in France,
    But we get a good cross section on here, so it’s a good forum for letting off steam.

  2. Its exactly the same as the Comres poll: CON 35 and LAB 35.

    This suggests still no crossover.

  3. That YouGov is remarkably similar to tonights ComRes.

  4. I am still annoyed that the chap who says he is Howard, isn’t. Some people on here will still think he is me and he can just change that by inputting a nickname on his record that makes him unique, as Prof Howard and I agreed to do.

    It was not playing cricket.

  5. @Alan – “If England left the UK, I’d love to see the arguments that we had to keep using the pound (or invite the other countries to join the “New Pound”).”

    One of the more things I found most amusing about the indy ref debate about keeping the pound was the misconception that somehow it was Scotland’s to keep.

    The pound sterling was English, and English alone, pre union. Scotland decided to join the union, and use English currency. So logically, if separated, England keeps it’s currency, and Scotland goes back to it’s currency.

    Although I’m sure divorce lawyers may have a different alternative.

  6. Ok this is very interesting

    YouGov/Sun:

    CON 35 (+1)
    LAB 35 (+1)
    LIB 8 (=)
    UKIP 12 (=)
    GRN 6 (=)

    Just like the ComRes poll, we see “others” have been squeezed for 2%. I can understand SNP moving to Labour, but where is the Tory VI from? I guess it’s just lost in the MOE and there’s some movement from UKIP, or no movement.

  7. Two polls on the same day showing a 35/35 Labour Tory tie.

    A few more of these and I may become a late subscriber to your pet theory.

    I await Friday’s Populus showing a 33/31 Labour lead of 2!!

    :-)

  8. That last post of mine should have been addressed to JimJam!

  9. Given that we’ve had two draws in a row on YG, coinciding with the Monday and Tuesday polls, I’d say (mischievously) that it’s good news for Labour.

  10. Tories recovering from lower 30s, clearly, but not at the expense of Labour, it would appear. Unless they benefit at the expense of Labour, they’re stuffed really. It has to be a blue up, red down zero sum game, otherwise they can’t win.

  11. I suspect there will be very few commenters here of lineage who will not realise that any squeeze on minor parties, that sends both Lab and Con higher, will not benefit Labour.

  12. CB11
    Snap

  13. BristolianHoward
    “It was not playing cricket.”

    At least it proves he was a true Englishman :-)

  14. I’m completely confused by this Howard business. There was The Other Howard, who seems to have disappeared, and Howard who was quite gentle and a stickler for the non-partisan policy, and who stopped commenting because of polldrums. Then two more appeared, and now there’s a fifth one pretending to be one of them but I can’t work out which one he’s pretending to be. Can someone explain please? Oreferably not another Howard.

  15. @PI

    As Rory Bremner used to say:

    “Don’t worry. I’m not going to hurt you!”

  16. Postageincluded,

    Are you suggesting we need a Howards end?

  17. @crossbat @bristolianhoward

    Remember Easter
    Remember Crosby
    Remember the Kellner 295!

  18. I note May 2015 just have moved Caroline Flint’s seat to the Labour column.

    Is that due to national polls, or has there been a recent constituency poll? (Ashcroft or otherwise).

  19. PI

    I am the ‘gentle’ one of whom you write and because I stick to the comments rule, that is why I am a bit cheeved (a self-dating use of colloquialism).

    The Other Howard is the chap who says he isn’t a Tory (he is to their right) but keeps telling us he will vote Tory and is sure Tories will win. I’ve never added them up but there must be at least 300 repetitions of that post from him in the last year alone.

  20. I think that Bristolian Howard is our pre-polldrums Howard because I always enjoyed his style & the same can be said of Bristolian Howard.

  21. @ Chris Lane,

    Danny Alexander will survive I think.

    You think Danny Alexander will survive? You, Chris Lane, he of “Lib Dems too high” fame??? Is there… some kind of rationale for this belief?

    @ Stardaz,

    I still think Labour will be the largest party but my question is would the DUP & what is left of the Lib Dems support a Tory minority Government if they won the by more votes as what happened in Feb 1974?

    The Tories are the preferred partners of the DUP. The Lib Dems are more ambiguous and it probably depends on Clegg’s survival and how well they weather the election. But the thing that really matters is the parliamentary arithmetic. Take the quite plausible scenario of Lab 35%, Con 36%. This would give seat counts around Con – 280, Lab – 285, LD – 25. There’s no way to get Con + LD + DUP + Ukip above 323 with these numbers, so there’s no point in a Tory minority government- it could never survive because the progressive bloc would immediately bring it down.

    So at that point it’s in the Lib Dems and the DUP’s interest to start talking to Labour, because Miliband is going to be Prime Minister and if they offer them C&S they might be able to get concessions.

    @ BristolianHoward,

    I suspect there will be very few commenters here of lineage who will not realise that any squeeze on minor parties, that sends both Lab and Con higher, will not benefit Labour.

    ??? I don’t think many of predicted large Ukip gains, and Labour are the ones facing the biggest threat from a minor party. How does it hurt Labour if both of the big two increase their VI at the same rate?

  22. :-)

  23. Amber Star

    I think we need a Ninja Howard the way he effortlessly snuck in and outdated your post before you made it.

  24. @ BristolianHoward,

    Are you the one who was living in the Netherlands who used to have a Lib Dem background, or is that another Howard?

  25. So is ProfHoward the same as The Other Howard???

  26. Wow, that previous question to Howard came out as gibberish. Let me try again.

    @ BristolianHoward,

    I suspect there will be very few commenters here of lineage who will not realise that any squeeze on minor parties, that sends both Lab and Con higher, will not benefit Labour.

    ??? I don’t think many of us predicted large Ukip gains, and Labour are the ones facing the biggest threat from a minor party. How does it hurt Labour if both of the big two increase their VI at the same rate?

  27. I can’t remember why I added the tt to my name, but I have a faint memory of being miffed that a doppelgänger was misrepresentkng me.
    BristolianH I applaud your gentle non partisan approach and can only hope it’s infectious
    I joined in the days when non partisan meant you couldn’t tell who voted for whom.

  28. @spearmint

    I’m with @chrislane. Danny Alexander will survive. I can’t give you a logical explanation as to why, it’s more of a heart over head thing.

  29. @ Pete B,

    Almost certainly not. You’ll know The Other Howard when you see him- his username has always been “The Other Howard” and he’ll either be posting about Tory majorities, his allotment, or how his predictive abilities always served him well in finance. ;)

  30. Perhaps I should change my name if there are too many Howards.

    I posted here over many years under the name Howard but at quite a low intensity (until recently) and I never really noticed Howard (Bristolian) and I don’t think Howard noticed me.

    BritolianHoward seems a good bloke especially now people are speaking of him being gentle and a stickler for the rule book. I feel a bit bad that he has had to change his name.

  31. @ Johntt,

    I joined in the days when non partisan meant you couldn’t tell who voted for whom.

    Did you join in the days when everyone had a background with their party logo on it? Because that was kind of a giveaway.

  32. No I am not The Other Howard.

  33. Spearmint.

    I’m not succeeding here. Does this help?

    If next poll shews Lab at 36 and Con at 36, this helps Lab.

    Add 1 to each, this helps Lab.

    and so on. In other words what CB11 thinks may not happen, is not bad for Labour (IMO)

    – but, just apply UNS, which, as the two main parties get higher VI, means that UNS will apply stronger, this favours Lab.

  34. Omnishambles,
    Perhaps we should just remember the Alamo instead.
    I wonder what the collective noun is for Howard’s?

  35. @ Spearmint

    I don’t think Bristolian Howard said it “hurt” Labour, he just said it didn’t benefit Labour.

  36. ProfHoward

    No need to change, like I have,and you have. It’s this other fellow who hasn’t, that niggles with me. He knows who he is (well, he isn’t me, nor is he you).

  37. Before then, spearmint , around 2008, and I didn’t add one to mine as there was an option to remain in-aligned
    I think the nature of the site was disrupted by that , but the traffic went up so who am I to complain ?
    I like the fact it gets so many more comments now, and it doesn’t seem to be descending to the depths of pre-May 2010 blinkered views

  38. @ Ann in Wales

    I wonder what the collective noun is for Howard’s?

    I’d like to propose a ‘Kendall’ of Howards. I hope it doesn’t betray the wrong kind of partisanship…

  39. @ BristolianHoward,

    Ah. I think I read it as a double negative or something. I get it now.

  40. @ann in wales

    A ‘swingback’ of Howards

  41. @ Johntt,

    Ah, you really are a UKPR veteran! I think Anthony agreed with you, since he got rid of them.

  42. Amber

    I think a greater VI for Labour, even if matched by Conservative, benefits Lab, due to the the way our constituencies are arranged. Try the various swingometers.

  43. So for a while there were probably two people posting as Howard who were unaware of each other because of infrequent posting. We all thought they were the same person. Now they are called Bristolian… and Prof…

    There is also The Other Howard, and possibly yet another Howard who surfaced more recently but has now disappeared.

    Is that about right?

  44. A collective noun for Howards? Not all Howards are the same!

  45. Pete B

    Indeed but he was here tonight and that is what riles.

  46. How ard can it be to sort out this Howard problem?

  47. Don’t upset the fan club of ‘The Other Howard’, gentlemen. Loud, elderly, right wing folk seem to spark almost a weird kind of deference in many British people, and it’s no exception on here. I personally find him a total bore, but there we are…I’m sure he’ll get over it.

  48. Continuing my bookmaker theme:

    Taking nos. 37 – 54 on the Labour target list ( Halesowen & Rowley Regis – Warwick & Leamington ), i.e. 18 seats, the bookies have Labour favourites to win 9 seats and joint favourites to win one. They are second favourites in the other 8.

    These are all presently held by the Tories, and all in England, with the exception of Arfon, which is held by Plaid, and thought likely to remain so by the bookies.

    There are nine target seats in the top 54 that are not Conservative-held. So removing them from the equation, we have 45 Tory-held English seats, where the bookies appear to be suggesting that about 38 will be lost to Labour. I am assuming that every seat below 37 or above 54 on the list will stay as before, or at least losses and gains will balance out.

    So Labour gains from Tories in England = 38 seats.
    But the Tories remain 4/9 to win the most seats overall, with Labour at 2/1.

    I will leave much more knowledgeable commentators to work that out. But I can see that the predicted gains by Labour in England will be counterbalanced by losses in Scotland to SNP, and possibly by an uneven distribution of Lib Dem losses.

    Labour at 2/1 still looks the value bet to me.

  49. BristolianH
    Perhaps there were three of you all along, and it wasn’t until one or more of you began to post more frequently that there was a problem? I suppose Anthony could possibly find out who was the first Howard by checking URLs but it would be an onerous task.

  50. Ladbrokes have it at 286 v 271, yes, bookies know nothing!
    Which with the Libs, DUP and UKIP makes about 3 seats short, or 3 seats over the threshold. I think it might be more comfortable than that. Good odds for a Tory majority at 5/1. Labour at 14/1 says it all really. Might be worth a shilling or two!

1 7 8 9 10