In this post back in January I wrote about the partisan effects of the different methodologies the different polling companies used, of how some companies tend to show consistently higher or lower scores for different parties. Since then I’ve been meaning to do a reference post explaining those different methods between pollsters. This is that – an attempt to do a summary of different companies methods in one place so you can check whether company A prompts for UKIP or what company B does with their don’t knows. As ever, this is from the published methodology details of each company and my own understanding of it – any mistakes are mine and corrections are welcome!

Phone polls

There are four regular telephone polls – Ipsos MORI, ICM, Ashcroft and ComRes/Daily Mail (ComRes do both telephone and online polls). All phone polls are conducted using Random Digit Dialing (RDD) – essentially taking phone numbers from the BT directory and then randomising the digits at the end to ensure the sample includes some ex-directory numbers, all polls will now also include some mobile phone numbers, though the pollsters have all said this makes little actual difference to results and is being done as a precaution. All telephone polls are weighted by some common demographics, like age, gender, social class, region, housing tenure, holidays taken and car ownership.

Ipsos MORI

Now the most venerable of the regular pollsters, Ipsos MORI are also the most traditional in their methods. They currently do a monthly political poll for the Evening Standard. Alone among GB pollsters they use no form of political weighting, viewing the problem of false recall as unsurmountable, their samples are weighted using standard demographics, but also by public and private sector employment.

MORI do not (as of March 2015) include UKIP in their main prompt for voting intention. For people who say don’t know, MORI ask who people who are most likely to vote for and count that equally as a voting intention. People who still say don’t know or won’t say are ignored. In terms of likelihood to vote, MORI have the tightest filter of any company, including only those respondents who say they are absolutely 10/10 certain to vote.

ICM

ICM are the second oldest of the current regular pollsters, and were the pioneer of most of the methods that became commonplace after the polling industry changed methods following the 1992 debacle. They currently do a monthly poll for the Guardian. They poll by standard demographics and by people’s past vote, adjusted for false recall.

ICM don’t currently include UKIP in their main voting intention prompt. People who say they don’t know how they will vote are reallocated based on how they say they voted at the previous election, but weighted down to 50% of the value of people who actually give a voting intention. In terms of likelihood to vote, ICM weight by likelihood so that people who say they are 10/10 certain to vote are fully counted, people who say they are 9/10 likely to vote count as 0.9 of a vote and so on. Additionally ICM weight people who did not vote at the previous election down by 50%, the only pollster to use this additional weighting.

Ashcroft

Lord Ashcroft commissions a regular weekly poll, carried out by other polling companies but on a “white label” basis. The methods are essentially those Populus used to use for their telephone polls, rather than the online methods Populus now use for their own regular polling. Ashcroft polls are weighted by standard demographics and by past vote, adjusted for false recall.

Ashcroft’s voting intention question has included UKIP in the main prompt since 2015. People who say they don’t know how they will vote are reallocated based on how they say they voted at the previous election, but at a different ratio to ICM (Ashcroft weights Conservatives and Labour down to 50%, Lib Dems down to 30%, others I think are ignored). In terms of likelihood to vote, Ashcroft weights people according to how likely they say they are to vote in similar way to ICM.

ComRes

ComRes do a monthly telephone poll, previously for the Independent but since 2015 for the Daily Mail. This is separate to their monthly online poll for the Independent on Sunday and there are some consistent differences between their results, meaning I treat them as two separate data series. ComRes’s polls are weighted using standard demographics and past vote, adjusted for false recall – in much the same way as ICM and Ashcroft.

ComRes have included UKIP in their main voting intention prompt since late 2014. People who say they don’t know how they will vote or won’t say are asked a squeeze question on how they would vote if it was a legal requirement, and included in the main figures. People who still say don’t know are re-allocated based on the party they say they most closely identify with, though unlike the ICM and Ashcroft reallocation this rarely seems to make an impact. In terms of likelihood to vote ComRes both filter AND weight by likelihood to vote – people who say they are less than 5/10 likely to vote are excluded completely, people who say they are 5/10 to 10/10 are weighted according to this likelihood.

Online Polls

Online poll sampling can be somewhat more opaque than telephone sampling. In most cases they are conducted through existing panels of online volunteers (either their own panels, like the YouGov panel or PopulusLive, or panels from third party providers like Toluna and Research Now). Surveys are conducted by inviting panellists with the required demographics to complete the poll – this means that while panels are self-selecting, surveys themselves aren’t (that is, you can choose to join a company’s online panel, but you can’t choose to fill in their March voting intention survey, you may or may not get randomly invited to it). Because panellists demographics are known in advance, pollsters can set quotas and invite people with the demographics to reflect the British public. Some pollsters also use random online sampling – using pop-ups on websites to randomly invite respondents. As with telephone polling, all online pollsters use some common demographic weighting, with all companies weighting by things like age, gender, region and social class.

YouGov

YouGov are the longest standing online pollster, currently doing daily voting intention polls for the Sun and Sunday Times. The length of time they have been around means they have data on their panellists from the 2010 election (and, indeed, in some cases from the 2005 election) so their weighting scheme largely relies on the data collected from panellists in May 2010, updated periodically to take account of people who have joined the panel since then. As well as standard demographics, YouGov also weight by newspaper readership and party identification in 2010 (that is, people are weighted by which party they told YouGov they identified with most in May 2010, using targets based on May 2010).

YouGov have included UKIP in their main prompt since January 2015. They do not use any weighting or filtering by likelihood to vote at all outside of the immediate run up to elections (in the weeks leading up to the 2010 election they weighting by likelihood to vote in a similar way to Ashcroft, Populus and ICM). People who say don’t know are excluded from final figures, there is no squeeze question or reallocation.

Populus

Populus used to conduct telephone polling for the Times, but since ceasing to work for the Times have switched to carrying out online polling, done using their PopulusLive panel. Currently they publish two polls a week, on Mondays and Fridays. As well as normal demographic weightings they weight using party identification, weighting current party ID to estimated national targets.

Populus have included UKIP in their main prompt since February 2015. They weight respondents according to their likelihood to vote in a similar way to ICM and Ashcroft. People who say don’t know are excluded from final figures, there is no squeeze question or reallocation.

ComRes

Not to be confused with their telephone polls for the Daily Mail, ComRes also conduct a series of monthly online polls for the Independent on Sunday and Sunday Mirror. It is conducted partially from a panel, partially from random online sampling (pop-ups on websites directing people to surveys). In addition to normal demographic weightings they weight using people’s recalled vote from the 2010 election.

ComRes have included UKIP in their main prompt since December 2014. Their weighting by likelihood to vote is slightly different to their telephone polls – for the Conservatives, Labour and Liberal Democrats it’s the same (include people who say 5+/10, and weight those people according to their likelihood) but for UKIP and Green I believe respondents are only included if they are 10/10 certain to vote. Their treatment of don’t knows is the same as in their phone polls: people who say they don’t know how they will vote or won’t say are asked a squeeze question and included in the main figures, people who still say don’t know are re-allocated based on the party they say they most closely identify with.

Survation

Survation do a regular poll for the Daily Mirror and occasional polls for the Mail on Sunday. Data is weighted by the usual demographics, but uses income and education rather than social class. Recalled 2010 vote is used for political weighting. Survation have included UKIP in their main prompt for several years. They weight by likelihood to vote in the same way as ICM, Populus and Ashcroft. People who say don’t know are reallocated to the party they voted for in 2010, but weighted down to 30% of the value of people who actually give a voting intention.

Note that Survation’s constituency polls are done using a completely different method to their national polls, using telephone sampling rather than online sampling and different weighting variables.

Opinium

Opinium do regular polling for the Observer, currently every week for the duration of the election campaign. Respondents are taken from their own panel and is weighted by standard demographics. Historically Opinium have not used political weighting, but from February 2015 they switched to weighting by “party propensity” for the duration of the election campaign. This is a variable based on which parties people would and wouldn’t consider – for practical purposes, it seems to be similar to party identification.

Opinium do not include UKIP in their main prompt (meaning they only appear as an option if a respondent selects “other”). They filter people by likelihood to vote, including only respondents who say they will definitely or probably vote. People who say don’t know are excluded from the final figures.

TNS

TNS are a huge global company with a long history in market research. In terms of public opinion polling in this country they are actually the successors to System Three – who used to be a well known Scottish polling company and ended up part of the same company through a complicated series of mergers and buy-outs by BMRB, NFO, Kantar and WPP, currently their ultimate parent company. At the last election TNS were the final company doing face-to-face polling, since then they have switched over to online. The sample is taken from their Lightspeed panel and is weighted using standard demographics and recalled 2010 vote. TNS do include UKIP in their main prompt, and also prompt for the BNP and Green. TNS filter and weight people according to likelihood to vote and exclude don’t knows and won’t says from their final figures.

Putting all those together, here’s a summary of the methods.

methods2

As to the impact of the different methods, it not always easy to say. Some are easy to quantify from published tables (for example, ICM and Ashcroft publish their figures before and after don’t knows are reallocated, so one can comfortably say “that adjustment added 2 points to the Lib Dems this week”), others are very difficult to quantify (the difference the choice of weighting regimes makes is very difficult to judge, the differences between online and telephone polling even more so), many methods interact with one another and the impacts of different approaches changes over time (a methodology that helps the Tories one year may help the Lib Dems another year as public opinions change). Rather than guess whether each pollsters methods are likely to produce this effect or that effect, probably best to judge them from actual observed results.

UPDATE:
TNS have confirmed they do prompt for UKIP, and also prompt for the BNP and Green – I’ll update the table later on tonight.


491 Responses to “Different pollsters methods – a summary”

1 2 3 10
  1. Many thanks, Anthony

  2. Lots of reading lol

  3. So which one is best?

  4. The one which gives your preferred choice the best result :-)

  5. Thank you. Worth careful study, but off the cuff, two quick questions:
    1. Online poll sampling … … conducted through existing panels of online volunteers
    How big are the panels?
    2. How confident are you that the VI distributions are the same for samples with the same demographics except that one sample has been prepared to join an on-line panel, and the other would never do that? What fractions of the voting population fall into the two camps?

  6. Yep
    The papers always have a flavour of the month(day)

  7. Another Magnum Opus Anthony, but of course, I’ll need magna tempus to absorb it. MAny thanks.

  8. Probably a silly question, but do any of the polling companies ask those who have recently changed their intended vote, what made them change? No doubt there are usually a multiplicity of reasons, but at least we could use that info as a means of retrospectively assessing if our guesswork has a basis in fact.

  9. May thanks for this, Anthony. With a little luck I’ll stop posting wrong things now!

  10. CatManJeff
    “The one which gives your preferred choice the best result :-)”

    That’s the Daily Mail school of poll reporting. Ignore your own commissioned poll if it does not give you the right answer and find a better one.

  11. YouGov/Sun: CON 34 (+1) LAB 34 (-1) LIB 8 (=) UKIP 12 (-2) GRN 6 (+1)

  12. YouGov tonight: CON 34%, LAB 34%, LD 8%, UKIP 12%, GRN 6%.

  13. @BristolianHoward

    You can always fall back on a phone straw poll for that purpose.

  14. When UKIP go down Con go up and vice versa.
    Its still 0-0 with neither side creating many chances. There are a lack of star players on either side.

  15. The YG pattern continues.

    Good solid Con and Lab scores, LD coasting, and UKIP and Green struggling.

  16. Its Leicester vs QPR

  17. CMJ

    They’ll not be interested in tonight’s YG then. On the other hand how about

    ‘Tories bounce back from freak weekend polling;.

  18. Dave

    Online poll sampling … … conducted through existing panels of online volunteers
    How big are the panels?

    It’s actually quite hard to find out, especially from those pollsters that don’t use their own panels. YouGov seems to be the largest of those that do – they say they have “over 400,000 respondents”, though some of these may not be terribly active. I think Opinium only has 30,000 however

  19. The Lib Dems locally used Survation to do a constituency poll. Apparently there were several of these (using different methodologies?) but the Lib Dems only published one of them. This was presumably the one that showed them in their best light (trailing Labour by one point). I am assuming that the unpublished polls gave Labour bigger leads. All the commentary on the published poll was left to the LDs. What trust can people have in such manipulation? Survation are a well respected polling organisation. I do not think they should allow their work to be distorted in this way.

  20. It looks more and more like UKIP`s misfortunes are keeping the Tory vote afloat.

    Labour are very steady, and should make no concessions to SNP. England gains are more important than dancing for AS and a Scotland recovery.

  21. Meaty post today, thanks

    Re: new YouGov,

    The UKIP squeeze continues. I predict more squeezing yet.

  22. “They’ll not be interested in tonight’s YG then. On the other hand how about
    ‘Tories bounce back from freak weekend polling;.”

    Swingback starts now, as Crosby is believed to have stated two week ago…

    “After Easter we’ll see Con VI surge ahead” is the new mantra. then two weeks later, they will say the polls are wrong.

  23. When is Easter this year?

  24. @BristolianH

    Tut. “Tempus” is neuter singular like “opus”, unless you’re talking about the weather, when it’s masculine.

    Good thing there’s no Raj any more, we’d never find people qualified to run it.

  25. “When is Easter this year?”

    Good Friday is 3rd April…

  26. @ Peter Crawford,

    Personally, my money is on “Swingback will start on May 8th.”

    The question of when/if the Tories start panicking is a good one. They’ve been remarkably disciplined since 2013, but with little more than a month to go and no firm leads, I wonder if it will start breaking down. (If Cameron’s spontaneous announcement wasn’t already the start of mass hysteria…)

  27. @amber
    “When is Easter this year?”

    When the Tory swingback starts. Let the polls be your guide.

  28. Interesting to see that the YouGov poll has seen a distinct swing to the Conservatives from Labour tonight. Too soon to talk of swingback – but could be the start of it.

  29. “magna tempus”

    My father would have been whipped hard for that!

  30. I was thinking it might be a “slow burn” Easter that would show up in a year or two. ;-)

  31. Peter Crawford

    Our colleague CB11 adds Con and Lab to give an indication of where the main parties stand. I think if it’s less than 70, he regards that there is little enthusiasm among voters for either of them.

    So apathy rules OK with tonight’s poll with the ‘gaffe count’ rising every day.

  32. @PI
    ‘Good thing there’s no Raj any more, we’d never find people qualified to run it.”

    Because knowledge of Latin is of vital significance in running India?

  33. Thanks Anthony, very useful post.

    It would be nice to also include sample size/MOE for each pollster, so I have added this link that Unicorn gave us a few weeks ago.

    http://blogs.lse.ac.uk/generalelection/interpreting-new-polls/

  34. “The question of when/if the Tories start panicking is a good one. They’ve been remarkably disciplined since 2013, but with little more than a month to go and no firm leads, I wonder if it will start breaking down. (If Cameron’s spontaneous announcement wasn’t already the start of mass hysteria…)”

    they won’t panic. They’ll believe in swingback, then they will say the polls are wrong like they were in ’92. They will think they will win, “deo gratia” and by divine right, until the last vote is counted on the morning of 8th May. Besides, the betting markets are euphoric about tory chances of being the largest party and patching together some deal, and the market is never wrong.

    the press is also promoting this sense of unshakeable certainty.

    I think Dave’s 3rd term remarks are trying to project this confidence. it has worked for tories in the past.

  35. PC

    The internet told me that one, as I don’t know Latin well enough. Come on, tell this pleb what the right answer is. I’m glad I didn’t go to your father’s school. Sounds a nasty place.

  36. Tories panicking?

    When do the Lib Dems start panicking they’re on 8%

  37. Smithson thinks it will be Liz Truss ,a very longshot.

    It will be Boris of course unless theres an event.

    The one after might be Pritti Patel -Mrs T reincarnate.

    Anyway the tories think they are going to win so dont panic.

    Off to do my homework set by Headmaster Wells .

  38. Great piece of work Anthony. Probably the only place where this information can be found all together.

    I’ve thought of another litttle wrinkle:
    You said that “all polls will now also include some mobile phone numbers, though the pollsters have all said this makes little actual difference to results and is being done as a precaution. All telephone polls are weighted by some common demographics, like age, gender, social class, region, housing tenure, holidays taken and car ownership. ”

    Would that include weighting to include things like the fact that those with a landline are likely to have an older age profile than those with mobile numbers? I’ve got both, but though I know quite a few people under 30, none of them has a landline.

  39. PI
    Oh you are in this snob Latin lark too. I thought it should be magnus tempus (intuitively) so I googled and that’s what I got. Blind faith.

  40. Thanks, Anthony. I imagine this post will become a frequently consulted resource. It’s very useful to have all the information in one place.

  41. @amber

    Saw that one coming!

    @peter crawford
    “then they will say the polls are wrong like they were in ’92”

    Like I said in the previous thread, the 2010 polls were out by a couple of %. And a couple of % is all you need to get the Kellner 295 (:

    They were 2% out the day before the election
    They were 2%+ out 6 weeks before the election

  42. “. Come on, tell this pleb what the right answer is. I’m glad I didn’t go to your father’s school. Sounds a nasty place.”

    magnum tempus…as the man, Postageincluded i think it was, said “tempus” is neuter like opus…so magnum is the right adjective.

    Dad was at a tough minor public school in the 50s!!

  43. It seems clear that Cameron’s announcement tonight wasn’t planned. There isn’t any clear line coming from backbenchers, Gove was unable to say that he knew in advance, and No 10 is unable to answer some pretty basic questions over the timing of the Tory leadership election, should they win in May.

    All in all, I’d say it was a mess. Tories big message is competence vs chaos, but that might have taken a bit of a knock tonight.

    Overall, I doubt this is a major impact event in terms of VI, unless things develop and party members speak out of turn. What it will do is create a distraction, when Tories should be focused on sending out the core message with relentless determination.

    At best, they’ve lost another day of news coverage to make their case. There are also hints of arrogance over the apparent presumption in the answer, which Labour and the media have already picked up on.

  44. RAF,
    Well there was that old general who sent the message saying”Peccavi”.I’ve
    probably spelt it wrong.

  45. RAF
    “Because knowledge of Latin is of vital significance in running India?”

    How else could Napier have reported the conquest of Sind? (peccavi)

  46. I am a little curious as to what research has been done recently on phone polls. In 2010 I would probably answer a call from an unknown number either on my landline or mobile, but now not so much, and I suspect I am not alone. Is there any weighing for that cultural change in our approach to communications, and whether it is more a tendency of the relatively young?

  47. “the 2010 polls were out by a couple of %”

    on what days?…which averages?

    the average (5 day i think) polls understated the lib dems, but got tory and labour pretty accurately, 6 weeks out…

  48. @ Couper 2802,

    The Lib Dems have all gone mad (as became obvious pretty soon into the coalition, to be honest) and believe they’ll hold most of their seats. I doubt they’ll admit they’re in trouble until after the election.

    The Tories aren’t quite that delusional, though. Cameron and his merry band, yes, but they also have some people who were bankers and such and who can presumably do arithmetic. I dunno, maybe Peter’s right and they convinced themselves it’s 1992 and the pollsters are just wrong- but even on that logic, they lost 40 seats in 1992.

  49. Do the Tories not usually get a weird ‘bank holiday bounce’ at Easter which fades immediately thereafter?

  50. I use two methods to track You Gov:

    1) Two sided CUSUM (sensitive to changes less than 1.5 SD)

    2) X Bar Control Chart (sensitive to changes greater than 1.5 SD)

    A few trends have been clear via 1) recently, but now two trends show on 2), meaning it is stronger.

    Firstly, the Conservatives have had ten data points above their 2015 mean.

    Secondly, UKIP have been below their 2015 mean for seven data points.

    UKIP and the Greens have grown since 2010, so should improve in that GE score. I suspect they will remain under pressure from now on, so my guess is they will end up with 10-12% and 4-5% respectively.

1 2 3 10