Personally I try not to get involved in knocking down some of the more outre polling conspiracy theories, especially since recently lots of them have been YouGov related. On UKPR I’ve always tried to educate about how polls work and what the differences are, rather than campaign for the methodologies I personally rate. Certainly I try to cover all of them fairly, rather than being the defender of my own employer YouGov.

However, when they start turning up in the mainstream media I suppose I need to discuss them. Earlier on I linked to a post from Peter Kellner rebutting what looks set to be a hatchet job against YouGov in the Telegraph tomorrow. From the questions they asked Peter, I’m really not hopeful of anything sensible emerging.

In terms of the recent results, YouGov have been showing some of the narrower leads, but actually the polls have not been as contradictory as they sometimes seemed. The more established pollsters have been showing pretty much the same story – YouGov was showing a 4 point Tory lead last week, it seems to have moved up to around about 7 now. Ipsos-MORI’s last poll had a 5 point lead. ComRes did not poll during the period of the real narroring leads, their only poll in March was post-budget and showed a 7 point lead, so the same as YouGov. TNS BMRB have, if anything, been showing smaller leads than YouGov.

ICM have tended to show slightly higher Tory leads than YouGov, but not by much, and nothing that can’t be explained by their different approaches to likelihood to vote, and the long standing contrast between the two companies’ Lib Dem scores.

The significant contrast has been with the newer online companies – Harris, Angus Reid and Opinium – who have all tended to show double point Conservative leads, though a methodology change has brought the Tory lead in Angus Reid’s most recent poll down into single figures. For all we know the newer companies may turn out to know something that more established pollsters do not, but given they have little or no UK track record, the burden of proof should probably lie with them.

Secondly we come to the subject of weighting – this is a bit long I’m afraid. It is very easy to paint weighting negatively, to someone outside the industry it can easily be made to sound dodgy: the raw data showed X, but then the pollsters “weighted” it to show Y. In fact a lot of polling is counter-intuitive, asking just 1000 people and then weighting them? For anyone with nefarious intent it’s always easy to make it sound suspicious.

One of my oldest pieces of advice here is do not fall in love with raw numbers. Unweighted data is not pure and unsullied – it is unrepresentative. The reason it is weighted is because pollsters can see its make up does not match the known demographics of the population, and weighting makes it representative. To give a simple example, we know that 52% of the adult population is female, so if a sample was only 50% female, all the women would be weighted upwards by 1.04 (i.e. rather than counting as one person, each female respondent would count as 1.04 people). The total weighted sample would then be 52% female, and it would be far more representative than the unweighted one.

There is a rather odd comment from Robert Winnett in his email to Peter about some Experian data not needing weighting because of its size. At the risk of boring regular readers who will have heard the tale many times before, size doesn’t imply representativeness. Famously in 1936 the Literary Digest carried out a poll with a sample size of millions, and George Gallup carried out a poll of a few thousand… but used proper representive sampling techniques. Gallup correctly called the 1936 Roosevelt landslide, while the Literary Digest confidently predicted a victory for Alf Landon. What makes a poll useful is how representative it is – an unrepresentative poll, an unweighted poll, however large, is worthless.

A seperate issue is degree of weighting. To fully understand that, we need to go back to basics about how pollsters do their samples and why they need to weight. Most phone pollsters use quasi-random sampling – they rely upon the laws of probability to generate a representive sample. If a pollster could obtain a genuinely random sample it might not actually need very much weighting, in practice though samples are not actually random because of huge non-response rates, meaning that much chunkier weighting is necessary. For example, in ICM’s last News of the World poll they needed to weight up 2005 Lib Dem voters to 1.41 and weight down 2005 Labour voters to 0.74

The alternative to quasi-random sampling is quota sampling – this basically equates to constructing a representative sample, rather than relying on probability to provide one. We know that 52% of adults are women and 48% are men, so you go out and interview 520 women and 480 men. Similarly, pollsters using quota sampling will try to interview the correct proportions of people in each social class and age group. This can result in very, very low levels of weighting. When MORI used to do face-to-face polling their weighting made hardly any difference at all, they didn’t need to weight much by gender for example, because their interviewers deliberately chose the right proportions of men and women to start with. With quota sampling, a lot of the work of getting a representative sample goes into drawing up and filling the quotas rather than in the weighting.

With YouGov’s method of sampling things are not as simple as the old face-to-face polling, or indeed as the way YouGov used to do things. Until last autumn YouGov used to invite people to specific surveys, so if they invited the correct number of people in each demographic group, not a lot of weighting was needed – all the work was done in defining and filling the quotas. Last autumn YouGov switched to inviting people to come and complete an unspecified survey, thus eliminating the problem of fast and slow respondents and allowing fast turnaround polls. The downside was that it allowed less tight control of who was invited to surveys, so more of the work needed to be done by weighting. In the months since then YouGov have been able to tweak the proportions of invites sent in order to meet the quotas more accurately in the first place and reduce the amount of weighting needed, so it is now less than it was back in January and February.

As a general rule, the less weighting you have to do the better, but what matters is that polls are weighted to the right targets, not how much weighting is needed to get there. The problem with large weightings is not that it skews samples, but that it reduces the effective sample size leading to more volatility. In practice however YouGov tends to be one of the less volutile pollsters so this does not appear to be a problem.

Moving on, the Telegraph ask a very odd question about whether Peter admits something was a rogue poll and whether he regrets publishing it, which suggests something of a misunderstanding of what a rogue poll is. A rogue poll means something specific. Too often it is used as a term of abuse, and this is wrong. When polls quote a margin of error (with a poll of 1000 people it is normally quoted as 3%), what they mean is that statistically 95% of the time the values in the poll will be within 3% of the “real” value. A rogue poll refers to the 5% of occassions when it will fall outside that margin of error.

This is an immutable part of statistical theory. 5% of ICM’s polls will be rogue polls. 5% of Populus’s polls will be rogue polls, 5% of MORI’s will, 5% of Angus Reid’s will, 5% of TNS’s will and 5% of YouGov’s will. No pollster is capable of defeating the laws of mathematics. Criticising a pollster for producing rogue polls is like criticising the lottery when it picks number 1 or number 49: it is the unavoidable functioning of the laws of mathematics.

The implication that pollsters shouldn’t release polls that they think might turn out to be rogues is even more bonkers. If a poll produces an odd result you certainly double check everything for possible human error and anything weird or wacky in the sample, but if everything checks out you must publish. Can you imagine the response if it was discovered that a pollster had refused to release a poll because they thought the figures were too nice to the Conservatives, or too nice to Labour?

Peter has pretty much addressed the rest, though there is one thing worth noting. As I’ve mentioned here several times, YouGov ran daily polls for several weeks before the Sun started publishing them in order to test out the systems and parallel test it with old style polling. These polls were never intended to be published. However, since Peter has confirmed it in his post I’m now free to say there was one instance during the test polls, straight after the “Tears for Piers” interview when it showed the Conservative lead right down to a single point.

UPDATE: And here is the article, it is very light indeed. It is pretty much the sort of thing I mentioned in the article above, describing “weighting” in ominious terms that would make the naive or uneducated think something was fishy (including putting weighting in inverted commas!). YouGov also apparently use an undisclosed formula to weight data, in the sense that they regularly undisclose it in a list of the weighting targets used at the end of every published table.


58 Responses to “YouGov and the Telegraph”

1 2
  1. It was a very shoddy article from the Telegraph.

  2. I’d like to make it even clearer that as an ex-pollster myself I would never suggest, nor have any truck withthe idea that Yougov are in ay way politically biased.
    All polsters try to produce the most accurate findings. Think about it their reputation and living depend on it.
    The only people who think any polling organisation would skew their figures to suit their clients or any views they may have are ignorant and/or biased partisans.

  3. AW,

    Typically shoddy journalism – what has happened to our press ? The broadsheets used to be so good, now they seem to think they have to chase headlines & soundbites at the expense of proper research and analysis.

    Could there be some political motivation behind this ? Could the Telegraph editorial team finally have twigged that they have alienated such a large part of their natural readership such that it is these days utterly fallacious to describe it as the “Torygraph” ?

    Maybe this article is an attempt to ingratiate themselves with CCHQ as they realise that the man they have been so vocally criticising for so long will shortly become PM ? The rather non-commital quote from an un-named source suggest that is, it has not worked !

    As another example of the recent change in heart at the DT look at Brogan’s blog article following Cameron’s speech earlier this week. “Eureka” moment ? As his own article puts it, “what took him so long ?”

  4. Despite what John TT recommends above, I’m afraid I do feel the need to let through even the most bonkers criticisms of YouGov, however ill-founded. There is one exception though, which is comments impugning the integrity of Peter Kellner.

    For obvious reasons, I don’t let libellous comments about anyone through since the owner of the website is just as liable to be sued. In the specific case of Peter there is obviously no chance of him suing me for bonkers comments on my site, but he gets the same respect other pollsters do. More to the point, I pay for this site out of my pocket, and I’m not allowing something I pay for to be used to personally insult friends and colleagues I like and respect.

    However though I’ve moderated a couple, I may as well respond to what they would being saying if I wasn’t moderating them.

    Comments saying pollster X produces figures good or bad for a particular party because they are baised anger me. Firstly they do so on a personal level – I know most of the senior players in the field, and while I disagree with some of their methodologies, I have not the slightest doubt of any of their professional integrities.

    Secondly, it’s intellectually lazy – no attempt to understand the methodological differences between pollsters, the different approaches and subtleties of their methods – nah, they must just be biased.

    Thirdly, it’s illogical – there is little or no money in political polling, pollsters make their money through commercial market research, political polling is a shop window to show off their accuracy, why would they bugger up their own companies success (and undermine their own professional integrity) to boost their party’s support by a percentage point or two.

    Fourthly and finally, it betrays a huge misunderstanding of how polling companies work – or at least, how a company the size of YouGov works. Peter is YouGov’s figurehead, but it’s not like he cranks all the figures himself. The panel team control the sampling, different people download the data and weight it, and run the tables, and then it goes to Peter (if he is in that day, many days he has nothing whatsoever to do with the poll). Peter would have to bind us all into his wicked conspiracy (and some of us, including the CEOs of the company, are Conservatives).

    The bottom line is that this blog is about understanding the methodology of polls. If you want to indulge in conspiracy theories and attacks on the professional integrity of individual pollsters, go and start your own blog.

  5. AW – I hope you appreciate that what I “recommend” is done with a motive to defend the principles you defend, rather than to encourage a change of policy.

    It’s your site, and (surprised to say this) the general level of debate hasn’t suffered as much as it might have been expected to given teh five-fold increase in posts. If anything, it’s become more humorous, which helps your reputationand that of YouGov.

    Attempting to improve lazy thinking is one of the main drivers of many people who join your blog and contribute. That’s a common motive on all sides, and long may it continue.

    If only there were a way of measuring lazy brain activity among the electorate and making a voting system more representative of what people would think if only they would be able to stop and think.

  6. Nice post Anthony.

    I must say I’m surprised that the Telegraph and Tories are so concerned with criticizing YouGov’s poll numbers. Usually sensible politicians don’t try to argue against the polls. What do the Tories hope to gain from criticizing YouGov? If YouGov polls are biased towards Labour, then the Tories can simply ignore them, and in the end if the YouGov polls are totally inaccurate wrt the election result, it’ll simply impact YouGov’s future business because no one will want to hire a company to do research if their results are not reliable.

    On the other hand if YouGov’s polls are more accurate then there is nothing to be gained by criticizing them, the Tories would then do better concentrating on the campaign against Labour and the Lib-Dems than attacking the messenger. I suspect this says a great deal about how rattled the Tory party is, several months ago it looked like this election was in the bag. The long phoney campaign might just mean that the Tories have peaked too soon. But if they are as rattled as they seem to be, pannicking won’t help them, it’ll hurt them, and this attack on a neutral pollster certainly does make them look panicky.

    On more point worth making about YouGov, it’s all very well to claim that their polls are biased in favour of Labour, but there is not really any evidence of this, and without evidence we should dismiss the claims. We will only know after the election. But if the argument is that there is systemic Labour bias, then how come in 2005 YouGov actually regularly gave Labour a smaller lead than the other pollsters? In the run up to the 2005 election one could equally have claimed that YouGov had a systemic anti-Labour bias because all the other pollsters were predicting a greater vote share for Labour. As it turned out YouGov were the most accurate pollsters in 2005.

    So the Tory position seems to be to attack pollsters when they produce polls they don’t like. Frankly I don’t think that’s a very sensible approach. They should ignore the polls, the Tories are not fighting the election against YouGov, if they try to do this they are wasting resources and frankly look more than a little immature.

  7. John TT – Don’t get me wrong, I appreciate it very much and would like to feel able to accept it!

    Alun – actually, I don’t think the Tories *are*. As Peter says, YouGov do lots of private work for the Tories quite happily and I have heard no such criticism from the party. The only Conservative the Telegraph managed to get to back up their claim that senior Tories were concerned was a single PPC.

  8. @Anthony

    Ah well maybe you’re right. I posted that after reading in the Telegraph article you link to:

    “Some polling experts, academics and senior Tories claim that YouGov polls predicting a hung Parliament are overstating Labour support.”

    If the Telegraph have only a single PPC to support this claim, then it’s just poor journalism, a simple attempt to sensationalize their article. After all “Some…senior Tories” is absolutely not the same as “a single PPC”. But then when did journalists ever let small things like facts ever get in the way of a good story.

    Not bad politics on the part of the Tories so much as bad journalism on the part of the Telegraph.

1 2