Whenever two high profile polls come out at the same time I see lots of confused comments asking about who is right, or how come the polls are showing such different things. Today we have the well publicised ICM poll showing Labour and the Conservatives neck-and-neck and the regular YouGov/Sun poll giving Labour a nine point lead. On the face of it these are very different results. How come?

Well, there are two main causes of variation in polls. The first is the differences in methodology between polls from different companies – are they online or telephone, how do they sample, how they prompt, what things they weight by, what they weight to, how they deal with turnout, what they do with people who say don’t know? All these things add up and have a party partisan effect, so one pollster may consistently produce figures that are better for Labour, or worse for the Lib Dems or whatever.

The second cause is normal random sample error. It comes from the fact that samples for polls are, to some degree, random. The margin of error is normally quoted as plus or minus three points, meaning that 95% of the time, the figures in the poll will be within 3 percentage points of the “true” figure. Now, in many ways this is a polite fiction – the formula assumes a genuine random sample, which no polls are, and ignore lots of other factors – but I would regard the plus or minus three points as just a good rule of thumb.

If you look at leads rather than shares in polls it is easy to forget just how large that 3 point margin of error is – imagine the real situation in the country was that the Labour party was on 38% and the Conservatives on 30%, and that the pollster conducting the poll had a method that was perfectly accurate. You would still get polls that has Labour varying from 41% to 35% and the Conservatives varying between 27% and 33% – that is, leads of between 2 or 14 points!

Anyway, let’s come back to today’s polls. ICM have a ZERO point Labour lead, YouGov have a NINE point lead. Let’s dissect that and see where the difference comes from.

Looking at YouGov first, because they conduct daily polls we actually have a pretty good handle of YouGov’s random error. If levels of support are actually pretty steady, the day-to-day ups-and-downs of YouGov’s tracker will mostly be down to normal variation within the margin of error. For example, the average Conservative scores in YouGov polls so far this month is a little over 31%. Below is how each day’s Conservative figure has deviated from that score.

So in four polls it was bang on average, in five further days it was one point above or below, on two days it was 2 points above or below. This is all what we would expect to see. We’d get a similar chart for Labour.

On average YouGov’s polls this month have been showing a Labour lead of SEVEN points, so I expect the sample variation in today’s YouGov poll is a bit on the Labour side, and two points of the difference in leads between ICM and YouGov is sample error in the YouGov poll.

Now let’s move onto methodology effects. Earlier this month I produced the chart below showing the different house effects of the main polling companies – the partisan effects each company’s methods have on their topline results.

As you can see, on average ICM show Labour leads that are about 3 points lower than those shown by YouGov. That’s an average – it too will vary from month to month. The cause is largely down to two differences between the companies polls – ICM weight their data according to how likely people say they are to actually vote. They also make some estimates about how people who say don’t know will actually end up voting, assuming that 50% will end up backing the party they did back in 2010. YouGov don’t do either of these things.

If we look at ICM’s detailled tables here we can see the effect these two approaches had. ICM started out with CON 33%, LAB 38%. After weighting it by how likely people said they were to vote this moved to CON 35%, LAB 37%. After reallocating don’t knows it became CON 36%, LAB 36%. There will be other, less tangible differences in methodology, but crudely speaking another five points of the nine point difference came from the different methodologies of ICM and YouGov.

That leaves 2 points difference to account for, which is all likelihood is down to random error in the ICM poll. Whereas we get 20 odd YouGov polls a month, we only get one from ICM so we don’t have the data to average out all ICM’s polling data this month and see if this particular poll is a bit too Labour or bit too Conservative. But because we can’t measure it, doesn’t mean sample error isn’t still there.

In summary, about half the difference between ICM and YouGov’s polls towards is down to methodological differences, about half is down to normal random sample error (YouGov a bit more Lab than usual, ICM probably a bit more Tory than usual). Case closed.

Except, it probably isn’t. You’re probably still asking “Which poll is right?”, is the Labour lead zero or nine? If polls are all over the shop, what possible use are they? The answer is that one single poll taken in isolation isn’t that much use, the solution is not to take them in isolation, look at the big broad picture, the underlying trends, the averages over many polls.

And, for the record, Labour have probably got a lead of around about 7 points, like it says up at the top right of the page.


299 Responses to “Zero points or nine points?”

1 2 3 6
  1. That’s settled then.

  2. UKIP have put a spanner in the works ! I am not sure anyone knows how they would perform at a GE. If they achieve 10% + around the country and take votes away from the Tories, then Labour could achieve a small majority with only 35% (ish) of the vote. If UKIP VI sinks in the run up to the 2015 GE, as Tory flirters return to the fold, then the outcome of the GE would not be known until the results were in.

    Perhaps YG should do one ‘mega’ poll per month with a sample of say 10,000 to see how this compares to their regular poll findings. Should make no difference ?

  3. AW

    Great post but what about the Ukip score, if you plot the ICM score for Ukip on a graph it looks outrageous, especially compared to other pollsters

  4. then the outcome of the GE would not be known until the results were in.

    -I think that’s normally how it is supposed to work.

  5. Good post Anthony.

    For what it’s worth, here are some lead calcs over the past 30 polls (to Sunday):

    Simple average: Con 31.3%, Lab 39.1% (lead – 7.8%)

    MAD: Con 31.1%, Lab 39.0% (lead 7.9%)

    Weighted MAD*: Con 31.3%, Lab 39.1% (lead – 7.8%)

    * After calculating out the outliers of the data set, I weighted the most recent five as 1, the next five as 5/6th, the next 4/6 and so on.

    The most recent week omits one Lab poll of 37, and one Con poll of 34.

  6. @RiN

    Per post above, UKIP at:

    Ave: 12.8%
    MAD: 12.8%
    Weighted MAD: 12.6%

    Lib Dems are 10.1, 10.0 and 10.0 respectively.

  7. Statgeek

    Yes but no other pollster has shown such large variations in Ukip scores, some are consistently high and some are consistently low but ICM is a rollercoaster

  8. According to the Guardisn, far from pinning blame on Labour for some imagined high death rate in the NHS, “the report will … contrast with much of the pre-publication coverage by stressing that mortality rates in all NHS hospitals have been falling for the past 10 years, with overall mortality down by an estimated 30%”

  9. Good stuff, Anthony, and I see your colleague Peter Kellner has also been looking at the sample size. ICM, once they strip out the dont knows etc have a very small sample of 444 – which he says gives a margin of error of 5%.

  10. Using an “argument from normality”, Labour have a fairly steady lead of 7-8 points over a period of some days: normally such a lead does not vanish overnight, unless either something quite unusual has happened to abruptly change people’s minds (I didn’t notice anything big) or just possibly it wasn’t really there to begin with. It’s just that most other pollsters have been finding leads in the range 5-10% or so of late, so I don’t think the lead is illusionary.

    I do suspect that the Labour lead is slowly decreasing,
    but I doubt it will go down with a bump like this.

  11. There’s evidence to suggest UKIP have become somewhat less popular. Things are going a little better economically, and no things European are particularly high on the media agenda.

  12. According to the Guardisn, far from pinning blame on Labour for some imagined high death rate in the NHS, “the report will … contrast with much of the pre-publication coverage by stressing that mortality rates in all NHS hospitals have been falling for the past 10 years, with overall mortality down by an estimated 30%”

    -That may be what it says but The Health Secretary Hunt immediately,prior to considering the real problems , took the opportunity to try and pass the buck for the NHS onto the Governments predecessors.

    A disgraceful bit of partisan politicking in relation to a subject which is actually a matter of life and death

  13. Europe is going to shake things up a bit. The most likely outcome I seem to remember being Labour coming first, with UKIP second and Conservatives third. This would be pretty sensational.

    For one thing, it’s going to make Labour and UKIP both look good, which given Labour’s terrible performance in 2009 they probably need, and then there’s the fact that it’ll probably shift UKIP VI northwards, taking votes from the Tories.

    With a third place finish and sliding VI, the 1922 Committee would very possibly have the knives out and Cameron should watch his back.

    Also interesting is what will happen to the Lib Dems. Their voters have largely abandoned them at Westminster and locally, but given their europhilia it may be that some support remains for them there. At best they might hope to snatch third, but at worst their voters run to Labour as a safer Europhile vote and LDs have another South Shields moment.

  14. Angus Reid shows large Labour leads than other companies but during the last Labour government, it showed larger Tory leads than over companies.

    Could it be that Angus Reid over estimates reaction against the government of the day?

  15. Hunt brought to task by the Speaker and members of the Coalition for his reinterpretations of the findings of the Keogh review.

  16. As a long time reader, but rare poster, thank you for that article Anthony. It’s depressing to see so much media attention paid to one poll that doesn’t even need to be rogue to be misleading.

    But what has bothered me for a couple of years now is all the people arguing Labour needs a lead of X% at this stage in the parliament to have a hope of winning (Jonathan Friedland wrote today that even if we accept the Yougov polls as accurate it’s a disaster for Labour as they should be getting 20 point leads).

    My question is why would anyone expect Labour to have such high leads? Unlike almost any other election I can think of, the party in power, the Conservatives, promised economic pain as part of their campaign (essentially arguing that Labour created the economic crisis and only cuts can fix it). So the usual argument that the economy is weak and our living standards have been squeezed so the opposition should be racing into a massive lead doesn’t appear to apply. The Tories are doing exactly what they said they would, so their voters have no real reason to complain about that now.

    Yes, Lab got (or got back) the soft LD support after the coalition formed, which boosted their support, but the only other reason for a big Lab lead would be if UKIP surges again. And that doesn’t really have anything to do with Lab, their leadership, performance or policies. I’m sure they’d rejoice if the right wing fractures permanently, but I’m equally sure they aren’t counting on it, and would privately expect the Tories to land somewhere between 34-37% at the election, and will hope that their own 37-40% proves enough for a majority.

  17. To be fair Bercow rebuked Hunt for his failure to answer questions rather than make a stump speech, not for misinterpreting the Keogh report.

    I did think that intervention from the Lib Dems was interesting, though. That’s a pretty big rift (and perhaps preparation for coalition building in 2015).

  18. Why would Hunt have to resign? He hasn’t broken the ministerial code, and he’s acting on orders from Number 10. It’s really not his fault at all if Lynton Crosby has lousy judgement.

  19. As I’ve said in a couple of previous threads, I am trying to steer people gently back towards obeying the comments policy, which has been too widely ignored over the last few months.

    Last week comments were very well behaved and non-partisan, but a couple of unusual polls seems to have pushed a few people back into bad ways.

    Can I remind people that the comment policy is for non-partisan comments. A good rule of thumb is that to a passer-by your comments should not read as if they are from a Labour support, or a Conservative supporter or whatever.

    Can I again thank people who have made a real effort over the last week, and note those few who haven’t. Again, if you don’t want to post in a non-partisan way, please don’t spoil it for others. I would much rather people moderated themselves and the policy worked by self- restraint rather than me putting people who are not making the effort on pre-moderation. It’s an arse-ache for me moderating the comments and an inconvenience for others who have to wait for their comment to appear.

  20. Had to go back a couple of threads and remind myself of a prediction from one of UKPR’s resident pundits:

    “MUch better, in my view to leave it to Hunt, who has an understated & non-confrontational style. He will make the points tell-I think DC might throw them away.”

    Don’t give up the day job!

  21. Anthony, you might want to just post that warning at the top of every thread. It would save you time!

  22. @MrNameless

    Have Labour ever polled well in the European elections?
    The Greens got 25% in 1989.if UKIP is in the running it will take votes from Labour

  23. @AW

    Can I put whatever comment I want providing I end it with a lol, LOL :) or anything similar Anthony?

    Joking of course – that was just for my dear Musical friend Paulcroft – lol (sorry)

  24. They got 44% in 1994 (under Margaret Beckett), but I suppose not. They’re leading most European polls though with about 27%. I suspect they may come second.

  25. Sine – only if you want to particularly annoy me ;)

  26. Ok i will try hard to be a good Tory Boy!

  27. I won’t

  28. Sweltering workers should be sent home by law when temperatures get higher than 30C (86F) to prevent potentially fatal accidents, a group of MPs has said.

    -Right that’s me out of the office and into the Garden then have to shift the cat off the sun lounger!

  29. Steve

    Which group of mps has said that, that the daftest thing I’ve ever heard

  30. Steve

    Cats have their human rights too you know.

  31. @STEVE – lol

    So are you retired Steve or are you collecting the maximum £26,000 from myself and others? :)

    Joking of course my musical friend!

  32. It is often said that a lot of Labour supporters don’t go out to vote and most people don’t want another coalition. If the polls look close , it might bring these people out to vote and Labour could end up with a bigger majority than is currently being predicted.

  33. Tomorrow should be interesting at PMQs
    Both sides seem to think they have red meat,Tories with the NHS and Labour with Lynton Crozzer ,Friend of Big Baccy.
    Will either affect VI? Probably not as only 4% of people,max, take any interest in politics between elections. But not us Habitues of UKPR,no we love it!

  34. @LIZH – but on the other hand :)

  35. @SINE NOMINE

    Would you care to elaborate without being partisan?

  36. @LIZH – yes sure – the exact opposite to what you suggest could happen.
    ie; More voters might very well come out to vote in 2015 and vote accordingly to prevent another coalition but also to prevent another party from gaining power that just might undermine 5 years of torcherous hard/unpopular work.

    Now that interpretation is entirely up the individual to decide which party it might be.

  37. @STEVE
    “Sweltering workers should be sent home by law when temperatures get higher than 30C (86F) to prevent potentially fatal accidents, a group of MPs has said.”

    —————-

    Hmm… One’s first thought is to wonder how often temps get above 30 degrees in the Commons.

    And one’s second thought is that if it isn’t often, they’ll probably turn up the heating. Maybe they can claim an extra allowance…

  38. @Spearmint

    “Anthony, you might want to just post that warning at the top of every thread. It would save you time!”

    Just above the comments box:

    “NB: Before commenting please make sure you are familiar with the Comments Policy. UKPollingReport is a site for non-partisan discussion of polls.”

    We are told every time we make a post, but I’m sure it gets ignored / unnoticed.

    @LizH

    I think Sine was making the point that you were speculating to the good of your preferred party, rather than either way. :)

    @AW

    Not sure if I got an answer to this (I asked in a previous post)…if I try to calculate the MoE for the average of a set of polls (e.g. five YG polls), do I use the average of the five polls’ populations, or the sum of the five polls’ populations?

    Both seem logically reasonable.

  39. Don’t wanna give them ideas but the hot weather is a great opportunity for a new boomer air conditioning payment…

  40. @ RiN

    There’s some temperature when workers can put down their tool in the UK. It’s actually temperature measured at the place of work (rather than Met Office figures).

    There’s one in the UAE too, but oddly the weather office has always reported 0.2 Celsius less.

  41. Statgeek – the sum of the populations. But like I said, treat the margin of error as rule of thumb, not a strict formula, because it really is a bit detached from reality.

  42. Laszlo – are you sure? I can find Health & Safety Regulations setting out minimum temperatures, but not a maximum (only that it be reasonable) – http://www.hse.gov.uk/contact/faqs/temperature.htm

  43. @AW

    It’s an unenviable task, explaining polling, it would seem. Even MoE, intended to accommodate the natural variation you get when just polling a small sample of the population carries its own sting in the tail, because there’s a five percent chance even the MoE will be exceeded.

    Never mind Trafalgar, we need a lot more stats in the National Curriculum. (Though to be fair, Trafalgar did help rescue my History “O” level…)

  44. @SINE NOMINE @STATGEEK

    “I think Sine was making the point that you were speculating to the good of your preferred party, rather than either way.”

    I wasn’t really speculating. It is widely accepted that Tories get most of their supporters out to vote and Labour doesn’t.

  45. @LIZH – now I wonder just why that is Liz?

    […I don’t think that was really in the spirit on non-partisanship. And it tends to be socio-economic – people from more deprived backgrounds, less certain unemployment, lower income, etc, tend to be less likely to vote. Young people are also less likely to vote. All those things also make you more likely to vote Labour – AW]

  46. LizH – it’s socio-economic, and probably won’t change. When an election looks close turnout goes up, but probably does across the board. I doubt there’s a huge difference in how effective the parties are at getting their vote out in marginal seats. The big difference between Labour and Con turnout is comparing the inner cities with rural England. In safe Labour inner city seats turnout is extremely low, in rural Conservative shires people still vote even if it won’t make any real difference. Turnout in those super-safe seats probably would rise in a close election… but would make not a jot of difference to seats won.

    The bottom line is that it’s good for a party to have a big lead in the polls. Any rationalisation of how a small lead could be good because X is always a little bit straw-clutchy.

    Equally, most companies (YG being the main exception) do factor in likelihood to vote in their methods, so if a party’s supporters were becoming more likely to vote it would show up and be taken into account.

  47. You are right AW, about high work place temperatures.

    I used to be Shop Steward, and the place of work exceeds 100 C in the summer (120 C in parts).

    There is no H and S law that applies as far as I know.

  48. @ Anthony Wells

    Then it could be local collective agreements. In my workplace staff were watching the thermometer last year, because at 30 Celsius they were allowed to call it a day.

  49. Jeremy Hunt’s health initiative/ attack has probably been wasted. No normal people will be watching the news; they’ll likely all be outside.

  50. HSE guidelines (not law) gives between 13 and 30 as acceptable, however, it leaves the details to local agreements. I know one workplace in Manchester (small) where it is 26 degrees.

1 2 3 6