Whenever two high profile polls come out at the same time I see lots of confused comments asking about who is right, or how come the polls are showing such different things. Today we have the well publicised ICM poll showing Labour and the Conservatives neck-and-neck and the regular YouGov/Sun poll giving Labour a nine point lead. On the face of it these are very different results. How come?

Well, there are two main causes of variation in polls. The first is the differences in methodology between polls from different companies – are they online or telephone, how do they sample, how they prompt, what things they weight by, what they weight to, how they deal with turnout, what they do with people who say don’t know? All these things add up and have a party partisan effect, so one pollster may consistently produce figures that are better for Labour, or worse for the Lib Dems or whatever.

The second cause is normal random sample error. It comes from the fact that samples for polls are, to some degree, random. The margin of error is normally quoted as plus or minus three points, meaning that 95% of the time, the figures in the poll will be within 3 percentage points of the “true” figure. Now, in many ways this is a polite fiction – the formula assumes a genuine random sample, which no polls are, and ignore lots of other factors – but I would regard the plus or minus three points as just a good rule of thumb.

If you look at leads rather than shares in polls it is easy to forget just how large that 3 point margin of error is – imagine the real situation in the country was that the Labour party was on 38% and the Conservatives on 30%, and that the pollster conducting the poll had a method that was perfectly accurate. You would still get polls that has Labour varying from 41% to 35% and the Conservatives varying between 27% and 33% – that is, leads of between 2 or 14 points!

Anyway, let’s come back to today’s polls. ICM have a ZERO point Labour lead, YouGov have a NINE point lead. Let’s dissect that and see where the difference comes from.

Looking at YouGov first, because they conduct daily polls we actually have a pretty good handle of YouGov’s random error. If levels of support are actually pretty steady, the day-to-day ups-and-downs of YouGov’s tracker will mostly be down to normal variation within the margin of error. For example, the average Conservative scores in YouGov polls so far this month is a little over 31%. Below is how each day’s Conservative figure has deviated from that score.

So in four polls it was bang on average, in five further days it was one point above or below, on two days it was 2 points above or below. This is all what we would expect to see. We’d get a similar chart for Labour.

On average YouGov’s polls this month have been showing a Labour lead of SEVEN points, so I expect the sample variation in today’s YouGov poll is a bit on the Labour side, and two points of the difference in leads between ICM and YouGov is sample error in the YouGov poll.

Now let’s move onto methodology effects. Earlier this month I produced the chart below showing the different house effects of the main polling companies – the partisan effects each company’s methods have on their topline results.

As you can see, on average ICM show Labour leads that are about 3 points lower than those shown by YouGov. That’s an average – it too will vary from month to month. The cause is largely down to two differences between the companies polls – ICM weight their data according to how likely people say they are to actually vote. They also make some estimates about how people who say don’t know will actually end up voting, assuming that 50% will end up backing the party they did back in 2010. YouGov don’t do either of these things.

If we look at ICM’s detailled tables here we can see the effect these two approaches had. ICM started out with CON 33%, LAB 38%. After weighting it by how likely people said they were to vote this moved to CON 35%, LAB 37%. After reallocating don’t knows it became CON 36%, LAB 36%. There will be other, less tangible differences in methodology, but crudely speaking another five points of the nine point difference came from the different methodologies of ICM and YouGov.

That leaves 2 points difference to account for, which is all likelihood is down to random error in the ICM poll. Whereas we get 20 odd YouGov polls a month, we only get one from ICM so we don’t have the data to average out all ICM’s polling data this month and see if this particular poll is a bit too Labour or bit too Conservative. But because we can’t measure it, doesn’t mean sample error isn’t still there.

In summary, about half the difference between ICM and YouGov’s polls towards is down to methodological differences, about half is down to normal random sample error (YouGov a bit more Lab than usual, ICM probably a bit more Tory than usual). Case closed.

Except, it probably isn’t. You’re probably still asking “Which poll is right?”, is the Labour lead zero or nine? If polls are all over the shop, what possible use are they? The answer is that one single poll taken in isolation isn’t that much use, the solution is not to take them in isolation, look at the big broad picture, the underlying trends, the averages over many polls.

And, for the record, Labour have probably got a lead of around about 7 points, like it says up at the top right of the page.


299 Responses to “Zero points or nine points?”

1 4 5 6
  1. Two polls at 9 and 6. Average of 7.5, which is pretty much on the YG average.

  2. @SINE NOMINE
    ” I presume if we now had a Labour Government the drop in unemployment by 57,000 would be hailed as ‘encouraging figures’ ?”

    ———-

    Probably Labour would. But we can see through all the party spin can’t we and look at what the figures really mean objectively?

  3. @Richard – yes I’m sure you are right :)

    I’m sorry but which ever side you are sat on 57,000 drop is not bad under the circumstances.

    each bit of news is now slightly more encouraging – although its very slow its all in the right direction – at least we can all agree on that – unless certain people actually don’t want to see the stats pick up until the safe installation of a Labour Government – funny old world politics at times!

  4. @Sine

    It depends what you mean by “not bad”. If unemployment is only coming back down to the levels they inherited and via lower-paid, less secure jobs, how good is that really?

    If a surgeon mistakenly cut off your leg, then said “great news!! We’ve found another leg for you that’s not as good as the old leg” you might not be happinated.

    A fair assessment has to take all into account. Eg long term unemployment went up. Impact on VI will likely be affected by the fuller reality…

  5. Colin

    I read your last post which was very good concerning the failers within the NHS. I agree that most of this failer amongst a small number of NHS Hospitals can be attributed to poor management and dysfuntional staff who were strangers to the words care and compassion.
    As to the political attack by both sides of the argument both Labour and Tory care about the NHS and want to promote a positive image of care within the service,
    when things go wrong they both display a not me gov approach.

    I would suggest that Labour and there supporters are more sensitive to this behaviour because the see the NHS as something thay created and have spent a great deal of time and effort in attacking Tory cuts only to discover that a great deal of the bad things that happened during the last decade, were under their watch, even if the minister responsible was unaware of it.

    Having said that a great number of hospitals under Labours watch did improve under their time in office and it would be churlish to say different.

    The good thing to come out the report is we may heading towards a more open NHS were criticism is allowed and hopefully a more caring and compassionate service will emerge from those few poorly managed hospitals.
    And even health ministers realise that a relentless positive spin when things go wrong is only holding back improvements and covering the cracks in poor service, some hope eh.

  6. “I would suggest that Labour and there supporters are more sensitive to this behaviour because the see the NHS as something thay created and have spent a great deal of time and effort in attacking Tory cuts only to discover that a great deal of the bad things that happened during the last decade, were under their watch, even if the minister responsible was unaware of it.”

    ————

    Lol, Rich and Sine have been plugging this line for a while, Roland did it yesterday too…

  7. @CARFREW – Sorry but surely the level of unemployment has to come back down to the point they inherited before it can go even further down – common sense really!

  8. @CARFREW – and your point is?

  9. @Sine

    Yes, that is true. It just doesn’t undermine what I said…

  10. @Sine

    I just thought it was a bit ironic…

  11. @CARFREW – everyone knew unemployment would go up before coming back down again especially with the cuts in public (tax funded) services.
    Are you saying it should really have come down by over 100,000 in one month before things start to look a bit better.
    That wouldn’t happen under any government of any colour!

  12. @ RiN and @ Sine Nomine

    “Laszlo would have been equally scathing about the unemployment figures if there was a Labour govt as would I, he believes that the current status quo is a bad one”

    Thanks Richard – yes, this is the case. (and the status quo is a bad one in my view, the whole debate about unemployment benefit is a compromise between the fire and the fire engine).

    Sine Nomine: I There were two points conflated in my post. One is that there is hiring and as far as I know I haven’t denied that there is recovery (but I still think it will run out of steam by mid Autumn). The second is that in spite of hiring, the main reason for the drop in the unemployment figure is the drop in the economically active population. It can be due to various reasons.

    It is simple really: every month a number of people enters and leaves the labour market and there are also unemployed. In order to reduce unemployment, the number of hiring has to exceed the difference between the entrants and leavers. In the latest ONS figures the main reason for the drop in unemployment is because of the drop in the economically active population. I don’t attribute it, it is in the figures.

    I’m pretty unorthodox about unemployment and unemployment benefit. I’m on the Left, but I’m not a Labour Party supporter and I’m quite happy to recognise anybody’s contribution to solving the current economic (and social) problems.

  13. @Sine

    No I am not saying that and trying to put words into my mouth will not help any.

    How does knowing the government are going to make a lot of people redundant eliminate it as an issue? Are you happy with any Labour policies you know will hapoen????

  14. Sorry AW, could not resist. You have to admit it was quite a good soundbite joke from Ed .

  15. @CARFREW – actually do we know ANY firm Labour Party policies that may or may not be in the 2015 manifesto?

    Bit difficult really isn’t it to pin anything down as yet!

    I liked one or two things the last Labour Government did – but on the economy, I like many other people saw it slipping down the pan.

  16. @Sine

    Stuff on which we can broadly agree!! I suppose Labourites will point to various policies but I have to say that for me, overall it’s not all that clear.

    (I could have pointed to past Labour policies instead though!!).

    I wasn’t happy with Labour’s economic policy pre-crash either…

  17. All getting a little partisan again and completely off topic really.

    The point is “what effect is an announcement of a large drop in unemployment likely to have on VI?”

    I suggest that it will add to a general sense of things improving in 2013 – along with the good weather, Wimbledon, the rugby, the cricket, the Royal Baby, improving growth figures, rising house prices and a general lack of “Oh my God the world economy’s on the brink of collapse!” stories.

    That general sense of improvement (call it the feelgood factor) seems to have delivered a slight improvement in the fortunes of the two governing parties.

    It may well that you hate sport, are a republican, have an aversion to heat, worry about a new housing bubble and think the growth and unemployment figures should be even better. You may have looked at the economic news from China and the EU and thought that the seeds are sown for a future deterioration in Britain’s fortunes.

    You may even be absolutely right about all of those things. Not for me to say. But you have to ask yourself if your own perception is really that representative of the average UK voter.

    Rightly or wrongly I think most voters pay enough attention to hear the positive noises, but not enough to hear the nagging doubts in the background.

    Eventually events will turn against the government again. Perhaps over the NHS, perhaps Crosby and Fags, perhaps the economy. Perhaps just another government minister being found with his fingers in the till (or somewhere ruder).

    For now I think the wind is fairly well set for the government. I suspect the recent improvement will be sustained and possibily even built upon, at least until the end of the Summer. The question is whether the improvement is on course to deny Labour victory in 2015. Which none of us really know.

  18. NEIL A
    All getting a little partisan again and completely off topic really.

    ———

    Lol, you too Neil?!!!

    We were discussing the effect on VI and your account leaves out the stuff Sine and I were discussing.

    In terms of VI, you may not be happinated if you lose a good job only to have it replaced with something lesser. People who depend on your income or who care about you may not be happinated either. Local businesses in that depended on the public sector may also be negatively affected. People who lose services as a result of the redundancies may be somewhat peeved.

    This is before we get to the fairness issue. Or cost of living eclipsing wage growth etc.

    So for the significant improvement on VI, the impact of economic improvement has to outweigh all this.

  19. @sine @carfrew 57000 drop caused partly by 87000 increase in economically inactive. No. of unemployed over 1 year highest since 1996.

    Re:NHS – Last night Keogh told cross party MPs: the 14 trusts have reduced mortality stats by 30-50% since 2005.

    I think the Tories (or Lynton) have realised that the greatest risk in 2015 will be NHS. ‘vote Labour to save the NHS’ Whether current strategy of attacking Labour will work yet to be seen. Labour still polling ahead on NHS.

  20. To clarify- unemployed for more than one year…

  21. @CARFREW – nice to agree a little :)

    @NEIL A – Good read Neil – and quite fair :)

    @everyone else – I thought DC was in good form for the last PMQs today – barnstorming even if you didn’t quite agree with everything he said – still good – well done David!

    Now – this is a polling site so basically how will it all play out in the polls before everything goes quite with the summer recess – one wonders.

    I think there is enough ‘good or reasonably good news’ on the economy and unemployment figures to notch up the VI for the Conservatives by 1 or even 2 points.
    So we may see the Cons on a regular 32-34% during the summer.
    But all interesting stuff coupled with the NHS & Unions/ party payments etc to possibly affect the polls.
    Wait and see time now!

  22. Neil A

    Quite right, there is a perception that things are getting better and the media is keen to promote this with misleading headlines and as you say the general public don’t know or care about the whys and wherefores, but if as many of us believe it turns pear shaped again how large will the backlash be, it could be dangerous to raise expectations only to see them dashed again. Imo that is likely to happen because the current policy in both parties is to reignite the rampant credit creation which was at the heart of the 2008 crisis, many here on both sides want to see that happen, myself I don’t think it will be possible indeed I hope that it’s not because if we do manage to get one more credit ramp out of this system it will only make the inevitable crash worse, and it must come because it’s not possible to have permanently rising debt

  23. Other Howard- people perceive policies partly on who proposes them. Ashcroft calls it the Tory effect. Policies on certain issues likely to prove less popular if those polled are told proposed by Tories. (a few more popular). It is also a matter of trust: in whose interests are you acting? People like me or your millionaire backers? That is why certain policies end up being so toxic.

  24. IpsosMori poll

    LAB 40% (+5)
    CON 29% (-2)
    LIB DEM 10% (nc)
    UKIP 12% (nc)

  25. @sine
    ipsos Mori labour lead up 7 to 11. (28,39)- but Cameron rating up and Ed down.

  26. Wheres that poll just appeared from.

    Good for Labour and not so good for Cons if true – boy what a rollercoaster this week -lol

  27. @Richard in Norway

    I quite agree rising debt is not the way forward which is why i have always said that the present Government has failed economically. Having said that the alternative is much much worse.

  28. TURK

    THanks.

    I agree with your last para very much.

    I give Hunt huge credit for adopting the mantle of patient care & sticking to it so determinedly.

  29. ipsos mori looks remarkably like lab on the up in the teeth of a massive dogwhistle and misinformation assault.

    Backfiring?

  30. @Paul Croft

    No his just stating the obvious (IMO) of course.

  31. @Neil A

    “It may well that you hate sport, are a republican, have an aversion to heat, worry about a new housing bubble and think the growth and unemployment figures should be even better.”

    So you’re looking for a youngish, overweight, pasty cynic, with no love of the monarchy.

    Not many of those about. ;p

  32. @ RiN

    BoE hasn’t yet published private borrowing figures, but I’d be surprised if it hadn’t gone up (even excluding the housing policy ones that may work).

    @ Neil A

    The call was mildly partisan, but:

    – good weather
    – falling utilities bills
    – wage increases
    – cheap holidays

    Increase support to government, so do the hopes for these. The rest (GDP, unemployment, inflation) are only proxy indicators for these.

    The relative performance of the government is dependent on setting the agenda. The coalition hasn’t been great in this, I think mainly because of the strategic advise from the other end of the world.

  33. “I give Hunt huge credit for adopting the mantle of patient care & sticking to it so determinedly.”

    …whilst abandoning minimum alcohol pricing and plain wrappers for fags, thus ensuring the NHS will continue to get plenty of punters.

  34. @ Reg,

    I’ll take your bet.

    What are our stakes? I was going to suggest that the loser has to find something nice to say about the winner’s party leader, but IIRC you’re not a Nick Griffin fan so you might not go for that.

  35. @SINE NOMINE
    ” nice to agree a little :)”

    ————

    Indeed. (Not everyone seems to notice whenever one agrees with the right though!!)

    @Chatterclass
    Yep, there’s the headline figure, which may affect perceptions and hence VI… As AW often reminds us, this can sometimes turn out to be temporary fluff.

    Then there’s the actual overall impact…

  36. New thread in the long grass…

  37. SPEARMINT
    @ Reg,
    “I’ll take your bet.
    What are our stakes? I was going to suggest that the loser has to find something nice to say about the winner’s party leader, but IIRC you’re not a Nick Griffin fan so you might not go for that.”

    ————–

    I’m not sure Reg said who Labour had to have a lead over. May I suggest the Libdems?

    I think the winner should get to mod the site for a day!!

  38. AW removed the post that males sense of the long grass. But it doesn’t matter really.

  39. Ipsos Mori looks remarkably like Lab on the up in the teeth of MoE. :P

    @ Laszlo,

    Do you really think the Government- well, I say Government but I mean Lynton Crosby- haven’t been setting the agenda? Both the Unite business and this Keogh dust-up were battles of their choosing. Their problem is more that Labour keep fighting them to a draw even though they’re picking the battles (although I suppose the jury is still out on Keogh).

    Frankly I think it looks terrible for everyone- the Tories are behaving more like an opposition than a government and the Opposition who should be the ones choosing the battles are letting Crosby lead them around by the nose- but I don’t think the problem is that the Government aren’t setting the agenda. They just seem to be having trouble finding any accomplishments of their own to talk about, instead of what Labour are doing wrong. (The explanation for this, I will leave to the reader.)

  40. LASZLO

    @” the main reason for the drop in the unemployment figure is the drop in the economically active population.”

    I don’t think so.:-

    “Comparing March to May 2013 with a year earlier,
    • there were 336,000 more people in employment,
    • there were 72,000 fewer unemployed people, and
    • there were 144,000 fewer economically inactive people aged from 16 to 64.”

    ONS 17/7/2013

    Note the word “and”.

    Actually in the last quarter, there was a rise in the number of economically inactive people :-

    “For March to May 2013, compared with December 2012 to February 2013,
    • there was a small increase in the number of employed people,( 16,000)
    • there was a fall in the number of unemployed people,(57,000) and
    • there was a rise in the number of economically inactive people aged from 16 to 64.(87,000)”

    ONS 17/7/2013

    …..again note the word “and”.

  41. My prediction for tonights/tomorrows Sun YouGov poll is:
    Con 33%
    Lab 38%
    LD 11%
    UKIP 10%

    Just for fun – anyone care to join me?

  42. Well l got the topics (and likely outcome lMO)of PMQ’s spats pretty much bang on.
    Good for me.

    That latest poll must knock the gilt off the gingerbread for Cammo ,it’s the silly season for the next two months and then the Rebeckah,Andy and Rupert show opening in late September….

  43. YouGov prediction?

    Con 29
    Lab 43
    UKIP 11
    LD 10

  44. I forgot the glory of your somewhat optimistic predictions Nick :)

  45. He started it.

  46. Dear Anthony,

    I notice that the swingometer is based on 600 seats, which is now clearly wrong, given that there the 2013 boundary review has been cancelled and nothing will happen until after the 2015 GE. If the swingometer comes under your remit, can it be changed?

    Yours, Chris Mockler

  47. Chris – um, it isn’t! There’s still a calculator up there for the provisional boundaries (which I guess I’ll take away at some point), but there’s also one for the current boundaries, and the two graphical ones are for the current boundaries

  48. Your first diagram is instructive. If you assume that the conservative % is steady over the period, then the successive poll results are just what you would expect if making a measurement by a method with a precision of +/- 3%. It is then fair to treat the results as repeated measurements of the same thing with a measure of that precision, and then combine the results to produce a value of 31 +/- 1.2%.
    But that abandons all hope of detecting any real changes of less than 3% in the Conservative share with any single poll. Similarly for Labour.
    With the error in the difference being sqrt(3×3 + 3×3) = 4.2% then any change in the difference which is less than that can be accounted for by the random errors in the polling.
    If you further assume that the Comres error in an individual poll is 5%, then the error in their difference figure is sqrt(5×5 + 5×5) = +/-7%.
    Thus the difference in the two polls of 11% is within the combined margin of error of the two polls. Not only that, such a conjunction of one poll at the upper end of its range, and the other at the lower end of its range, is bound to occur from time to time.
    So “move along there, nothing to see here”. An 11% difference is not only possible, it is from time to time inevitable, even with no actual change in either C. or L. share.

  49. If the average “true”poll lead of labour over the conservatives is currently only 7% then it will be interesting to see what impact if any the conservatives get from the feelgood factors likely to be around in the next couple of months – royal baby / England demolishing the Aussies and potentially falling unemployment. Given your margin for error comments there will be polls that show labour behind.

    It look as if Ed Milliband is actually going to have to do something positive rather than concentrating on doing nothing wrong if he is to win the next election.

1 4 5 6