On margins of error

Last night’s YouGov poll had topline figures of CON 35%, LAB 41%, LDEM 9%, UKIP 7%. The Labour lead of 6 points is unusually low, but as ever, this in itself doesn’t actually mean anything. A series of low Labour leads is meaningful, it would show a narrowing of the gap. Just one or two can be explained through normal sample variation.

I say this repeatedly – yet anytime there is a poll that is an outlier from the average I see Twitter filled with otherwise quite sensible people speculating on what might have caused the movement. The answer is almost always “normal sample error”.

I think part of the problem is people forget the degree of normal volatility we should expect from polls. The quoted margin of error for polls is normally plus or minus 3 percent. This is actually quite tenuous – the 3 point margin of error refers to a genuine random sample, when actual polls are never purely random (most involve some elements of stratification or quota sampling, and even an attempt at a random sample wouldn’t be random because of non-response), there are design and weighting factors, actual voting intentions are based on smaller samples once won’t votes are excluded and so on. However, taking plus or minus 3 points as the margin of error is a good enough estimate for our purposes.

If we look at YouGov’s fourteen polls so far this month, the average Conservative score is 33. All fourteen polls have been within 2 points of this. Eleven have been within 1 point of this.

The average Labour score has been 43. All fourteen polls have been within 3 points of this, and thirteen of them have been within 2 points of this. Eight have been within 1 point.

The average Lib Dem score has been 9 points. All fourteen polls have been within 2 points of this, thirteen have been within one point of it.

In other words, while there have been polls showing twelve point leads and polls showing six point leads, all of the polls have actually been within the normal margin of error of CON 33, LAB 43, LDEM 9 and the distribution around that has been very much what you would expect from normal sample variation.

115 Responses to “On margins of error”

1 2 3
  1. @Colin

    The revised 2011/12 deficit of £119bn is now £39 bn less than the Governments’s legacy deficit of £158 bn for 2009/10.


    Which goes to show people, how much easier it was to reduce the deficit in the earlier days when the cutting thing hadn’t gone as far and they still had Labour’s growth.

    But now they’ve tanked the economy, corporation tax receipts down, welfare bills up… because the cuts don’t cut the deficit as much as cut our income and increase expenses like welfare. Still, at least there’s only eighty percent of the cuts to go…

  2. @Socal – “I think it’s funny that that’s the outrageous and offending word here.”

    I think you’ve missed something here. ‘Pleb’ has become the central point of the debate as mainstream media outlets over here don’t air the work ‘[email protected]’. When the BBC first reported this on the evening news, they didn’t even refer to any swear words, leaving ‘plebs’ as the only offending item.

    And – “Now that all these cuts have happened….” – minor technical point, but they haven’t. I understand 80% of the cuts remain to be enacted.

    I think your central point is valid though. We are seeing the pain, but the central economic and political objective (the deficit) is heading in the wrong direction. This is where Cameron and Osborne made a serious strategic error in my view, when they defined themselves purely in terms of the deficit.

    Apart from being a complex economic issue on which to base a party identity, if if goes well, you’re left with nothing much to promote, and if it goes badly, you’ve defined yourself as a failure. [snip]

  3. This ‘pleb’ comment is interesting as you can read across with what’s happening with Romney’s ‘appraisal’ of 47% of his fellow US citizens.

    There does seem to be a contempt for the majority of the populace underpinning some of these atitudes from the right. It used to be the lumpen proletariat that the elite were in equal measures scared of/despises. Now it’s most of the hard working electorate.

    That’s the perception anyhow, and as the dark lord says…


    @”Which goes to show people, how much easier it was to reduce the deficit in the earlier days when the cutting thing hadn’t gone as far and they still had Labour’s growth.”

    Does it?

    It might show all sorts of things-some of which might be totally unconnected to your “Labour growth” thing.

    That’s the thing about the numbers thing.

  5. @Martyn

    Thanks for your comments about Hunter S…. and for seeing the mental image of him gnawing the top off a pill bottle. Nixon popping pills might explain the 20 hour working days, the paranoia. “He has enough stamina to be President” as William P Rogers said of him when Nixon was standing against George Romney for the GOP nomination.

    I did find The Generation of Swine in a remaindered bookshop down the Charring X Rd a few years ago. I agree about snapping out of a digression being the best part sometimes. I actually came to genuine Gonzo journalism late in life and then realised where all the milk-sop imitations had come from. Reading The Kentucky Derby is Decadent and Depraved for the first time was a series of disorienting double takes…this is ‘literature’ and someone printed it?

    What can you say… he had discipline in the army, and they had the sense to push him towards a successful career in sports journalism after a chaotic and delinquent upbringing. Then the sixties was going on, and again he was a nasty brutish reflection for their counter-culture, but he was going to cane their stash for all he was worth. It was going to end badly and he knew it.

    I agree with you substantially, but the comedown scene on the plane in Fear and Loathing, when they are trying to ingest everything remaining in the holdall always makes me laugh. Maybe It would have done Nixon good to have gone on a crazy sideways spree like that just once.

    Thanks again for your reply.

  6. @Carfrew

    “Why do Tories keep going on about mid-term blues and reclaiming protest votes come the election? Eh?? What mid-term blues? What Protest vote? They’re only down about 3% on the election. It’s kinda hard to suffer a significant protest vote when you’re not far off core vote to begin with. Even if they recover all their “protest vote”, they’re still only back to 36% anyway.”

    A very good point and one that has occurred to me too. It’s arguable, such were the highly propitious political circumstances that existed for them in May 2010, that the Conservatives dredged up just about all they can really hope for in the modern era in terms of popular vote. Let’s put it this way; if you were a potential and/or likely Tory voter at the last General Election, what excuse could you possibly have had to vote for another party or sit on your hands? There wasn’t one, was there, so it’s a reasonable assumption that the Tory army in its entirety turned out and that 36-37% is just about as good as it’s ever going to get for them.

    Now, depending who you believe, they’ve lost between 3%(ICM and YouGov) or 6% (Populus, Survation, IPSO/MORI etc etc) since May 2010; not exactly mid-term desertion, more a slight erosion at the core.

    As some others have already observed, what sort of political earthquake is going to occur between now and May 2015 that’s going to float them back up to anywhere near 40% of the popular vote?

  7. Alex

    Re: Deficits and Bond Rates.

    I don’t claim to be expert on the causal effects, but I can check historical data like the next person.

    And the last decade’s worth of data sys to me that, for large countries with a sovereign currency, there is no correlation whatsoever between Govt debt and bond rates. None at all.

    That much has been true of Japan for 20 years – its bond rates have been almost 1-to-1 inversely proportional to it’s exploding debt. But Japan is a special case of course with relatively little dependence on the international bond markets because if it’s strong domestic savings.

    But it’s also true of the US over the last decade. There has been a consistent fall in their 10yr bond yields for a decade, even as their debt has exploded. But of course the US is a special case, being the world reserve currency.

    But it’s also true of Canada. Over the last 20 years their Govt debt grew, fell then grew again. Their bond rates steadily fell throughout that period.

    And it’s true of the UK. As our deficit exploded in 08 with no sign at that time of a plan to reel it in, our bond rates continued their decade long fall, with barely a blip.

    I don’t claim to understand the mechanisms, but the concept that debt=bond Armageddon is demonstrably untrue. Which makes a political programme based on that equation questionable at best.

  8. Sys=suggest

  9. @Crossbat11,
    ‘Let’s put it this way; if you were a potential and/or likely Tory voter at the last General Election, what excuse could you possibly have had to vote for another party or sit on your hands? There wasn’t one, was there, so it’s a reasonable assumption that the Tory army in its entirety turned out and that 36-37% is just about as good as it’s ever going to get for them’

    I am not quite with you on this because the polls at the time did suggest that ‘Cleggmania’ did hit the Tory vote as well as Labour.Had there been no debates in 2010 it’s quite likely the Tories would have managed 39 – 40%.Moreover, it may be one reason the Tories will not be too upset if there are no debates next time!

  10. “Pretty poorly for a mid term opposition. Cameron had 32% and didnt win outright.”
    If we’re referring to the lead.. erm.. when?

    The highest lead I can find, just after the crash, is YouGov/The Sun with a 26% lead.
    But that was after the largest financial crisis since the 1930s.

    But the interesting thing about these massive leads is that the Conservatives never topped 49%, most of the 2008 massive leads were in the 45-47 VI range for Cons – so 4-ish% above Labour now.
    If we were to increase Lab’s share by 4% (to put it in the 47% high) and reduce Cons by 4%, that’d only give a lead of 18% – 4% behind the Con 2008 leads when they were on 47%.
    So the Con leads were also down to Lab unpopularity, not Con popularity.

    But 2009 leads, which shrunk to somewhere between 14-19% – still whopping leads compared to Labour’s current leads.. still only had Cons on 37-42% VI (the 19% had Cons on 41%).
    So again, due to Lab unpopularity, not Con popularity.

    So even comparing to just Cameron leads us to ‘It’s more complex than that’, as opposed to just looking purely at leads.

    What also makes things different now is that Labour haven’t really benefited from Con>Lab transfer, as would have been required if both Labour and LibDems were in opposition (say Cons had scraped a majority).
    If that were the case, we’d probably see both Cons and Lab on low 30s at this point.
    But we’re dealing massively with Lib>Lab voter transfer, which has completely changed the context of the next general election.

    So assuming that Miliband is holding back the Con>Lab transfer (i.e he’s Kinnock), then Labour would be on much bigger leads now.
    But I think that’s a massive assumption given the collapse in Labour support in 2010 to historically low levels – I think a lack of trust in them is probably holding them back.

    So “context is key” – Labour may win the next GE ‘by default’ because of a united centre-left, rather than what they’d have to do traditionally to win. [1]

    [1] This is, of course, very dangerous in the long-term for Labour if the LibDems (or other centre-left third party) emerge to claim the centre-left.

  11. @Graham

    I agree to some extent about the TV Debate induced (first one, less sure of other two) Cleggmania effect in May 2010, but it subsided to a greater or lesser extent as the campaign went on and, by the actual polling day, I’m not sure how big a factor it really was. Were there that many natural or potential Tory voters amongst the Lib Dems by polling day? Possibly, but I’d tend to call them floating voters who are potentially available to all or none of the other parties. Whatever it was, they obviously couldn’t bring themselves to vote for the main Opposition/alternative Government party come election day. With a particularly unpopular incumbent Government in place at the time, that was extraordinary and unprecedented.

    All this feeds into the polling evidence that the Tories have by far the smallest pool of potential voters to call upon. I wonder if things like the recent Mitchell incident make that pool even smaller?

  12. @Crossbat11,

    I agree with your central point, but I also believe that some voters who early in the 2010 campaign were inclined to switch direct from Labour to Tory in the end – largely due to Cleggmania – stopped halfway and voted LibDem.

  13. ALEC

    @”The sanguine reaction today may also reflect a general belief that Osborne will do what is necessary, so perhaps credit needs to go to Osborne for giving markets sufficient confidence to ride through the borrowing level disappointments.”

    Bloomberg recently surveyed a bunch of UK debt buyers.

    These are some quotes from their report :-

    “Investors whose companies manage $5.2 trillion said they may keep buying British debt even if Cameron pulls back from the largest budget reduction since World War II, according to a Bloomberg News survey. U.K. bonds returned 26 percent since he took office in May 2010, beating a 14.8 percent gain from Treasuries and bunds’ 14.5 percent advance, based on Bank of America Merrill Lynch indexes….

    ….“The key is to have a clear sense that there’s a credible effort to put the U.K. fiscal situation back on an even keel,” Adrian Grey, a fund manager at Insight Investment Ltd., who helps oversee $295 billion, said in a phone interview. “Over time, that has to involve retrenchment, but it doesn’t have to be done by next week. If it’s one or two years more on the back end, and there’s a clear indication of the reason for spending more, that’s not a big problem.”…

    …..Grey was one of 16 investors, among the biggest owners of gilts, who participated in the survey. Ten said a shift in government strategy on the fiscal plan may not trigger a decision to reduce holdings. Six of the respondents, who oversee less than $1 trillion, said they’d sell on a change in plan……

    …..“The government was right to set out a multiyear plan to deal with the deficit,” said Michael Amey, a portfolio manager at Pacific Investment Management Co. in London. “Now that growth has continued to disappoint, there is a good case for higher spending in public-sector investment.”…

    …..“A change in stance could lead to an increase in risk premia and a loss of confidence by markets,” said Dagmar Dvorak, a director of fixed-income and currencies at Baring Asset Management in London, which has $50 billion. “I would probably see it as a trigger to reduce my exposure.”….

    …..“If the government were to loosen the pace of fiscal consolidation, that would ordinarily be met by an increase in the supply of gilts,” Mitul Patel, a fund manager at Henderson Global Investors Ltd. in London, which has $103 billion under management, said in a phone interview. “However, if the BOE are also buying significant quantities of government bonds at the same time, that would probably limit the extent of the rise in gilt yields.”….

    …..“U.K. fiscal policy has little bearing on gilt performance” and the global economic outlook is among the biggest reasons to change position, said Michael Riddel, a London-based fund manager at M&G Group Plc, which oversees about $324 billion….”


    There seems to be a range of views-there is no definitive “Bond Market” view-but insofar as it exists, it is the importance of the existence of a deficit reduction plan, with continuing intent & commitment, but appropriate flexibility.
    The role of QE in supporting demand for UK gilts has also been a significant factor-and one can only presume that , as GO continues to let automatic stabilisers & extended time frames compensate for slow growth, BoE will continue with an active monetary policy.

  14. @ Colin

    “However, if the BOE are also buying significant quantities of government bonds at the same time, that would probably limit the extent of the rise in gilt yields.”….
    That’s the salient point from all those you’ve selected regarding why the market is holding up.

  15. Which is why there is no possibility of an exit from QE

1 2 3