On margins of error

Last night’s YouGov poll had topline figures of CON 35%, LAB 41%, LDEM 9%, UKIP 7%. The Labour lead of 6 points is unusually low, but as ever, this in itself doesn’t actually mean anything. A series of low Labour leads is meaningful, it would show a narrowing of the gap. Just one or two can be explained through normal sample variation.

I say this repeatedly – yet anytime there is a poll that is an outlier from the average I see Twitter filled with otherwise quite sensible people speculating on what might have caused the movement. The answer is almost always “normal sample error”.

I think part of the problem is people forget the degree of normal volatility we should expect from polls. The quoted margin of error for polls is normally plus or minus 3 percent. This is actually quite tenuous – the 3 point margin of error refers to a genuine random sample, when actual polls are never purely random (most involve some elements of stratification or quota sampling, and even an attempt at a random sample wouldn’t be random because of non-response), there are design and weighting factors, actual voting intentions are based on smaller samples once won’t votes are excluded and so on. However, taking plus or minus 3 points as the margin of error is a good enough estimate for our purposes.

If we look at YouGov’s fourteen polls so far this month, the average Conservative score is 33. All fourteen polls have been within 2 points of this. Eleven have been within 1 point of this.

The average Labour score has been 43. All fourteen polls have been within 3 points of this, and thirteen of them have been within 2 points of this. Eight have been within 1 point.

The average Lib Dem score has been 9 points. All fourteen polls have been within 2 points of this, thirteen have been within one point of it.

In other words, while there have been polls showing twelve point leads and polls showing six point leads, all of the polls have actually been within the normal margin of error of CON 33, LAB 43, LDEM 9 and the distribution around that has been very much what you would expect from normal sample variation.


115 Responses to “On margins of error”

1 2 3
  1. A 10 point Lab lead looks pretty solid, at the mo.

    DC has tested the effect of a harder line on EU with his veto that wasn’t a veto last December. Surely, DC and the Cons must be tempted to adopt a more ant-EU line and secure some if not all of the UKIP votes as Con votes?

  2. How does this lead for Labour compare historically? Didn’t both Edward Heath and Tony Blair have leads of 30%+?

  3. Pretty poorly for a mid term opposition. Cameron had 32% and didnt win outright. Blair had over 40 iirc.

  4. @AW

    It’s amazing how such basic maths needs constant restating.

    One thing to add, of course, is that the Lab and Con VI are not independent values. A lower than average Lab VI is very likely to be accompanied by a higher than average Con VI. So the moe for the Labour lead is more like 6%.

  5. Kinnock had a 25 point lead midterm. 15 points maximum is very low.

  6. “I see Twitter filled with otherwise quite sensible people” – I think that’s a variant on the oxymoron. :D

    Anyway reported from previous thread as relevant:

    There are two problems here. The first, as Anthony implies but is too polite to say, is that by fixating on the lead rather than the individual Party poll rating, people effectively double the normal random variation. This then gives them an excuse to go round proclaiming that the sky has fallen in or UFOs have landed or whatever. No, it’s just the gentle ebb and flow of sampling mathematics.

    But there is another recurring problem, which Colin has already picked up on, which is our old friend Da Yoof. Normally Labour have their strongest lead in the under-25s but in this sample the Conservatives lead 40% to 35%. The 6% for the BNP looks a bit unusual too.

    Of course this can work both ways. If you look at yesterday’s poll:

    http://d25d2506sfb94s.cloudfront.net/cumulus_uploads/document/p9pvi7r9ky/YG-Archives-Pol-Sun-results-190912.pdf

    the Labour-Tory split for 18-24s was 71% to 16%. This is probably as bad the other way (I think YouGov did do a decent size sample of just their under-25s a while back, but forgot to note the results of the VI).

    As I explained the last time we got a 6 point lead (and obviously no one took a blind bit of notice) YouGov’s under-25 sample is particularly volatile for a number of reasons and has a disproportionate effect on the headline VI. I suspect that the sampling people are playing about with their algorithms trying to minimise this and this may lead to even greater short-term variation as they try out various solutions. Today’s sample is actually quite large for the Under-25s, but of course it hay not be representative of them.

  7. There are several factors being overlooked., IMO…

    – this is a new govt, and we are only into its third year
    – this is a coalition, so comparisons with past govt and VI may not be appropriate
    – perhaps the LDs have suffered more than their govt partners
    – Lab left govt with an historically low VI.

  8. …and of course no modern party has increased its share of the vote while in government (SNP excepted, which I would argue is a special case).

  9. @Mike N – “A 10 point Lab lead looks pretty solid…”

    Correct. Looking at Con/Lab VI, Con has perhaps picked up touch since the hieght of the omnishambles, and Lab a touch more, but it is the lead is which captures attention.

    Looking back over the last three months, in two week chuncks on YouGov:

    9.4% > 9.5% > 10.1% > 9.3% > 10.1% > 10.1%.

    It is noticable that it is ten weeks since we saw an instance of a 7 point lead, and five weeks since we saw an 8.

    YouGov is painting a perfectly beastly picture when you join the dots to include those three 6s.

  10. Other issue I’ve noticed about polls is, even when there are shifts in the polls outside of margins of error, they can still be pretty random with no apparent connection to any particular event. Then it can equally randomly shift the other way.

    I generally wait for changes in polls leads about about 10% before I attempt to put a change in voting patterns down to any reaction to a specific development. Most of the time, all polls show is how arbitrary people are when asked the question.

  11. Those who constantly refer to 20+ leads enjoyed by Oppositions in years gone by appear to fail to take account of the changes in methodologies in the intervening years. I would suggest that if polling companies adhered to the methods used up to the mid-1990s several would by now have come up with 20% Labour leads.

  12. In North Korea the polls are always spot on!!

  13. Interesting to see reaction to Andrew Mitchell’s ‘[email protected] plebs’ outburst at Downing Street police. All in all, even his friends are saying it’s a bit of a disaster, and plays right back into the image of a party of entitlement with little regard to the plebian population around them. Not good, and a terrible image setter for the Tories.

    Meanwhile, further bad news on the deficit, with the August figures showing no reduction from the record 2011 figures. The only bright spot was a £7b reduction in the estimate for the full year deficit 2010/11. While welcome, this arguably shows that Labour policies were better/less bad for the deficit, and that as these have unwound and coalition policies increasingly kick in, the deficit is heading in the other direction.

    Of course, there are many different factors to take account of and much partisan point scoring to be done, but it’s increasingly clear that Osborne’s targets will be missed and he will not be able to present himself as a successful deficit cutter to anything like the extent he had hoped by 2015.

    Quite how damaging this is politically remains to be seen. I’m not altogether sure how much voters as a whole will bother about the deficit, but the political impact may well be more related to the limited room for maneuver on stimulus and tax cut measures that Osborne finds in the run up to the poll. Altogether it’s shaping up as another bad day for the government.

    However, even with yesterday’s poor retail sales figures for August, there are many forecasters now predicting Q3 GD growth – some of these posting really quite bullish predictions. I’m not so sure, but perhaps better news is on the way.

  14. NICKP

    “and of course no modern party has increased its share of the vote while in government (SNP excepted, which I would argue is a special case)”
    ____________

    But the UK government is just that…a special case!!
    Coalition governments are not the norm for Westminster so when the Conservatives (although in government) ditch the Lib/Dems in 2014, in all probability they will increase their share of the vote from 2010 which in its self was a weird election because of the banking issues etc!

  15. @Mike N – I would pick up your first point ( this is a new government) as being particularly significant.

    What I have found highly significant (I won’t say surprising – I haven’t been in the least surprised by this, and indeed predicted as much pre GE) is the fact that there is already a sense of drift evident within and around government.

    In recent weeks there has been a rash of articles and comment along the lines of ‘what is the point of David Cameron’, and with one or two notable policy area exceptions, there is a clear sense of a government that has run out of ideas. I hasten to add that this analysis is not a partisan point on my part – I’m simply regurgitating comments that have come mainly from friends of the government and internal Tory party commentators.

    This is the kind of condition you expect from a second, or more likely third term government, yet here we have it on the half way stage of the first term.

    I really don’t see this as simple mid term blues. Many first term governments were deeply unpopular at times, but I can’t recall any of them having such a tired and lacklustre appearance. Sections of the electorate may have disliked them intensely, but most were able to maintain a sense of political direction.

    I don’t feel this is a result of coalition, but much more a comment on the Tory party and Cameron as a leader.His own backbenchers still don’t know what he stands for and they give the appearance of a rudderless vessel bobbing about in a storm.

    Perhaps this is the last hurrah for the Tories as a party of government – who knows – but the smell of decay is already beginning to attached itself to this administration, even if the polling deficit appears relatively modest in historic terms.

  16. @Graham – “… if polling companies adhered to the methods used up to the mid-1990s several would by now have come up with 20% Labour leads.”

    I share that suspicion.

    I have seen references on other sites to a Peter Kellner article for YouGov, which laboured the point that if opposition parties don’t chalk up 20+% leads, they do not go on to win a general election. I haven’t tracked down the article in question, so don’t know if that is really what he said, or whether he attached any particular significance to methodological changes.

    The 1992 GE would be a example of polling companies on average overestimating Lab (+5%), and underestimating Con (-4%):

    h
    ttp://www.research-live.com/comment/general-election-2010-did-the-opinion-polls-flatter-to-deceive?/4003088.article

  17. I’m amazed that people are still comparing the polls today with those of yesteryear, the methadology has altered considerably since then.

    On top of that there is a diversity of political parties to choose from, UKIP, SNP etc that didn’t exist then, the rise of the ‘others’ will temper the lead the major opposition party has.

    The Mitchell outburst will undoubtably have some impact, (short term) because of the contrast with the way that the public is feeling such sympathy with the police over the tragic deaths of the two WPC’s. as gaffes go its a whoppah.

  18. Alec
    I’ve tried to draft several replies to your quite reasonable post but every time I drift in partisanship.

    [Snip]

  19. We should really consider binging a class action for dereliction of duty against bystanders outside Downing Street if none of them had their iphones at the ready.

    Guys and gals in the control room on the other hand will be having a good old laugh watching it on continous loop.

  20. All this stuff about historic mid-term leads is really silly. There is no precedent for the current situation and for Labour to have a consistent lead just over two years into this Government, given where they started from in 2010, is remarkable.

    From being around half of the combined con/ld vote at the last GE they are now generally level-pegging or ahead.

    Isn’t that enough?

  21. Leak from the police or security due soon?

    (Rubs hands with glee at thought)

    I don’t think Mitchell’s outburst (alleged) will affect voting intention one jot but it makes him look silly.

  22. paulcroft

    What is interesting is the abrupt departure of the anti-Tory vote from LD to Lab (almost) immediately the coaltion was formed.

    You might speculate about that happening but to see the polls reflect it so clearly is fascinating. If we now see the Tory vote split with UKIP we could see the vote on the right split three ways. Tory, Lib and UKIP and the left/anti-Tory line up with Lab.

    I don’t think many would have predicted it happening so starkly.

    Of course we haven’t had a general election, so it’s quite possible that it won’t happen at all!

  23. Yes, I too have had the impression that we are back to a taste of the mid-90’s situation with Major, or the last year or two of the last Labour government, the feeling of waiting for something to happen, rather than making it happen.

    About the polling deficit – some of those historical mid-term leads were a bit silly, in hindsight. I suppose they included many of those who were temporarily so fed up with the incumbent that they would choose to vote for someone else, without considering too deeply what the alternative was. I think these people were only marginally inclined to vote for their new (and temporary) favourite.

    When elections then became closer, magically the lead then dropped because these people started to think a bit harder and switch back. I like to think that the current polling leads accounts for some of these “temporary supporters”, and thus we get a figure that is lower, but less likely to sink when the GE gets closer.

  24. Well Plan A surely merits a very special in political and economic history.

    I remember the words of a former PM describing the Chancellor as “brilliant, brilliant, brilliant”.

  25. very special “place”

  26. In 1997 the vote was split 43/31/17

    It’s possible that because of the collapse of the LD vote we could see Milliband the un PM-like geek with nasal whine score bigger than that.

    I might be indulging in wishful thinking, but I can only see Labour’s lead growing and Con’s falling from now on.

    Biggest landslide ever?

  27. Alec
    I will have a go at replying within the boundaries regarding Mitchell.

    I am a retired Met Police Officer and have on occasions performed duties at Downing Street and Parliament.
    I have also worked in some of the most troubled areas of inner London and consequently have had my fair share of abuse both verbal and physical.

    There are enough (redacted) in the Real World without the inhabitants of Westminster joining in.

    What a (Doubly redacted)

  28. I have to say that Mitchell was one of the few leading Tories I had a bit of time for and one of the last that I thought would behave in the way that he allegedly has. I don’t know him that well, although he’s a local MP in Sutton Coldfield, but when I’ve seen him interviewed, more often than not in his former role as Secretary of State for International Development, he’s usually come across quite sympathetically. I thought he appeared to be a fully paid up member of the human race and was doing some good work in his ministerial role.

    However, if he is guilty as charged, then it does show him in a particularly poor light. I don’t quite go along with Jim Murphy’s twitter reaction, but Mitchell’s behaviour does play into the image of boorish arrogance that is often associated with the Tory Party. To some extent it’s the stuff of Cameron’s nightmares considering the assiduous work he’s done to try and detoxify his party’s brand.

    In the light of the surge of sympathy for police officers in the wake of the fatal shootings in Manchester earlier this week, not a good time to for a Cabinet Minister to be seen publicly belittling a member of the force, I fear.

  29. @Crossbat – I too had viewed Mitchell as one of ‘the good guys’ but the stream of commentary on this tells a different story. His record is patchy apparently, with his friends admitting that his leadership of Davis’ leadership campaign was a disaster and he also has a long standing reputation for boorish rudeness. He’s a former banker who lives a variously extravagant lifestyle (according to some) so there was much amusement within Tory circles at his recent positioning as champion of the poor. Some of the things done with the aid budget recently have tended towards business investment, rather than aid, and I suspect this is Mitchell’s doing.

    These incidents fall into the politics box marked ‘Deeply Funny’, and I hope people on all sides can laugh at the utter insanity of it all. It ranks alongside the Mrs Bigot episode, and while all parties occasionally go through this, it remains as entertaining as ever.

    I’m considering establishing a new political party called The [email protected] Plebs Party. Might even invite Mitchel to be our honorary chair.

  30. This poll is clearly a bit of an outlier – as was the recent 15% Populus poll. Overall Labour lead for Yougov is probably around 10%.

  31. “I’m considering establishing a new political party called The [email protected] Plebs Party”

    Count on my vote.

  32. Alec:

    Sounds fun: have you already got some lovely Plebs lined up or is it BYO??

  33. I sense this Mitchell story will not fade away quickly. I actually think this episode is very damaging.

    It is damaging for DC if he does not dismiss Mitchell from the role of Chief Whip. However, this is an opportunity for DC to be seen by joe public as in control, butch and much more importantly cognisant of the fact that politicians are servants of the public. Failure to dismiss Mitchell will encourage the view that Tories see workers as beneath them.

  34. I wonder if Boris still thinks swearing at a police officer should be an offence, as he stated at party conference http://m.guardian.co.uk/politics/2011/oct/04/boris-johnson-people-swearing-police-arrested?cat=politics&type=article

  35. Some people might even be tempted to think that the class war continues.

  36. Not sure the Mitchell story is that big…For one,there is no video footage.And to think that Mitchell will be fired for this is just wishful thinking.Bigger fish have done far worse in this administration and been promoted for it

  37. Interesting that if a ‘posh bloke’ calls a policeman something it’s reprehensible, but anyone else can make fun of them.

    Yup, the class war continues. :)

  38. A labour of Sisyphus Anthony, respect all the same.

    :)

  39. Mitchell story no big deal.
    DC might be secretly pleased that a DD backer has hit a little trouble

  40. Statgeek
    “…anyone else can make fun of them.”
    Is “them” policemen or “posh blokes”?

    “Yup, the class war continues”
    Aye, so you’re one of those “tempted to think” to whom I was referring?

    And in the spirit of things…;-)

  41. ALEC

    @”The only bright spot was a £7b reduction in the estimate for the full year deficit 2010/11. While welcome, this arguably shows that Labour policies were better/less bad for the deficit, ”

    That revision was to 2011/12 -not 2010/11.

    In fact revisions to deficits for 2007/08, 2008/09
    and 2009/10 respectively were all upward revisions of £0.5 billion, £0.5 billion and £0.9 billion .

    Deficit 2010/11 was also revised upward by £1.1 bn

    So-Labour’s legacy of deficits was worse than previously thought.

    THe downward revision of 2011/12 deficit by £6.7 bn takes it to £119.3 billion;.

    This is £7 billion lower than the last Office for Budget Responsibility (OBR) forecasted net borrowing for 2011/12 of £126.0 billion and takes it back to within £3bn of OBR’s original forecast in the 2010 Budget.

    The revised 2011/12 deficit of £119bn is now £39 bn less than the Governments’s legacy deficit of £158 bn for 2009/10.

    It is also £1bn less than the last OBR forecast of £120bn ( excl*) for the current year 2012/13.

    This however , will simply make the prospective over-run for deficit this year look even worse.

    *Care will have to be taken for 2012/13 numbers which include a special item of £28 bn relating to PO pension transfer

  42. Heard Peter Kellner on radio this morning as I was going out-thought he said UKIP will come first in UK 2014 EU elections, beating Cons.

    Interesting noises about UKIP/CON pacts for next GE. Farage to spell it out at their Conference apparently.

    It’s that Referendum thing again Dave :-)

    Actually the way the EU Commission is sounding a referendum is looking like a cert -unless we want to join the Federation of European States.

  43. Yes in/out referendum looks very likely sometime in the next Parliament when negotiations completed.
    Reckon all 3 main parties will commit to one at the next GE, I guess voters have to decide who to believe.

  44. Not a good week for apologies. Clegg’s is now unravelling, with the revelation that the LD’s were cynically preparing to ditch the pledge well before the election.

    http://www.guardian.co.uk/commentisfree/2012/sep/21/nick-clegg-tuition-fees

    Staggering ineptness – raising a totemic symbol of all that ex-LD voters hate about the coalition, to no conceivable benefit to the LDs. Unless of course this was about shoring up his own position with the remainder of the LD party.

    But I’d expect the LD conference to be dominated by rehashing of the pledge, which ain’t going to be good for LD VI.

  45. It’s the word ‘pleb’ that’s the problem. If he’d said ‘f*cking jobsworths who need to learn their place’ the public would’ve probably agreed with him, in the particular circumstances.

    And folks who are linking it to the deaths of the WPCs in Manchester need to get a grip. Have we all forgotten the police conduct at Hillsborough already? Peter Tomlinson’s death? The way they pander to the Murdoch press?

    We need the police when they do a good job, we deplore them when they don’t; & every one of us gets annoyed when they are acting in a way which we believe is petty. Most of us keep it to ourselves though because we don’t think that arguing with them over pettiness is going to turn out well for us!
    8-)

  46. @ Colin

    Here’s what the public see, as opposed to the analysis which you have posted:

    UK government borrowing rises to record August high; problems for George Osborne as the figures mean the deficit has widened 22% so far this year – compared with target of 4.6% reduction.
    8-)

  47. @colin – thanks for the detail and the correction. I had only skimmed a report of the announcement. Having said that, I think @Amberstar hits the nail on the head. The governments only easily understandable political objective is currently traveling backwards, with lots of second order problems potentially flowing from this failure. Not great.

  48. Picking up on Robin’s comment, the problem for ell the three main parties is hhow they can convince the electorate that anything in their mainfesto is sacrosanct. Basically, the parties can put whatever they like in them and then ignore the ‘pledges’ when negotiating a coalitions agreement. I suggets this could be an issue for the LD party.

  49. I am struck by how far the interpretation of the polls is influenced by qualitative considerations. Some people think that the Con Lab gap will persist because the government looks tired, Others think the gap may reverse as people start to become serious with an approaching election and stop simply trying to vent their discontents. So these predictions rely on some understanding of why people are voting as they are and how their replies are related to underlying motives and perceptions.

    If this is how we interpret the polls are we right to do this? And if we are, would polls ideally be supplemented by ‘qualitative data’ , the results of focus groups and the like on which the political parties do seem to place a lot of weight? We don’t seem to quote these at all. Is that because they are not made public?

    @Robin

    ‘One thing to add, of course, is that the Lab and Con VI are not independent values. A lower than average Lab VI is very likely to be accompanied by a higher than average Con VI. So the moe for the Labour lead is more like 6%.’

    Oddly this does not seem to be so for the last 24 polls given in the margin to Anthony’s post. The correlation between the lab and con VI is .052 i.e. effectively zero. This seems counter to common sense and it would be interesting to know why it is – perhaps I copied the figures wrongly. But your main point would presumably still stand even if the two VIs were independent of each other. To the best of my memory the variance of a difference between two variables is the sum of the variance of the two variables concerned.

  50. Mitchell denies swearing and denies using the word “pleb”.

    A diplomatic protection officer has gone on record stating that the verbatim quotes which appeared in the Sun are a correct report of what was said.

1 2 3