Two new polls out today, both good for Labour. Populus this morning had toplines of CON 31%, LAB 38%, LDEM 9%, UKIP 14% (tabs here). Lord Ashcroft’s weekly poll has topline figures of CON 27%, LAB 34%, LDEM 11%, UKIP 15% (tabs here).

The Ashcroft poll comes after a poll last week that showed the Conservatives 2 points ahead, and has naturally provoked some comment about volatility. In one sense it’s fair comment – Ashcroft polling has been volatile. In another sense it’s not – Ashcroft’s polling hasn’t necessarily been any more volatile than you should expect, it’s just that we sometimes have slightly unrealistic expectations of how accurate a poll of 1000 people should be!

The standard margin of error on a poll of 1000 people is plus or minus 3 points. However, voting intention figures aren’t based on the whole sample, only on those who give a voting intention – in a phone sample of 1000 that’s typically 500 or so people, giving a margin of error of plus or minus 4 points. I should add that the margin of error is based upon what the margin would be in a pure random sample. This is very much a polite fiction – no voting intention polls are actual pure random samples. Many are from internet panels, even quasi-random phone polls aren’t actually random because of low response rates. Weighting effects would also change the actual margin of error.

Looking at Ashcroft’s nine regular polls to date, the average level of Labour support has been 33%, and all nine polls have been within 2 points of this. The average Lib Dem support has been 8.5%, and all nine polls have been within 2.5% of this. What’s made them look erratic is the level of Tory support, which has averaged 29%, but has varied between 25% and 34% – two of Ashcroft’s Tory scores have differed from the average by 4 points, one by 5 points. This assumes that there hasn’t been any genuine movement in Tory support, when it’s possible there has. Ashcroft’s highest Tory score came in his first poll in mid-May, at a time when ICM also showed a Tory lead and YouGov a neck-and-neck. Ashcroft’s lowest Tory score came just after the European results when UKIP had a post-European election boost.

Bottom line is that while Ashcroft’s polls look erratic, they probably aren’t much more erratic than we should expect from topline figures based on 500 people. There isn’t anything strange about their methodology, nothing odd going on, it’s just the normal limits of how precise polling with a given sample size can be. And it’s a useful reminder of why we shouldn’t read too much into individual polls, and it’s the underlying trend and average that count.

542 Responses to “On volatility in polling”

1 7 8 9 10 11
  1. @Sheep
    Those are fair points both, but I intended my analysis more as an active demonstration of where Ashcrofts Apologia falls down. Those were the 2 pollsters he used to demonstrate that his polls were no more volatile than the rest. I think he failed, because the same data under a marginally more sophisticated analysis contradicted his conclusions..

    Obviously with few weekly Ashcrofts so far, 8 data points is all we have, and only YG and Pop did 8 plus polls in the same period. If we revisit this in 6 months time we’ll be able to make a comparison with the monthly ICM. It wouldnt make much sense now.

    Neverthelless, if we want to quantify the relative volatility of pollsters I’d stand by my method, which compares like with like and does so in an appropriate way. Ashcroft’s choice of measure is odd and arbitrary -why measure volatility only by the most extreme results? That goes against my statistical intuitions, especially as he used datasets of different size, as Roger Mex pointed out..

  2. Alec,
    There is an interesting article in the comments section of the Guardian today,
    regarding the self-employed which I think supports your view of the strength
    or lack of it,of the economic recovery.

  3. @Bigfatron – “I’m struggling to find definitive figures to confirm or deny this”

    Markit Monthly Household Finance Index. It’s a survey, so needs to be treated with some caution, but it does fit the prices/earnings gaps, which steadily narrowed, briefly crossed over (barely) and then widened sharply again.

    On investment, yes, Q1 saw good growth, but yesterday’s BCC survey of 7,000 companies suggests exports and investment have both fallen back in Q2. We’ll need to wait and see if this is reflected in official Q2 stats, but the BCC said in their press release – “Unless investment and net exports make bigger contributions to growth, the recovery could stall…” .

    Incidentally, the BCC also found a significant cooling in domestic sales in the key service sector – again, possibly providing some further evidence to suggest consumers are becoming more constrained.

  4. @Floating

    Glad it made sense, and was helpful — I usually overcomplicate.

    I had a look at my figures from 2012 when Labour Vi was at it’s highest: of those Others stating a VI labour had 35-40%, double what anyone else was getting.

    May come back to this if real llife gives me a break.


  5. @Colin – thank you.

    We are at peace.

  6. “Re: Con & Lab top & Bottom polling
    According to my calculations, the figures for this Parliament are, for all opinion polls:

    Con : Top 44% Low 23%
    Lab: Top 45% Low 28%

    Also, Lab have polled twice in the 20?s (both in the month following 2010 election) and the Tories 106 times (Yes, one hundred and six!), all since 2012 omnishambles budget.”

    This is interesting.

    However if you exclude the post-election coaltion ‘honeymoon’ period in 2010, and take the polling from 2011 onwards, it looks even worse for the Tories.

    When was the last time they scored above 36% in any poll?

  7. Colin and Alec at peace.

    I expect the two state solution to be agreed within days.

  8. I think i can answer my own question – March 2012. Over two years ago.

    They’re going to be needing to poll 39%+ in the GE for a majority. I quite happily would chew my own arm off if that happened.

  9. Someone upthread (@Spearmint?) said something about this latest YG poll also being very poor for Labour, due to the 36% score.

    I wonder. This would give them a 1% bigger margin than the 2005 GE result, which would I suspect be viewed as a pretty spectacular performance by Ed if this was the eventual result.

    I guess the judgment comes from the notion that the gap must close in the next 9 months or so, and that with a lower LD score, Lab should be further ahead. But then why don’t people say with a lower LD score, Cons should be closer?

    This poll, in itself, remains in my view very good for Labour. The key question is what today’s poll means for next years election – that’s an altogether different question.

  10. UK’s deficit on trade in goods and services was estimated to have been £2.4 billion in May 2014, compared with £2.1 billion in April 2014.

    The trade with the EU seems to be the problem this month and the UK bought some aircraft which cost £433 million

    Assuming June is similiar, then net trade will take off about 0.2 from the GDP figures and if GDP is going to be 0.9% that is going to need some heavy spending from consumers or another good quarter for investment – Markit is saying yes that happened, BCC saying not so much.

    We will have to wait and see

  11. ciderman


    those figures would presumably look very different if one then excluded the sort of artificial “honeymoon” period the Tories enjoyed after the election?

    Maybe the last three years would look very different?

    I also suppose one would need to build it regularity of higher figures as jus the occasional spike would be less persuasive than a consistent run of them.

  12. I guess to a certain extent volatility can be removed by the methodology used but possibly at the expense of a more accurate poll.

    If I had to guess I’d say ICM are the least volatile but perhaps that comes down to them being the ones who most heavily add in voting intention back to the party they voted for in 2010. Therefore whatever actual voting intentions are recorded you still have a large chunk of “don’t knows” to even out the final numbers.

    I think Ashcroft is wrong to defend his results as much as he has done. I think his defence of his polling methods is valid and absolutely fine but I think he has to accept the results so far have been all over the place. He’s probably just had bad luck with samples but you don’t need to be an expert in statistics to see his polls so far have been more volatile than any of the other polling companies.

  13. @Floater – Markit surveys c 600 firms, BCC 7,000, although we know nothing of sampling methodology or response rates from either.

    On general matters –

    This looks positive at first, until you realise that they seem to be talking of strong expansion overseas. The bit about falls in profits due to the strong pound also provides a counter to @Bantam’s point about currency movements and prices.

    The difficulty we have is that George Osborne’s stated policy objective is to maintain a weak pound to encourage exports.

    I said it was a poor policy at the time, and we are seeing precisely what can go wrong with it. If you want to encourage exports, it’s far better to promote investment and productivity, rather rely on suppressed wages and an artificially low currency.

  14. @ Alec

    I was only talking about domestic goods and food prices in this country benefitting directly from the pound. Exports are a different thing altogether, 80%ish of our exports are service related, insurance etc and aren’t (so far) being affected, strong growth is still occuring. On manufacturing, most products produced have some element of imported parts or materials involved so the effect is watered down. I wouldn’t want to see the pound much higher though, it’s a real balancing act.

    Regarding suppressed wages, I would rather see fuller employment first and then see wages float up when the demand begins to the supply of labour.

  15. ‘Markit surveys c 600 firms, BCC 7,000,’

    i didn’t know the Markit sample was so small.

    I know Williamson of Markit was using their overall PMI score for the UK of 57 to predict a GDP increase in Q4 2013 of 1.5% which was miles too high, it came in at 0.7%

    He has changed his interpretation now and for the same score of PMI 57, Markit are giving a prediction of 0.8% for Q2 2014

    But i don’t know how they can be so confident in predicting GDP -, they don’t include the self-employred or the retail sector or the governemt sector or the energy sector or the extraction sector. It just seemd a bit of a guess to me.

  16. ‘most products produced have some element of imported parts or materials involved so the effect is watered down’

    I read somewhere that each pound sterling of car export cost 2 pound in car parts import. = I can’t source that fact though.


    ‘ I would rather see fuller employment first and then see wages float up when the demand begins to the supply of labour.’

    i would agree with that of course, it is better to have fuller employment and lower wages, which is what we have now, than higher unemployment and higher wages.

    I believe though that voters would be more optimistic about their personal finances if they had higher wages and they wouldn’t care so much about the higher unemployment that went with it, as long as it wasn’t them personally unemployed.

    This makes most voters sound selfish, but i think it is true, not every voter but generally..

  17. @Floater – I posted on the Markit methodology a couple of days ago. They ask respondents to answer whether a particular indicators has got better, worse, or stayed the same. There is no question regarding scale.

    They then score ‘the same’ responses by 50% – eg if you have a sample of 100 and all responses say ‘the same’, this gives a score of 50 = no change.

    ‘Worse’ responses are scored as zero, while ‘better’ responses as 1.

    I think the effect of this is to magnify small changes in a small sub set of a given sample, which may explain why Markit usually get the timing of GDP movements about right, but get the level of the peaks and troughs wrong. They consistently seem to be more extreme than real output data.

    For example, playing with numbers, if we have a sample of 100, with 84 saying no change and 16% reporting a very marginal improvement, we get a headline score of 58, which is claimed to be equivalent to annual GDP growth of up to a booming 8% – all because 16% of respondents feel marginally better off.

    I don’t think there is anything wrong with Markit’s approach. They seek to provide quick and regular snapshots to guide market opinion. It’s a bit like Lord Ashcroft’s VI polls, and I think the problem really lies in the interpretation of the results. News organizations increasingly report the PMI data as fact, when it really is only a limited scale, short time frame indication, heavily subject to distortion from sample variation and the method of calculating the index.

  18. Latest YouGov seems to have UKIP and LibDems a bit high – Labour a tad low.

  19. ” I think I can answer my own question – March 2012. Over two years ago”

    Yes, and again, this was just before 2012 Budget which, IMO will be the defining event of this Parliament even if it was not as a cathartic shambles as Black Wednesday.

  20. @Alec

    ‘news organizations increasingly report the PMI data as fact, ‘

    I have noticed this as well. i think Markit people have been clever here, they produce their numbers on the 1st, 2nd and 3rd of the month, when other economic news is absent and news organsiations desperate for copy and short of time print it verbatim

    It is a simple easy to use number,then one side of A4 with 2 short summing up paragraphs, filled with eye-catching quotes and they even supply the headline – brilliant business strategy i tihnk.

  21. Today’s low Tory VI is due to unusually high Tory -> Ukip flux. So two bad polls for them in a row for two completely different reasons- I still predict a return to 33-34% tonight.

    @ Alec,

    I’m more concerned with the VI figures than the lead. For reasons that are not entirely clear, YouGov has the two main parties higher than most of the other pollsters, so going by an average of polls a 36% with YouGov equates to something along the lines of 33/34%.

    It’s quite a bad poll for Labour by the standards of previous YouGov polls in this Parliament, and it’s a bad poll in comparison to the Tory cap, which appears to be around 36% with YouGov (post-Ominshambles). When Labour aren’t comfortably above that I get uneasy.

    To be clear, I think Labour will still be ahead in the national polls come the election and I believe they are on course to win a narrow majority. But it’s no longer good enough for them to tread water: they now need to win some of their Ukip and Green defectors back. It shouldn’t pose insurmountable difficulties for them, but it’s a worse position than they were in last summer.

  22. @Spearmint – understood and all making sense.

  23. We are in such a different landscape to previous elections and polling. Looked back to run up to ’92, and it appears in the short campaign period Labour dipped and LibDem rose. Surely no one actually expects a LibDem revival of that magnitude next year.

  24. Speermint,

    Tread water is fine for now imo

  25. @spearmint : getting back Green defectors (and LibDem to Green switchers) will not be easy if Labour continue to do things such as agree to emergency legislation on surveillance law. EU decision was in April, so why the sudden rush? Legislation should be scrutinised and not rushed through parliament unless there is a real emergency.

  26. ALEC

    @”The difficulty we have is that George Osborne’s stated policy objective is to maintain a weak pound to encourage exports.”

    I would be interested in seeing the policy statement you refer to, since the CoE cannot dictate Sterling’s XR.

    In fact the last Budget & GO’s bullish comments on the economy, have both helped to push sterling higher.

    Of course the prime driver is the BoE-whose “guidance “on interest rate rises in the future is consistently read by markets as a warning that it won’t be long before they start going up.

    All of this is making life harder for UK exporters-but helping keep imported inflation down , which is to the benefit of UK’s retail customers.

  27. If Labour made a solid commitment to be fiscally responsible, and not undo years of austerity pains by dramatically increasing borrowing on Day 1, they would steal this election.

    The floating voters are among those still capable of remembering Brown’s mass tax, borrow and spend programme and will always remain wary of what Labour would do if they were again given the keys to the Treasury.

    So why don’t Labour make such a commitment?

  28. Interesting bit in DP today.

    Frances O’grady was on re the strikes. After outlining her list of grievances, AN asked why the unions supported Labour on the pay freeze issue, since they had a similar policy.

    She replied-yes but they would listen & negotiate.

    She alswo said that Labour would ensure the Living Wage for Public Sector workers. AN suggested this was an aspiration-not a policy of compulsion. She repeated, that Labour would “listen”.

    EM is going to have some serious “listening” to do when he walks into No 10.

  29. @Colin – 2010 – 2012 saw a number of speeches in which GO expressly stated that we had to get the deficit down to avoid interest rates rising and help maintain a competitive (ie devalued) currency for exporters. It was a central plank of his export led recovery policy, back when ‘the march of the makers’ was going to be our saviour.

  30. That’s the thing see Colin, the Tories represent the interests of business and Labour represent the interests of ….well, labour !

  31. ALEC

    I understand that. And getting the deficit down has achieved that end-though many would argue that BoE Base Rate, & Asset Purchases have been the main driver.

    Now, however, as the economy picks up & BoE starts talking of Monetary Policy tightening, it is a different set of circumstances.

  32. EWEN

    You rather make my point for me :-)

  33. @Jimjam

    Of course, I’m not saying UKip won’t fall back. Just that there’s one potential driver (Immigration) for it not falling back quite as much as some might expect (and of course another in how UKip seem to benefit from electoral exposure), and another possible driver in that Lab might benefit a bit more from an actual fallback (i.e. if the voters had already migrated to Lab first before moving to Ukip)…

  34. @Steve2

    That’s a misreading of the political situation. There are hardly any voters floating between Labour and the Tories. Even when there have been big movements in the polls, such as after the Omnishambles and the so-called ‘veto’ the numbers of direct switchers numbered maybe 1% of the electorate.

    The real floating voters Labour need to worry about are the ones floating between voting Labour or not voting at all. YouGov data from the past 12 months shows these people to number more like 4% of the electorate, and Labour’s support has dropped away markedly among such people.

    As it happens, Labour already are making a big deal about being ‘fiscally responsible’ and a fat lot of good it’s doing. The small share of the electorate it’s aimed at are too cynical about politicians to believe them, and it risks demoralising their core vote and activist base.

  35. @SCOUSER

    It is also worth recalling (yet again) that “Brown’s mass tax, borrow and spend programme” was precisely the same programme which Osborne/Cameron were committed to… until Lehman Bros etc.

  36. @steve2 Labour have already made the commitment and have asked for the rules to be changed so that the OBR can cost the Party’s manifesto. Osbourne has refused.

  37. Not sure what your ‘point’ is Colin.
    I was merely stating the underl*ing political divide in British politics, which no one seriously questions. Despite the Tories recent ludicrous attempt to describe themselves as the Workers’ party and sensible attempt to be ‘business friendly’.
    Smiley things all round !

  38. Missed out “Labour’s” (sensible attempt).

  39. @ Steve2
    ‘If Labour made a solid commitment to be fiscally responsible, and not undo years of austerity pains by dramatically increasing borrowing on Day 1, they would steal this election. ‘

    So if Labour promised to do just what the tories would do and have been doing then they would win the election? That seems to be your logic.
    But labour have been opposing the tories on every issue up to now.

    It may be that Labour would also say that they would do lots of nice cuddly things – but given they would be presumably sticking to ‘promise nr 1’ then where would the money come from.

    The question remains [whether people would] believe them?

    I am unsure how a poll that shows labours lead dropping by 4 points is seen as good for them.

  40. That 39%+ estimate for a Conservative OM seems a quite reasonable target for what they need. Under foreseeable circumstances, that looks highly unlikely. Conservative VI just seems stuck at low 30’s. They’ll be relying on things going very badly for everyone else, particularly Labour.

    It makes what happened in 1992 seem almost a plausible outcome, if it were proposed before the event.

  41. @ ALEC
    ‘I find the timing of new anti terror laws amusing.’

    Except they are not new anti terror laws.
    The laws are currently on the statute book and have been since a 2006 EU retention of data directive (which followed terror attacks in UK and Madrid)
    As a result of meddling by some ‘liberalists’ in Ireland and Austria the dim judges of the EJA have ruled against it.

    ie the ECJ have ruled against an EU directive.

    It remains free for all EU nations to enact whatever anti terror surveillance legislation they want. That’s what we are doing now, reinstating the EU 2006 law. Norway is doing the same (note Norway is not in the EU but it had enacted the directive), although Sweden is not.

    So it is not a new law, its making sure the law we already had stays. There is absolutely nothing to get worked up about in any of this – except to complain about the dim ECJ judgement.

  42. Interesting to see stock market wobbles today over Portuguese banks and talk of Eurozone periphery contagion – takes yer back, doesn’t it?

    I posted a couple of days ago about an analysis suggesting traders are now more heavily leveraged than at any time in the past – well beyond the previous 2007 record. This analysis suggested such a peak always comes before a market crash, with an unexpected incident often being the trigger.


  43. EWEN

    @”Not sure what your ‘point’ is Colin.”

    …that O’Grady certainly agreed with you that the Labour Party is there to support the pay demands of today’s strikers.

    She didn’t describe those workers , as you did Ewen, as “Labour”-but as “Public Sector Workers”. She didn’t mention Private Sector workers at all.

    ie the TUC wants a Labour Government so that it will “listen” & “negotiate” , and subsequently concede to Public Sector workers’ demands.

    It is obviously clearer to you & Frances O’Grady, than it is to me , that this is Labour Party policy for Government.

    The expectations of Public Sector workers from a Labour Government are high, and I was merely musing on whether EM & EB share them.

    [I’m not sure, however, whether that’s much to do with public opinion and polling or just a recipe for a partisan back-and-forth, ahem… AW]

  44. @Hookeslaw – I understand. At the time I posted, it wasn’t clear what this story was about.

  45. I could try and make it to do with them Anthony , by looking at OP results on perceptions of fiscal competence, and musing on whether the TUC aspiration is helpful to Labour prospects or not.

    I think there is an issue here which touches on Polling & the GE result.-but I obviously failed to convince you of it.

  46. Colin
    Sorry, I should know by now you don’t really do irony.
    Can’t you see that both of our Great Parties of Govt will try to steal the other’s clothes on occasion ? That doesn’t alter the the underl*ing reality.
    It is also the reason the LDs are stuffed, cos they are “neither fish, flesh, fowl or good red herring !”

  47. EWEN

    I don’t understand what that means.

    Very simply & for the last time:-

    Today on TV, Frances O’Grady, TUC Gen. Sec stated that the Public Sector workers on strike today wanted a Labour Government, because, she believed, that a Labour Government , would “listen” to their demands for a pay rise & implementation of the Living Wage.

    Both EM & EB have been at great pains recently, to counter Polling opinion which still blames Labour more than Cons for “spending cuts”.

    So-I ponder whether the TUC’s expectation as expressed by O’Grady today , is realistic , and calculated to enhance Labour VI ( which she wants) -or to jeopardise it.

    In view of AW’s note to me, its best if we don’t exchange viewpoints.

    But that’s what I was on about :-)

  48. @ Chatterclass,

    getting back Green defectors (and LibDem to Green switchers) will not be easy if Labour continue to do things such as agree to emergency legislation on surveillance law.

    While I agree with you on the principle, anyone who has managed to either miss or tolerate every other position they’ve taken on civil liberties for the past four years (and really, the past seventeen years, since Cooper is unreconstructed New Labour on this and Miliband either agrees or doesn’t care) is unlikely to be put off by this one.

    At the end of the day parties are a package deal, and civil liberties just don’t have high salience. Which is why all three main parties now have indistinguishable positions*.

    * I would have thought the Lib Dems would want to preserve liberalism as their USP, but then I thought that about not bombing Syria as well. Clegg makes interesting life choices.

  49. Are there votes in promising to reduce the deficit?
    I’d say: Definitely maybe.
    Are there votes in promising the living wage?
    I’d say: Definitely.

    I think an ealier post from you pointed out that there would be only a small constituency which followed the HOC debate and proposed legislation on slavery and human trafficking.
    Supposing that that consists say of one in ten ABC+ voters and one in twenty CDE voters: 4 to 5% of the electorate? That might be a constituency actively concerned to elect a government with serious concerns to legislate and invest in a program which, in a wider context, is at the heart of the immigration debate. It would be nice to know, possibly from polling, how that would add up in VI salience.

1 7 8 9 10 11