Two new polls out today, both good for Labour. Populus this morning had toplines of CON 31%, LAB 38%, LDEM 9%, UKIP 14% (tabs here). Lord Ashcroft’s weekly poll has topline figures of CON 27%, LAB 34%, LDEM 11%, UKIP 15% (tabs here).

The Ashcroft poll comes after a poll last week that showed the Conservatives 2 points ahead, and has naturally provoked some comment about volatility. In one sense it’s fair comment – Ashcroft polling has been volatile. In another sense it’s not – Ashcroft’s polling hasn’t necessarily been any more volatile than you should expect, it’s just that we sometimes have slightly unrealistic expectations of how accurate a poll of 1000 people should be!

The standard margin of error on a poll of 1000 people is plus or minus 3 points. However, voting intention figures aren’t based on the whole sample, only on those who give a voting intention – in a phone sample of 1000 that’s typically 500 or so people, giving a margin of error of plus or minus 4 points. I should add that the margin of error is based upon what the margin would be in a pure random sample. This is very much a polite fiction – no voting intention polls are actual pure random samples. Many are from internet panels, even quasi-random phone polls aren’t actually random because of low response rates. Weighting effects would also change the actual margin of error.

Looking at Ashcroft’s nine regular polls to date, the average level of Labour support has been 33%, and all nine polls have been within 2 points of this. The average Lib Dem support has been 8.5%, and all nine polls have been within 2.5% of this. What’s made them look erratic is the level of Tory support, which has averaged 29%, but has varied between 25% and 34% – two of Ashcroft’s Tory scores have differed from the average by 4 points, one by 5 points. This assumes that there hasn’t been any genuine movement in Tory support, when it’s possible there has. Ashcroft’s highest Tory score came in his first poll in mid-May, at a time when ICM also showed a Tory lead and YouGov a neck-and-neck. Ashcroft’s lowest Tory score came just after the European results when UKIP had a post-European election boost.

Bottom line is that while Ashcroft’s polls look erratic, they probably aren’t much more erratic than we should expect from topline figures based on 500 people. There isn’t anything strange about their methodology, nothing odd going on, it’s just the normal limits of how precise polling with a given sample size can be. And it’s a useful reminder of why we shouldn’t read too much into individual polls, and it’s the underlying trend and average that count.


542 Responses to “On volatility in polling”

1 2 3 11
  1. @ Anthony
    …in a phone sample of 500 that’s typically 500 or so people, giving a margin of error of plus or minus 4 points.
    ————-
    I don’t think this is what you meant to write.

  2. Fixed – Thank you :-)

  3. There might 6 or a dozen polls with a Labour lead and in the middle of them one with Conservatives just ahead or level pegging yet our wonderful media will go banners on the one and almost ignore the rest. Am I the only person to recognise this or am I oversensitive?

  4. The 1980’s dossier and cover up story is one that has wide traction across a wide range of the public including those not inclined to follow current affairs – increasingly rare in the modern world with fewer buying papers and watching the news.

    This story does not look good for the Tories and perhaps rekindles the feeling of them as somewhat toxic, corrupt and self-serving. It has been Labour MPs leading the fight for openness on this – Tom Watson and Simon Danczuk which must have helped their fortunes.

  5. “Bottom line is that while Ashcroft’s polls look erratic, they probably aren’t much more erratic than we should expect from topline figures based on 500 people. There isn’t anything strange about their methodology, . . .”

    I beg to differ – surely part of his polls’ volatility is down to conducting them at weekends, which all other pollsters avoid like the plague.

  6. @Anthony

    Is there ever a time when a sudden and discernible movement in the polls actually represents a genuine shift in public opinion or are we always able to explain everything away by sampling error and margin of error volatility?

    If it’s the latter, then all of us on this site are condemned to discuss a political form of virtual reality. Constant debates about the thermometer rather than the temperature.

    I’m always loathe to cite some genuine movement in the polls through fear of looking a complete ass when they suddenly revert to type or move in an opposite direction, but three of the last four polls (Opinium on Sunday, Populus and Ashcroft today) suggest something might be going on, don’t they? Ditto the TNS Scottish Referendum poll over the weekend.

  7. Amber – damn, I thought I’d changed that before anyone saw it.

    Far Easterner – nothing usual at all about that. Telephone polls are normally done over weekends. The regular ICM and MORI monthly polls are both typically done over the weekend.

  8. trend, trend trend.

    Of course Populus changed their methodology as did YG early in this parliament, although not much impact in the second case.

    So best, biggest ow whatever score on Populus or YG should be ‘since method change’.

    Think Cons Res as well but not sure.

  9. Thanks Anthony.

    So I think the message for Ashcroft is that he needs to increase his sample size.

    Not sure if you have seen Dan Hodges’ twitter feed.

    https://twitter.com/DPJHodges

    Dan Hodges
    ?@DPJHodges I like Lord Ashcroft. But he should pull his weekly poll. It’s getting silly now.

    Retweeted by Dan Hodges
    Adam Boulton @adamboultonSKY · 4h
    @DPJHodges Well one of his polls is bound to be right

    Perhaps polls should be reported in ranges so that small sample sizes show the larger margin of error, or BPC should advise that MOE needs to be stated when reporting any poll result.

  10. @Mike B

    I don’t buy papers finding I get enough news from the BBC and lurking on UKPR. However, I guess you are right that the papers will splash Tories narrowing the gap or gaining the lead whenever they can. How far do you think that is their right wing bias and how far a liking for NEWS (and continuing labour leads are not that) aided by an ignorance of what individual polls amount to?

  11. @ MikeB,

    You’re not, but

    a) Crossover is objectively more newsworthy, and

    b) The reporting on polling has a minimal impact on anything that happens in the real world.

    There is only one poll that matters, and they’re going to have to report it accurately. If they want to pretend things are neck and neck before then to sell papers, that’s dishonest and it’s lousy journalism, but it’s not really hurting anything but their credibility.

    On this one I’m with the Conservatives: let the free market sort it out. People who want accurate reporting on polling can get it here, from Anthony, for free.

  12. I think the problem for the BBC is often simply lazy journalism.

    They take their lead from newspapers, which have an agenda and will report not just what “someone says” but even more idiotically what someone SAYS someone says.

    So is that is that Miliband is under pressure, blah, blah, blah then no other source is required apart from to give the name of the newspaper and then bang on about it.

    The same happens if there is something like an EU issue with Cameron but far less often.

  13. @ R&D,

    I think that’s necessary but not sufficient to explain what’s going on at the BBC. That Cardiff University study was pretty damning.

    Fortunately Miliband seems impervious to all influence, which in the long run is probably going to be a problem because the PLP and the Shadow Cabinet struggle to get through to him either, but it’s been pretty helpful in opposition.

  14. The media loves a horse race. The story of the parliament that labour has enjoyed a lead of between 3 and 10 points since march 2012 is dull.

    The story that labour have had a 3-6 point lead for 15 months is even duller.

    X-over is newsworthy… I think there’s a slight bias too…the lobby are, with a few exceptions, southern public schoolboys who haven’t quite got used to the idea that the gawky, “body awkward”, as the economist described him, goofball that is Ed Miliband might actually become PM.

  15. If Ashcroft were polling say once a month and getting similar volatility to that which he’s getting once a week, I think the random volatility would be a lot less of an issue because it would be easier to assume that it was down to intervening movements in public opinion over the space of a month.

    ICM also have a similar sample size and approach Ashcroft in terms of volatility. That’s been less of an issue up to now, because people have often been rather generous in taking ICM polls at face value and attributing the movement as down to public opinion.

    I think the lesson from Ashcroft is to remind us that the ICM poll movements should also always be viewed initially as being down to fluctuations in smallish samples, until a run of several months starts to suggest a trend.

    Incidentally, both Ashcroft and ICM would be even more volatile if they did not reallocate don’t knows – since this assumption in effect increases the sample size and builds in some regression back towards the fixed 2010 election result.

  16. @Spearmint

    Misreporting or selective reporting of polls does matter because (a) good polls create the momentum which parties crave and vice versa and (b) in a FPTP system being seen as in contention is absolutely crucial.

    Many countries seem to agree, since it’s quite common for reporting of polls to be banned for a couple of weeks before elections.

  17. @Charles @Spearmint

    I include the electronic media (BBC, Sky but ITV and Channel 4 seem better). Of course polls are snapshots not predictions another mistake that is often made.

  18. YG C34 L37 LD9 UKIP13

    Not really showing the big Labour leap there.

  19. I used to love teaching O level Sociology on the Meejah and ‘news values’ and ‘gate-keepers’ and all that kind of stuff. Up until about 15 years ago the papers and the telly seemed to choose the news agenda fairly impartially outside of party political stuff, now I ain’t so sure. A certain bald chief political correspondent has certainly fallen a long way from the standards of the blessed John Cole !

  20. Mrs Thatcher stopped Sociology from being taught in state schools. I felt quite proud to be one of the enemy within alongside the miners.

  21. @MikeB

    Rightly or wrongly polls will only make news (beyond the paper they were commissioned by) when they can either feed a particular angle or show something unexpected.

    The talk about Miliband has been driven by poor poll ratings and much discussed in recent weeks, so when more evidence comes about to back this up it becomes a story.

    CON leads have been few and far between since 2012 so when they come up it is newsworthy, particularly now that the GE is fast approaching. If they become a regular thing eventually they will cease to be quite so newsworthy, though all polls will of course start to be reported more heavily over the coming months.

    Ashcroft does seem to be rather volatile but I guess that is one of the things you get with phone polls, which also have their advantages. It sounds silly but a busy, and generally warm, summer weekend like this one may fuel that volatility with people watching the tennis/ football or out and about rather than bothering to answer phone polls.

  22. @Ewen

    Hondootedly!

  23. @ Phil Haines,

    I see your point and raise you 1992.

    If anything the papers are encouraging voter participation rather than discouraging it by making the race appear closer than it is. Except for the Greens, I’ve not seen any serious challenger hurt under FPTP by a lack of media coverage of their potential to win- if anything the press try to inflate their prospects because an upset is a story (eg., the obsessive Ukip coverage even though the establishment don’t want them to succeed).

  24. It’s interesting that Lord A’s polls are only now ‘getting silly’ when they show a big shift to Labour.

    AW has neatly explained why they swing around, so now we know to get a bit less excited when big moves occur.

  25. EWEN.
    John Cole would not want to have the beatification, being a Cromwell admirer.

  26. @Alec

    “It’s interesting that Lord A’s polls are only now ‘getting silly’ when they show a big shift to Labour.”

    Mind you, they were “bang on the money” when they revealed that 2% Tory lead a week or so ago. :-)

    @Neil A

    “YG C34 L37 LD9 UKIP13
    Not really showing the big Labour leap there.”

    What are you talking about, man; that’s a 33% increase in the Labour lead from yesterday!!!

  27. @JimJam

    “So best, biggest ow whatever score on Populus or YG should be ‘since method change’.
    Think Cons Res as well but not sure.”

    A trifle gnomic for me or have you just had one nightcap too many tonight. :-)

  28. AW: I was a little surprised that confidence intervals (presumably you were working on the basis of 5%) didn’t even get a passing mention in your post. Essentially, for those that don’t know about confidence intervals, that percentage of polls will have results MORE than the MOE away from the proportions in the population as a whole. If all polls were actually within the MOE published, _that_ would be a statistical fluke.

  29. CB – just looked up Gnomic to see what you meant .

    I plead guilty … i think.

  30. @Crossbat

    “If it’s the latter, then all of us on this site are condemned to discuss a political form of virtual reality. Constant debates about the thermometer rather than the temperature.”

    We are, on this site, in the realm of quantum politics. As Heisenberg said (in the context of the subatomic particle): you can know where it is, or you can know where it’s going, but never both at the same time.

  31. Phil Haines,
    What a lovely idea,that banning of polls a fortnight before an election,including
    hopefully those spoilsport exit polls that spoil all the drama of”I being the
    returning officer “etc.

  32. @MOG

    I don’t know about you but as I get older I feel like the quantum thingy quite often.

    @All

    The really annoying thing about volatility is that it so often looks like a trend. If it just looked more random it’d be easier to spot.

  33. @Ann

    Couldn’t agree more. There’s a natural “herd instinct” of doing what you’re told others are doing. It means people would actually have to make their own mind up!

  34. They should just say

    “I the returning officer hereby confirm the exit poll was more-or-less right”

    And also ban speeches by the various candidates.

  35. Or not bother to count the votes at all. Just accept the exit poll. We wouldn’t have to wait up all night then listening to David Dimbleby droning on.

  36. CB11
    ‘What are you talking about, man; that’s a 33% increase in the Labour lead from yesterday!!!’

    Wrong! It’s actually a 50% increase!!

  37. Ed Miliband is “body awkward”? He really isn’t. He is actually tall, slim & imposing. What he does have, when you meet him, is a rather unnerving type of eye contact. He looks directly at you & gives the impression that he is reading you; for comparison, I’d previously only encountered this ‘1000 yard stare’ (as my son calls it) in goalkeepers, fighter pilots & snipers. Perhaps that’s what the press find ‘weird’ about him….

  38. @Ewen Lightfoot

    That’s odd, I did sociology at school mid-80’s. Didn’t want to and found it boring and pointless.

    The big question with a poll out of line with everything else recently announced is always “is it the start of a trend, or an outlier”. By the time we find out, the poll in question is forgotten. How irritating. I can see why people say that the most important poll is the one where people actually mark crosses on bits of paper. We never ask if that’s part of a trend, perhaps we should.

  39. @Ewen

    I certainly can’t imagine John Cole thinking an opposition leader pulling a goofy face as he ate a bacon butty to be newsworthy, or talking over the announcement of the Labour leadership result and then, to top it all off, getting his maths wrong.

  40. Can i just say Rosie and Daisy are the bees knees,and could i ask are you from the North East as your humour is very similar to mine.I am northerner and see a lot of political views that match your own.

  41. I don’t necessarily disagree with your sentiments Shaun but wise to keep it a little bit toned down. We do pride ourselves on relatively civil and non-partisan discussion!

    You raise a valuable point though which is why I hope Anthony will merely use the scissors rather than the deletion ray. The fate of the Tory party after a potential defeat in 2015 is marked with a large question mark. There is no immediately obvious successor as leader and they’ve stilled not settled on whether to moderate or hit the accelerator.

    I’ve never met Ed Miliband but I have noticed the rather deep stare he does and can see why it might unnerve someone. However he is not built weirdly – he’s not got Jacob Rees-Mogg’s wiriness or a Hattersleyesque figure but he’s surprisingly tall and very “clean” looking.

    As a fun aside, when I met Neil Kinnock (at this event opening with a great story about Woy Jenkins: http://youtu.be/iOIvRN6usws) I was quite struck with how short the man is – surely can’t be over 5’9″. He exudes a huge amount of confidence though – possibly too much which I guess results in the occasional Sheffield Rally.

    (He quotes me at 26:53 – End self-aggrandisement here)

  42. Thanks for the timely reminder that we have to look at some kind of underlying trend. It’s still a bit odd that the underlying Labour Lead seems to have jumped back to nearly 4% where it was stuck for most of June. Possibly this is to do with the historic paedophile allegations in which case I don’t think there will be much lasting impact.

    Will the voters really think in May “Well Labour wrecked the economy last time and their leader isn’t up to the job of being PM, but some Tories behaved appallingly in the 1970s and 80s so we’ll let Labour in”?

  43. Mike – your comment on media avidly reporting on any stray poll reporting a tory lead or level pegging whilst ignoring all the others is something many of us have noticed. It is not just the mostly right wing print media, even the BBC are,at it. So whilst the Ashcroft poll from last week pretty widely reported, little doubt this week’s Ashcroft will be conveniently ignored.

  44. @MR N

    Phew, for a moment I thought Neil had blown your cover, young man.

  45. amber

    “I’d previously only encountered this ’1000 yard stare’ in goalkeepers, fighter pilots & snipers. ”

    I also find the odd, fully-armed sniper or two keeps a dinner party in good order Amber.

  46. @Rosie&Daisie

    I also find the odd, fully-armed sniper or two keeps a dinner party in good order.
    —————
    LOL :-)

  47. AMBER
    ” I’d previously only encountered this ’1000 yard stare’ (as my son calls it) in goalkeepers, fighter pilots & snipers.”
    You’ld have found it in boxers and karate fighters. As a boxer you are trained to hold continuous central vision on the eyes of your opponent and pick up his (or her) lateral action from head and eye movement and your own peripheral vision. Just saying.

  48. The “thousand yard stare” is the very opposite of focused eye-contact, and has very sad origins:-

    http://en.wikipedia.org/wiki/Thousand-yard_stare

    No politician should want to be known for his/her “thousand yard stare”.

    Intriguing YouGov this morning , adding to the status quo-confusion & angst :-)

  49. can we keep the partisan carp out of this website…I think [stuff], but i tend to just concentrate on the polling here.

    Clearly the tories have a real job on their hands. they have to overturn what seems like a 2-3% deficit and get a 3-4% margin over labour to have a chance of being the largest party…all in less than 10 months.

  50. JAMES

    just pressed “report” on that one.

    Cons have a mountain to climb as you say.

    Still think that most votes is a possibility-leaving another coalition as their only route to a majority. A LD collapse ( not to mention Con backbenchers !) will make this very very difficult though.

1 2 3 11