Tony Twyman, who died last year, was the man behind much of the mechanics of TV and radio viewing figures, most notably as technical advisor for BARB viewing figures. In broader market research he is more widely known for coining Twyman’s Law – “Any figure that looks interesting or different is usually wrong”. The point is, of course, that strange and unusual things in a single poll are more likely the result of sample variation or error than some amazing shift in public opinion, and you should be cautious of them before getting excited (My colleague Joe Twyman likes quoting it without attribution in the hope people will jump to conclusions… not so fast!).

Anyway, today we have a classic case. Two polls that look interesting when compared to recent averages, but which are both probably no more than the result of normal sample error.

Today’s twice weekly Populus poll had figures of CON 32%, LAB 37%, LDEM 10%, UKIP 13%, GRN 4% (tabs). The five point Labour lead is the biggest Populus have shown since November, their 37% share the largest any company have shown since November. Labour resurgence?

Lord Ashcroft’s weekly poll however had figures of CON 34%, LAB 28%, LDEM 8%, UKIP 16%, GRN 8% (tabs.) A six point Conservative lead, by far the best poll for the Tories from any company for several years (the largest Tory leads up to now were the last two MORI polls, which had them three points up). Tory surge?

Of course the actual answer is that there is probably neither a Labour nor a Tory surge, that both of these changes are probably just down to sample error and that people should watch the overall trend across multiple polls, not get overexcited about individual polls. If the figures in one poll look strange or unusual, it’s probably wrong.

In some ways it’s quite nice they come on the same day, as it should stop people getting too excited over an outlier in just one direction. On the other hand, it does tend to produce lots of confused comments about how polls can be accurate when they are showing both a five point Labour lead and a six point Tory lead. Bottom line for those who are confused, part of it is down to pollsters using slightly different methods (in this case, the way Populus weight their polls tends to produce a bigger share of the vote for the main two paries than does Ashcroft). A bigger chunk will be simple margin of error – polls are not precision instruments and no one who understands them would claim they are. They are randomish samples of about 1000 or so people. The quoted margins of error are about plus or minus 3% (though given response rates, weighting effects and that polls are not pure random samples, that’s a bit of a polite fiction). That means if the real position was Labour and Conservative tied on 33%, you would expect to see the Conservatives ranging from 30% to 36% and Labour from 30% to 36%, and while the results would tend to be clustered around the middle of that range, random variation could reasonably vary between a 6 point Tory lead and a 6 point Labour lead. Taken alone and in isolation, it does mean an individual voting intention poll isn’t that useful… which is why you shouldn’t look at them alone and in isolation – watch the trend.


200 Responses to “Contrasting Populus and Ashcroft polls”

1 2 3 4
  1. Thanks Anthony. We could do with you on the telly for the next 16 weeks. I suspect we will see nowhere near such sensible comment.

    When’s the next poll?

  2. Thank you .. This makes it clear for even a fool like me

  3. Can I just agree about Mr Wells being on the TV

  4. Mr N

    “When’s the next poll?”

    I assume it’s the YG one that Anthony’s colleague was tweeting as a Green teaser.

  5. Mr. Wells was indeed on the telly over the weekend, commenting on Sheffield Hallam with rather poorly green-screened hair.

  6. “They are randomish samples of about 1000 or so people.”

    ~2000 for the Populus poll, only ~1000 for the Ashcroft. The Populus certaincy squeeze does reduce it to ~1000, but the Ashcrof squeeze takes it down to ~500. In polling, the difference between a 2000 sample and a 1000 sample is a moderate change to the error, but the difference between a 1000 sample and a 500 sample is massive.

  7. They do both look like outliers of course. But the sample sizes involved, and magnified by the certainty squeeze, means that Ashcroft will produce more of these outliers than Populus.

  8. Two-poll average:

    Con 33.0%
    Lab 32.5%
    UKIP 14.5%
    Lib 9.0%
    Green 6.0%
    Others 5.0%

    Swing back? (joking!)

  9. An accurate reflection of the poll from the Paper that shall not be named

    “Two very different UK polls – we should ignore them both
    One UK poll released today has the Conservative party ahead by six, another poll has Labour leading by five. While the temptation to focus on one’s preferred political flavour may be strong, it is always best to look at the trends instead”

  10. Comres polling on ‘issues’ for ITV:

    The National Health Service has overtaken immigration as the top concern for voters, the latest ComRes poll for ITV News has revealed.

    Interesting list.

    http://www.itv.com/news/2015-01-12/itv-news-index-poll-reveals-the-nhs-is-now-the-top-concern-for-voters/

  11. Today’s twice weekly Populus poll had figures of CON 32%, LAB 37%, LDEM 10%, UKIP 13%, GRN 4% (tabs).

    Uh oh!!

    Lord Ashcroft’s weekly poll however had figures of CON 34%, LAB 28%, LDEM 8%, UKIP 16%, GRN 8% (tabs.)

    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    …………PHEW!!

  12. It is Comedy Results but an 11 point jump for the NHS would make sense given the A&E stuff that’s been in the news this week. In theory it will benefit Labour, but we’ll see. The Populus today could have implied that, but the subsequent Ashcroft has thrown a big doubting question mark up.

  13. Hate to say it but the Ashcroft poll looks q. dodgy to me. I don’t understand how you get from 22% Con 20% Lab to 34% Con 28% Lab

    The cross breaks look weird.

    37% Con 29% Labour in the South East is believable.

    39% Con 23% Labour in the Midlands strains credulity.

    Unless there’s some change in the political geography of England, where the Midlands is more of a Tory stronghold than the South East. bizarre set of figures…

    The midlands cross break is incredible to me. On those figures that basis the Tories should make substantial gains in the Midlands, while holding everything they have.

  14. All we need now is a YouGov showing a UKIP lead.

  15. 22% Con 20% Lab related to the unweighted base in contrast to the weighted base

  16. STATGEEK

    It’s not a joking matter, some of us suffer from extremely high blood pressure and take every poll seriously. ;-)

  17. @Peter Crawford

    No, it’s from a squeeze down to respondents that said they were ‘absolutely certain to vote’. Hence the reduction in sample size from ~1000 to ~500. Populus do the same, but they start from ~2000, and go down to ~1000.

  18. New Statesman’s May2015 blog has an explanation of the Ashcroft poll.

    Whether it is correct or not, I’ll leave for you number crunchers to decide.

    http://may2015.com/datablast/why-the-tories-arent-suddenly-6-points-ahead/

  19. @Jayblanc

    But the weird thing is the the Labour VI shows more certainty to vote than the Con VI. So the squeeze should have improved the Lab % relative to the Con %. But somehow it went the other way… something is wrong with the data processing.

  20. @Hal

    I think you’re misreading the crosstabs, read page 2 again.

  21. Peter Crawford

    Ashcroft’s poll seems not to have a separate London region – it must be included in SE.

  22. Jayblanc,

    Oh, maybe I am misreading. How do you get the 515 weighted base on page 5?

  23. GRAHAM.
    Thank you for the ORC factoid in the last thread; OCR is an examination body!

    AW: thank you for your post.

  24. Lol. Just lol.

    All eyes on tonight’s YouGov, eh, Anthony?

  25. Easy explanation for the two polls being so different….

    They were commissioned for FOX NEWS!

    Peter.

  26. 39% Con 23% Labour in the Midlands strains credulity.

    I still hold to this…

    Con 39% Labour 23% would have the tories destroying labour in nearly all the seats defined as “the midlands”…

    It doesn’t seem at all plausible to me.

  27. What was the timing of the two polls?

    Have the Tories ‘benefitted’ from the terror in France?

  28. A Green surge is the Twitter rumour for tonight’s YG …..

  29. Jasper22,

    Knowing Twitter someone is probably wearing Green Serge tonight!

    Peter.

  30. How likely is it hat somone from the Sun may have leaked the YG poll?

  31. Graham

    a YG source?

    No detail however.

    Strange….

  32. Well this is all very interesting.It would seem that the Midlands will be the main
    battlegrounds in the GE.Interestingly also so in the civil war.Now there’s an
    interesting analogy.See the financial times for their comparison between DC and EM.

  33. Anyone else heard the Murdoch papers are going to launch a huge attack on EM in the next few days? Source is a well connected journalist friend.

    I guess we should not be surprised as there will be a lot of this to come.

  34. Jasper22,
    Well that is absolutely splendid news.

  35. Blimey, only just dropped in to UKPR after a pretty dreadful commute from Muslimham or, as we know it around these parts; Birmislamabad, and what do we see; an Ashcroft poll showing a Tory lead of 6% only hours after a Populus poll showing Labour 5% ahead. Does anyone else ever get the feeling that these opinion polls like to tease us a little?

    Tonight’s YouGov will be a tie; I’m sure of it!

    :-)

  36. JASPER22
    Anyone else heard the Murdoch papers are going to launch a huge attack on EM in the next few days? Source is a well connected journalist friend.

    I guess we should not be surprised as there will be a lot of this to come
    __________

    We were all warned about it over the past 4 months by our missing resident PRESSMAN.

  37. Is there some correlation between te fact that th Midlands is now a Tory stronghold and the fact that Birmingham is apparently 100% Muslim?

  38. Also is theresome reason why my iPad apparently cannot understand the word the?

  39. Who/what is PRESSMAN??

  40. @Jasper 22

    “Anyone else heard the Murdoch papers are going to launch a huge attack on EM in the next few days? Source is a well connected journalist friend.
    I guess we should not be surprised as there will be a lot of this to come”

    You mean another one? When did they stop the huge attacks on EM? How will anyone be able to tell?

    If EM is so unelectable why does the Press keep having to remind the electorate of this supposed fact?

  41. CROSSBAT

    At least in Glasgow Govan the muslims are well integrated. They can be found chanting “Allah Allah we are the muslim boys” during Rangers matches.

  42. JASPER22
    Who/what is PRESSMAN??
    ______

    He used to post on here up until around December spewing out how the Murdoch empire was gunning for EM. He was quite comical and usually received a comical response.

  43. @ Wolfie,

    The two polls had identical field dates.

    Ashcroft’s polls are always volatile and Populus’s methodology is always dodgy, which probably explains the discrepancy, along with normal MoE variation.

    (Although when YouGov pops up tonight with a 5% lead for the Greens we may start to wonder what’s going on.)

  44. Anthony

    They are randomish samples of about 1000 or so people. The quoted margins of error are about plus or minus 3% (though given response rates, weighting effects and that polls are not pure random samples, that’s a bit of a polite fiction).

    It’s important to remember that the quoted margin of error is always a minimum – what it would be with a perfect sample. So in real life it’s always going to be bigger than quoted. But in this particular case, like a lot of telephone polls, the sample the percentages are based on a much smaller sample after the Don’t Knows etc have been removed. So in this case the margin is at least 4 not 3 points – whatever Ashcroft teewts.

    Jayblanc said that he thought that Ashcroft was using a strict likelihood to vote filter – only including those who say they are 10 out of 10 certain to vote (as MORI does). He doesn’t mention this anywhere (the methodology description with his polls is fairly scanty) and in any case the figures don’t quite match. But clearly something odd is happening.

    I’ve been playing about trying to work out how the Ashcroft figures are adjusted with Likelihood To Vote and I think I have worked out something that matches. Unfortunately it looks like another spreadsheet problem. The best fit I can get is if LTV=10/10 votes are weighted by 0.9, LTV 9/10 by 0.8 and so on. In other words:

    Adj for LTV = SUM({No for LTV=n} * (n-1)/10)
    for n = 7 to 10

    while it would be more usual for it to be:

    Adj for LTV = SUM({No for LTV=n} * n/10)
    for n = 7 to 10

    It doesn’t make any difference to the published figures as far as I can tell (not all the information is available). It’s possible some polls have been out in the past if this is a repeated problem.

    Some pollsters might prefer to include the n = 5 to 10 range (which would alter the figures), but that’s a matter of judgement. But it does look as if the formula is slightly wrong.

    It’s not as important as the Doncaster mistake (nor indeed the Sheffield and Thanet ones about which nothing was said), but it would nice of them to fix it sometime.

  45. Is there some correlation between te fact that th Midlands is now a Tory stronghold and the fact that Birmingham is apparently 100% Muslim?

    I guess all the non-Muslims must have fled Al-Birmingham and – no doubt in revulsion at the beatings given out by the Islamic Religious Police there – have gone over, en masse, to the Tory party.

    It might be worth looking up Fox News and finding out what has happened. They seem to have their finger on the pulse there….

  46. My response to the Populus and Ashcroft polls is that I’m now about 95% confident that if we had a General Election tomorrow, Labour would poll somewhere between 28 and 37%.

    I think they’re tremendously enlightening for this reason.

  47. Election Forecast UK is making a prediction that will have some happier than others:

    Latest forecast update:
    Con 284
    Lab 280
    SNP 33
    LD 28
    UKIP 3
    PC 2
    Green 1
    Others 1 (not sure if this is Respect, the Speaker or what)
    NI 18

    In that prediction, we’re looking at a minimum 3-party coalition, a minority, or a Con / Lab pact.

    It’s looking increasingly messy.

  48. “It’s looking increasingly messy”

    Well, only if the GE was tomorrow and that was the result.

  49. @Oldnat

    I can’t believe I used valuable seconds of my life and Internet bandwidth for that link.

1 2 3 4