Tony Twyman, who died last year, was the man behind much of the mechanics of TV and radio viewing figures, most notably as technical advisor for BARB viewing figures. In broader market research he is more widely known for coining Twyman’s Law – “Any figure that looks interesting or different is usually wrong”. The point is, of course, that strange and unusual things in a single poll are more likely the result of sample variation or error than some amazing shift in public opinion, and you should be cautious of them before getting excited (My colleague Joe Twyman likes quoting it without attribution in the hope people will jump to conclusions… not so fast!).

Anyway, today we have a classic case. Two polls that look interesting when compared to recent averages, but which are both probably no more than the result of normal sample error.

Today’s twice weekly Populus poll had figures of CON 32%, LAB 37%, LDEM 10%, UKIP 13%, GRN 4% (tabs). The five point Labour lead is the biggest Populus have shown since November, their 37% share the largest any company have shown since November. Labour resurgence?

Lord Ashcroft’s weekly poll however had figures of CON 34%, LAB 28%, LDEM 8%, UKIP 16%, GRN 8% (tabs.) A six point Conservative lead, by far the best poll for the Tories from any company for several years (the largest Tory leads up to now were the last two MORI polls, which had them three points up). Tory surge?

Of course the actual answer is that there is probably neither a Labour nor a Tory surge, that both of these changes are probably just down to sample error and that people should watch the overall trend across multiple polls, not get overexcited about individual polls. If the figures in one poll look strange or unusual, it’s probably wrong.

In some ways it’s quite nice they come on the same day, as it should stop people getting too excited over an outlier in just one direction. On the other hand, it does tend to produce lots of confused comments about how polls can be accurate when they are showing both a five point Labour lead and a six point Tory lead. Bottom line for those who are confused, part of it is down to pollsters using slightly different methods (in this case, the way Populus weight their polls tends to produce a bigger share of the vote for the main two paries than does Ashcroft). A bigger chunk will be simple margin of error – polls are not precision instruments and no one who understands them would claim they are. They are randomish samples of about 1000 or so people. The quoted margins of error are about plus or minus 3% (though given response rates, weighting effects and that polls are not pure random samples, that’s a bit of a polite fiction). That means if the real position was Labour and Conservative tied on 33%, you would expect to see the Conservatives ranging from 30% to 36% and Labour from 30% to 36%, and while the results would tend to be clustered around the middle of that range, random variation could reasonably vary between a 6 point Tory lead and a 6 point Labour lead. Taken alone and in isolation, it does mean an individual voting intention poll isn’t that useful… which is why you shouldn’t look at them alone and in isolation – watch the trend.


200 Responses to “Contrasting Populus and Ashcroft polls”

1 2 3 4
  1. Regarding reference to the Ashcroft pole in The Daily Telegraph, the same article mentioned the Populus pole.

    Please remember this is a site for non-partisan discussion of poles.

  2. @ Alec

    Can it be that there use to be lot’s of false positive diagnoses? Can it be that use to be diagnosed as Alzheimer’s disease is now classified somwhere else? Do we have a lot of false negatives now?

  3. Lot’s = lots

  4. LASZLO
    Lot’s = lots
    _____

    Presumably ” somwhere” = somewhere?

    Sorry I couldn’t resist…………….. ;-)

  5. “Please remember this is a site for non-partisan discussion of poles.”

    Well I think they should be as welcome here as we are in Benidorm.

    [Actually I’m not surenow if that wasn’t partisan as I s’pose it suggests I am not a UKIP supporter…………..]

  6. Interesting that the BBC reports on the new cartoon on the cover of Charlie Hebdo but refuses to show it – even the words being spoken by Prophet Muhammad [peace be upon him………]

  7. Alec (re: deflation)

    It seems that most of the downward pressure on inflation are consumed goods.

    Petrol and food. People aren’t going to hold off buying these in the expectation they will reduce further. If it was deflation across the board that would be very concerning as you say. Right now anecdotally it seems people have a few pounds more in their pocket and are spending them on stuff they want.

    There might be a delay to investment in oil exploration until prices recover

    I think interest rates are going to remain very low for a long time, the Eurozone won’t sort itself out anytime soon which will have a knock on effect here.

    I doubt QE will be on the table unless Europe really throws a wobbly. We at least have the leeway to go back to that if needed.

    […]

  8. @R and D,

    Sometimes I think with the news less is more.

    They have reported the story, and if anyone wants the fine details or view the image they can find elsewhere easily.

    There are numerous times in the past where I think a story had details that are controversial or graphic, and the coverage became more sensationalism than news reporting. For example, such issues include massacres, where I want to the outline of the story, but filming rows of dead bodies becomes sensationalist and more than news reporting.

  9. @Number Cruncher

    The percentages are the proportion that each region in the YG polls is weighted to. The old numbers are per my page on YG’s polling:

    http://www.statgeek.co.uk/polling/

    The new numbers are a mystery to me. I’m not aware of London suddenly becoming 10% of the UK (less than 6.5 million in London?). Perhaps YouGov are putting in some synthetic population weightings to correct flaws?

  10. Haley,

    I didn’t make it clear but myreference to Medication was not to do with treatment of the condition but rather treatment of things like heart disease in earlier life which improve general health and blood flow, from Statins to Warfarin.

    The indication seems to be that people who are healthier in their forties to sixties are less likely to see mental deterioration in their seventies and eighties.

    Peter.

  11. @ Statgeek

    Not exactly what you were posting about, but I enjoyed the Yes Minister clip you inserted at the bottom of that page.

    Tutorial #1 for all poll watchers…

  12. Statgeek – haven’t changed the weighting targets for the regions, and we’re still weighting to the same old figures (and I’ve just gone and checked the last three sets of tables, and they all have the usual 12.8% for London as they should be)

  13. @Anthony

    Hmm. Looking at poll 9th of January (weighted):

    London: 183
    Total Poll: 1684
    10.87%

    12th Jan:
    London: 170
    Total: 1649
    10.3%

    Maybe I’m missing something, but that data used to come to 12.8%

  14. @ Statgeek

    You are posting the unweighted proportions… The numbers this year are:

    10.3
    10.9
    11.6
    13.1
    12.7
    12.2

    And as Anthony says, the weighted proportions are:

    12.8
    12.8
    12.8
    12.8
    12.8
    12.8

  15. I have sen studies suggesting a link between higher education level and lower risk of dementia so perhaps improvements in education and literacy are partly responsible.

  16. Ye Gods…where’s the blush smiley?

    I’m so in the habit of taking note of the unweighted samples…the grey row was all I saw. Sorry to all…maybe I can take solace in the fact that I have some sort of flu bug and will blame it all on that.

    Going back to bed.

  17. Umm, this is descending into partisanship based on Alan’s (clearly) debatable original post and Lefty’s riposte.
    I’m tempted to weigh in on Lefty’s team but a) it’s boring b) it’s against the rules and c) I doubt there is a single poster or lurker on here who will take any notice.

  18. The Tory V Lab deficit debate seems broadly to be.

    Tory; Deal with the deficit through Austerity then the economy will grow.
    Lab; Let the economy grow without Austerity and the deficit will fall.

    In the Lab view if the deficit is 10% and we don’t cut it but grow the economy by 10% the deficit will be the same in cash terms but fall to 9% of the larger GDP.

    I think the Tories tightened to much and didn’t clamp down on tax evasion enough but I think the Lab idea that we would have seen high enough GDP growth and tax revenues to see growth slash the deficit not much more convincing.

    I suspect that’s why when you look at both Partys plans for the next five years and compare them to either UK GDP or overall public spending or borrowing in percentage terms there really isn’t that much between them.

    Peter.

  19. @GUYMONDE
    Bravo mon brave, absolutely correct in every detail. (Except the team you would have chosen.)

  20. “we’re beginning to get perilously close to deflation”

    I am fascinated by the idea that 0.5% inflation is ok and 0.1% deflation is a problem.

    Inflation and deflation both come in good and bad varieties, depending on the cause. If there’s a negative oil shock, a rise in prices is good because it’s the price mechanism responding to the increased scarcity of oil. If there’s a positive oil shock, low inflation and even deflation can be good because it’s the same thing. On the other hand, inflation caused by excess overall spending (as in the 1961-1992 period) or deflation caused by a shortfall of spending (as in the 1920s/1930s or 2009) are “bad” in that they are caused by something that can and should be avoided.

    It’s like sweat: you can be sweating because of exercise or because of a high fever. Healthy people can sweat and unhealthy people can sweat, and you can’t tell from the fact that someone’s sweating alone whether they’re healthy or not. The Eurozone has a very dangerous fever; the UK still has a cold, but our sweat is mostly the result of vigorous healthy exercise.

    So deflation in the Eurozone is a sign of a bad thing (not enough demand) and deflation in the UK/US is a sign of a good thing (a fall in oil prices largely caused by the structural weakness of OPEC). The Bank of England is doing the right thing by sitting on its hands, though if growth drops it should be ready to reverse gears and increase QE.

  21. cmj

    Don’t really disagree [and I just said it was “interesting”]

    My only worry is if it ever becomes a slippery slope, with a select group of people attempting to determine what we are allowed to do within our own borders when we ourselves have no laws against what is being done.

    It is only in that respect that I do have some concern.

  22. Bill Patrick,

    “So deflation in the Eurozone is a sign of a bad thing (not enough demand) and deflation in the UK/US is a sign of a good thing (a fall in oil prices largely caused by the structural weakness of OPEC).”

    Unless the lack of demand in the EZone is because they are averse to Government, Corporate and Household debt driven growth, while we here thinking things are getting better because the Oil price has fallen are driving our economy forward with another credit binge!

    Peter.

  23. CATMAJEFF

    ” For example, such issues include massacres, where I want to the outline of the story, but filming rows of dead bodies becomes sensationalist and more than news reporting”
    ________

    In 16 weeks time I think we will see a lot of this sort of reporting in Scotland after the polls have closed. ;-)

  24. The intervention of IFS into the fog of the Deficit reduction political debate is most welcome.

    At least IFS have tried to quantify the Con/Lab difference. It is more than we would have got from either Party.

    On the question of “deflation”-UK is probably not going to experience it. The lower inflation is due to retailer price discounts & commodity price falls. It is probably short lived . This is good for UK consumers while it lasts.

    In EZ, on the other hand deflation reflects falling demand.Yields on German bonds — negative rates for two year bonds & zero for five-year — tell you that investors expect inflation to be close to zero for a long time.

    ECB is desperately trying to find a way to launch QE in a system with no connection between monetary union & fiscal sovereignty.
    The daftest idea I have read yet received this comment from the FT :-
    “The absolute worst variant of QE, currently under discussion, is one where each national central bank buys the debt of their own governments, which would effectively be the end of a single monetary policy. If that happens, you will wish that you never asked for QE in the first place.”

    Putting procedure & the intense antipathy of Germany aside, the most damning commentary on the proposed ECB monetary stimulus would seem to be that in a system of different capital markets to UK/USA , inflexible employment laws & structural professional cartels, where lenders currently cannot find willing borrowers , it is probably a damp squib in the making.

    Serious commentators are suggesting that EZ might be one of the few situations where a “helicopter money drop” might be the only answer.!

  25. I love politics, particularly its ability to transform peoples lives for the good, and there are dark arts, even skullduggery, involved in bringing about that greater good. The Good Friday Agreement in Northern Ireland was one such example and I’ve just been watching Tony Blair’s questioning on the “On The Runs” issue by the Parliamentary Select Committee. The Old Master was in fine form, and whatever you say about the man, who’s reputation and legacy have been routinely and lazily trashed since he left office in 2007, it’s nice to be reminded occasionally of what a masterly politician he was and, in many ways, still is.

  26. ROSIEANDDAISIE

    The new cartoon was shown briefly on BBC News at lunchtime.

  27. AW

    Point taken.

  28. Peter Cairns,

    I don’t see the credit binge in the UK, and the kind of bad deflation that the Eurozone is undergoing will make their debt problems worse, not better. It’s basic commonsense: if your debts are denominated in unadjusted prices, and your nominal income is falling, your debt-to-income ratio will tend to rise, with potentially appalling consequences. This is one of the reasons why financial instability and nominal income instability are correlated.

  29. “we’re beginning to get perilously close to deflation”

    We’ll be following the EU into deflation eventually – slowed down at the cost of the welfare bill by McDoom’s tax credits – but the oil price drop will distort things for a bit.

  30. to expand

    deflation is a reduction in *measured* economic activity (which is bad) but because it’s measured in money if a primary cost goes down (e.g. oil) then you can have exactly the same economic activity but with a lower monetary value (which isn’t bad) plus the lowered costs would increase economic activity back to where it was soon enough anyway (if people thought it was permanent and maybe even if they didn’t)

    so for example if someone invented fusion there’d be a massive nominal deflation at the same time as a huge jump in prosperity

    the bad kind of deflationary spiral i.e. the one we’re in but slightly behind the EU is to do with the contraction of demand caused by (imo) banksterism and mass immigration

  31. MrJones,

    “demand caused by (imo) banksterism and mass immigration”

    And just how does mass immigration cause a drop in demand and why hasn’t it happened here?

    Peter.

  32. Rob Wainwright, , head of Europol ,appearing in front of the Home Affairs Select Committee, said ‘We are talking about 3,000 to 5,000 EU nationals who are potentially a threat.”

  33. Colin

    I suspect Germany would like to be in control of the helicopter.

    (Just like they want to be in control of which bonds get purchased)

    Bill Patrick

    Nice summary.

  34. Colin,

    ” ‘We are talking about 3,000 to 5,000 EU nationals who are potentially a threat.”

    Sounds a lot but with with 500 million citizeans in the EU, some 7.3% of world population it is about 1 in 100,000 which is a bit like a needle in a haystack!

    Scotland share would be about fifty “Potentially” dangerous individuals.

    Like I mentioned earlier quoting the absolute number be it 5,000 Jihadists or £170bn of debt over a parliament without pitting it in context it tells us very little.

    It’s like looking a one poll without the trend!

    Peter.

  35. Great comment on PBS. An extremist wants to kill you.A moderate wants an extremist to kill you.

  36. @Crossbat11

    Being non-partisan here, I thought TB sounded commanding too.

    When you look back you remind yourself of the way both he and MT reigned supreme in the political arena.

    I don’t know if it was good or bad to be honest, but none of the current leaders come even close that sort dominance. When you scan the current Parliamentary intake, I can see no contenders either.

    Maybe they were the last giants of an older political age, and the next great leaders will be those able to cobble together an increasingly diverse electorate.

  37. @Peter

    And just how does mass immigration cause a drop in demand and why hasn’t it happened here?

    Don’t you know?

    I thought it was obvious mass immigration is blame for everything.

    ;-)

  38. Catmanjeff,

    “I don’t know if it was good or bad to be honest, but none of the current leaders come even close that sort dominance. When you scan the current Parliamentary intake, I can see no contenders either.”

    Not even Stewart Hosie?

  39. https://pbs.twimg.com/media/B3DcBDZIEAAZDhg.jpg

    Not sure that Tony Blair could have a crowd that large that would cheer for him.

  40. @Rosie and Daisie

    Don’t really disagree [and I just said it was “interesting”]

    My only worry is if it ever becomes a slippery slope, with a select group of people attempting to determine what we are allowed to do within our own borders when we ourselves have no laws against what is being done.

    It is only in that respect that I do have some concern.

    I think we just need a period of calmness, as the current atmosphere is creating more heat than light.

    I understand the cartoon has been shown (thanks for that @TOH), so I guess it’s matter showing any sensitive material in moderation and context, not pasting it everywhere provocatively.

    I’m sure a sensible way forward will emerge.

  41. @Bill

    I’ve never heard of him.

    Was he in Celebrity Big Brother?

  42. PETER CAIRNS

    I don’t think its a needle in a haystack-EU Security & Intelligence seem to have many of them identified. The problem seems to be manpower in keeping tabs on all of them all of the time. As one of the Spooks said recently-there are more of them than there are of us.

    I’m not sure what you mean by “context”. But if your use of threats headcount vs population headcount is a guide-then I think you are seriously misguided.

    A phrase you hear often is -they only have to get it right once-we have to stop them every time. And each one of them , with the right logistical & financial support , can, if they so wish kill hundreds.

    If you have followed debate on this issue recently you will know that intelligence people say the days of the Big Spectacular attack are over. The modus is now targetted attacks with sophisticated weaponry and many footsoldiers who are ready to die.

    As we have seen-if the targets are chosen with care, their significance for a nation’s moral & internal cohesion can far outweigh the mortality rate.

  43. Catmanjeff,

    If there is justice in this world, he IS Big Brother.

    (I have nothing against Stewart Hosie.)

  44. ALAN

    Don’t think so.

    Their representative on ECB board always objects to the proposal to buy sovereign bonds .

    The idea of giving the stuff away, for free, is not calculated to appeal to the Bundesbank. Isn’t going to happen, I think.

  45. So the Labour MPs voted with the Tories on Austerity today. A couple of comments

    1. Will cause more problems for Labour in Scotland. It probably won’t make it into the news bulletins but is on social media. I watched the debate and I noticed Ed Balls being extra-nice to the SNP. Saying ‘I’m sure you have more in common with us than the party opposite’ which made me think he is preparing for possible SNP support after GE. BTW a Tory MP was talking about a Labour Tory grand-coalition after the GE

    2. Difficult to keep the anti-Austerity parties out of the debates. SNP/Greens/PC looking very co-ordinated and friendly

    As I understand it – anti-austerity is no cuts as long as can pay debt interest and to have a flexible medium term solution to balancing books like New Zealand.

  46. “And just how does mass immigration cause a drop in demand and why hasn’t it happened here?”

    Upward pressure on living costs + downward pressure on incomes = reduction in disposable income so economy shifts from discretionary spending to necessities

    Example just for illustration:
    step 1) one person getting 2K with 1K going on necessities and 1K left for discretionary spending
    step 2) two people getting 1K each so it all goes on necessities
    step 3) shift from 1K necessities + 1K discretionary -> 2K necessities

    You could say that would just lead to a shift in demand with a reduction in discretionary demand and an increase in the demand for necessities but if discretionary spending has a higher velocity (which i think it does) then that would lead to an absolute reduction also.

    In a nutshell: the discretionary demand that used to exist is being sucked into the black hole of housing costs instead.

  47. @ Colin: “Serious commentators are suggesting that EZ might be one of the few situations where a “helicopter money drop” might be the only answer.!”

    That will actually be the effect of falling energy and food prices, which will leave more money in EZ consumers’ pockets.

    Unusually, I believe there’s a good chance the EZ could be poised for a period of rapid growth. Some of the erstwhile basket case economies (e.g. Spain) have seen a steady improvement in competitiveness, while the whole EZ benefits from the recent depreciation of the euro.

    It’s a bit ironic that in large part the EZ has successfully reduced deficits and debt, avoiding a credit binge, thereby incurring massive schadenfreude from those UK observers who champion debt reduction in the UK. Meanwhile, we have record indebtedness, an unsustainable balance of payments deficit and resulting GDP growth which leads to crowing over our supposedly superior economic performance. Hubris or what?

  48. “That means if the real position was Labour and Conservative tied on 33%, you would expect to see the Conservatives ranging from 30% to 36% and Labour from 30% to 36%”

    OK, I think I have realised the source of my confusion. I think by “real position”, what the author means is the MEAN of the distribution of all potential polls from, say, Ashcroft that potentially could have been conducted on that day.

    But I think we need to understand exactly how unlikely a poll like this would be.

    If we’re saying this is a distribution where 95% of all potential polls fall within the range 30% to 36%, we’re also saying that this is a distribution where 97.5% of all polls would be ABOVE 30%, to make it one tailed.

    But in Ashcroft’s case we have a poll that is at 28%.

    So “back of a fag packet” calculations, for a one tailed normal distribution, the 2.5% cutoff represents 1.96 standard deviations, so we can say that 1.96 standard deviations are equal to 3%. This makes a single standard deviation 1.5306. Ashcroft’s poll is a whole 5% below out expected mean, which makes it 3.27 standard deviations below the mean of our distribution. From our Standard Normal Curve, we can see that the probability of a poll being 3.27 standard deviations below what we expect is 0.5-0.4995, or 0.0005.

    When we say “outlier” we need to understand just what this means, it means that the probability of this poll occurring by chance, from a distribution with a mean of 33% and a standard deviation of 1.5306 is 1 in 2000.

    That’s an extraordinary unlikely event. So I think we have to assume something else is going on here.

    The same applies to the Populus poll, by the same calculation, we see that it’s 2.61 standard deviations out, with a probability of 0.0045. That has a 1 in 222 probability of being by chance.

    If Ashcroft conducted a poll every day for 5.5 years, then we’d expect the sort of outlier we see in this polls to occur once. He’s not conducted anywhere near that amount of polls, and it’s not the first time he’s shown really bizarre results. When we consider the mistake made during the polling of Miliband’s constituency, I’m inclined to think that there has been some sort of methodological error here, rather than it simply being a freak poll.

    Populus on the other hand have tended to show larger Labour polling, and I don’t accept that their poll average for Labour is 33%, they haven’t given Labour a poll number of 33% throughout 2014, and have only shown Labour on 34% twice. So their 37% is more consistent with their own polling anyway.

  49. Jus to add to my last post, in the past six months, Populus’s average polling for Labour is 36.02%, and they’ve shown Labour with 34% on three occasions (yes I said twice in the previous post, my mistake), at 35% on 19 occasions, 36% – 17 times, 37% – 16 times, 38% – 5 times.

    So I dispute that the Populus poll is an outlier based on Populus’s methodology.

    Also, just having a quick look at Ashcroft’s polling, he tends to show Labour lower than 33%, I haven’t checked all his polls, but the last few have been, for Labour:

    28%, 32%, 30%, 29%, 29%, 31%, 31%, giving an average of 30% for Labour.

    So Ashcroft’s poll is also not actually THAT much of an outlier.

    It would be a more interesting question to ask, why are Ashcroft’s polls giving Labour a 30% average, and Populus’s giving Labour a 36% average.

    Because that’s a huge difference.

1 2 3 4