On cherry picking

I often read articles in the media where polling is casually misrepresented or misinterpreted and ponder writing something… but usually pass it by. While it annoys me, it’s not really enough to make a big fuss about. It does not warrant the much sought after “UKPR Crap media reporting of polls award”.

On the other hand, it bothers me that these things seep into the consciousness – if they sneak past once, then other people quote it and before you know it becomes accepted wisdom. My initial reason for starting bloging about polls in 2004 was because media reporting of them was often so slapdash and shoddy, and I should perhaps try to chase up more of this.

For that reason, here are two that I’ve spotted in the last few days. I hasten to add that they are not particularly unusual in these errors… they just happen to be the culprits I’ve spotted most recently!

In an article in the Telegraph Christina writes that the polling figures prove right her belief that the Conservative party is not struggling amongst women, citing an article by Nadine Dorries in ConHome here that claims “However, according to a You Gov survey the number of women intending to vote for the Conservative Party today in comparison to during the general election is down by just one point. The greater dip is in fact with men with a figure of minus three.”

As it happens, YouGov don’t show that at all (which should be slightly embarrassing for Odone, as it suggests she took what Nadine Dorries said without actually looking at the polling). The polling that shows the Conservatives down 1 point amongst women, but 3 points down amongst men does exist, but is actually aggregated data from Ipsos MORI’s political monitors this year.

Of course, it really would be a bit petty to haul someone over the coals just for attributing poll findings to the wrong company. The problem here though is that YouGov data looking at differences between men and women also exists, and shows a different picture to MORI. While MORI have the Conservatives having lost marginally more support amongst men than women, YouGov show the Conservatives having a significant advantage amongst women back in 2010, which has subsequently disappeared. I looked at the two sets of data in more detail here.

Two lessons here. First – don’t believe what people writing about polls claim they show, go and look at the figures yourself. Secondly, polls may have conflicting messages, so don’t assume the first thing you find tells the whole story, keep looking (and certainly if there are conflicting messages, don’t cherry pick the one that agrees with you!)

The second case is this story from PoliticalScrapbook, which claims that YouGov are saying the “Liberal Democrats slump to three percent with 18-24 year olds”. In one of YouGov’s daily polls last week in the 18 to 24 cross break the Liberal Democrats were indeed at 3 points, but this was not typical. YouGov’s daily polls typically include around 120 people under the age of 25. A sample size this low has a margin of error of plus or minus 9 points, in the case of the poll that had the Lib Dems on 3%, the sample had only 87 people under 25, a margin of error of plus or minus eleven points. Suffice to say, one should not give much weight to any individual cross break with a sample size this low.

A brief glance over the last couple of weeks of YouGov polls shows the level of Lib Dem support amongst under 25s varying between that 3 points at the low end, and 16 points at the high end… exactly as one would expect to find in a cross-break with a small sample size and a high margin of error. To take the 3 points as typical is misleading at best, though of course, that’s not to distract from the fact that the Lib Dems have lost a LOT of support from younger people.

Lessons here are that one should give due scepticism to figures from cross-breaks in polls, and pay particular attention to the sample sizes. A poll of 1000 people may have a margin of error of 3 points overall, but small cross breaks have much larger margins of error and are much more volatile. Secondly, be careful of cherrypicking data, in any stream of data there will be outliers at either end, and there is a temptation for those looking for sensational stories to pick out and highlight those outliers as if they are deeply meaningful. They aren’t.

UPDATE: Tonight’s YouGov poll for the Sunday Times has topline figures of CON 35%, LAB 41%, LDEM 9%. As usual, I’ll update properly tomorrow.


44 Responses to “On cherry picking”

  1. Still, it’s nice to dream of a world where the Lib Dems are on 3%.

  2. Nothing to say but “Yeah, what he said”.

  3. I completely take your point about margins of error Anthony, and I’m sure most if not all regulars here will understand the statistical logic behind it.

    However, not many people (including journalists and politicians) understand statistics very well, and I couldn’t see any disclaimer about margins of error on the link you gave.

    You can’t really blame innumerate people for quoting published figures.

    Perhaps the published tables should carry a potted version of what you wrote above as a standard feature. People still may not read it, but at least it takes away the excuse of ignorance.

    If I’ve missed a disclaimer somewhere in the published figures, please accept my apologies.

  4. Quick question folks.

    “YouGov’s daily polls typically include around 120 people under the age of 25. A sample size this low has a margin of error of plus or minus 9 points, in the case of the poll that had the Lib Dems on 3%, the sample had only 87 people under 25, a margin of error of plus or minus eleven points.”

    What’s the trick for working out a margin of error on a sample? I’ve worked with 1000-sized samples and confidence levels of 95% etc, but am unused to thinking in small samples.

  5. Statgeek –

    The simplest trick is to google “margin of error calculator”.

    :)

    (What’s the point of working out the formula yourself when people have already coded it into php for you!)

  6. I have a general rule that I ignore any opinion poll put on the front page of a newspaper. This is because newspapers sell copies by putting eye-catching stories on the front page, and no-one sells copies by reporting “The latest YouGov poll is in line with other recent polls and doesn’t really tell us anything new”. It’s more sensational to report a sudden rise or fall in the polls, which, when you take into account margin of error and seemingly ad-hoc arbitrary shifts in polling, rarely amounts to anything significant.

    Like most scientific activities, it’s rare to make an expected discovery in one go and more likely it will be found over a series of findings. So a more accurate poll report of a significant shift would be:

    Monday: Don’t bother with this one, nothing outside of statistical fluctuations today.
    Tuesday: Hang on a second, that’s a better than expected result for [Party] two days running. Not drawing any conclusions just yet, but we’ll keep an eye on it.
    Wednesday: Oh, maybe not. Gone down a bit.
    Thursday: Hold on, they’re back up again. Still within margins of error, but that’s three out of four.
    Friday: Okay, this time it’s just outside the margin of error. Could be something.
    Saturday: Nope, back in margin of error again.
    Sunday: Wait, it’s up again. Let’s give it a couple more days.
    Monday: Ah, up again.
    Tuesday: And it’s stayed there. Okay, we can now assume, on the balance of probabilities, that support from [Party A] has gone up a few %, but we’re not sure how much.

    Not holding my breath though.

  7. Today’s YG – 35C 41L 9LD Approval -28

    Nothing to see here, we’ll have to wait until tomorrow to cherry-pick something we like from the other questions polled. :-)

  8. CHRIS NEVILLE-SMITH

    Don’t knock it!

    It helps to pay Anthony’s salary, and without him, we would all be considerably less well informed.

  9. Best weekend poll for Labour for a while! It has really odd how the Labour lead has apparently shrunk at the weekend but grown again during the week.

  10. @Chris Neville-Smith

    http://en.wikipedia.org/wiki/Publication_bias

    Regards, Martyn

  11. Regarding errors due to sample size.

    I’d suspect it went as something like 1/Sqrt(N) however as the relative voting %ages aren’t completely independent this may not hold perfectly true. As a rule of thumb it shouldn’t be too far off.

    Reminds me of certain cross breaks from North of the border that get posted here with a sample size of under 100.

  12. A message from The Great Leader :-

    Europe will need 10 years to clean up its finances and emerge from the current debt crisis, German Chancellor Angela Merkel declared on Saturday.

    Resolving the debt crisis “is a path which calls for much effort, and along which we will have to advance step by step,” Merkel said, the day after the end of a G20 summit in Cannes, France, which was dominated by the issue.

    “The debt crisis will not be solved all in one go, (and) it is certain that it will take us a decade to get back to a better position,” she added in comments posted on her website.

    For that to happen, all eurozone nations would have to legally limit their debt levels, she stressed in the weekly web posting.

    “Everyone in Europe must make an effort to achieve all that is required,” Merkel said.

    h ttp://www.thelocal.de/nationa…

  13. @StatGeek

    Here is a nice article about margin of error for subsamples.

    Regards, Martyn

  14. @Colin

    Rough translation: mama smack

    Regards, Martyn

  15. Rough translation: None of us have a clue…. but we’ve all had a wonderful time posing with each other in Cannes.
    8-)

  16. Is anyone else concerned that Germany basically told Greece they couldn’t have a referendum?

    I wonder if that’s why we aren’t allowed to have one?

  17. Yes I am very concerned about that. Greece isn’t a baby, it’s an independent country with a remarkable, long & proud history and it should on no account be told by Merkel, Sarkozy or anyone else whether to have elections, referenda or anything else. It’s a very worrying precedent.

  18. I’ve never really been able to take Cristina Odone seriously (unlike Anthony I’m not on first name terms), something confirmed by a recent column attacking unmarried mothers. That’ll be the Cristina Odone who had a baby outside wedlock seven years ago then.[1] Not that I mind the birth, just the shameless “do as I say, not as I do” attitude.

    However anyone really wanting to confirm themself as an irony-free zone, need do nothing more that cite Nadine Dorries as a reliable source of information.

    I think Pete B is granting far too much tolerance to those “many people (including journalists and politicians) [who don’t] understand statistics very well”. We wouldn’t accept those same people saying that they couldn’t be bothered to learn how to read and write; we would be contemptuous of MPs who said they weren’t interested in what laws meant; if political articles had to explain what an MP is every time the term was mentioned, we’d think the papers were written for 8 year olds.

    And yet when anything statistical or mathematical comes up, ignorance becomes not only acceptable but almost praiseworthy. How often have you heard some celebrity titteringly admit that they barely passed Maths GCSE as if that made them more interesting? A journalist attributing a quote from Shakespeare to Churchill would be shamed – if s/he mixes up a median and an average who cares?[2] And yet only the second is important to how the country works.[3]

    I know there are some people who have real problems grasping numbers, but they are probably as rare as severe dyslexics. For most people it seems to be either a learned panic response (perhaps due to poor education) or, for far too many in the Establishment, a sort of wilful ignorance. If they can’t be bothered to understand it, therefore it can’t be important. As Anthony points out, With the computational powers of computers and the knowledge available on the Internet, there’s no excuse for not finding something out for yourself.[4]

    All you really need then is some clear thinking to think through what the numbers you are using actually mean and the willingness to look things up. Admittedly neither of these actions seem much practised in the media.

    [1] Which used to be a fact available on her now notorious Wikipedia entry, but which has now been purged. But nothing ever dies on the internet.
    [2] However since the sole purpose of political journalists seems to passing on dubious Westminster gossip from people they were at Oxbridge with, this doesn’t seem to matter. (I can also think of lots of people who are better at passing on gossip too, but they don’t seem to have joined the right dining clubs).
    [3] And if you thing it doesn’t, which would you rather earn – twice the average or the median wage?
    [4] Well obviously if you’re a !Kung bushman hunting in the Kalahari you might find it a bit difficult at the moment.

  19. @PeteB

    We are not having a referendum in this Parliament for the same reasons we didn’t have a PR referendum – Cameron doesn’t want to leave the EU (never ask a question if you’re not willing to accept the answer “no”) and Clegg doesn’t want to insist on one despite it being in the manifesto.

    The interesting question is not will there be a ref in this parliament, it’s whether the Blue, Red or Yellow manifestos will contain a commitment to have a referendum and – the crucial point – if so, what will the question be?

    About the disturbing aspects of preventing the Greeks having a referendum, it does need to be politely pointed out that the Greeks were delaying an extremely fragile deal that would (from memory) have written off ~$180billion of Greek debt, transferred ~$8billion to them immediately to prevent them running out of money, preserved the Greek state from immediate bankruptcy, and prevented (delayed?) the catastrophic collapse of every Greek bank and the consequent loss of life savings (thru bank collapse or forced devaluation) of every citizen with money in a Greek bank.

    One may opine that for that amount of money, asking the Greeks politely for an actual decision now instead of going “um” and futzing around for at least three months was, all in all, not that unreasonable.

    Regards, Martyn

  20. @ Pete B

    “Is anyone else concerned that Germany basically told Greece they couldn’t have a referendum?

    I wonder if that’s why we aren’t allowed to have one?”

    I doubt it. I think if Germany wants war again, Europe can stand back and we’ll simply unleash Israel on them. Problem solved (not that I advocate this route).

    I think the problem here is not Greece having a referendum but the way they went about doing this. Papandreou spent months negotiating with the EU for a bailout. He never bothered to mention during negotiations that whatever they came up with would be subject to a referendum. It was only until after they hammered out a deal that he supposedly accepted that suddenly, he sprung this on them. The time to have a referendum was a while ago, not now.

  21. @Martyn,

    I basically agree with what you say about maths and statistics, but I would make the point that statistics is an extremely arcane art, and understanding and expressing statistical data properly is something very, very few can do.

    English courts of law have almost no grasp of statistics, and I’ve sat through several consultations between (highly educated, intelligent and forensically-minded) barristers and scientific expert witnesses which left me wanting to cry in frustration.

    I suspect politicians and businesspeople are equally dumbfounded by stastistics. Journalists are not remotely unusual in their ignorance. I’d even volunteer that I personally didn’t understand the statistics module I was compelled to take at university, and like most of my year relied on a hard-working and bookish Yorkshire lad to help us through our exam.

  22. @ Amber Star

    “Rough translation: None of us have a clue…. but we’ve all had a wonderful time posing with each other in Cannes.”

    Lol.

  23. @ Martyn

    “We are not having a referendum in this Parliament for the same reasons we didn’t have a PR referendum – Cameron doesn’t want to leave the EU (never ask a question if you’re not willing to accept the answer “no”) and Clegg doesn’t want to insist on one despite it being in the manifesto.”

    I think your answer is pretty convincing. There’s a general legal strategy/rule of thumb. At a trial, never ask a question you don’t know the answer to. Whenever politicians decide to sponsor referendums, they are basically doing that of the public.

  24. @ANTHONY WELLS & MARTYN

    Thanks! Both good places to start.

  25. @NeilA

    You’re not wrong: Roy Meadow got an innocent woman put in jail for infanticide thru rubbish statistics. It was a gutwrenchingly appalling case and the RSS went absolutely ballistic (well, they wrote a strongly worded letter, but it’s ballistic for them) over it.

    @StatGeek
    You’re welcome

    @SoCalLib
    You’re welcome

    Regards, Martyn

  26. @ Amber Star

    “The very idea of holding the Summit in an iconic playground of the idle rich says pretty much everything we need to know about the objectives of this meeting… i.e. to main wealth & luxury for the few whilst preaching austerity to the rest of us.”

    I never knew that Cannes was the playground of the idle rich. Though I guess it makes sense. Maybe the next G20 summit should be held in Detroit. It’ll be a lot cheaper and will bring business to those most deserving.

    @ Neil A

    “English courts of law have almost no grasp of statistics, and I’ve sat through several consultations between (highly educated, intelligent and forensically-minded) barristers and scientific expert witnesses which left me wanting to cry in frustration.”

    It can go beyond people’s heads sometimes. Lawyers are typically not scientifically gifted or even neccessarily mathematically gifted.

  27. Breaking 2 rules here – linking to the paper we must not name, and carrying over from last thread, but this may answer Socal Liberal’s question:

    http://www.guardian.co.uk/commentisfree/cartoon/2011/nov/06/davidcameron-g20

  28. @ Aberdeen Cynic

    “Breaking 2 rules here – linking to the paper we must not name, and carrying over from last thread, but this may answer Socal Liberal’s question:

    http://www.guardian.co.uk/commentisfree/cartoon/2011/nov/06/davidcameron-g20

    1. Lol.

    2. Why can’t we name that paper? Is there any reason why at luxury hotels in London and Edinburgh and on Virgin Atlantic Upper Class, they don’t offer it as a newspaper reading choice?

    3. I doubt Cameron looks that bad shirtless or in speedos on the beach. In the one shirtless photo of Cameron, he was wearing a swim suit (I don’t think he’s a Speedos kind of guy) and didn’t appear to have man-teets. Doubtless, there are some politicians who, if Prime Minister, would probably be much better looking in shirtless pics on the beach. But that doesn’t turn Cameron into Alex Salmond.

    4. I’m not sure why there was such opposition to giving up the pound (I think it was because of the pound’s high value….ironically, the Euro seems almost as strong as the pound now) but in hindsight, you guys were extremely smart to keep it.

  29. Good Morning from such a beautiful morning on Bournemouth beach.

    I am not an Ed fan, but am I the only one who thinks he is growing in stature?
    His article, reported on BBC website this morning, mentions 1945, 1979 and 1997 as turning points in political culture, and then he says that 2011 is another such time, with finance capitalism in crisis, and protesters on the streets.

    A telling point, I think, supported by a leading Banker today.

  30. On Greece bailout, referendum etc.
    I think that the idea for a referendum in order to validate the October bailout deal was indeed a very unusual one, but on the other hand I sense that it was the only option left to Papandreou. His party (technically I should say “our party”, since I am still a member, but internal party life has been suspended for months due to the crisis and besides I have been very busy with the French socialist primary and did not really care about anything else, which might also be wrong), his party then has been left with 152 MPs , after a number of defections, and other MPs threatened to do the same, leaving him with no majority (151 is OM in Greece). And since no other party seemed willing to share this tough decision (since it is accompanied by draconian austerity measures), Papandreou had to threaten with the referendum, so that everyone in Greece see what the Merkel-Sarkozy reaction would be and come into their senses, i.e. understand that there is no other way out. And indeed now we are talking here about a unity government from all main parties (except radical left ones) that will close the deal. This all makes sense, but it also shows (in conjunction with what happens in Italy, which I find very disturbing, despite my visceral dislike for Berlusconi who seems really finished now) that EU is in reality Frau Merkel. So, beware UK citizens: do not join the Euro unless you want Frau Merkel to tell you when, what and for whom to vote, what to do with your money, how many hours to work, when to sleep, what to eat and what to watch on TV!!!

  31. @Martyn,

    Roy Meadow is a very good example of both doctors and lawyers/courts misunderstanding statistics.

    In terms of an example of miscarriages of justice, I am less comfortable with it however. I have personally come across several cases of Induced Illness, and during my training I have watched covert videos of women actually harming their children in hospital.

    Interestingly, the medical and legal establishment in the USA has pretty much rejected the criticism of Meadow out of hand and still proceeds on the basis that his overall approach to such cases is correct.

    It is one of the areas of law that makes me wonder about the use of juries. There is a natural inclination on the part of lawyers and judges to “dumb down” any case that is complex and academic in nature, so that the jury can understand the issues. I wonder if such cases should be heard by specialist courts so that that the evidence can be presented and challenged in the proper, rigorous technical language that is appropriate.

  32. Virgilio

    Thanks for that interesting commentary.

    I had the feeling throughout, that Papandreou was just trying to make his people & his Parliament accept responsibility.
    I think history ( assuming he will shortly be history) will judge him kindly -at least on this episode.

    May I respectfully point out re :-

    ” unless you want Frau Merkel to tell you when, what and for whom to vote, what to do with your money, how many hours to work, when to sleep, what to eat and what to watch on TV!!!”

    It wasn’t “their own money-Greece doesn’t have any-only other peoples money.

    I don’t think anyone would have an issue with Greek working hours, provided their standard of living was geared to them-and not in need of support from other peoples working hours.

    ………but I agree that interference in Greek politics ( as seems to have been the case) was a step too far.

    Greeks need to be allowed to make their own mistakes, like everyone else.

  33. @Colin,

    Is it just me, or is it very enlightening to see Virgilio’s comment echoing virtually exactly the sort of thing that Tory eurosceptic rebels used to say at the time of the Maastricht Treaty.

    Then it was roundly condemned as paranoid, backwardlooking, the-Germans-bombed-our-chippie hyperbole.

    Now it is pretty much established fact, as expressed by a socialist from mainland Europe.

    I refer back to my comment on a previous thread about the “slander” eurosceptics used to endure!

  34. @Coiin
    I certainly agree that my (half-)compatriots were very irresponsible in dealing with their finances, and that Greek political class’s only concern was how to buy votes by distributing money that were not theirs . But the attitude of major Eurozone powers is like the one of the parents that spoil their children for years in the most exaggerate manner, and then threaten them with the most terrible punishments if they do not learn to behave overnight. Greece is in the EU since 1981 and in the Eurozone since 2000. How come that no one had noticed their irresponsibility before 2009? But all this was OK since with that money that came from the sky Greeks bought BMW and Mercedes cars, Siemens electrical appliances, and so on and so forth. And is it really reasonable to expect from them to change in a few months all their thought patterns, habits, behaviors etc without a major political turmoil, whose first victim is precisely Papandreou, one of the few Greek politicians that was against all these manifestations of clientelism and corruption?

  35. NEILA

    It’s been a while coming-yes.

    And its no comfort that it is vindicated, given the circumstances. I see DC talking about six years to get out of this EZ crisis-at least that’s better than Merkel’s decade I suppose.

    It has enormous implications for UK.
    DC’s stance of -stay out of EZ-stay in EU & retrieve powers looks sensible on the face of it if you are a eurosceptic realist rather than an out & out europhobe.

    But I fear that it is going to unravel, because I don’t trust Merkozy. I see the EZ “caucus” which GO voiced fears about being very much on their agenda.

    The EU could become a club in which we have no say whatsoever, because we don’t belong to it’s “inner sanctum”. Indeed it is the EZ which Merkozy are trying to save -that is their priority-everytning else is subordinate to it.

    Is EU membership without EZ membership a viable & practical option for UK going forward?

    Is it in our interests-and if it is , how can those interests actually be protected?

    The 10 non EZ members of EU are actually 2 or 3 ( UK . DK, Swe ) who will never join , plus 7 or 8 who are waiting to qualify.

    Do the latter still want in?-can they actually change their minds legally, even if they wanted to?

    Does membership of a club of 3 attached to a decision making caucus of 24 ( using QMV) make any sense at all?

    …..will “UK Out” be the only sensible policy -because that’s what the 24 want?

  36. VIRGILIO

    Good questions.

    Merkozy conveniently forget that they too breached the Stability Pact when it suited them.

    I wonder how long it will be before the resentment which you voice on behalf of Greeks at the ruling dupoly of Germany & France, spills over into Italy-Portugal-Spain…….?

    Is the EU actually governable now?

    Is it able to reach decisions which meet the approval of ALL its members?

    When Obama observed that there are “too many institutions” in EU he was dead right. Van Rompuy / Barosso/ Merkel/Sarkozy/EFSF/ECB…… its a Dog’s Breakfast .

  37. @Neil

    And of course Sally Clarke, the mother in the case involving Roy Meadows, died of alcoholic poisoning.

    Did she drink because of guilt, or grief at the death of her two babies? I guess only she knew.

    Interestly, according to Wiki, her father, who complained about Meadows to the GMC, is a retired senior police offer. Obviously he believed in his daughter’s innocence.

  38. “Did she drink because of guilt, or grief at the death of her two babies? ”

    Or as a result of the trauma of being prosecuted and jailed for their deaths?

    The entire episode was an utter disgrace. The idea that Meadows should have been anywhere near a witness box is a joke – much of what he said on oath as a supposed ‘expert’ witness could only have been said by someone who was either dishonest or incompetent. His statistical statements are now a case study in incompetence (and are often used as such in the teaching of particular statistical principles), likewise his apparent lack of awareness (and ongoing denial) of the demonstrable natural causes of SIDS. Even given our current (very incomplete) knowledge of these, multiple infant deaths within a family are *expected* to occur quite regularly, somewhere in the country.

  39. @Robin

    I agree with you.

    As a children’ social worker in the 90’s, one of my cases involved a family where 2 of the 4 children had died of SIDS.

    People were pretty concerned when the mother became pregnant again. Fortunately the medics in the case did not fully subscribe to the Meadows theory and we worked with the parents to look at ways of reducing the risks ie smoking was banned in the house etc.

    Thank god the baby thrived.

  40. @Robin/Valerie,

    I absolutely agree that Meadow’s statements in court were wrong, and that for a man with his supposed grasp of the subject (and with the power that comes from having a reputation such as he enjoyed at the time) his erroneous evidence was indeed a disgrace.

    But it is worth remembering that the first appeal bench that reviewed the case held that even without any of Meadows’ evidence, the case against Ms Clarke was strong enough to support a conviction. It was only the emergence of some disregarded pathology results that clinched the second appeal.

    As someone who worked in the field at the time, and still does, my concern is that the Meadows/Clarke example is routinely trotted out, usually alongside expressions like “his now discredited theory of MSbP”.

    Parents do, quite frequently, cause deliberate harm to their children and sometimes kill them. The Meadows case has made gettting justice for those children much, much harder (which is another reason his behaviour was a disgrace).

  41. @ Virgilio

    Who won the presidential primary?

  42. @ROGER MEXICO

    A median is an average.