Last month Chris Elliot, the Guardian’s readers’ editor, quoted a letter from a reader saying there “seemed to be a cultural problem among Guardian reporters that it is of no consequence if you completely misunderstand or mis-report the figures in a story […] I hope that you can urge on the editor some training of reporters on basic understanding of statistics”. Chris Elliott said he had organised three sessions with external statistical experts for Guardian journalists in the past year (and Nigel Hawkes at Straight Statistics reveals he was one of them).

The Observer’s readers editor should probably do the same. Earlier this month the Guardian’s front page story mentioned an open-access voodoo poll on the Royal Medical Journal’s website that had been touted round Twitter as if it was meaningful. The Observer this weekend was on a similar subject, but was worse – hanging a whole story on very dubious figures.

The story is titled “Nine out of 10 members of Royal College of Physicians oppose NHS bill”, and claims that “a new poll reveals that nine out of ten members of the Royal College of Physicians – hospital doctors – want the NHS shake-up to be scrapped.”

The story is based upon an open access survey created by and linked from a website campaigning against the heath bill, callonyourcollege.blogspot.com, and again, bandied around Twitter. The survey was open access, so there could have been no attempt at proper sampling and contained no demographic information that could have been used to weight it. It should go without saying that a survey from a website campaigning against the NHS reforms and co-ordinating opposition to it amongst the Medical Royal Colleges is more likely to be found and completed by opposed to the bill (in much the same way that a poll carried out on, say, the Conservative party’s website, might be considerably more supportive).

Any poll actually measuring the opinion of members of the RCP would have needed to randomly sample members, or at least contact members in a way that would not have introduced any skew in those likely to reply. For all we know this may have also shown overwhelming opposition – but we cannot judge that from an open-access survey liable to have obtained an extremely biased sample.

Once again, I would urge any journalist thinking of including any polling figures in a story to look at this guidance from the British Polling Council, particularly on how to judge whether to take a poll seriously or not. If these had been looked at, the Observer should never have got to this point…

Who conducted the poll? Was it a reputatle, independent polling company? If not, then regard its findings with caution

In this case, the poll was not conducted by a polling company, but by a group lobbying against the bill they were asking about. This should have been the first alarm bell.

How many people were interviewed for the survey? The more people, the better — although a small-sample scientific survey is ALWAYS better than a large-sample self-selecting survey.

In this case, the number of people interviewed is not mentioned. It could be high, it could be low. But note Peter’s other point… this was a self-selecting survey anyway…

How were those people chosen? If the poll purports to be of the public as a whole (or a significant group of the public), has the polling company employed one of the methods outlined in points 2,3 and 4 above? If the poll was self-selecting — such as readers of a newspaper or magazine, or television viewers writing, telephoning, emailing or texting in — then it should NEVER be presented as a representative survey.

This was a self-selecting poll of doctors directed there from a site campaigning against the legislation. There is no way it should have been presented as a representative survey.

UPDATE: Credit where it is due. Denis Campbell, one of the authors of the piece, wrote about the same poll on the Guardian’s rolling blog the next day, but this time caveated it with “But that was to a website run by anti-Bill doctors and a self-selecting rather than scientific poll, so may not reflect opinion precisely.” In a perfect world I’d hope that journalists would spurn non-representative polls completely, but progress nonetheless.


46 Responses to “The Observer and Voodoo polling”

  1. Are you perchance therefore suggesting that there is a majority of physicians IN FAVOUR of the Governments health bill. But ‘voodoo’ polls are trying to cover this up?

    At the risk of incurring your wrath due to the ‘unscientific’ nature of my straw poll, But at my major teaching hospital in Scotland I could not find a single medic who supports the Governments Health Bill.

    The right seems to be in denial that most of those involved in health care, plus most patients dont want this rotten bill to become law.

  2. Dodgy, those Grauniad hacks.

    Speaking of which, Levenson today is dynamite. Looks like jail for Coulson for perjury, serious problems for Brooks and bribery of officials meaning real trouble for News International in the States.

    Can’t wait for Levensoin to look at the power of Murdoch over politicians and undue influence. Be embarrassing for most PMs going back to Thatcher…and if Coulson DOES go down, will we get that promised apology from the present PM (?)

  3. Anthony W

    Shocking stuff, hey. Those damnable rogues at the Observer and Guardian at it again. Why anyone ever believes a thing those scoundrels say is quite beyond me.

    Before I accuse thou of protesting just a little too much on this, have you any experience of other newspapers using similar underhand tactics to misrepresent support for certain causes that they either support or oppose? The Express on European issues, the Mail on immigration, the Sun on crime, perhaps, or do you think they’re always just telling it as it is, whereas the left wing charlatans in the press are twisting the truth to serve their dastardly aims?

    Just a thought.

  4. Kevin – no, not at all. The only actual polling I’m aware of NHS staff (YouGov for 38 Degrees- tables here) found strong opposition to it. Obviously that was NHS staff as a whole, rather than doctors, but I’ve no evidence to suggest that doctors’ views differ from NHS staff as a whole (though equally, I’ve no evidence to suggest they don’t).

    It’s not the particular issue, but the use of biased polling I object to. Journalists should treat polls with due scepticism whether or not they agree with the findings and too often they seem to accept without question things that support an agenda, reserving scepticism for those things they disagree with (not, I should say, a failing that is limited to journalists at all, but one it is probably more important for their profession to overcome!)

  5. Crossbat1 – I certainly do! I found a cracking example from the Daily Mail the other day, I was saving it up for a rainy day but I think it wrenches away my “worst ever reporting of an opinion poll” award from the previous holder, the Daily Mirror.

    http://www.dailymail.co.uk/news/article-2040783/Conservative-UK-Almost-half-Britons-oppose-gay-marriage.html

    It’s from last September, but FullFact flagged it up in their evidence to the Leveson inquiry. The Mail headline says “Conservative UK: Almost half of Britons still oppose gay marriage” and goes on to say “Most people still oppose gay marriage and the adoption of children by same-sex couples”

    It was based on Eurobarometer data showing that 45% of people in the country opposed gay marriage. What the Daily Mail did not mention at all (until they were eventually made to write an apology – see the bottom of the article) was that 46% of people supported it… so actually rather than most people still opposing gay marriage, marginally more people supported it than opposed it.

  6. And the Express – the Express’s voodoo polls are a thing of wonder! http://ukpollingreport.co.uk/blog/archives/2890

    The best newspaper on reporting polls is normally the Times, because when he was there Peter Riddell understood these things, maintained good standards and did his best to make sure such things didn’t find their way into the paper. Danny Finkelstein there also actually understands polling (as does John Rentoul on the Indy, though their budget means the questions they ask are normally the god-awful agree/disagree statement grids). Sam Coates, who has written up the Times’s polling since Peter retired, is also good at it, so their write ups of Populus’s polls don’t usually contain the sort of hyperbole that I’m rude about elsewhere.

  7. I find it somewhat strange that GO has said we (the UK/government) has run out of money.

    I don’t intend arguing whether there is or isn’t any more money…or indeed what can be done…but I do wonder how joe public (and indeed the ‘markets’) will view this announcement.

  8. Poor grammar…”have” not “has”

  9. How can we run out of money as we print billions to buy gilts every few months?

  10. “voodoo poll”

    Definitely a case of jounalistic error, however, the “a new poll reveals… ” quote from the Observer was the only one time that they used the word “poll” instead of “survey”.

    The context of the article makes it clear that we are talking about finite numbers:

    “This huge opposition to the health bill we see in our survey flies in the face of the rather placatory approach… [from] the RCP,” said Dr David Wrigley

    The “survey” seems to have been used as a way to alert RCP officials to the need for a ballot of their 26,500 members – rather than as an attempt to influence/misinform public opinion.

  11. Yes Kevin, your poll is completely pointless unless you do it properly.

  12. I must say I saw the Observer piece, followed the link to the ‘call on your college’ website and thought there’s Anthony’s Sunday ruined. ;)

    All you can say in defence is the that the website itself is trying to get the Colleges to consult with their members and so are trying to show widespread opposition rather than produce a scientific result. They also ask for membership number, so that should prevent both non-valid and duplicate votes.

    However, as with the last time this was discussed, the newspaper has been lured with the cachet of fake exactitude, when vague terms such as ‘overwhelming response’ would have made just as good a story and kept Anthony’s blood pressure down.

  13. Billy,

    The article is unambiguous in presenting the findings as being representative of the wider membership of the RCP.

    “Nine out of 10 members of Royal College of Physicians oppose NHS bill”

    “a new poll reveals that nine out of ten members of the Royal College of Physicians – hospital doctors – want the NHS shake-up to be scrapped”

    “The findings, showing that 92.5% of RCP members want the health and social care bill withdrawn”

    I’m not really criticising what the campaign are doing. It’s a democratic country and they are campaigning against a policy they dislike, whatever works for them. It is journalists who are insufficiently sceptical when such things land on their desks, especially when it coincides with the paper’s stance.

  14. Someone really needs to take GO aside, and explain that every time he says “The country is out of money”, skittish traders put the bond rates up a tick and the stock market down a tick.

    We know he does it to lower expectations so we can have ‘better than expected’ results, but he’s running the risk of making self fulfilling predictions.

  15. Excellent blog entry. The latest RCGP and RCPCH surveys are similarly flawed and are quoted repeatedly by several respected science/statistics journalists on twitter and elsewhere.

  16. Isn’t there a period before the Budget when the Chancellor is ‘required’ not to say anything in case he alerts taxpayers etc to his plans?

  17. JAYBLANC
    `Someone really needs to take GO aside, and explain that every time he says “The country is out of money”, skittish traders put the bond rates up a tick and the stock market down a tick.`

    I do wonder whether Osborne has news of impending credit ratings cuts,or is it due to not being able to cut fuel duty as Diesel has risen to 150 p/litre…Noone can accuse Osborne of talking up the economy

    Meanwhile his other achievement of linking Cameron to Coulson forever seems to be paying dividends…Is there any news from Scotland about the Tommy Sheridan trial?

  18. @AW

    Your criticism is valid – I hope the journalists concerned do get to read it.

    “… when it coincides with the paper’s stance.”

    Perhaps one of the pro-camp will run a headline on the YouGov data – “14% of Voters Still Support the Bill.” :)

  19. I get the impression these bogus statistics are produced specifically for, or even requested by, journalists who want back-up for whatever political angle they are taking on a story.
    Radio and TV give airtime to dodgy statistics and numbers every day. Last week a very small protest outside No. 10 by the SWP , who were outnumbered by producers, reporters and cameramen, was filmed in close-up to obscure its numerical insignificance. News reports quite deliberately exaggerated the number of ‘protesters’ and withheld information about who they were.

  20. The ultimate voodoo pool was the “97% of climate scientists…” poll, in which the grand total of 79 scientists were polled, and asked leading questions which even sceptics would have answered yes to.

  21. Incidentally – I have found a properly conducted, properly sampled and weighted survey of doctors on the reforms here:

    http://www.kingsfund.org.uk/current_projects/the_nhs_white_paper/impact_of_the_health.html

    Alas, it was done in 2010, so will be woefully out of given how far the debate has progressed since then. If newspapers actually want to know what doctors think, they need to recommission something like that survey.

    Mike N – you are thinking of “Budget Purdah”, back in the day Chancellors disappeared completely for a couple of months before the budget and didn’t give interviews. It was formally done away with in 1993, though Chancellors obviously still don’t announce actual budget measures beforehand (whether they brief the media on them, or trail them in advance is a different matter).

    Long gone are the days when a chancellor could resign over making a minor hint to a journalist about what might be in the budget on his way to give the speech.

  22. AW
    Thanks – 1993!? That long ago.

  23. Whether the Guardian poll is a voodoo poll or not, it reflects today`sRoyal College of Physicians EGM fairly accurately.
    89% of the attendees felt that the bill would damage the NHS and 79% want the college to call for the bill to be scrapped.Now the college is to ballot it`s entire membership which is in excess of 25000 members.

  24. Well done to Nick Clegg for finally publicly stating what his position is on NHS reform.

    http://news.sky.com/home/politics/article/16177797

    If Clegg and his LD colleagues show that they are equal partners in this coalition government and not just the providers of votes to get Tory policies through, this should help them with their polling numbers.

  25. Dr Kevin Law

    Welcome.

    If Anthony permits I will summarise what I said earlier.

    Three weeks ago today I had a triple heart bypass in the Jubilee hospital. I have never seen such excellenceof skill, training, equipment, staffing, commitmen, morale or compassion. in any kind of activity.

    It should be good, they have 2m catchment population with the worst cardiac disease in the world.

    I was visited in hospital by the mothr of a GP in Lanarkshire who was concerned about the possibility that the changes in England would be folowed in Scotland . I was able to assure her that I had sought comfort from my former MSP, Jim Mather,on the same question and he gave me an unequivocal assurance not only as a minister in the then minority government, but as someone who knew the Health Secretary personally. So long as Nicola Stugeon is Health Secretary, the NHS in Scotland is going in the opposite direction.

    If and when the NHS in England shrinks and expenditure is replaced by payments through insurance, there would be adverse effects from Barnet consequentials, but initially, the extra administration and consultancy fees would prouduce aditional funds which the Scottish government can use for other things than Health if it wishes.

  26. @R Huckle
    “Well done to Nick Clegg for finally publicly stating what his position is on NHS reform.”

    I am a bit skeptical about Clegg’s motivations. I don’t understand why he has supported the Bill until now and why he does not support the publication of the Risk Register. He abstained and most LDs either voted against or abstained on the motion of publishing the NHS Risk Register.

  27. Dr Kevin Law

    Is your MP a SLD who is prepared to vote for the gadua privatisation of the NHS in England, but who would not dare propose such changes if they affected his own constituents?

    SLAB MP’s at least supported policies for England that they would also defend for their own constituencies. The English nationalists got very angry about that.

  28. From abtternhs blog

    Today’s Extraordinary General Meeting of the Royal College of Physicians saw heavy criticism of the foundering Health and Social Care Bill.

    186 Fellows of the College attended the EGM, which passed the following motions. Only the first is binding on the College, as the papers for the EGM were sent out before the other motions could be added to the agenda papers.

    1. 80% voted in favour of the College surveying its membership about the Health and Social Care Bill; 16% voted against, with 4% abstaining.

    2. 89% agreed that the Health and Social Care Bill will harm the health and healthcare of the population (8% voted against, 3% abstained)

    3. 79% agreed that the College should publicly oppose the Health and Social Care Bill (18% voted against, 2% abstained)

    At the EGM, consultant neurologist Dr David Nicholl announced the results of an informal online survey of members and fellows of the College.

    Over 92% of 878 fellows and members who responded to the survey said they believed the RCP should publicly call for the withdrawal of the Health and Social Care Bill.

    Dr David Nicholl commented:

    “I’m delighted at the result of today’s EGM. The College has made major efforts to improve this deeply flawed Bill, but the time has come to state publicly that overall the Bill will do damage to the NHS and the health of our patients. College Officers now realise the strength of feeling among members and fellows.”

    Dr David Cohen, Consultant Stroke Physician, London, added:

    “Today’s EGM recognised the huge gap between the spin that Ministers have tried to give to the Bill and what the Bill actually says. “The proposed “reforms” will actually deliver less care at a higher unit cost. Asking consultancy companies, that have to make a profit for shareholders, to commission healthcare and manage the chaos of contracts that the Bill would create is going to cost far more than the current system, and waste enormous amounts of taxpayers’ money that should be used to improve the health of our patients..”-

    ENDS – Notes to Editors

    Some of the speeches made at the EGM are available from [email protected] or [email protected] .

  29. ANTHONY.
    Many thanks for your work on the polling.

  30. @R Huckle – “Well done to Nick Clegg for finally publicly stating what his position is on NHS reform.”

    I have had a post put into moderation on this (and quite rightly so – completely inappropriate tone) but I can’t agree with your analysis.

    Clegg has previously voted for this policy he now disagrees with, and I’m struggling to see how such a switch could improve his supporter numbers.

    Opponents of the bill, who he now agrees with, will wonder why he didn’t say these things when he last voted for them, while suporters of the bill will be irritated further by his behaviour.

  31. @John B Dick

    Perfect illustration of the confusion as to what health is. Is it wonderful operations or do we consider that not getting heart disease in the first place is a better idea.

  32. @Wolf

    “not getting heart disease in the first place is a better idea.”

    Obviously that, but to achieve it it’s probably not best to give prominent roles in public health policy to the purveyors of junk food.

  33. The extent of the (alleged) cover up of phome hacking by the Met is really breathtaking. Prescott and Paddock have been giving evidence.

    Mulcaire had apparently got the details of the Bulger killers new identities amongst other things.

    Extraordinary. Really does take the breath away.

  34. Smukesh,
    A poll at the RCP EGM is also voodoo polling. Attendees of such occasions are a self-selecting body of activists whose opinions may, or may not, reflect the views of the membership as a whole. The only way to find out is by polling the whole membership. Even then, of course, only those who feel strongly are likely to vote.

  35. “Once these final changes have been agreed, we believe conference can be reassured that it has finished the job it started last March and the Bill should be allowed to proceed,” the [Nick Clegg’s] letter adds.
    —————————–
    IMO, Clegg is doing a smoke & mirrors thing to his own members so they don’t hold a vote against the Bill at conference.
    8-)

  36. @ Billy Bob

    Did you see this?

    http://www.guardian.co.uk/world/2012/feb/27/julia-gillard-beats-kevin-rudd?INTCMP=SRCH

    I’m kinda surprised actually. His own caucus must really dislike him. She’s extraordinarily unpopular among the electorate and the electorate would overwhelmingly prefer him to her. Yet she did extremely well in the vote (apparently better than past leaders who were challenged).

    Given the wide margin of her victory and his decision to challenge her, I have to wonder if he didn’t get the feau commitment of a number of people who he thought would vote for him over her and think he had enough votes to win only to see that those commitments were fake. The Australians invented the secret ballot, I wonder if they have it in their own party leadership ballots.

  37. The Coulson and Brooks back story looks to me as a very
    toxic brew indeed.

  38. @SoCalLiberal – “His own caucus must really dislike him.”

    One analysis I read had Rudd as a policy genius, but hopeless at implementation… whereas Gillard is very effective – but weak on policy formation. Pity they can’t work together.

    The position was the reverse when she replaced him in 2009, with polls showing Gillard more likely than Rudd to pull Labour back from a losing position against Abbott at the GE. She was also able to totally outsmart Abbott when it came to coalition building.

    There was a bit more background in Saturday’s paper:

    h
    ttp://www.guardian.co.uk/world/2012/feb/24/labor-australia-leadership-julia-gillard-kevin-rudd?intcmp=239

    Incidentally there has been some speculation that had Rudd had won this latest “spill” it could have precipitated an immediate election – if he was unable to command the confidence of the assorted Greens and Independents.

    h
    ttps://theconversation.edu.au/what-happens-if-kevin-rudd-wins-the-leadership-spill-5528

  39. Wolf @ John B Dick

    I have never smoked or eaten a deep fried mars bar, nor am I overweight, but my father had diabetes and so have I.

    We need both.

  40. @ R Huckle,

    But I think Clegg has made me more confused about his position. He supported the original bill, it was govt policy, but then (apparently) his objections called a halt for the “I’m listening (honest)” exercise. Then the new bill comes back, which Clegg says he supports, and begins to make it’s way again through parliament.

    But now Clegg is supporting yet another set of amendments and is saying the bill should not pass unless he gets those (which obviously he will otherwise he couldn’t oppose such a major plank of government policy). So he’s now against the bill in it’s current form. I am COMPLETELY unclear what his position is.

    If the Deputy PM can’t decide whether he’s for it or not, it’s not surprising that according to polling most people are against the bill, and another sizeable chunk just don’t understand it.

    I do wonder whether this letter will work, or whether (like with many Labour activists with Iraq) LD activists will now lose faith with their leader.

  41. I am expecting good news in 5 minutes time as we have had no early tweets today.

  42. of course, no right wing newspapers would never do the same thing would they?

  43. @Dougie – you don’t know what a Royal College does, do you?

    And as one of the thing these ‘activists’ passed at the EGM was a full poll of members, we will soon see.

  44. I had an email exchange with one of the authors of this article on Sunday making these points and he said “Given the article clearly stated the provenance of the survey, how can it fairly be called “misleading”?” When I pointed out that:
    “this is not a “poll” and it doesn’t tell us anything about the views of 90% of RCP members” he said “we’d have to agree to differ”.

  45. it is an English NOT Scots Health Bill.
    One should always follow the Money not the poll.