Back in 2012 I wrote about the Observer reporting an open-access poll on a website campaigning against the government’s health bill as if it was representative of members of the Royal College of Physicians. I also wrote to the Observer’s readers’ editor, Stephen Pritchard, who wrote this article about it.

The Guardian today is making the same error – they have an article claiming that seven out of ten junior doctors will leave the profession if the new junior doctor’s contract goes through. The headline presents it as representative of all junior doctors and it is referred to as a poll and a survey in the first two paragraphs. Only in the final, seventeenth paragraph is it revealed that it wasn’t conducted by any reputable market research organisation, but a self conducted survey of members of a Facebook group, the Junior Doctors Contract Forum, which is campaigning against the new contract (the Telegraph had a similar article earlier this month that appears to be based on the same data).

We cannot tell if efforts were made to limit the poll to actual doctors or to make it representative of junior doctors in terms of career stage, age, region and so on – it doesn’t really matter, as it is fatally undermined by being conducted in a forum campaigning against a contract. It would be like conducting a poll on fox hunting in the Countryside Alliance’s Facebook group and presenting that as representative of the countryside’s views on foxhunting. The flaw should be screamingly obvious.

Questions along the lines of “If thing you oppose happens, will you do x?” are extremely dubious anyway. The problem is that respondents to opinion polls are not lab rats, they are human beings who seek to use polls to express their opinion, even when it’s not exactly what the question asks. From a respondent’s point of view, if you are filling in a survey about something you oppose, you’re are likely to give the answers that most effectively express your opposition. Faced with a question like this, it’s far more effective to say you might leave your job if your contract is changed than say you’d meekly accept it and carry on as usual.

We see this again and again in polls seeking to measure the impact of policies. For example, before tuition fees were increased there were lots of polls claiming to show how many young people would be put off going to university by increased fees (such as here and here). After the rise, they miraculously continued to apply anyway. Nobody wants to tell a pollster that they would just swallow the thing they oppose.

I don’t doubt that many or most junior doctors are unhappy with the new contract, but you can’t get a representative poll by surveying campaigning groups, and you shouldn’t necessarily believe people telling pollsters about the awful consequences that will happen if something they don’t like happens. It’s a lot easier to make a threat to a pollster that you’ll resign from your job than it is to actually do it.

UPDATE: While I’m here in voodoo polling corner, I should also highlight this cracking example of a voodoo poll in the Daily Mail. It claims “One in three women admit they watch porn at least once a week”… but it seems to be an open access poll of Marie Claire readers, certainly it is in no way representative of all women in terms of things like age. It contains the delightful line that “Out of the more than 3,000 women surveyed, 91 per cent of the survey’s respondents identify as female, eight per cent identify as men and one per cent is transgender.” I don’t know how to break it to them, but you probably can’t include the 8% who are men in a survey of 3000 women.

Just catching up on a couple of polls over the last few days.

Friday’s two Westminster voting intention polls from YouGov and Populus were YouGov/Sun – CON 33%, LAB 36%, LDEM 10%, UKIP 14% (tabs) and Populus – CON 34%, LAB 35%, LDEM 9%, UKIP 14% (tabs).

There was also a YouGov/Channel 4 Scottish poll in the week, showing very little change from the previous YouGov referendum poll in March. YES is on 37%(nc), NO is on 51%(-1) (tabs. Excluding don’t knows this works out at YES 42%, NO 58% – exactly the same as a month ago. This, incidentally, produced some superbly inept reporting from the Daily Mail, well deserving of my much sought after “Crap Media Reporting of Polls” award: Campaign against independence soars to 16 point lead. Apparently there has been a “surge” in support for the Union following “growing anger over Putin praise”. That’ll be a surge from 58% to 58% then.

There is also a new YouGov poll of European voting intentions, conducted for the Green party. Topline figures there are CON 22%, LAB 30%, LDEM 9%, UKIP 27%, GRN 8% (tabs)

Finally there was a Survation poll of London (tabs) which had toplines for the European election in London of CON 21%, LAB 39%, LDEM 13%, UKIP 20%, GRN 7% and for the London local elections of CON 26%, LAB 42%, LDEM 14%, UKIP 11%.

Last month Chris Elliot, the Guardian’s readers’ editor, quoted a letter from a reader saying there “seemed to be a cultural problem among Guardian reporters that it is of no consequence if you completely misunderstand or mis-report the figures in a story […] I hope that you can urge on the editor some training of reporters on basic understanding of statistics”. Chris Elliott said he had organised three sessions with external statistical experts for Guardian journalists in the past year (and Nigel Hawkes at Straight Statistics reveals he was one of them).

The Observer’s readers editor should probably do the same. Earlier this month the Guardian’s front page story mentioned an open-access voodoo poll on the Royal Medical Journal’s website that had been touted round Twitter as if it was meaningful. The Observer this weekend was on a similar subject, but was worse – hanging a whole story on very dubious figures.

The story is titled “Nine out of 10 members of Royal College of Physicians oppose NHS bill”, and claims that “a new poll reveals that nine out of ten members of the Royal College of Physicians – hospital doctors – want the NHS shake-up to be scrapped.”

The story is based upon an open access survey created by and linked from a website campaigning against the heath bill, callonyourcollege.blogspot.com, and again, bandied around Twitter. The survey was open access, so there could have been no attempt at proper sampling and contained no demographic information that could have been used to weight it. It should go without saying that a survey from a website campaigning against the NHS reforms and co-ordinating opposition to it amongst the Medical Royal Colleges is more likely to be found and completed by opposed to the bill (in much the same way that a poll carried out on, say, the Conservative party’s website, might be considerably more supportive).

Any poll actually measuring the opinion of members of the RCP would have needed to randomly sample members, or at least contact members in a way that would not have introduced any skew in those likely to reply. For all we know this may have also shown overwhelming opposition – but we cannot judge that from an open-access survey liable to have obtained an extremely biased sample.

Once again, I would urge any journalist thinking of including any polling figures in a story to look at this guidance from the British Polling Council, particularly on how to judge whether to take a poll seriously or not. If these had been looked at, the Observer should never have got to this point…

Who conducted the poll? Was it a reputatle, independent polling company? If not, then regard its findings with caution

In this case, the poll was not conducted by a polling company, but by a group lobbying against the bill they were asking about. This should have been the first alarm bell.

How many people were interviewed for the survey? The more people, the better — although a small-sample scientific survey is ALWAYS better than a large-sample self-selecting survey.

In this case, the number of people interviewed is not mentioned. It could be high, it could be low. But note Peter’s other point… this was a self-selecting survey anyway…

How were those people chosen? If the poll purports to be of the public as a whole (or a significant group of the public), has the polling company employed one of the methods outlined in points 2,3 and 4 above? If the poll was self-selecting — such as readers of a newspaper or magazine, or television viewers writing, telephoning, emailing or texting in — then it should NEVER be presented as a representative survey.

This was a self-selecting poll of doctors directed there from a site campaigning against the legislation. There is no way it should have been presented as a representative survey.

UPDATE: Credit where it is due. Denis Campbell, one of the authors of the piece, wrote about the same poll on the Guardian’s rolling blog the next day, but this time caveated it with “But that was to a website run by anti-Bill doctors and a self-selecting rather than scientific poll, so may not reflect opinion precisely.” In a perfect world I’d hope that journalists would spurn non-representative polls completely, but progress nonetheless.