Voodoo polling corner

Back in 2012 I wrote about the Observer reporting an open-access poll on a website campaigning against the government’s health bill as if it was representative of members of the Royal College of Physicians. I also wrote to the Observer’s readers’ editor, Stephen Pritchard, who wrote this article about it.

The Guardian today is making the same error – they have an article claiming that seven out of ten junior doctors will leave the profession if the new junior doctor’s contract goes through. The headline presents it as representative of all junior doctors and it is referred to as a poll and a survey in the first two paragraphs. Only in the final, seventeenth paragraph is it revealed that it wasn’t conducted by any reputable market research organisation, but a self conducted survey of members of a Facebook group, the Junior Doctors Contract Forum, which is campaigning against the new contract (the Telegraph had a similar article earlier this month that appears to be based on the same data).

We cannot tell if efforts were made to limit the poll to actual doctors or to make it representative of junior doctors in terms of career stage, age, region and so on – it doesn’t really matter, as it is fatally undermined by being conducted in a forum campaigning against a contract. It would be like conducting a poll on fox hunting in the Countryside Alliance’s Facebook group and presenting that as representative of the countryside’s views on foxhunting. The flaw should be screamingly obvious.

Questions along the lines of “If thing you oppose happens, will you do x?” are extremely dubious anyway. The problem is that respondents to opinion polls are not lab rats, they are human beings who seek to use polls to express their opinion, even when it’s not exactly what the question asks. From a respondent’s point of view, if you are filling in a survey about something you oppose, you’re are likely to give the answers that most effectively express your opposition. Faced with a question like this, it’s far more effective to say you might leave your job if your contract is changed than say you’d meekly accept it and carry on as usual.

We see this again and again in polls seeking to measure the impact of policies. For example, before tuition fees were increased there were lots of polls claiming to show how many young people would be put off going to university by increased fees (such as here and here). After the rise, they miraculously continued to apply anyway. Nobody wants to tell a pollster that they would just swallow the thing they oppose.

I don’t doubt that many or most junior doctors are unhappy with the new contract, but you can’t get a representative poll by surveying campaigning groups, and you shouldn’t necessarily believe people telling pollsters about the awful consequences that will happen if something they don’t like happens. It’s a lot easier to make a threat to a pollster that you’ll resign from your job than it is to actually do it.

UPDATE: While I’m here in voodoo polling corner, I should also highlight this cracking example of a voodoo poll in the Daily Mail. It claims “One in three women admit they watch porn at least once a week”… but it seems to be an open access poll of Marie Claire readers, certainly it is in no way representative of all women in terms of things like age. It contains the delightful line that “Out of the more than 3,000 women surveyed, 91 per cent of the survey’s respondents identify as female, eight per cent identify as men and one per cent is transgender.” I don’t know how to break it to them, but you probably can’t include the 8% who are men in a survey of 3000 women.

Full tabs for the YouGov/Sun on Sunday poll are now up here. The slightly larger sample than usual was to make sure they had a good sub-sample of Sun readers, which the Sun used in yesterday’s analysis of the poll to look at what their own readers thought. The Express, however, has decided to report the Sun reader crossbreak as a national poll – obviously it wins the coveted UKPR crap media reporting of polls award. Just to be crystal clear UKIP are not in second place in this poll. The headline figures for this poll were CON 33%, LAB 34%, LDEM 8%, UKIP 15%. The figures quoted in the Express relate only to respondents who read the Sun.


Just catching up on a couple of polls over the last few days.

Friday’s two Westminster voting intention polls from YouGov and Populus were YouGov/Sun – CON 33%, LAB 36%, LDEM 10%, UKIP 14% (tabs) and Populus – CON 34%, LAB 35%, LDEM 9%, UKIP 14% (tabs).

There was also a YouGov/Channel 4 Scottish poll in the week, showing very little change from the previous YouGov referendum poll in March. YES is on 37%(nc), NO is on 51%(-1) (tabs. Excluding don’t knows this works out at YES 42%, NO 58% – exactly the same as a month ago. This, incidentally, produced some superbly inept reporting from the Daily Mail, well deserving of my much sought after “Crap Media Reporting of Polls” award: Campaign against independence soars to 16 point lead. Apparently there has been a “surge” in support for the Union following “growing anger over Putin praise”. That’ll be a surge from 58% to 58% then.

There is also a new YouGov poll of European voting intentions, conducted for the Green party. Topline figures there are CON 22%, LAB 30%, LDEM 9%, UKIP 27%, GRN 8% (tabs)

Finally there was a Survation poll of London (tabs) which had toplines for the European election in London of CON 21%, LAB 39%, LDEM 13%, UKIP 20%, GRN 7% and for the London local elections of CON 26%, LAB 42%, LDEM 14%, UKIP 11%.

Saturday round up

Yesterday saw a cracking example of poor newspaper reporting of polls and a worthy recipient of the much sought after “UKPR crap media reporting of polls award”. Regular readers will recall Ipsos MORI’s monthly poll for the Standard, which showed topline figures of CON 29%, LAB 38%, LDEM 10%, UKIP 15%. This was a nine point Labour lead and if repeated at a general election then a uniform swing would give Labour an overall majority of 94 seats.

While in the context of all the other polling we’ve seen lately it wasn’t a fantastic poll for Labour (the lead was pretty typical and there were some less than positive finding on whether people thought Miliband was ready to be PM), on the whole it was still a poll showing the Labour party with a steady lead over the Conservatives which if maintained would give them very substantial gains and put them back in government with a healthy majority.

And how did the Daily Star report this poll? As “Ed Miliband Facing Election Wipeout“. The Star went on to say “The latest Ipsos MORI survey said just 40% [sic] of voters now supported Labour, the party’s lowest ratings in a year. Only a quarter of people think he is ready to lead the country.”

Now, I do try to be charitable to journalists – it’s a hard job with tight deadlines. They are not responsible for the headlines put above their work, and often I’ve seen a sensible and nuanced poll write up with a wrong-headed and simplistic headline. The rest of the article suggests that the “warnings of wipeout” at the next election was probably referring to the apparent criticism that Miliband had faced from Tony Blair and David Blunkett, not the poll. Nevertheless, the overall impression the article leaves, and the failure to point out that while Labour had dropped in the polls, they were still leader by a substantial and potentially election winning amount really does create the false impression that the polls are showing Labour heading for defeat.

Meanwhil the YouGov poll on Friday morning had topline figures of CON 33%, LAB 40%, LDEM 10%, UKIP 11%. I’ve been very wary over the last week or so about claims that the YouGov poll was showing Labour’s lead falling, but this is now the third single digit lead in a week, so perhaps there is something there. Keep an eye out for the YouGov/Sunday Times poll later on tonight or tomorrow morning to see if the trend continues. We should also be due the fortnightly Opinium poll for the Observer.

Finally Lord Ashcroft released some new polling of ethnic minority voters on Friday. Most of the poll doesn’t tell us much new, merely underlining the difficulties the Conservatives face with ethnic minority voters, but that these difficulties are not the same across the board (Hindu voters in particular seem to be far more well disposed towards the Conservatives than voters from other ethnic minority groups).

There was an interesting question towards the end though. A question and problem for the Conservatives to tackle is why they do so badly amongst ethnic minority voters. One the strongest predictors of NOT voting Conservative is to be a member of an ethnic minority. Some of this is due to socio-economic factors, or due to ethnic minority voters being more likely to work in the public sector, but even accounting for these factors ethnic minority voters are less likely to vote Tory. The obvious explanation for this is that the Conservatives are still associated with anti-immigrant language and policies and that ethnic minority voters assume that the Tories are “not for them”. I was being interviewed about this by the BBC a few months back and menioned Enoch Powell when talking about that historical legacy and the interviewer asked, reasonably enough, whether people really did still remember and were influenced by a speech given 40 years ago. Well, Lord Ashcroft asked that – 58% of voters said they had heard of Enoch Powell and knew what he said, rising to 64% of the black carribean community.

Last month Chris Elliot, the Guardian’s readers’ editor, quoted a letter from a reader saying there “seemed to be a cultural problem among Guardian reporters that it is of no consequence if you completely misunderstand or mis-report the figures in a story […] I hope that you can urge on the editor some training of reporters on basic understanding of statistics”. Chris Elliott said he had organised three sessions with external statistical experts for Guardian journalists in the past year (and Nigel Hawkes at Straight Statistics reveals he was one of them).

The Observer’s readers editor should probably do the same. Earlier this month the Guardian’s front page story mentioned an open-access voodoo poll on the Royal Medical Journal’s website that had been touted round Twitter as if it was meaningful. The Observer this weekend was on a similar subject, but was worse – hanging a whole story on very dubious figures.

The story is titled “Nine out of 10 members of Royal College of Physicians oppose NHS bill”, and claims that “a new poll reveals that nine out of ten members of the Royal College of Physicians – hospital doctors – want the NHS shake-up to be scrapped.”

The story is based upon an open access survey created by and linked from a website campaigning against the heath bill, callonyourcollege.blogspot.com, and again, bandied around Twitter. The survey was open access, so there could have been no attempt at proper sampling and contained no demographic information that could have been used to weight it. It should go without saying that a survey from a website campaigning against the NHS reforms and co-ordinating opposition to it amongst the Medical Royal Colleges is more likely to be found and completed by opposed to the bill (in much the same way that a poll carried out on, say, the Conservative party’s website, might be considerably more supportive).

Any poll actually measuring the opinion of members of the RCP would have needed to randomly sample members, or at least contact members in a way that would not have introduced any skew in those likely to reply. For all we know this may have also shown overwhelming opposition – but we cannot judge that from an open-access survey liable to have obtained an extremely biased sample.

Once again, I would urge any journalist thinking of including any polling figures in a story to look at this guidance from the British Polling Council, particularly on how to judge whether to take a poll seriously or not. If these had been looked at, the Observer should never have got to this point…

Who conducted the poll? Was it a reputatle, independent polling company? If not, then regard its findings with caution

In this case, the poll was not conducted by a polling company, but by a group lobbying against the bill they were asking about. This should have been the first alarm bell.

How many people were interviewed for the survey? The more people, the better — although a small-sample scientific survey is ALWAYS better than a large-sample self-selecting survey.

In this case, the number of people interviewed is not mentioned. It could be high, it could be low. But note Peter’s other point… this was a self-selecting survey anyway…

How were those people chosen? If the poll purports to be of the public as a whole (or a significant group of the public), has the polling company employed one of the methods outlined in points 2,3 and 4 above? If the poll was self-selecting — such as readers of a newspaper or magazine, or television viewers writing, telephoning, emailing or texting in — then it should NEVER be presented as a representative survey.

This was a self-selecting poll of doctors directed there from a site campaigning against the legislation. There is no way it should have been presented as a representative survey.

UPDATE: Credit where it is due. Denis Campbell, one of the authors of the piece, wrote about the same poll on the Guardian’s rolling blog the next day, but this time caveated it with “But that was to a website run by anti-Bill doctors and a self-selecting rather than scientific poll, so may not reflect opinion precisely.” In a perfect world I’d hope that journalists would spurn non-representative polls completely, but progress nonetheless.