I hope most of my regular readers would assume a Daily Express headline about a “poll” showing 80% of people want to leave the EU was nonsense anyway, but it’s a new year, a new election campaign, and it’s probably worth writing again about why these things are worthless and misleading as measures of public opinion. If nothing else, it will give people an explanation to point rather overexcited people on Twitter towards.

The Express headline is “80% want to quit the EU, Biggest poll in 40 years boosts Daily Express crusade”. This doesn’t actually refer to a sampled and weighted opinion poll, but to a campaign run by two Tory MPs (Peter Bone and Philip Hollobone) and a Tory candidate (Thomas Pursglove) consisting of them delivering their own ballot papers to houses in their constituencies. They apparently got about 14,000 responses, which is impressive as a campaigning exercise, but doesn’t suddenly make it a meaningful measure of public opinion.

Polls are meaningful only to the extent that they are representative of the wider public – if they contain the correct proportions of people of different ages, of men and women, of different social classes and incomes and from different parts of the country as the population as a whole then we hope they should also hold the same views of the population as a whole. Just getting a lot of people to take part does not in any way guarantee that the balance of people who end up taking the poll will be representative.

I expect lots of people who aren’t familiar with how polling works will see a claim like this, see that 14,000 took part, and think it must therefore be meaningful (in the same way, a naive criticism of polls is often that they only interview 1000 people). The best example of why this doesn’t work was the polling for the 1936 Presidential election in the USA, which heralded modern polling and tested big sample sizes to destruction. Back then the most well known poll was that done by a magazine, the Literary Digest. The Literary Digest too sent out ballot papers to as many people as it could – it sent them to its subscribers, to other subscription lists, to everyone in the phone directory, to everyone with a car, etc, etc. In 1936 it sent out 10 million ballot papers and received two point four million responses. Based on these replies, they confidently predicted that the Republican candidate Alf Landon would win the election. Meanwhile the then little known George Gallup interviewed just a few thousand people, but using proper demographic quotas to get a sample that was representative of the American public. Gallup’s data predicted a landslide win for the Democrat candidate Franklin D Roosevelt. Gallup was of course right, the Literary Digest embarrassingly wrong. The reason was that the Literary Digest’s huge sample of 2.4 million was drawn from the sort of people who had telephones, cars and magazine subscriptions and, in depression era America, these people voted Republican.

Coming back to the Express’s “poll”, a campaign about leaving Europe run by three Tory election candidates in the East Midlands is likely to largely be responded to by Conservative sympathisers with strong views about Europe, hence the result. Luckily we have lots of properly conducted polls that are sampled and weighted to be representative of whole British public and they consistently show a different picture. There are some differences between different companies – YouGov ask it a couple of time a month and find support for leaving the EU varying between 37% and 44%, Survation asked a couple of months ago and found support for leaving at 47%, Opinium have shown it as high as 48%. For those still entranced by large sample sizes, Lord Ashcroft did a poll of 20,000 people on the subject of Europe last year (strangely larger than the Express’s “largest poll for 40 years”!) and found people splitting down the middle 41% stay – 41% leave.

And that’s about where we are – there’s some difference between different pollsters, but the broad picture is that the British public are NOT overwhelmingly in favour of leaving the EU, they are pretty evenly divided over whether to stay in the European Union or not.


Compare and contrast

A couple of weeks ago someone sent me a link to a “poll” in the Tab (which, one understands, is some form of newspaper for students) that claimed to show Conservatives were in the lead amongst students. Nonsense of course, it was an open access voodoo poll with no attempt to get a meaningful or representative sample (hell, 10% of the sample were Cambridge students!). Of course, it was only a poll in a campus newspaper so I didn’t bother writing rude things about it, the only other media I found foolish enough to cite it were Vice and Breitbart.

Just for the record though, today’s Independent has a properly conducted poll of students by YouthSight (we’ve met them here before, under the name of Opinionpanel). This was a panel based survey amongst undergraduate full-time students, recruited via UCAS and validated through an ac.uk email address, weighted by type of university (Russell, pre-1992, post-1992, specialist), year of study and gender. In contrast to the voodoo poll above, it shows Labour with a solid lead amongst students who say they are likely to vote – Labour 43%, Conservatives 24%, Lib Dems 6%, Greens 14%, UKIP 5%. Compare and contrast.


-->

A couple of years ago I wrote a piece giving advice on how to report opinion polls, or rather, how not to. Look specifically at the third point on being careful of extremely small sample sizes in cross-breaks.

There was a cracking example of the media failing on this front on BBC Look East this week, which has done the rounds on Twitter. The clip is here, the offending portion starts at the three minute mark. It claims to show the results of a poll of the Eastern region that put UKIP on 44% of the vote.

The figures come from page 36 of this ComRes poll. It wasn’t a bespoke, properly weighted poll of the Eastern region. It’s a crossbreak on a normal national poll. The figures are based upon only only 58 respondents, giving a margin of error of plus or minus 13 points. The figures are not even accurately quoted, the Lib Dems are actually on 7%. The were no caveats about sample size offered (the youtube clip from UKIP cuts out suddenly, but at the moment the full programme is on iplayer). This is truly appalling reporting of polls – they is no way that such a tiny cross-break should be reported out of context as if it were a representative poll.


Time for some bad poll reporting, or more specifically, bad poll headlining (Nicholas Watt’s actual article is eventually perfectly clear about the details of the poll). Tonight the Guardian report that “Labour support up 14 points after Miliband’s energy pledge”. Now, one might very well interpret that as meaning Labour’s share of support in the polls has risen fourteen points since Ed Miliband made his pledge on energy prices. Of course, this isn’t the case. Labour were up in the high thirties before conference and now they are in the high thirties – perhaps a tad higher, it’s still unclear. What the poll actually shows is that amongst middle class people who say they are struggling to make ends meet Labour are up 14 points since the general election in 2010. Given the vast majority of Labour’s increase in the polls happened in the tail end of 2010 or after the omnishambles budget in 2012, it’s fair to assume this was not the result of Ed Miliband’s energy pledge.

That said, 14 points is a big increase considering Labour are only up about 8 or 9 points overall. Once
Peter’s actual article and the tables are out it will be interesting to see the contrast between those people who are struggling and those who are doing well (Though its worth considering that correlation will not only work one way – people who feel badly off may be more likely to support Labour, but I suspect people who support Labour are also more likely to say they are struggling. Poorer people will already be more Labour anyway, the interesting contrast will be the changes). It’s not up on the Progress website yet, but presumably will be in the next few days.

Today’s papers also have some ropey poll reporting from a different source in the the Telegraph. It reports a poll of Countryside Alliance members, but headlines as if it were representative the views of rural voters as a whole. Again, the problem is the headline, Steven Swinford’s actual article is fine. Needless to say, the membership of the Countryside Alliance is not interchangeable with the entire population of rural areas, for reasons which I would hope were blindingly obvious (it’s a pressure group, so it attracts more politically active and engaged people. It grew from the campaign against the hunting ban, so it attracts more pro-hunting people. It doesn’t restrict it’s membership to people actually from rural areas, etc, etc). The Speccy has got very excited about the same poll because it shows 13% of Countryside Alliance members saying they’d vote UKIP… so, roughly the same proportion of people as in the country as a whole. If anything, one might have expected a more rural and conservative demographic to be more supportive of UKIP than the population as a whole, in fact, they seem to be exactly the same. It strikes me a bit as a “Pope in no more Catholic than anyone else shocker”.

Finally, while I’m picking on people, I might as well waste a few pixels being horrid to the Daily Express, which today claims 98% of people think Britain should close its doors to all new immigrants. It seems almost superfluous to point out that almost any survey in the Express is complete tripe, like making the effort to write that things in the National Enquirer may be untrue. Perhaps so, but I feel the need to point it out occasionally – it would hardly be fair for me to pick upon upon the motes in the eyes of the Guardian and the Telegraph and ignore the forest sprouting from the Express. Express “phone polls” are premium rate numbers they put in the paper, to get people to ring up to vote yes or no (multiple times if they wish), presumably after reading a foam-flecked Express rant on the subject in question. There is obviously no attempt to get a representative sample and they always show around 97%, 98% in agreement with whatever the Express’s line is. On the Express’s old website they used to have a wonderful archive of them but they don’t seem to be put up online anymore, presumably to stop people laughing at them.


You may remember my blog post about the Observer reporting a voodoo poll as if were representative of members of the Royal College of Physicians a couple of weeks ago. Back then an open access poll hosted on a website campaigning against the government’s NHS bill found 92.5% of respondents wanted the RCP to “publicly call for the withdrawal of the Health and Social Care Bill”, and this was reported as being representative of the RCP’s membership.

I dutifully wrote a letter to the Observer’s readers’ editor, Stephen Pritchard, and he addresses the report in his column for the Observer today. Mr Prichard writes “we know opposition among hospital doctors is extremely high, but readers have a right to expect that things that we proclaim to be polls are properly conducted, using scientifically weighted samples of a population or group” and, as I have before, points journalists to Peter Kellner’s British Polling Council guide for journalists on how to report opinion polls. Full marks to the Observer for addressing the matter seriously.

Meanwhile, the RCP has since commissioned a ballot of its whole membership, professionally carried out by Electoral Reform Services. The ballot managed a 35% response rate. It found that 69% of members were opposed to the bill, but that only 49% thought the RCP should seek the withdrawal of the bill, with 46% saying the College should work constructively with the government to try and improve it.

That’s 49% who wanted the RCP to call for the Bill to be withdrawn, not 92.5%. That, dear readers, is an example of why voodoo polls are bunkum.

(Nigel Hawkes at StraightStatistics also has a post welcoming the RCP conducting a proper survey of their membership, rather than touting voodoo polls here)