I’m writing a longer piece rounding up public opinion towards Brexit, but rather than let that get dominated by a big rant, I thought I’d write a separate piece about self-selecting newspaper polls apparently showing changing attitudes to Brexit. Several newspapers in the North East and the Black Country have had open-access “polls” on their websites that have had more responses supporting remain than leave. These have been widely picked up on social media by people who support Britain remaining in the EU, who have given them rather more weight that they probably would do to most open-access polls on newspaper websites.

I have spent a decade making the same post about self-selecting polls – the phone in polls in papers, press-the-red-button polls on TV, click here to vote at the bottom of newspaper articles – are bunkum. That venerable old sage Bob Worcester has been doing it for decades before me, coining the term “voodoo polls” to refer to them.

However, they keep on coming back. This time it’s a little different because of the people doing it. I keep seeing otherwise intelligent and learned people on social media quoting them. Even people who recognise that they are not a robust way of measuring public opinion sometimes suggest they must mean “something”.

They really don’t.

To roll over the old arguments again…

1) Polls are only meaningful if they are representative, and self-selecting polls are not. To be meaningful, a sample needs to mirror whatever population it is trying to measure. If it has the same gender balance, age balance, class balance, etc, then it should have the same balance of opinion. Properly conducted polls ensure this is the case using the sampling (normally using randomisation and/or demographic quotas) and weighting (adjusting the sample after fieldwork to ensure it does indeed have the right number of men and women, old and young, rich and poor). Self-selecting website & newspaper polls do not do this.

More specifically, if you look at the professionally conducted polls asking how people would vote in a referendum now, they are almost all weighted to ensure they are representative in terms of how people voted last time. For example, look at the ComRes/CNN poll, the most recent poll to ask how people would vote in a referendum today. The first question ComRes actually asked was how people voted back in June, this ensured the sample was representative, that it matched the actual figures and did not have a skew towards people who voted Remain in June or who voted Leave in June. This means we know ComRes’s sample accurately reflects the British population, and if it had shown people would vote differently now, we would have known there had been a genuine change in opinion. Without controls on this, without an attempt to make sure the sample is representative, it could just be the case that a sample is full of people who voted Remain in June to begin with.

2) Getting a large number of responses does not make a poll meaningful. Self-selecting “polls” often trumpet a large sample size as a way of suggesting their data have some validity. This is false. If you conducted a poll entirely of, say, Guardian readers, then sheer numbers would never make it representative of the whole country. This was famously tested to destruction in 1936. Back then the best known “pollster” was the Literary Digest magazine, which conducted mail in surveys. They sent out tens of millions of surveys based on subscription directories, phone directories and so on, and received literally millions of responses. Based on those they confidently predicted that Alf Landon would win the 1936 Presidential election. George Gallup, the pioneer of modern polling methods, had a far smaller sample, but made it properly representative – as a result he accurately predicted Roosevelt’s landslide victory. It was the birth of modern polling but, alas, not the death of newspapers conducting voodoo polls.

3) Self-selecting polls reflect the views of those who are interested and have an axe to grind. Any poll where people can choose to take part, rather than the polling company controlling who is invited, will attract those people with strong views on the subject and under-represent those with only a limited interest (indeed, this is a problem that even proper polls are not immune to, given low response rates). It probably explains some of the apparent shift from the pre-referendum newspaper “polls” to the current ones – before the referendum it was angry Leave supporters with the motivation to take part in “polls” on obscure newspaper websites; now the Leave supporters have what they want, and it is the frustrated Remainers hammering away on website “polls”, grasping for evidence that public opinion has moved their way. To measure public opinion properly, you need to represent all the public, not just those who are the most fired up.

4) Self-selecting polls can be orchestrated and fixed. Open access polls don’t usually have any limits on who can take part (do you actually live in the Black Country?) or any method of preventing multiple votes. They can never stop people sharing or distributing the link to like minded people to encourage them to vote (indeed, given they often exist purely as clickbait, they are very much intended to be distributed in that way). This doesn’t have to be an organised attempt, as much as people sharing links on social media with a “vote here and show people that not everyone wants Brexit”. Again, properly controlled polls have measures preventing such manipulation or skews.

Almost no new information is entirely useless. If we lived in an information vacuum then these sort of things would perhaps be an interesting straw in the wind, a pointer to something that might be worth proper investigation. We do not live in an information vacuum though – there have been numerous properly conducted opinion polls using properly representative samples over the last six months (I will write something on them later, but in the meantime John Curtice has collected them at the beginning of this article.) and these have painted a different picture. Properly conducted polls in recent months have consistently shown very little net movement in whether the British public want to leave or remain in the EU.

There are only three obvious ways of resolving the conflict between the picture painted by professionally conducted national polls and the self-selecting website polls. One, that that has been a vast shift of opinion in favour of Remain in the North East and the Black Country, but it has been balanced out by a big shift in the other direction elsewhere in the country so at a national level it evens out. Two, that professionally conducted polls with representative samples (polls that even on their very worst days, still end up within a handful of percentage points of the actual result) have somehow completely missed a 45% swing in public attitudes to Brexit. Or three, that open-access polls on newspaper websites that let any old bugger vote multiple times without any attempt to get a sample that is politically or demographically representative are an utterly useless way of measuring opinion.

I know it’s the third explanation, and deep in their hearts, I think most of those people sharing them know it’s the third explanation too. The kindest advice I can give to those who would like Britain to remain in the EU is that they need to change public opinion, not grasp at voodoo polls to kid themselves that it already agrees with them.


21 Responses to “Self-selecting newspaper polls do NOT show Britain has turned against Brexit”

  1. Excellent, as always, Anthony.

    Perhaps a more severe punishment is required for perpetrators of these “polls” (and the new abomination of similar “polls” on Twitter)?

    IIRC, a previous YG poll suggested that a representative sample of the GB population favoured lethal injection as the most appropriate method, were capital punishment to be restored.

    Thanks for your coverage of 2016 polling, and looking forward to lots of “Full Scottish” polls that you doubtless plan to do in 2017. :-)

  2. AW

    @”I have spent a decade making the same post about self-selecting polls”

    Prepare yourself for more decades Anthony-and for a Year of people on either side of the Brexit divide wanting them to “mean “something”.

  3. Many thanks AW what good sense you come up with every time you take the trouble to give us a thread on a particular aspect of polling.

  4. I wonder if maybe you’ve missed the point of these kinds of polls. A lot of people just use them to troll people they disagree with, especially when they go the other way to the one the poll author expected.

    Anyone with a programming background will be even more sceptical of any online polling whatsoever given their vulnerability to a good script and a few vpns.

  5. “Any poll where people can choose to take part, rather than the polling company controlling who is invited, will attract those people with strong views on the subject and under-represent those with only a limited interest ”

    Quit so. But it seems to me that proper Opinion Polls don’t usually measure intensity of opinion, which can sometimes become a factor. (Please correct me if I’m wrong) In this way alone, could the voodoo polls -both before and after the referendum – be a ‘straw in the wind’?

  6. AW

    Excellent post, as always, Anthony, but I suspect you’re being too optimistic with:
    before the referendum it was angry Leave supporters with the motivation to take part in “polls” on obscure newspaper websites; now the Leave supporters have what they want

  7. BARBAZENZERO

    You didn’t complete AW’s sentence :-

    ” ………….and it is the frustrated Remainers hammering away on website “polls”, grasping for evidence that public opinion has moved their way.”

    :-)

  8. @PatrickBrian

    Quit so. But it seems to me that proper Opinion Polls don’t usually measure intensity of opinion, which can sometimes become a factor. (Please correct me if I’m wrong) In this way alone, could the voodoo polls -both before and after the referendum – be a ‘straw in the wind’?

    Polls are weighted by likelihood to vote, so in a sense they do take into account intensity of opinion. If someone is so strong in their view, aren’t they more likely to vote in favour of that?

  9. COLIN
    You didn’t complete AW’s sentence

    Well spotted. Given the result of the June referendum it is unsurprising that many remainers still want to remain. After all, the leavers moaned for more than four decades after the previous one on the topic.

  10. Sound comment indeed on the dangers of self-selecting newspaper polls.

    However, in reply to CMJ on previous thread:

    Cornwall CC elections will indeed be interesting in May. In reporting one of the Cornwall by-elections in September, “Britain Elects” pointed out that it’s quite common there, for wards to have so many candidates, that they are often won on remarkably low shares of the votes. In that month, the LD candidate won with 25% in Newlyn & Goonhavern, a ward they did not contest last time around. In the same month, they also took Four Lanes with 35% – once again, a ward they did not contest the last time around.

    In those conditions, even a small swing to the LD’s could bring a lot of wards into play, come May.

    What I’ll be interested in, will be whether any of this extends also to other areas of the South West – Devon, Somerset or Bristol, perhaps?

  11. Catmanjeff

    Fair point, but it’s not quite the same. I’ve voted in every election I’ve been qualified for, so have a very high likelihood to vote, but I’ve voted for several different parties, and sometimes I care very deeply, sometimes not that much at all. If those voodoo pollls tell me anything (and I was barely aware of them before AW’s post) it’s that a significant number of Remainers are not about to give up and unite behind Mrs May’s vision. But I suppose we knew that anyway!

  12. One of Anthony’s other regular complaints has been about the tendency of some journalists to draw inaccurate conclusions from proper polls.

    Even on here, we can all slip into simplistic interpretations (which coincidentally happen to suit our political stance :-) ) from time to time.

    Dorling, in a paper on the LSE site, explores justifiable alternative reasons for the Brexit vote being the way it was based on polling.

    While inevitably anglo-centric, it’s worth a read (or, at least a quick glance) unless you already know why people voted as they did.

    http://blogs.lse.ac.uk/politicsandpolicy/brexit-inequality-and-the-demographic-divide/

    “Whilst many factors will have contributed to the Brexit vote, there is some sense in the phrase follow the money. Although there is generally a stark age divide amongst voters concerning the European Union, the same can potentially be said for divides along the lines of the spread and centralisation of wealth. The EU referendum has brought deep divisions in Britain to the surface – it appears economic inequality and its accompanying despondent effects on democracy were some of those.”

  13. Anthony,

    Just because;

    “Polls are only meaningful if they are representative, and self-selecting polls are not.”

    “Getting a large number of responses does not make a poll meaningful.”

    “Self-selecting polls reflect the views of those who are interested and have an axe to grind.”

    And

    “Self-selecting polls can be orchestrated and fixed!”

    Doesn’t mean there not true!!!!

    We are in the post Truth era now, so as experts freely admit to not knowing everything, we are free to assume they know nothing and simply believe what we want to!

    I have personally decided to believe that there is a conspiracy amongst Astronomers to say the Earth revolves around the Sun!

    As most Astronomers receive state funding it’s clearly in there interests to keep telling us that we are not the centre of the Universe to keep the money coming.

    As to Archaeologists and Dinosaurs…what a scam!!!

    Peter.

  14. @Peter Cairns

    I must confess that the whole “post truth” idiom seems a little bit odd to me, at least in its occurrence at the present moment.

    Maybe back when John Smith and John Major were facing each other across the despatch box might be a more appropriate end of the “pre-post truth era”. At least since then, it seems to me we’ve had a increased and increasing emphasis on politics as a competition of managerial competency, rather than one of divergent utopian ideals. Given that, the disagreements have moved away from a generally-agreed present, with disputes about the better way to proceed from there, and towards a generally-agreed future with disagreements about the the better stewards to achieve it. Since there is less debate between the parties about the desired direction of travel, this leaves only with the competence of the driving on which to disagree. Ground truth, being the measure of the present incumbents’ stewardship, takes the place of philosophical ideals as the battle ground for political sparring.

    It seems that there has actually been a widening of political philosophies since the ramifications of the financial tumult of 2007 (I struggle to label true price discovery as a crisis!) have become increasingly apparent. The “supply-side contraction in the choices presented to voters” as so eloquently put in Oldnat’s reference has eased somewhat since then, I would suggest.

    Maybe voters have become so inured to politicians predominantly bickering about the true reality that their credulity for the tolerable range of possible “truths” has become almost limitless? I don’t know. It does seem to me that there is nothing new about recent political discourse in principle, the only possible distinction is that the discrepancy between the many competing truths may have simply “trumped” (ho ho) previous attempts in magnitude.

  15. Popeye,

    I tend to agree with most of that but for me the Managerialism you talk about is sort of the Political Consensus that the likes UKIP called the Liberal Elite Consensus.

    Both Left, Blairite and Right, Major through to Cameron agreed on the facts and accepted neoliberalism, but they argued over the best way forward.

    What has happened is a rise of those who don’t accept the facts…
    So Immigrants are taking our jobs and we are being ruled from Brussels even though their is little evidence for it…they don’t need any!

    Peter.

  16. New thread

  17. Dear Anthony. Slightly off point – but I really need to mention those text ‘surveys’ which inevitably ping into your mobile phone inbox after you’ve used an NHS service, had a parcel delivered, had your car fixed, used Amazon, etc etc. Self selecting, lacking independence, and just awful. Do they really think that NHS service users are going to give a reliable response to the people who provide the service they depend on? Totally worthless at best and mildly sinister at worst.

    Oh. And a happy New Year to all.

  18. I used to run such customer feedback surveys for large organizations. You don’t read the top line satisfaction figures. You read the changes from period to period. And these are definitely meaningful. Furthermore the “surveys” have a primary function which is resolving problems, and providing an opportunity for customers to raise them. All negative responses would be fed back to relevant staff, and action expected where appropriate (eg contacting the person concerned to get more details).

  19. @John Chanin
    So you’re quite comfortable when when large organisations quote topline satisfaction figures based on dodgy information gathering then? And please tell me how rating rating your experience from 1 to 5 via text message provides any meaningful feedback? And yes, Longitudinal analysis is clearly valuable. But it still should be based on sound methodology and not cod science.
    I used to work in the NHS Modernisation Agency which developed a more serious method: the discovery interview. https://www2.rcn.org.uk/downloads/professional_development/mental_health_virtual_ward/treatments_and_therapies/chd-discovery-interviewspdf.pdf
    A brilliant form of information gathering. But improving care is a complex business which can’t be solved by a text

  20. PS These messages normally arrive before you are properly able to assess your experience anyway! ie too soon

  21. The non-national newspapers do not attract people solely from a particular region or political persuasion. I cast votes on newspapers I would not buy. I do not do so with the intention of making it look as though a certain style of voter thinks this or that on a particular issue – although some might intentionally do so. I simply click because the option is there. Furthermore, I often arrive at a regional newspaper after putting in a search term: persons from the entertainment world frequently originate from outside the south, so if they are interviewed, it’s inevitable that if one clicks on the story, the newspaper is viewed. A side-banner on the page can stimulate one to do a bit of clicking n flicking. There we go. Anyone who had access to a website can arrive at and click on a poll. I am presuming there is no ISP-bar based on non-UK folks attempting to click?

    In short, I wouldn’t take open-access polls seriously. Have an election if you’d like – or even a 2nd Ref, so long as the Remainer-millionaires foot the bill!