Voodoo polling corner

Back in 2012 I wrote about the Observer reporting an open-access poll on a website campaigning against the government’s health bill as if it was representative of members of the Royal College of Physicians. I also wrote to the Observer’s readers’ editor, Stephen Pritchard, who wrote this article about it.

The Guardian today is making the same error – they have an article claiming that seven out of ten junior doctors will leave the profession if the new junior doctor’s contract goes through. The headline presents it as representative of all junior doctors and it is referred to as a poll and a survey in the first two paragraphs. Only in the final, seventeenth paragraph is it revealed that it wasn’t conducted by any reputable market research organisation, but a self conducted survey of members of a Facebook group, the Junior Doctors Contract Forum, which is campaigning against the new contract (the Telegraph had a similar article earlier this month that appears to be based on the same data).

We cannot tell if efforts were made to limit the poll to actual doctors or to make it representative of junior doctors in terms of career stage, age, region and so on – it doesn’t really matter, as it is fatally undermined by being conducted in a forum campaigning against a contract. It would be like conducting a poll on fox hunting in the Countryside Alliance’s Facebook group and presenting that as representative of the countryside’s views on foxhunting. The flaw should be screamingly obvious.

Questions along the lines of “If thing you oppose happens, will you do x?” are extremely dubious anyway. The problem is that respondents to opinion polls are not lab rats, they are human beings who seek to use polls to express their opinion, even when it’s not exactly what the question asks. From a respondent’s point of view, if you are filling in a survey about something you oppose, you’re are likely to give the answers that most effectively express your opposition. Faced with a question like this, it’s far more effective to say you might leave your job if your contract is changed than say you’d meekly accept it and carry on as usual.

We see this again and again in polls seeking to measure the impact of policies. For example, before tuition fees were increased there were lots of polls claiming to show how many young people would be put off going to university by increased fees (such as here and here). After the rise, they miraculously continued to apply anyway. Nobody wants to tell a pollster that they would just swallow the thing they oppose.

I don’t doubt that many or most junior doctors are unhappy with the new contract, but you can’t get a representative poll by surveying campaigning groups, and you shouldn’t necessarily believe people telling pollsters about the awful consequences that will happen if something they don’t like happens. It’s a lot easier to make a threat to a pollster that you’ll resign from your job than it is to actually do it.

UPDATE: While I’m here in voodoo polling corner, I should also highlight this cracking example of a voodoo poll in the Daily Mail. It claims “One in three women admit they watch porn at least once a week”… but it seems to be an open access poll of Marie Claire readers, certainly it is in no way representative of all women in terms of things like age. It contains the delightful line that “Out of the more than 3,000 women surveyed, 91 per cent of the survey’s respondents identify as female, eight per cent identify as men and one per cent is transgender.” I don’t know how to break it to them, but you probably can’t include the 8% who are men in a survey of 3000 women.

217 Responses to “Voodoo polling corner”

1 2 3 5
  1. Wouldn’t surprise me to see the Daily Mail report “Shock as 8% of women are men”

  2. You maybe have to exclude lots of the 91% “who identify as women” too, if they lack the necessary definitive features, by that reasoning!

  3. Sometimes I wonder whether these sorts of polls are deliberately ill-conceived clickbait.

  4. @Polltroll

    Oh, some definitely are. My favourite of all time is this one – http://www.express.co.uk/news/uk/213821/99-of-you-say-Get-us-out-of-Europe

  5. Jack Sheldon

    Though, since the Express seems just to make up “news” stories, the idea that “99%” agreed with them may be little different from “9 out of 10 cats prefer ….”.

  6. This looks interesting

    NumbrCrunchrPolitics tweet
    Goodwin/Hix/Pickup (#EURef):

    REMAIN 50.6
    LEAVE 49.4

    1st-12th Oct
    Fieldwork by YouGov

  7. I didn’t even bother to read the Guardian story because I knew the sort of data it would be based on. As it happens there are already serious problems with the retention of junior doctors and their recruitment into fields such as general practice, but this sort of crying-wolf story ends up being counter-productive. All it does is to give the government the excuse to do nothing and hope something will turn up.

  8. One of the interesting things about Anthony’s list of EU referendum polls[1] is that we may already be seeing the possibility of ‘house effects’. Two of those showing small Leave leads are the two YouGovs, the third is Survation (always the most UKIP-friendly of pollsters). In contrast the four (online) ICMs all show modest 3-6 point Remain leads and the ComRes 19 points. That’s the only phone poll here (using the revised question) and also shows a much lower percentage of unsure voters than the others[2].

    A possible reason for the differences may be the reaction of Conservative voters. In these ICM polls they’re split about 41% each. However YouGov polls are different, in the large poll Leave leads among Conservatives by 51% to 33% – a similar pattern to what we have seen in a long line of Yes/No polls. Now we also know from such polls that Tory voters switch round when offered the possibility of a Cameron-endorsed re-negotiated deal, and this makes the overall result fairly strong for Remain. The question is whether a similar thing would happen with other pollsters or whether Conservative YouGov panellists tend to switch like that because they have been asked similar questions before and so distinguish more between bad now-EU and good new-EU with added Dave.

    Of course it’s probably true that, until we see what any such ‘re-negotiation’ produces, all polls are pretty academic. An awful lot of people are not going to believe that the UK should ‘Leave whatever’ or ‘Remain whatever’, they will want to see the promised details. And both sides are studiously trying to ignore the fact that with the EU even referendum results aren’t final, especially close ones.

    [1] Anthony appears to have missed out the YouGov one included in the September Eurotrack poll:


    though the GB figures aren’t that different from the large sample YouGov done the previous week. (Remain 38%, Leave 41%, WNV 4%, DK 17%).

    [2] The same is true of other phone polls using the old Yes/No questions, from ComRes and MORI, though these have an even bigger lead for Yes. The low percentage for unsure voters is particularly odd as VI polling normally has these higher in phone polls than online.

  9. Did any of you guys follow the Canadian Election?

    Spent my entire Monday night watching it (proves that I am an elections nerd). Was texting back and forth with a Canadian who I’d worked with last year on a CA State Senate campaign who had voted NDP. He claimed that his riding was so right wing that only the Tories could win it. The Tories just barely held the seat against the Liberals. He did not think any party would get a majority. I predicted the Liberals would get a majority (even though I didn’t quite believe it). So I get a free coffee out of it.

    It’s kind of amazing to think about. 4 and a 1/2 years ago, the Liberals were basically voted off the island in Canada (or basically onto the Island since the only ridings they seemed to hold were on or near islands). They were in third place for the first time, left for dead, many of their top leaders and cabinet contenders defeated and out of office, and people were drawing comparisons of them to the UK Liberals a century ago. But suddenly, just one election later, they’re not only back, they’re back with a sizeable majority and a victory that is in fact greater than what they last received in their last majority victory. It was nice to see the left exceed their polling in the actual election this year after seeing the opposite occur repeatedly this year (Israel, UK).

    One uniform lesson that we can all learn from that election (since every country is honestly different), when your campaign is hitting the skids, bringing out the Ford Brothers to conduct rallies for you (and criticize your opponent’s soft-on-crime stance) is not a good idea.

  10. Socalliberal.

    It doesn’t help a sitting PM if the best celebrity endorsement you can get is the lead singer of Nickelback!

    I’d be interested to know how much deadwood was cleared out from the Liberals in the last election. The sad death of Jack Layton cannot have helped the NDP either I suppose.

  11. Okay, so here’s a political development that everyone seems to have glossed over: the launch of the Women’s Equality Party. It’s already got 45,000 members (more than UKIP, and sneaking up on the Lib Dems), and yet people have barely given it a moment’s notice.

    I suspect this is because the WEP is a campaign group masquerading as a political party. While gender equality (which is a far more neutral term, by the way) is a serious issue and a worthwhile crusade to pursue, is contesting elections really the best way of getting that message out? If anything, getting hammered in elections just tells the establishment that people are happy with current equality legislation, which is far from the truth in many cases.

    I suspect the WEP may become the first British political party with more members than voters.

  12. PollTroll,

    I noticed the story about the WEP yesterday. They describe themselves as ‘apolitical’ in which case my response is “why are you a political party then?”

    They don’t seem to have any firm detail on branch or constituency structures, selection procedures, or campaign software ie. NationBuilder.

    The aims are laudable but I’m not sure they’re going about it effectively. 45,000 members is enough to mount a hostile takeover of the Lib Dems if they wanted, or enough to have a serious impact on the internal politics of any other party. But electorally, on their own, I think they’re of limited potential.

  13. @Polltroll

    I think you’re exactly right.

    They are a “political party” in the same way that a candidate standing for the “Save Our Cottage Hospital Party” is in a political party.

    They’d be better off channeling their energies into pushing for better opportunities for women within the existing political parties.

  14. Anthony,

    I’m writing my dissertation on the topic of reporting of opinion polls and the effect it can have on the actual outcome of elections. Do you have any advice as to worthwhile reading material on the subject?

  15. @ Anthony W, you could simply have an ongoing thread and archive on your page called “Voodoo Polls – Daily Mail/Daily Express” as I think they seem to produce one every couple of days. We could post and add our favourites.

    Rather like a “pollster” version of Hoax Busters. Any takers??

  16. They’d [the Women’s Equality Party] be better off channelling their energies into pushing for better opportunities for women within the existing political parties.

    The Labour Party has already implemented most of the things which WEP are pushing for.

    But there are lots of women who don’t vote because they think all the existing political parties are alike. If WEP can do something to get more women registered & turning out to vote, it’s surely a good thing.

  17. I see all the male posters are ‘no, no, no. you don’t want a woman’s equality party, you’d be far better of joining the male dominated mainstream parties’

    Seriously what is going on England? Lab/Lib/Con Party Leaders – Male
    Deputy Leaders – Male
    Mayor Candidates – Male

    In Scotland we have ‘Women for Independence’ it’s not a party but a campaign group, most of it’s members are in the independence supporting parties, SNP, Green, SSP. They run campaigns such as a successful campaign against a new woman’s prison being built, and help the under-privileged for example they set up Uniform Bank – school uniforms for children whose parents can’t afford them.

    And we have Women 50/50 who are unaligned. Women 50/50 want 50% representation of public bodies and boards so more than just politics.

    So, in Scotland we have followed the course of campaign groups not a political party

  18. @ Mr Nameless

    They [WEP] describe themselves as ‘apolitical’ in which case my response is “why are you a political party then?”

    I think your comment was tongue in cheek but I’m going to answer anyway.

    WEP are trying to avoid positioning themselves anywhere on the left/right/centre political scale.

  19. @Couper

    The reason I think they’d be better off joining the male-dominated mainstream parties is because those parties would hopefully become less male-dominated in the process.

    Consider if the opposite occurred – if large numbers of women left the mainstream parties and joined the WEP. I don’t think that would improve the chances of women reaching senior political positions in this country.

    As it happens I don’t agree with some of their specific proposals, but I support their objectives. I just don’t think its the best use of their time and energy.

  20. @Neil A

    I was joking. I agree that a pressure group would be best.

  21. 100% of the people writing this note think that the rise of the WEP is another symptom of dissatisfaction with the big two parties and rise of such as SNP, UKIP, Plaid, Greens etc as discussed on an earlier thread.

  22. Does anyone know of a woman police officer who has actually solved a major crime? Not being contentious but it may be that there are cases similar to the discovery of DNA where a woman did all the work but men took the credit.

  23. @Wolf.

    That’s a misunderstanding of how major crimes are solved. I’ve worked on dozens of major crime cases, most of which were “solved” collectively by teams of officers. Never once was the team involved all-male and rarely was the leadership of the team all-male either.

  24. Membership numbers are about as meaningful as Facebook likes.

  25. I notice we’re going to have the first by-election of this Parliament in Oldham West and Royton.

    It should be the first test of whether the Corbyn people put in a Corbyn-friendly candidate or not. (I understand that Mr Meacher was in the Corbyn camp but of course he would have been selected by his local CLP 45 years ago, and the shape of the membership would have changed in the intervening years).

    Do the Lab people on this board have any idea if lots of new members joined the Oldham West and Royton CLP to vote in the leadership election?

    Do the Kippers on this board know if UKIP are going to push hard to capitalise on Corbyn’s hard left foreign policy in this seat (Lab’s chief point of vulnerability)? UKIP really need a scalp in a Labour area.

    I’m assuming the Conservatives and LibDems will put up paper candidates and won’t campaign too hard so as not to distract from the main drama.

  26. The Commons has just passed the EVEL legislation. All ammendments defeated and the margin on the main vote was 312 to 270 a clear majority of 42. Many will think that at last English MP’s have something approaching parity

  27. @CANDY,

    I agree it would be interesting to have a straight Labour, UKIP fight but Conservative associations are not programmed to stand back and the Lib Dems can’t afford to miss any by-election opportunities (no matter how remote their chances). The public may of course ignore them and vote heavily for UKIP but I wouldn’t bet on it.

  28. @TOH

    And about time too. I can’t see how the proposition can be reasonably opposed.

  29. RMJ1

    The wording is not what I wanted but it is a step in the right direction and I totally agree, it’s been long overdue.

  30. @TOH


    Such as 533 MPs to 117?


    Sorry I don’t recognise your figures. The previous situation was totally unjustifiable which Tam Dalyell (Labour) recognised in the 70s.

  32. It’s a difficult day for people who see themselves as British, that’s for sure.


    Would that the rest of the EU had adopted the UK position from the start with regard to Syrian refugees and funded it accordingly. The acceptance of such huge numbers of refugees (the EU programme doesn’t scrape the surface), May then have not been necessary. We will nevertheless, be accepting very large numbers of EU migrants for many years to come if we remain EU members and the population projections remain valid.


    I do not understand your post. I have always seen myself as British, and English. All that has happened is that an injustice has been at least partially corrected IMO.

  35. Statgeek

    As far as MPs were concerned, there was parity before.

    No English, Welsh, Scottish or Northern Irish MP could vote on matters that had been legislatively devolved away from Westminster.

    That England was stuck with allowing elected politicians from elsewhere to vote on their domestic issues was only because their MPs opposed a devolved English Parliament.

    On many issues, the only difference it will make is that a governing party without a majority of English MPs won’t be able to force its will on England, against the wishes of English MPs.

    Better than nothing for England in such a scenario, I suppose, but they only have the position of “spoilers”. They can’t initiate and legislate for what they see as good things for England.

    That would be for some time in the distant future, when England had a majority from one party, while the UK Government was a party that had no English majority, but made up the numbers from the other nations.

    In the meantime, any problems are likely to rise over cash. Without a sensible process, whereby England had its own Block Grant, or raised its own taxes and borrowed to cover much of its domestic expenditure, we are likely to see a confusion between “English” or “E&W” measures and the Barnett consequentials.

    That looks to be a recipe for problems, rather than their solution.

  36. OLDNAT

    As I posted above it’s better than nothing although still unfair to English MP’s for the reasons you lay out. As I said hopefully this will be corrected in future, perhaps when the matter is reviewed in a years time.

    I don’y think it’s Englsh MP’s who didn’t want an English parliament , I think it’s the English voters who show no appetite for the additional, and IMO the unecessary expense.

  37. @OLDNAT

    In an ideal world, each of the nation’s in the UK would have its own assembly but, because politicians are unpopular, creating more would be suicidal for the party that set it all up. The legislation is far from perfect, but is a step in the right direction.

  38. TOH

    “although still unfair to English MP’s for the reasons you lay out.”

    No. I pointed out that English VOTERS may feel that it was unfair their sole Parliamentary member, couldn’t vote on whether a school in West Lothian could be refurbished while s/he could for a school in West Derbyshire. [1]

    No MP from any part of the UK (including West Lothian) can vote on such an issue – in exactly the same way that no MP (including Heather Wheeler) can vote on an issue devolved to Derbyshire County Council.

    I can understand that people have grievances – especially when they haven’t bothered to find out the differences between administrative and legislative devolution, or other esoteric matters [2].

    But there is nothing at all unfair about the current role of English MPs. What you are actually complaining about is the fact that your voters (and MPs) don’t want an English Parliament, but just to add in those from the annoying other bits of the UK – who increasingly, refuse to do as they are told!

    [1] Under “administrative devolution” (aka the Acts of Union) Scotland ran its own domestic affairs – and funded them – with little interference from Whitehall and even less from Westminster, unless we needed them to pass a law, having carelessly misplaced our own Parliament. After 1945, the SoS for S acted as a Governor-General, and no one in West Derbyshire (then or now) cared as to which school in West Lothian he did or didn’t refurbish.

    [2] Even after the indyref, Scots who, as a result, are probably more aware of the constitutional niceties than most, find it difficult to get their heads around the Byzantine squirmings of Westminster. It’s unsurprising, therefore, that your average Daily Mail reader has less awareness of constitutional complexities than a brain damaged mollusc who was consigned to a secondary modern.

  39. I haven’t seen a legislation that has taken away so much from the legislative and gave it to the executive.

    I think it’s worth to actually read the law passed rather than just saying things from an ideological stance. It is a huge power-grab of Westminster, so I’m rather surprised, among others’) ToH’s flagmatic acceptance.

  40. @Polltroll

    I think you’re absolutely right about the WEP. Single-issue parties have never been a force in British politics, unless that single-issue happens to be something local and extremely salient – e.g. Kidderminster Hospital and Dr. Richard Taylor; the national movement that grew out of that, ‘National Health Action’, have made little headway at all. And before someone says UKIP, UKIP started to succeed (outside the European election context, which is different) only when they stopped being a single issue ‘leave the EU’ party and started projecting their messages about immigration and other related issues as well.


    EVEL is a big change in standing orders, but will have little practical effect this side of the 2020 election. There is no chance whatsoever of an English (or English and Welsh only) majority blocking or amending government measures that have the support of the whole House. My prediction is that Tory backbenchers will come to understand that shortly and start pitching for more. Labour and the SNP have been extremely vocal about this but the SOs that have been introduced do little to effect their position when they aren’t in government.

  41. @ TOH

    I do not understand your post. I have always seen myself as British, and English. All that has happened is that an injustice has been at least partially corrected IMO.

    Do you understand the Sewel convention & the potential scope of it?

  42. RMJ1
    “The acceptance of such huge numbers of refugees (the EU programme doesn’t scrape the surface), May then have not been necessary.”

    I am afraid your conclusion is noti true. The wave of migration taking place is derived from demographic change,monitored in UN and OECD reports, which will see populations increase far beyond the capacities of their economies to cope, or industrial and urban development to absorb the education young who make up a majority of the migrants whom the EC and IOM calculate will continue to move illicitly. They will do so because of the lack, related to other lags in development and development administration, in the regulated and legitimate mechanisms for migration to the EU. The Agenda programme, beyond the immediate quota allocations to the Schengen counties, is aimed at legitimation of the process, based partly on setting up support and registration systems as part of the provision of decent reception and residential camps or facilties such as those now being discussed for Turkey and Niger. This needs to be made part of the institutional structure for coping with the illicit element in a continued migration, already seen in the 200,000 annual net immigration to the UK, both from the EU and from developing countries predicted to continue indefinitely, in present projections at least to the middle of the century.

  43. @ Lazslo

    I think it’s worth to actually read the law passed rather than just saying things from an ideological stance. It is a huge power-grab of Westminster, so I’m rather surprised, among others’) ToH’s flagmatic acceptance.

    Exactly; & then look at the ‘instruments’ which exist to address imbalances – like the Sewel convention – which have rarely been used but which will now be employed more often, as the only means of redress against an ‘out-of-control’ executive.

    This EVEL “mess” bodes ill for all the parliaments of the UK.

  44. @ Colin

    Very sensible. I don’t know if it’s practical for the remaining three quarters of a million on the road and on medium term, but very sensible.

    The extreme right, as you called them, however, sooner or later has to be dealt with.

  45. More on the immigration:

    The mean age of a huge proportion of Africa is less then 20 years of age.

    Without manufacturing, there is a huge enducement of immigration. And as we are members of the same species, and as the notion of nations is a historical-political creation …

    So, there is plenty to think about, rather than just saying slogans.

  46. @ Amber

    Indeed, and there will be a bout of more secondary legislation not discussed in the HoC …

    But the proposal is really worth the 10 minutes to read.

  47. The West Lothian question cannot be resolved it is part of the constitution of the Union….therefore its only resolution is in the dissolution of the Union. This latest amendment merely creates and new inequity – where a Minority Voter in Scotland or Wales or Ireland has more political power than a Minority voter….

  48. @Laszlo

    The SO changes take away nothing from the legislature. A majority of the whole House – as now – will still be able to amend or defeat any bill. All that will happen is that English (or English and Welsh) MPs must consent to the bill, or relevant provisions within the bill, before it can become law.

    In fact I’d argue the changes take power from the executive and give it to the legislature. Because a future government could potentially not have a majority in England (or England and Wales), but a majority of the whole House; in such circumstances the English (or English and Welsh) MPs would gain significant powers that they don’t currently have.

  49. LASZLO

    “Sweden will receive as many as 190,000 refugees this year, according to official estimates, more than double the projected number envisaged just three months ago, placing unprecedented strains on the country’s immigration services as politicians struggle to agree a way forward.
    “It’s as if we have a land border with Turkey,” said Anders Danielsson, head of the Migration Board, as he announced the figures on Thursday.
    “The current refugee situation is unprecedented in modern times, and the housing situation is critical.”
    At the end of July, the Migration Board said the rate of new arrivals was falling and it expected a total of just 74,000 in 2015. The previous peak number of asylum seekers in Sweden was in 1982, when 84,000 fled the Balkan wars.
    The numbers are “not sustainable for Sweden”, said migration minister Morgan Johansson, ”

    FT today.

    We await with dread the background to the Trollhattan.killer.

  50. RMJ1

    “is a step in the right direction.”

    Token gestures are that sometimes!

    The “Scottish Grand Committee” [1] was also held up by Westminster as an example of caring about Scotland.


    In practice it was simply an opportunity for MPs to have another forum in which to make a speech that they hoped their tame local press would pick up. Your English Grand Committee is an equally posturing opportunity – with the notable exception that, if there is ever another Labour Government reliant on Scots & Welsh MPs to govern England, you can say “Yah. Sucks. Boo”.

    Any predictions as to when that is going to happen?

    Westminster/Whitehall is long experienced in giving tokens to those who feel aggrieved about something, hoping that they will go away and forget about it. It is a tactic that has often worked.

    [1] EVEL has provoked that rarest of events – Jake Rees-Mogg made a procedural error! He wanted to know if English MPs would resign from the Scottish Grand Committee. Silly fellow. The Scottish Grand Committee membership is restricted to MPs from Scottish constituencies (as similarly the English and E&W ones will be). He meant the Scottish Select Committee which is, of course, packed out with MPs from England.

1 2 3 5