Voodoo polling corner

Back in 2012 I wrote about the Observer reporting an open-access poll on a website campaigning against the government’s health bill as if it was representative of members of the Royal College of Physicians. I also wrote to the Observer’s readers’ editor, Stephen Pritchard, who wrote this article about it.

The Guardian today is making the same error – they have an article claiming that seven out of ten junior doctors will leave the profession if the new junior doctor’s contract goes through. The headline presents it as representative of all junior doctors and it is referred to as a poll and a survey in the first two paragraphs. Only in the final, seventeenth paragraph is it revealed that it wasn’t conducted by any reputable market research organisation, but a self conducted survey of members of a Facebook group, the Junior Doctors Contract Forum, which is campaigning against the new contract (the Telegraph had a similar article earlier this month that appears to be based on the same data).

We cannot tell if efforts were made to limit the poll to actual doctors or to make it representative of junior doctors in terms of career stage, age, region and so on – it doesn’t really matter, as it is fatally undermined by being conducted in a forum campaigning against a contract. It would be like conducting a poll on fox hunting in the Countryside Alliance’s Facebook group and presenting that as representative of the countryside’s views on foxhunting. The flaw should be screamingly obvious.

Questions along the lines of “If thing you oppose happens, will you do x?” are extremely dubious anyway. The problem is that respondents to opinion polls are not lab rats, they are human beings who seek to use polls to express their opinion, even when it’s not exactly what the question asks. From a respondent’s point of view, if you are filling in a survey about something you oppose, you’re are likely to give the answers that most effectively express your opposition. Faced with a question like this, it’s far more effective to say you might leave your job if your contract is changed than say you’d meekly accept it and carry on as usual.

We see this again and again in polls seeking to measure the impact of policies. For example, before tuition fees were increased there were lots of polls claiming to show how many young people would be put off going to university by increased fees (such as here and here). After the rise, they miraculously continued to apply anyway. Nobody wants to tell a pollster that they would just swallow the thing they oppose.

I don’t doubt that many or most junior doctors are unhappy with the new contract, but you can’t get a representative poll by surveying campaigning groups, and you shouldn’t necessarily believe people telling pollsters about the awful consequences that will happen if something they don’t like happens. It’s a lot easier to make a threat to a pollster that you’ll resign from your job than it is to actually do it.

UPDATE: While I’m here in voodoo polling corner, I should also highlight this cracking example of a voodoo poll in the Daily Mail. It claims “One in three women admit they watch porn at least once a week”… but it seems to be an open access poll of Marie Claire readers, certainly it is in no way representative of all women in terms of things like age. It contains the delightful line that “Out of the more than 3,000 women surveyed, 91 per cent of the survey’s respondents identify as female, eight per cent identify as men and one per cent is transgender.” I don’t know how to break it to them, but you probably can’t include the 8% who are men in a survey of 3000 women.

217 Responses to “Voodoo polling corner”

1 3 4 5
  1. Wow rereading that last post autocorrect had a bloody field day. Free cookies to anyone who can decipher that mess.

  2. @Candy

    What’s the cost of the House of Lords? Surely that’s a fair swap for an English Parliament and there’s about 1000 of them.

    @Amber Star

    The snide comments aimed at Holyrood is very much part of LiS’s problem.

  3. Not sure about the comment about Wales having few differences. Having lived in South and North Wales I have found huge differences. Even within SouthWales itself Pembrokeshire is as different to The Rhonda valley as the Rhonda is as different to Wiltshire. Infact I would suggest parts of Pembrokeshire have more in common with Wiltshire than the Rhonda. Similar in North Wales, Wrexham has little in common with Ruthin, etc

    Can’t speak for Scotland as I have not lived there.

    I would quite like an English Parliament, with the same powers as Scotland, but would never vote for the break up of England into Regional assemblies. I used to live in Hereford, it was bad enough when we joined up with Worcestershire, all the organs of County Council Government were moved to Worcester, at considerable expense and rghtly or wrongly many in Herefordshire felt the poor relations. Atleast with an English Parliament I still think you would still have localism with major decisions, tax, spending etc taken at the English Parliament level.

  4. AW…….Your advanced swings enter map….what a beautiful sight. :-)

  5. Swingometer !

  6. @ Courper 2802

    The snide comments aimed at Holyrood is very much part of LiS’s problem.

    Really? Get a sense of humour.
    Anyway, it was aimed at MPs, not at the parliament itself.
    And when did I get promoted to being an official spokesperson for LiS?

  7. Couper
    Good point regarding the Lords, it’s abolition or at the very least significant reduction would go a long way towards covering the costs of a federal Britain.

    Regarding Wales that’s kinda my point, Wales and Scotland both contain very diverse areas both economically, culturally and politically. What I was saying was compared to England they are very uniform hence the need for a serious look at how England is governed. England is vastly different, you’d be forgiven for thinking your in a different country at times.

    Regarding devolution you say that major decisions would be taken at an English parliament level but for many (myself included) that’s precisely the problem, I don’t want to be under the thumb of the south east. Look at the north east for example, at present the Tories have only three seats there, how is it fair that they have policies forced upon them that they never come anywhere near close to voting for. The south is bigger than the north thus the north will nearly always have to follow the south’s lead winless we get regional government.

    Forgive me for possibly being presumptuous but it seems that everyone opposed to english devolution is either a Tory voter, lives in the home counties or lives in one of the wealthier Tory voting areas of the regions, English devolution is hardly aimed at these people since they are the precise type of people who don’t see much of a problem. The one way I can demonstrate this is the number of times I’ve heard people where I live claim (perhaps jokingly) that they would rather be ruled by Holyrood than Westminster. This is the level of disatisfaction. In parts of England.

  8. NeilJ

    The idea that ‘localism’ (whatever that is and you can find at least 5 definitions) is better prosecuted by an English Parliament as opposed to a regional assembly or a sub regional combined authority is- to be polite- for the birds!

    We have asymmetrical devolution in UK- another failure of the early Blair period when he could have got anything through.

    For a broadly symmetrical devolution we would need elected parliaments in the English regions and Wales and NI to run alongside the devolved parliament in Scotland. (Cue howls from cybernats due to notion of Scotland being a region)…

    For possible localism we are better served by the city regional/ subregional scale: the local level is hopelessly astrategic and parochial and the district/ city scale inefficient at managing competing neibourhood claims. The city regional scale as well as being functional in an economic development sense also takes the overall view in terms of housing and transport needs whilst at the same time being close enough to local scale to be sensitive to difference.

  9. Rob Sheffield
    what I mean by localism is taking the decisions at the lowest possible political level.
    The regional assemblies proposed in the past gave very limited powers responsibilities that were much better done at Town, Unitary council or county council level.

    Putting power in the hands of Regional Governments would likely result in the peripharies losing out. For example if there was a West Midlands regional assembly, most of the power would be held by the Representatives from the big West Midland city areas. The Shire counties, Herefordshire, Worcester, Shropshire would have little say in how or where the money was spent. The inner city areas have very real problems, but so do rural counties, putting them ll together is not likely to result in an equitable solution. Especially as it will mean taking power and money away from more local councils

    An English Parliament would make the decisions re tax spending etc. The full UK Parliament would make decisions on defence and Foreign Policy. This would require FFA for all Parliaments, assemblies, with perhaps Scotland and England having to give some of their money to Wales and Northern Ireland to ensure fairness.

    Regional assemblies would be a recipe for disaster. But if they are to happen they should be given similar powers to the Scotttish Parliament with local councils taking the decisions they currently take.

  10. @rivers10,

    That’s because devolution is largely seen as about who is best to acquire and distribute money, rather than who is best to raise it.

    Poorer regions want a method by which cash can be squeezed out of London and the South East and spent on them. Richer areas of those poorer regions don’t need the money so much, so are less concerned.

    An interesting parallel is the USA, where there is far more right of local self-determination and the process actually works in reverse. Richer areas declare independence from poorer ones (usually suburbs breaking off from cities) in order to protect their wealthy taxpayers from funding the schools and welfare spending of their poorer neighbours.

    If that was the system here, then I expect positions would be reversed. Of course, not being American, we would ensure that any devolution doesn’t stop the transfer of money from richer to poorer areas.

    In other words, we want a federal parliament to take Surrey’s money from them, and a local parliament to make sure the people of Surrey have no say over how its spent.

  11. CON: 36% (-3)
    LAB: 32% (-2)
    UKIP: 12% (+5)
    LDEM: 10% (+1)
    GRN: 3% (-1)

    Ipsos MORI / 17 – 19 Oct


  12. Migrant crisis, I suspect.

  13. Double digits! Lib Dem majority in 2020 confirmed.

  14. Reversiontothemean.com

  15. Good afternoon all from Central London. +18 foretasted for London today..who said summer was over?


    Fantastic work on the new updated swingometer. It really will help us to better forecast seat projections for Westminster now that we can tinker about with seats in each of the nations individually. You really do spoil us. :-)
    “This version allows a
    combined projection, factoring
    Scottish and Welsh polling
    into a GB projection.
    Enter figures from a GB poll in
    the top box, and either Scottish
    or Welsh (or both) into the
    lower boxes. Clicking Calculate
    will then produce a combined
    projection basing the swing in
    England (or other nations
    without a poll) on the figures
    implied by the GB and nation


  16. @Rivers10:
    “My opinion on this matter is only reinforced by the fact that the only arguments against my proposals seem to be one of two things.
    1) It’s too expensive.
    2) Don’t break up England, I like England.
    Neither of which are particularly satisfactory arguments.”

    I would have thought that in terms of practical sustainability, the second reason is the *only* reason which really does make sense for drawing dividing lines on the political map?

    Surely what is fundamentally needed for democracy to function adequately is for it to be based around a coherent demos, a population that generally sees itself as a coherent whole and thus agrees to be bound by the collective will of the group. If the result is that such that the natural demos is of a different size in terms of population and/or geography in different historical areas of the UK, why does that matter? If the people of the entire nation of Scotland see themselves as a coherent socio-political unit, but the people of the city of Chester equally see themselves as a distinct demos, surely their absolute or relative sizes are not an issue, only the feeling of distinctness of the people themselves?

    Drastically different sizes sub-units may well give need for careful thought on how the various parts of any official federal structure would interact. But in terms of defining the units themselves, don’t we see all around the world that the imposition of artificial boundaries (which don’t match the natural feelings of the population) only ever leads to democracy functioning less well, however neat or pleasing the artificial boundaries look to statisticians?

  17. AW……. M goi. :-)

1 3 4 5