Voodoo polling corner

Back in 2012 I wrote about the Observer reporting an open-access poll on a website campaigning against the government’s health bill as if it was representative of members of the Royal College of Physicians. I also wrote to the Observer’s readers’ editor, Stephen Pritchard, who wrote this article about it.

The Guardian today is making the same error – they have an article claiming that seven out of ten junior doctors will leave the profession if the new junior doctor’s contract goes through. The headline presents it as representative of all junior doctors and it is referred to as a poll and a survey in the first two paragraphs. Only in the final, seventeenth paragraph is it revealed that it wasn’t conducted by any reputable market research organisation, but a self conducted survey of members of a Facebook group, the Junior Doctors Contract Forum, which is campaigning against the new contract (the Telegraph had a similar article earlier this month that appears to be based on the same data).

We cannot tell if efforts were made to limit the poll to actual doctors or to make it representative of junior doctors in terms of career stage, age, region and so on – it doesn’t really matter, as it is fatally undermined by being conducted in a forum campaigning against a contract. It would be like conducting a poll on fox hunting in the Countryside Alliance’s Facebook group and presenting that as representative of the countryside’s views on foxhunting. The flaw should be screamingly obvious.

Questions along the lines of “If thing you oppose happens, will you do x?” are extremely dubious anyway. The problem is that respondents to opinion polls are not lab rats, they are human beings who seek to use polls to express their opinion, even when it’s not exactly what the question asks. From a respondent’s point of view, if you are filling in a survey about something you oppose, you’re are likely to give the answers that most effectively express your opposition. Faced with a question like this, it’s far more effective to say you might leave your job if your contract is changed than say you’d meekly accept it and carry on as usual.

We see this again and again in polls seeking to measure the impact of policies. For example, before tuition fees were increased there were lots of polls claiming to show how many young people would be put off going to university by increased fees (such as here and here). After the rise, they miraculously continued to apply anyway. Nobody wants to tell a pollster that they would just swallow the thing they oppose.

I don’t doubt that many or most junior doctors are unhappy with the new contract, but you can’t get a representative poll by surveying campaigning groups, and you shouldn’t necessarily believe people telling pollsters about the awful consequences that will happen if something they don’t like happens. It’s a lot easier to make a threat to a pollster that you’ll resign from your job than it is to actually do it.

UPDATE: While I’m here in voodoo polling corner, I should also highlight this cracking example of a voodoo poll in the Daily Mail. It claims “One in three women admit they watch porn at least once a week”… but it seems to be an open access poll of Marie Claire readers, certainly it is in no way representative of all women in terms of things like age. It contains the delightful line that “Out of the more than 3,000 women surveyed, 91 per cent of the survey’s respondents identify as female, eight per cent identify as men and one per cent is transgender.” I don’t know how to break it to them, but you probably can’t include the 8% who are men in a survey of 3000 women.

217 Responses to “Voodoo polling corner”

1 2 3 4 5
  1. ItIt’s certainnly wrong to assume that all those born in England and now ersident in Scotland voted ‘No’. I, for one, voted ‘Yes’.

  2. @John B

    Me too!

  3. Argentine election tomorrow. Should be interesting. Scioli will win the first round, the only question is whether the gap will be large enough to avoid a run-off. Outcome of a run-off between Scioli and Macri is less obvious.


  4. Some may have seen Iain Macwhirter’s article in the Independent EVEL isn’t about Scotland. It’s about locking Labour out of power in the UK, where he makes similar points (though better written!) that some on here have made.


    Peat Worrier looks at what might happen if the unlikely scenario of “a majority of MPs from across the UK who are prepared to back a new Labour government over the Tory minority”.


    “Caught between the frying pan of EVEL and the fire of governing in England without power, any Labour leader worth their salt would – and could – use their UK majority to consign the “English vetoes for some English laws” experiment to the scrap heap. If you look at the text of the Standing Orders changes passed on Thursday, you’ll see that EVEL is not an “England only” matter. It is for the whole House of Commons to decide how to structure its work. That will be as true in 2020 as it was this week.”

    For those actually interested in what EVEL can achieve (as opposed to the symbolism), these are worth a read.

  5. That seems to me expressly the purpose of EVEL. To paint a future Labour government into the corner of legislating to take it away.

    How effective that trap will be will depend on whether that future Labour government has a majority of MPs in England (as is usually the case). If so, the political risk of the repeal would be small.

    If a Labour minority in England calls in the Celts to repeal EVEL then this will be used by the Tories and their media supporters as a tool in the same way that the prospect of a Labour-SNP coalition was used.

    However, the fact that its pretty nakedly a political move doesn’t alter my view that there is an important principle at stake.

    My preferred option, like most of those here who responded to my “pop quiz” on alternatives to EVEL, would be a federal UK. I personally think an English parliament would be fine, but could also live with Heptarchy style regional parliaments in England, so long as they had equal and equivalent powers to Scotland (ie the end of England as single country).

    @Neil A
    That is a scandalously unjustified assumption sir. Indeed I am one of the only ten people in the UK who bothers to vote in both local and European elections

    I can go better than that, I vote in General election, European elections, Local elections and referenda.

  7. Neil A

    Except the whole point of Peat Worrier’s argument is that there is no need for an incoming Lab Government to be involved in “legislating to take it away”.

    There have only been a few months since 1945 when a Labour Government didn’t also have a majority of English MPs [1]. and you would have to be predicting the total collapse of the SNP in Westminster elections – and Labour to be the only beneficiaries – for that situation to be replicated.

    Under the current parliamentary arithmetic, EVEL will have had precisely zero opportunity to demonstrate its effectiveness in “protecting” England, so it’s hard to see how it might have generated any level of passion in its support.

  8. Forgot to add the footnote!

    [1] The role of Scottish/Welsh Lab MPs in forcing measures on England is nothing to do with Lab not having a majority in England. Their value lies in their not particularly giving a damn about what happens in England, and thus happily voting with the Lab leadership. to defeat rebel English Lab MPs.

  9. @Neil A

    “That seems to me expressly the purpose of EVEL. To paint a future Labour government into the corner of legislating to take it away”

    Only if Labour play that game though. They could just legislate to have a federal parliament after all (they might even get support on that basis as providing a real solution to the problem.) That would then put the ball back in the Tories court, who really don’t want a federal parliament for England.

    Of course this comes to naught as the Labour strategist are as daft as the Tory strategists and will thus miss the open goal presented by their opponents supposed chess wizardry.


    … You win this round, Howard…

  10. @Oldnat,

    Whether you call it legislating or something else, doing away with through the votes of Scottish MPs would involve some political risk in English marginals.

    I don’t disagree that a UK Labour government would be likely to have a majority of English MPs. Also, reversion to the mean suggests to me that at some point there will be a “collapse” of SNP seats and the LiS will be the main beneficiaries – assuming independence hasn’t already made it moot.

    Lastly, we all know the phenomenon that involves people placing greater value on things they already have than on things they are being offered. I think that may apply to EVEL too, to some degree. I expect that’s the hope of the Tories in any event.

  11. Neil A

    “Whether you call it legislating or something else, doing away with through the votes of Scottish MPs would involve some political risk in English marginals.”

    1. I suspect that you couldn’t be bothered actually to read Peat Worrier’s argument! (or Iain Macwhirter’s article)!

    2. What’s your obsession with “Scottish MPs”? Don’t your media tell you that Wales and Northern Ireland have MPs at Westminster as well?

    As to “reversion to the mean” – would you care to put a date on when the Liberal Party will, once again, hold three-quarters of the Scottish seats?

    I agree with you that some folk in England are hoping that EVEL may be of some significance. In the early 20th century, some Scots hoped that the Scottish Grand Committee would be of some significance too.

  12. It seems that BBC London is as useless at reporting EVEL as they are on many other things regarding the UK constitution.


    “Housing Bill set to be first test for English votes plan”, they say. However since a majority of MPs in the HoC and a majority of English MPs are from the same party, and thus it is no “test” at all.

    There is no suggestion that the BBC is biased – simply that their reporters are wholly incompetent, and not worth their salaries.

  13. @Oldnat,

    I did read the articles, of course. They simply express opinions that are similar to your own, rather than present any information that I wasn’t aware of. There’s nothing in either article that invalidates my belief that a post 2020 HoC vote by a Labour government to repeal EVEL, where the casting votes came from outside England would be politically risky. Peatworrier even predicts the political attacks that would result from the media, whilst dismissing their effect of course.

    I am using “Scottish MPs” as a sort of shorthand, as “Scottish, Welsh and Northern Irish MPs” is a bit of a mouthful and Scottish MPs outnumber the other two put together by some margin, and the effect of EVEL (or EWNIVEWNIL if you want to stay pernickety) is stronger for Scottish MPs than others. Also, the dynamics of Scottish political relationships with England are a bit more volatile – there is no prospect of either NI or Wales voting for a divorce anytime soon.

    As to reversion to the mean, you and I both know that this expression means that the trend will usually move in the direction of the long term average – not that it is destined to return exactly to what it had previously been.

    Maybe you’re right. Perhaps the SNP will take those last 3 seats in 2020 and then hold all 59 for the next 300 years. Stands to reason.

  14. Neil A

    Peat Worrier’s description of the conflict was one of the reasons why I thought his post was worth linking to.

    Politics is always about taking decisions in the conditions that currently prevail – and guessing as to those that might in the future.

    On a polling site, it normally makes sense to link to some polling evidence for such future possibilities.

    Of course, there may be some return to Scotland returning a set of MPs “in the direction of the long term average”. That means a majority of non-Labour MPs so I suspect that what you actually hope for/anticipate is a return to a situation in which Scottish MPs are of the pattern that your experience makes you comfortable with.

    Your use of Scotland to include Wales and Northern Ireland may justifiably annoy those from those nations. “Non-English” uses 3 more keystrokes than “Scottish” – so hardly “a bit of a mouthful”. :-)

    Your last sentence is just silly.

  15. Amber
    “I actually think it’s John Chanin & Pete B who’ll suffer a loss by not reading my jolly fine comments. :-)”

    As I’ve been dragged into this, I’ll just say that I didn’t mean that all posts by Scottish posters should be ignored, but simply that posts about specific Scottish issues should be confined to threads about Scottish polls, rather than dragged into every single thread at enormous length.

    And by the way, while I might disagree with some of your comments you are not one of the posters that I had in mind.

  16. ON
    My previous post seems to have been moderated, but I will just say that any reasonable person would have taken John C’s comment as a jocular remark.

  17. Nothing much changes around here I see.


    I agree that Tristran’s comments were equally unacceptable and i should have included him in my post.

    As to the history this is disputed as you probably well know. I think Candy’s version is more correct. The use of “little englander” typifies your unpleasant comments. Candy is almost certainly correct in stating that the percentage of the UK population in England will rise as she indicates.

    You do yourself down when you insult people and it’s not in the spirit of this site..

  19. Colin

    Nothing changes. If you don’t agree with the usual suspects you get abuse. Off for a long walk with my wife as the weather looks set fair today. Have a good one yourself.

  20. PeteB –
    “Let’s survey the posters here with the following question.
    Discussions about Scotland should be restricted to threads about Scottish polls, yes or no.”

    The comments policy here isn’t a democracy, it’s a dictatorship (hopefully a relatively benign one) and that is the request I have previously made. That said, “EVEL” does obviously have an impact on, um, England, so I can hardly complain – AW

  21. Seen a poll that suggests UKIP voters would favour a military coup by 44% to 40%.

  22. Blimey. Perhaps all polls are voodoo polls now we have established they are all wrong.

    Nationalism does seem to be gaining where the young are leaving. Perhaps because nationalism is an old person’s sport? certainly it is the root of a lot of the world’s evils.

  23. Heaven portend that you’d let facts get in the way of good Scotland bashing thread

    Scotland’s population highest ever and ?Migrants to Scotland tended to be younger than the general population

  24. couper2803

    Isn’t that the problem? The older indigenous population are worried about the younger incoming migrants affecting their way of life (and voting for smaller pensions?)

  25. @Couper,

    I don’t think Candy said that the headline figure of Scotland’s population was falling. Simply that England is growing at a very, very fast pace and Scotland isn’t.

    The figure in the article, net migration of less than 18000 in two years, that if we were talking about England would be more or less margin-of-error territory.

    The point, that Scottish political weight in the UK will decline over time, is valid I think.

  26. @Neil A

    That’s why Scotland will eventually chose Independence. I think it is an advantage not to be over crowded and if we can attract youngsters, which we are doing then great, but we don’t have to measure ourselves against England.

    Every week on QT & other programs there is endless discussions on immigration, but we don’t have that problem in Scotland, other than a bit of leakage in public opinion from the English agenda. So lots of immigration is not a completely good thing according to English audiences.

  27. Good morning all from Westminster North.

    On the subject of EVEL I have to say I prefer [stuff, but will probably follow the comments policy and leave my own politics at the door – AW]

    However I personally think the whole EVEL thing is a storm in a tea cup and I can’t foresee a scenario where sitting Scottish MP’s can not actually vote on something that would have a detrimental impact on Scotland.


    “Off for a long walk with my wife as the weather looks set fair today.”

    Mind and wrap up because its a bit chilly today and make sure you let other people know where you are going and what time you expect to be back home.


    “The comments policy here isn’t a democracy, it’s a dictatorship”

    UKPR (United Korea Propaganda Report) My ole chums in North Korea are very interested in your website Domain. ;-)

  29. @Couper,

    I agree it is a long-term driver towards independence, but more for the reasons given by Candy than yourself.

    Scotland being independent probably wouldn’t shield them from inward migration, if there was a community of migrants wishing to go there. It is probably Scotland’s geography, climate, economic structure and the proximity of England (as an alternative, and vastly preferable, destination) that does that. An independent Scotland would be very likely to have open borders with rUK and there would probably be no difference seen in migration whether in or out.

    However, it is the balance of power between Scotland and England that would lead put pressure on the Union, as it has already.

  30. Also I wouldn’t set too much store by the mean age of new migrants. At least, I’d want to dig into the figures a bit more. I suspect that family reunifications, for example, bring a lot more mature migrants into England than Scotland – on account of the much greater existing community applying for visas.

    It may also be that the older migrants to England reflect a higher rate of qualified professionals, as we have a far greater issue with finding doctors, nurses, teachers, engineers etc on account of our high living costs.

    I don’t know that any of those factors are actually true, but I’d want to exclude those possibilities before concluding that it is simply a case of the average age being lower without any other differences in the profile of migrants coming in.

  31. @ Couper 2802

    Heaven portend

    It’s generally heaven forfend but maybe you did mean portend. It would be nice to know whether you meant to coin a new version of the phrase or whether predictive text changed your intended comment.

  32. @Neil A

    I think we are agreeing, independence will solve the population imbalance & once Scotland is independent the relationship with England will be the same as that between England and the Republic of Ireland ie We won’t really care how many immigrants are in England.

    I have a big house in by the beach with great views, in a lovely suburb walking distance to great shopping, cafés restaurants and pubs. I am always thankful to the Scottish weather otherwise I could never afford to live here, in a warmer country my house would be worth millions.

  33. @Amber Star

    Yes it’s the curse of predictive text.

  34. I see Simon Danczuk has issued his ‘come and get me’ plea to the producers of ‘I’m a Celebrity’

    Also, it’s worth noting that, if all trends continue as projected, then in a centuries time Scotland and Wales will most likely be the most populated regions in the UK. A significant chunk of England will, after all, be beneath water.

  35. NickP – “Nationalism does seem to be gaining where the young are leaving. Perhaps because nationalism is an old person’s sport? certainly it is the root of a lot of the world’s evils.”

    Justin Trudeau said that nationalism indicated “smallness of thought” :-)

    Seriously though – most of the populations where the majority is nationalist – Eastern Europe, Russia, appear to be worried about facing a possible extinction event. And they’re trying to shut their doors because they know they don’t have enough young and can’t reproduce quickly to repopulate and believe that the strangers will outnumber them in short order. It’s irrational in the short term, but may not be in the long term.

    Since 1992 the Baltic region (Lithuania, Estonia and Latvia) has lost one in five of it’s people, mostly young. It’s a disaster for them. They haven’t got enough people to adequately defend themselves, when the existing middle aged and old drop out of the stats in about 30 years, the population will be so low, some in NATO will be wondering whether it’s a waste of money defending them.

    Is the same fear behind the reason that Scotland is going nationalist (though the nationalists are not a majority there yet)? Don’t know. But their weight is dropping – in the boundary review coming up, a lot of the seats that are over-represented (i.e. have sub par levels of constituents) are in Scotland – Scotland will lose seven seats, Wales will lose ten seats but the populous South East will lose just one seat.

    So political weight is moving south.

  36. As for EVEL from a Westcountry perspective, can I briefly point out that down here we are very relaxed about Scottish independence, the Barnett formula, etc. We’re fed up with being dictated to by London, not by Edinburgh or Cardiff..

  37. @Millie

    Living in West Yorkshire, London and the South East feels as alien, distant and irrelevant to me as Edinburgh does.

    A potential English Parliament should not be in effect a London Parliament.

  38. Millie, Catmanjeff
    Sounds like you two are very much in agreement with what I was saying two pages ago, any solution to the UK’s constitutional woes basically has to break up England (for administrative purposes only)

    One of todays most ignorant acts is to refer to what “the English” want. Now lumping any nationality into one group and assuming they all want the same thing is dubious at the best of times hence I try to refrain from saying things like “what the Scots or Welsh want” but compared to the English the Scots or Welsh could have all rolled out of a cloning facility.

    No UK politician can claim to speak on behalf of England, no UK politician can even claim to speak on behalf of MOST of England and until Westminster realise that the devolution question will drag on and on.

    To me there is only one satisfactory solution, massive devolution to the English regions (on par with what Scotland, Wales and NI have) My opinion on this matter is only reinforced by the fact that the only arguments against my proposals seem to be one of two things.

    1) It’s too expensive.
    2) Don’t break up England, I like England.

    Neither of which are particularly satisfactory arguments.

  39. @Rivers10.

    Seems odd to say that no one can claim to speak for England. Plenty of people feel they can speak quite categorically for Scotland…

  40. Rivers
    Didn’t the North-East (of England, Oldnat) overwhelmingly reject a Regional Assembly about 10 years ago?

  41. @Pete B

    That was as part of a ‘bolt on’ onto the current constitutional settlement, so was a different matter.

    In addition, I suspect that as a nation are in a massively different place regarding how people view devolved powers than back then.

  42. CMJ
    I take your points, but i think it would be a hard sell to the voters. I looked it up, and though the government had chosen the North East because they thought it was most likely to go for a regional assembly, nearly 80% voted against. It would take a lot to overcome that sort of resistance.
    There is a lot of intra-regional rivalry, and many would rather be ruled by London than by local rivals. For instance Sunderland would not want to be ruled from Newcastle, Liverpool from Manchester, Black Country from Birmingham etc etc.

  43. Pete B
    The North East did reject an assembly back then but aside from the glaringly obvious “that was then” argument, which one shouldn’t underestimate. Support for Scottish independence back then never exceed one third of Scots (things changed there pretty quickly) not to mention we have since had devolution dragged up to the forefront of issues and 5 (soon to be 10) years of an unpopular Tory government (in the North at least, North East especially)

    But I think the more significant argument is that have you looked at what powers the North East assembly was offered? Most seem to be under the impression they were offered a similar package to Scotland and Wales, as it happens it was just a glorified local council, they were offered less powers than the London assembly currently has. For those that don’t believe me here’s the exact powers mentioned in the draft bill.

    Promotion of economic development
    Promotion of social development Promote health, safety and security of the community
    Reduce health inequalities
    Enhance individual participation in society
    Improve the availability of good housing
    Improve skills and the availability of training
    Improve the availability of cultural and recreational activities
    Improvement and protection of the environment
    Additional functions and duties that the Secretary of State thinks appropriate

    Nobody is going to get excited by that offer. Frankly I’m amazed as many people voted for it as they did. I’m an ardent advocate of devolution but if I was told that’s all your ever going to get my own thoughts would be “not worth it, I’ll vote no”

  44. Pete B
    Quick follow up post in response to your “inter regional rivalry” point. Its a valid point hence I’ve always liked the idea of (in the event regional assembles are offered) a regional “capital” where the regional government meets is voted on by said regions populace. Several cities in the region are put forward (caveats being they have to be reasonably centrally located within the region and have the infrastructure to host the regional government and respective government departments) and then voted on with people unable to vote for their own city.

  45. Neil A
    “Plenty of people feel they can speak quite categorically for Scotland…”

    And I’d go on the record as saying those people are probably exaggerating their mandate a wee bit. But as I said compared to the disparate voices of England, Scotland and indeed Wales are remarkably uniform.

  46. I’ve now updated the fancy graphical swingometer (the one with separate swings for England, Scotland and Wales) for 2015, including adding UKIP, here:


    It will be out of date if the boundary changes go ahead of course, but people kept asking for it!

  47. Rivers that seems quite a wide-ranging list of powers, even if they’re a little vague. I wonder what proportion of the electorate were aware of the precise powers on offer? I suspect that many would be voting on general principles such as ‘do we want another layer of government?’.
    Although your proposed method of choosing a regional capital seems very reasonable, I still think it would lead to resentment in many cases (such as the ones I gave earlier). Perhaps a compromise candidate on the model of Canberra might work better (such as Warrington in the North West), but we’re getting a bit bogged down in detail here.
    Also, I am sure that you are aware that the EU wants nations to devolve powers to regions so that it can more easily bypass national governments. So long as you are aware of this consequence of devolution, that’s fine.

    AW – nice job

  48. @Rivers10 – “the only arguments against my proposals seem to be one of two things 1) It’s too expensive.
    2) Don’t break up England, I like England.”

    It really is too expensive!

    If an English Parliament was set up with the same level of representation as the Scottish parliament (129 MSPs for about 5.2million people), we would have about 1300 MPs in it!

    That’s a colossal amount. Even if you broke it down into regional parliaments the total new elected bods (complete with expenses, pensions and their very own staff all paid for by the taxpayer) would be about 1300.

    And they’d all demand top pay [Snip – I don’t think this topic of conversation is going to end up with civilised, non-partisan discussion. Do you? AW]

    Oh. And we don’t want to break up England :-)

  49. If an English Parliament was set up with the same level of representation as the Scottish parliament (129 MSPs for about 5.2million people), we would have about 1300 MPs in it!

    Another 1300 MPs taking no responsibility for anything & moaning about Westminster. What a glorious prospect!

  50. Candy
    Well there’s quite a few things to adress there. First of all I’ve always advocated trimming some areas of government to make way for the assemblies, for example Westminster could see its number of mps reduced reflecting it’s reduced power. Also there are over 18,000 local councillors in England, having lost there status as the only type of local representitive England gets it would be unreadable to see them reduced. A 10% cut per authority equalling 1800 councillors, surely that’s worth at least a dozen or so assembly members?

    Then moving on to your 1300 figure, who says it has to be equal to Scotland. Scotland has 1 map per 40,000 people (reflective of the tiny electorates in the islands) Wales not burdened with this problem has one assembly member per 50,000 people. That ratio would see the figure for England drop to 1000 from the get go.

    But even if we ignore other reductions in government and take your 1300 figure and work on the very generous presumption that the wages, pension, expenses, staff of each assembly member costs 500,000£ (it would almost certainly cost half that) then the cost per year works out at 650 million which in total budget terms is small change for ending our constitutional woes and saving the union. But as I said with practically no effort and some natural reductions elsewhere it would cost less than half that figure. That is good value and is how all federal nations are governed.

1 2 3 4 5