Voodoo corner

As we get closer to the election I expect we’ll also get a flurry of what Bob Worcester calls “Voodoo polls” – open access polls that do not make any attempt to gather a representative sample.

In yesterday’s Sun and Times, and today’s Indy, we have a “poll” from a website for mothers, that claims to show mothers rushing to support Cameron and intending to vote in record numbers.

As far as I can tell, it was just an open access poll on a website for mothers. It will, therefore, be skewed towards mothers interested in politics who would have been likely to take such a poll(hence the high figure of them saying they will vote), probably towards mothers with an interest in the sort of issues discussed on websites for mothers, and certainly by the demographics of the people who read the site. We know that last one for sure, since the poll gave some demographic breakdowns of respondents which show that stay-at-home mums and part time workers were over-represented, full time working mums were under-represented, and low income mums and single parents were grossly under-represented. With that income skew it should be no surprise they were very Conservative!

Philip Webster in the Times can be partially forgiven, since he did at least add the caveat that the survey “cannot be regarded as in any way conclusive and is not an opinion poll […] is not representative because it is self-selecting and the organisers accept that people responding to such a survey are more likely to be politically engaged than those who do not.” Unfortunately, he then went on to report the figures as if they did mean something – far better to have done what one editor who mailed me about it on Tuesday, and one veteran political editor who I spoke to today did, and not touch it with a bargepole.

If you see something in the papers proporting to be an opinion poll and it is not from an established polling company, read it very carefully. How did they gather the sample, who are they claiming it represents the views of, and what measures did they take to make sure the it matched the demographics of that group. If the answer is they took whoever came along, and didn’t take any measures at all to make it representative, I think you know how useful it is.


62 Responses to “Voodoo corner”

1 2
  1. Thirty years ago, this “poll” would not have been given space in a tabloid. Today it is in a so-called broadsheet.

    Reminds me that this will be the first “Twitter Election”.

    Oh dear. Oh dear. Oh dear.

    Just how much more simplified – and consequently, almost without fail, incorrect – can information become?

  2. Well, strictly speaking it wouldn’t, since the internet hadn’t happened yet, but press coverage for voodoo polls is certainly nothing new. I bet Bob Worcester could wheel out some horror stories from the 1980s.

  3. Thanks Anthony-thought you would say that.

    Are these subgroup findings equally unreliable?

    ” Among low income mothers some 26 per cent said that they would vote for the Conservatives with only 22 per cent voting Labour. Those stating that they were on family incomes of less than £20,000 found Mr Cameron more likeable than Mr Brown. Among those who stated that they were single mothers, 24 per cent said they would vote Conservative and 22 per cent Labour. ”

    From The Times.

  4. Yep, subgroups within the whole will also be skewed.

  5. Thanks Anthony

  6. Off-topic but in the absence of having anything to say about voodoo polls, I’m wondering about the “known unknowns” again. Anthony said that one of them was the emergence or otherwise from recession this week.

    Given how quickly this disappeared from the news agenda, I’m wondering if a more significant “known unknown” will be the April announcement of Jan-March GDP. If this is also weak then bad news for Labour (assuming no election by then) but if it’s relatively strong then this could be good news for Labour and this week’s 0.1% figure would be forgotten.

  7. Leslie – I suppose it is. A more accurate description of my four “known unknowns” was four “unknowns that Anthony Wells knew were coming”. At the time I expected the exit from recession to come along sooner or later and we’d see what affect it had.

    I didn’t really consider the figures that showed us out of recession being so close to zero that the next lot could go either way.

  8. Perhaps the word ‘poll’ should be confined to specialist polling meeting the quality standards determined by expertise in political polling and its pitfalls.

    The survey of the views of mothers of dependent children is very informative as long as it is not taken as a poll. Just as the views of a large number of bankers, financiers and investors provided information. Devising a filtered feed from these surveys into a view meeting polling standards should be possible.

    After all, as individuals we are doing just that. Saying to ourselves ‘given this, and allowing for that, and bearing in mind the other, it’s interesting’. Certainly we rate this kind of survey information higher than anecdotal.

    Perhaps we shouldn’t?

  9. Hatfield girl – I think you’ve missed the point. There’s nothing wrong with a poll of mothers of dependent children, nothing at all.

    A poll of mothers of dependent childen however would need to be representative of that universe. So it would need to have the correct proportion of single mothers, the right proportions of working mothers and non-working mothers, the right spread of childrens ages, the right proportions of mothers in various income brackets, etc. It would need to be sampled in a way that did not lead to an obvious skew.

    There’s nothing wrong with polling subgroups of society, but for it to be meaningful, measures need to be taken to make it representative of the group it is measuring.

    If anything voodoo polls are worse than anecdotal. At least anecdotal evidence is readily identified as such, voodoo polls tend to have a spurious pretence of validity because they have large numbers, when often they are completely wrong.

    The classic example of “huge numbers” vs “represenative sample” is the US 1936 Presidential election, where George Gallup really proved proper polling. The Literary Digest carried out what passed as polls back then – a huge postal survey of 10,000,000 people that got something like 2 million replies, but made no attempt to make them represenative. George Gallup interviewed a few thousand people, but used proper quota sampling and weighting.

    The Literary Digest predicted Alf Landon would win big, Gallup predicted a landslide for Roosevelt. Gallup was right, and the moral is that polls that do attempt to get a represenative sample are worthless, however large.

    (Oh, and just spotted the implication of your comments, no, even if it was weighted, it would likely be useless because it was self-selecting. It would probably still have respondents who were more politically aware. While I’ve no reason to think it happened in this case, self-selected open polls are also hugely open to people fixing them or just piling in. As we get closer to the election I guarantee people will set up some big open access internet polls that will inevitably predict majorities for UKIP or the BNP, as their activists tend to pile in on them.)

  10. A voodoo poll story

    In the early 70′, the Daily Record (Scottish equivalent of the Mirror) ran a set of ” political polls” to which you could respond by filling in a coupon in the paper, and handing it in to the newsagent.

    In those days, the SNP had a small electoral base and was pretty Poujadist in nature. Consequently, instead of increasing the Record sales, a number of newsagents simply cut out the coupons before returning unsold papers.

    Scottish Independence often was the preferred option of well over 50% responding to the “polls”!

  11. Maybe newspapers could join some sort of sub-group of the BPC by which they agree to publish disclaimers about some of their stories?

  12. I was thinking about classic ethnographic research really, which is very like what the mothers’ survey is, though usually on a much smaller scale. The researcher seeks out a respondent and asks questions that enable the building of a model of whatever is being investigated. With that model in mind further respondents are sought, though rarely was it possible to do what you list, (there were a few studies carried out in eastern England). Obviously this is not polling. But it could and perhaps should, inform polling inputs as the poll is devised. I wish I hadn’t missed the point, because there is something in this information gained from the mothers’ survey kind of interviewing that ought to be able to be useful to polling design and interpretation. Still, not my field, thank you for explaining.

  13. From Mike Smithson

    “There has been nothing official from the firm but I’m hearing a lot of murmurings to the effect that [YouGov] will be starting an election daily tracker – possibly next month.

    Apparently this will operate on weekdays only at first stepping up to seven polls a week as we get closer to the big day.”

  14. Poor old Anthony will have to a synopsis every day, and we will be exhausted by the GE. Perhaps a continuous thread.

  15. Hatfield Girl – with qualitative research (like ethnographic research) it isn’t nearly so much of an issue, and a representative sample isn’t really necessary in the same way. It isn’t my field either, so I can’t really expand much. Obviously though qual research doesn’t lead to quant answers like X% think this or Y% think that.

    Oldnat/Glenn Otto – I couldn’t possibly comment, though once we get into the election campaign there will be probably be threads on new polls every day anyway – certainly in the immediate run up to the last election most days saw at least one new poll arrive.

  16. This voodoo poll reporting doesn’t surprise me – newspapers often use figures from their own online ‘polls’ as evidence of national opinion on a whole host of issues but do normally say for these that it was an online poll where no effort was made to make it representative; they should have done that in this case as well.

    @ Oldnat

    I did wonder if YouGov might do a daily tracker close to the election like they did for the conference season – would be interesting to see what trend that shows. I was also wondering if the other polling companies would have more regular polls commissioned as well? Like Populus? and also, Harrison’s poll aaaages ago, was that really just a one off?

  17. **Harris

  18. Glenn Otto / ALW

    At least on this site most of us, most of the time, will be able to either understand the polls – or at least learn from each other.

    I might just avoid other sites – just to avoid screaming at the monitor too many times!

  19. With a high number of participants like this would it not be true to say that people who bothered to take an active part in this poll are actually more likely to take an active part in a real election, making the skewing rather less than it might at first appear?

    It may not be that the single and low income mums would be so severely under represented as they all seem to have internet access these days and with the folks in that category I know…spend quite a bit of time using it…albeit that in at least a couple of cases they are more likely to be on “Warcraft” than on “Webmums” !

  20. Dear Sir
    I have looked on this site before and everytime i come across the same comment
    “non-partisan” please can someone tell me,if its non-partisan then why is there a banner in the middle of the last editorials saying”ken livingstone stop the right “.
    Will there be a “stop the left” banner as i belive the “looney left” has been in power since 1997.
    glenn

  21. Will there be a “stop the left” banner as i belive the “looney left” has been in power since 1997

    ————-

    Then you must be a minority of 1.

    I can’t imagine anyone thinking that Tony Blair or Gordon Brown are left wing.

    Failed Conservatives, yes. But left wing?

  22. Glenn

    And also on this site you will find an advert saying “Be at the heart of the Party Join Conservative Way Forward Now”

    The loony left again?

  23. @m with Glenn, except that I think the loony left have been in since 1992! :)

  24. Glenn,

    The adverts are presumably paid for – they don’t have to be non-partisan – its how Anthony makes money for this great site.

    David in France,

    Glenn certainly isn’t in a minority of one – only the left think that Gordon Brown is anything other than loony left. Suppose you’ve just given yourself away there – not that we didn’t know anyway.

  25. There’s been a lot of coverage linking Cameron with a site called Mumsnet – perhaps he gets a percentage of the profits.
    He seems to be targetting Tony Blair’s market among women.

  26. Neil

    Most arguments are actually semantics.

    For those that define Mussolini as centrist, New Labour along with Cameron, Clegg et al are clearly on the far left.

  27. Mussolini was a socialist I thought? ;)

  28. Neil A

    Only if you also thought he made the trains run on time! :-)

  29. “its how Anthony makes money for this great site.”

    He makes Money… Where our share, we do all the work.

    Peter.

  30. The ‘loony left’ appears to be alive and well. George Osborne wrote this today for the Guardian – “But perhaps most significantly, the crisis has finally put to rest the assumption, which underpinned Labour’s entire system of financial regulation, that individual behaviour is always entirely rational and that market prices always reflect intrinsic values.”

    Well ‘pon my soul – the next Tory Chancellor saying that markets don’t work! Whatever next?

  31. New Angus Reid poll on attitudes to the economy (tables are available)

    “In descending order, Britons express moderate or complete trust in the following political leaders to do what is right to help the economy: Mervyn King, Governor of the Bank of England (41%), David Cameron, leader of the Opposition (40%), Nick Clegg, leader of the Liberal Democrats (35%), Vince Cable, Liberal Democrat Shadow Chancellor (also at 35%), Gordon Brown, Prime Minister (30%), George Osbourne, Conservative Shadow Chancellor (29%), and Alistair Darling, Chancellor of the Exchequer (23%).

    The highest level of distrust in this file goes to Darling (62%) and Brown (61%).

    Britons trust the Conservative Party more than Labour to rein in the national debt (47% to 22%), and control inflation (41% to 28%). The contest is tighter but still favourable to the Tories for who is better to end the recession (35% to 29%), and create jobs (35% to 32%).”

  32. No seriously, didn’t Mussolini start off as the editor of a far-left newspaper or something? I remember reading that he gradually mutated from Socialist to Fascist in the 20s.

  33. Neil A

    The history of the 20s/30s is littered with examples of people who lurched from one extreme to the other.

    In the context of this discussion, however, it’s where people end up that matters.

    Mussolini and Brown are identical in having abandoned Socialism. The fact that their policies on restriction of civil liberties are scarily close, might give pause for thought!

  34. Fascism (like National Socialism and Communism) is a far-left philosophy, despite what students and others seem to think.
    Right-wing philosophy is all about individual freedom.

    Oldnat – I always wonder how many “Don’t Knows” are involved in these polls. Many people I know either wouldn’t trust any of them or just plain wouldn’t know or care. Having said that, I staggered that presumably 39% do not distrust Brown.

  35. Pete B

    I said earlier that these arguments are entirely semantic. You are perfectly free to describe the concentration on “individual freedom” as right wing, as opposed to Liberalism.

    I am perfectly free to prefer using definitions more commonly used in political theory, and to decline to debate this further.

  36. I am so tempted to join this debate and show the historic correlation between leftism and fascism (correct spelling, no Americanisations here), but I will refrain as I don’t think Anthony would like it.

  37. Oldnat – I have no wish to debate this further either except to suggest that you might have meant ‘liberalism’ rather than ‘Liberalism’ which implies the modern Liberal Democrat party.

    Just to clarify, I believe that most modern ‘right-wing’ parties espouse liberalism with a small ‘l’, whereas 19th Century parties such as the Tories were more into paternlism. I do detect paternalism in most modern parties – i.e. the feeling that ‘we know better than the common people’.

  38. Apropos of nothing :-)

    A favourite aphorism on the Scottish blogs has been

    “One dimensional politics is only for those for whom the concept of a flat earth is too complicated.”

  39. @Pete B

    I believe the terms “left” and “right” come from the French parliament where traditionalists and monarchists were once seated on the right hand side, and liberals and republicans on the left.

    The right stands for traditional values and authority against the left’s desire wealth redistribution, individual liberty, and political equality.

    Contrary to popular belief there is no such thing as right wing economics. The right were once bitterly opposed to the free market economics which they now claim as their raison d’etre.

    Over the last few centuries the right has opposed parliament (in favour of absolute monarchy), religious freedom, abolition of the slave trade, allowing poor people to vote, allowing women to vote, ending racial segregation in the US, boycotting apartheid South Africa, and so on. That’s hardly a glowing endorsement of their support for individual freedom.

    I am sure you will mention Stalinism in response to this, but only a small minority of deluded extremists ever had favourable views of Stalin whereas the aforementioned policies were once part of mainstream centre-right politics.

  40. Jakob

    I refer the Honourable Member to my aphorism above! :-)

    Politics is multi-dimensional. To try to reduce it to a single dimension would strain the capacity of a cosmologist to explain string theory without recourse to the holographic principle.

  41. Off message I know but one of the known unknowns seems to be unravelling.

    The current run on Greek debt is a BIG issue and could be followed by the same in Ireland, Spain, Portugal and Italy – then, I’m afraid, we are next.

    IF (and I hope it doesn’t) this comes to pass and we have panicking bankers, tumbling markets and a run on the pound as a backdrop to the election it would have a significant impact.

    The GDP figures to be issued in April are likely to be weak – work stopped for 2 weeks in Jan for the snow!

  42. Alec:-

    ” Tory Chancellor saying that markets don’t work!”

    That is not what you quoted him as saying-which is that markets don’t allways work.

    Your chosen paraphrase would lead to the conclusion-find something else-from which the choice would probably be of the loony left variety.

    In fact Osborne refers specifically to the problem of asset values . This can-& I have no doubt will-be addressed under a different regulatory regime-but without abandoning “markets”.

    Of much more interest to me is the growing divide in thoughts on the structure & purpose of banks. I think that is where we will see radical change if Obama & a Con administration in UK come up with a joint approach to segregation of “boring” banking & “gambling” banking.

  43. @ OLDNAT
    “Politics is multi-dimensional. ”

    I do agree.
    However too many dimensions would tend to obscure the wood with trees-smear out core principles etc.

    I’m happy with a two dimension approach & use The Political Compass test online.
    It made me think a lot about my own political orientation.

  44. @Oldnat – your poll report numbers are very bad for Labour. Unless there is some rapid movement in economic policy ratings for them I think it’s all over. Their only hope was to have an improving economy that they can get some credit for and regain some support. While there will be some better news it is beginning to look like it’s too little, too late.

    @Jamess “The GDP figures to be issued in April are likely to be weak – work stopped for 2 weeks in Jan for the snow!” I wouldn’t bank on it. Poor weather has been a real bonus for gritter drivers, condom manufacturers and pizza delivery firms, to name a few. I’m not sure what long term effect the weather will have, if any. There are other reasons why the figures might be poor, but then there are also reasons why they could be better. The manufacturing surveys in December showed a slower than expected output in November (that suppressed the Q4 output figures) but also suggested a very sharp rise in orders. This suggests the weak pound and global recovery is beginning to help UK manufacturing exports and there are already some signs of firms taking on workers and attracting overseas investment. Such a scenario would greatly reassure bond markets, but only so long as the government was also seen as credible on debt reduction. All the countries you quote are in the Euro zone – they cannot adjust their currency. This is UK’s biggest asset at present as it will help to rebalance the economy to a degree. Brown deserves some credit at least for keeping us out, but on all the other economic issues it looks like he is running out of road.

  45. “Brown deserves some credit at least for keeping us out”

    Absolutely agree Alec.
    In the teeth of a mad Blairite enthusiasm for it too.

    Perhaps TB doesn’t feel quite so interested in the Euro after failing to get that cushy top job.

  46. @NEIL & OLD NAT
    We all agree Benito M started of as a socialist. However, he is boring because he was just “not very nice”. He was not utterly evil. Now, a reel horror who was very much a left wing interlectual rather than ultra right in his views was Joseph Gobbals, yet he loved Hitler more than any of them.

  47. For reel read real. I need a little hand on my watch and spell checker on my laptop.

  48. Anthony:

    My local Conservative Candidate is spinning “Key findings from ORB Poll of Scottish Voters (Dec 09)” as ” Bad News Polls for Labour, SNP & LibDems”.

    I’m quite sure if it had the usual analysis from yourself you could find some bad news for the Conservatives too.

    Is it a Voodoo poll or just a voodoo interpretation?

  49. It certainly won’t be a voodoo poll – ORB are a reputable outfit – but if it’s on a party leaflet the interpretation of it probably isn’t particularly charitable to other parties.

  50. Alec

    Can you quote specific instances of companies recruiting staff to produce exports. The BBC seems to be full of the opposite.

1 2