Or at least, questions that should be treated with a large dose of scepticism. My heart always falls when I see questions like those below in polls. I try not to put up posts here just saying “this poll is stupid, ignore it” but sometimes it’s hard. As ever, I’d much rather give people the tools to do it themselves, so here are some types of question that do crop up in polls that you really should handle with care.

I have deliberately not linked to any examples of these in polls, though they are sadly all too easy to find. It’s not an attempt to criticise any polling companies or polls in particular and only he who is without sin should cast the first stone, I’m sure you can find rather shonky questions from all companies and I’m sure I’ve written surveys myself that committed all of the sins below. Note that these are not necessarily misleading questions or biased questions, there is nothing unethical or wrong with asking them, they are just a bit rubbish, and often lead to simplistic or misleading interpretations – particularly when they are asked in isolation, rather than part of a longer poll that properly explores the issues. It’s the difference between a question that you’d downright refuse to run if a client asked for it, and a question that you’d advise a client could probably be asked much better. The point of this article, however (while I hope it will encourage clients not to ask rubbish questions) is to warn you, the reader of research, when a polling question really should be read with caution.

Is the government doing enough of a good thing?

A tricky question. This is obviously trying to gauge a very legitimate and real opinion – the public do often feel that the government hasn’t done enough to sort out a problem or issue. The question works perfectly well it is something where there are two sides to the question and the question is intrinsically one of balance, where you can ask if people think the government has not done enough, or gone too far, or got the balance about right. The problem is when the aim is not contentious, and it really is a question of doing enough – stopping tax evasion, or cutting crime, for example. Very few people are going to think that a government has done too much to tackle tax evasion, that they have pushed crime down too low (“Won’t someone think of the poor tax evaders?”, “A donation of just £2 could buy Fingers McStab a new cosh”), so the question is set up from the beginning to fail. The problems can be alleviated a bit with wording like “Should be doing more” vs “Has done all they reasonably can be expected to do”, but even then you should treat questions like this with some caution.

Would you like the government to give you a pony?*
(*Hat tip to Hopi Sen)

Doesn’t everyone like nice things? There is nothing particularly wrong with questions like this where the issue at question is something controversial and something that people might disagree with. It’s perfectly reasonable to ask whether the government should be spending money on introducing high speed rail, or shooting badgers, or whatever. The difficulty comes when you are asking about something that is generally seen as a universal good – what are essentially valence issues. Should the government spend extra on cutting crime, or educating children, or helping save puppies from drowning? Questions like this are meaningless unless the downside is there as well – “would you like the government to buy you a pony if it meant higher taxes?“, “would you like the government to buy you a pony or could the money be better spent elsewhere?

How important is worthy thing? Or How concerned are you about nasty thing?

Asking if people support or oppose policies, parties or developments is generally pretty straightforward. Measuring the salience of issues is much harder, because you run into problems of social desirability bias and taking issues out of context. For example, in practical terms people most don’t actually do much about third world poverty. Ask people if they care about children starving to death in Africa you’d need a heart of stone to say that you really don’t. The same applies to things that sound very important and worthy, people don’t want to sound ignorant and uninterested by saying they don’t really care much. If you ask about whether people care about, are concerned about or think an issue is important they will invariably say they do care, they are concerned and it is important. The rather more pertinent question that is not always asked is whether it is important when considered alongside all the other important issues of the day. The best way of measuring how important people think an issue is will always be to give them a list of issues and pick out those they consider most important (or better, just give them an empty text box and ask them what issues are important to them).

Will policy X make you more likely to vote for this party?

This is a very common question structure, and probably one I’ve written more rude things about on this site than any other type of question. There are contexts where it can work, so long as it is carefully interpreted and is asked about a factor that is widely acknowledged to be a major driver of voting intention. For example, it’s sometimes used to ask if people would be more or less likely to vote for a party if a different named politician was leader (though questions like that have their own issues).

It becomes much more problematic when it is used as a way of showing an issue is salient. Specifically, it is often used by campaigning groups to try and make out that whatever issue they are campaigning about will have an important impact on votes (and, therefore, MPs should take it seriously or their jobs may be at risk). This is almost always untrue. Voting intention is overwhelmingly driven by big brush themes, party identification, perceptions of the party leaders, perceived competence on the big issues like the economy, health or immigration. It is generally NOT driven by specific policy issues on low salience issues.
However, if you ask people directly about the impact of specific policy issue on a low salience issue, and whether it would make them more or less likely to vote for a party, they will normally claim it does. This is for a number of reasons. One is that you are taking that single issue and giving it false prominence, when it reality it would be overshadowed by big issues like the economy, health or education. The second is that people tend to just use the question as a way of signalling if they like a policy or not, regardless of whether it would actually change their vote. The third is that it takes little account of current voting behaviour – you’ll often find the people saying a policy makes them more likely to vote Tory is made up of people voting Tory anyway, people saying a policy makes them less likely to vote Tory are people who wouldn’t vote Tory if hell froze over.

There are ways to try and get round this problem – in the past YouGov used to offer “Makes no difference – I’d vote party X anyway” and “Makes no difference – I wouldn’t vote for party X” to try and get rid of all those committed voters whose opinion wouldn’t actually change. In some of Lord Ashcroft’s polling he’s given people the options of saying they support a policy and it might change their vote, or that they’d support it but it wouldn’t change their vote. The best way I’ve come up with doing it is to give people a long list of issues that might influence their vote, getting them to tick the top three or four, and only then asking people whether the issue would make them more or less likely to vote for a party (like we did here for gay marriage, for example). This tends to show that many issues have little or no effect on voting intention, which is rather the point.

Should the government stop and think before going ahead with a policy?

This is perhaps the closest I’ve seen to a “when did you stop beating your wife” question in recent polls. Obviously it carries the assumption that the government has not already stopped to consider the implications of policy. Matthew Parris once wrote about a rule of thumb on understanding political rhetoric, saying that if the opposite of a political statement was something that no one could possibly argue for, the statement itself was meaningless fluff. So a politician arguing for better schools is fluff, because no one would ever get up to the podium to argue for worse schools. These sort of questions fall into the same trap – no one would argue the opposite, that the best thing for the government to do is to pass laws willy-nilly without regard for consequences, so people will agree to the statement in regard of almost any subject. It does NOT necessarily indicate opposition to or doubt about the policy in question, just a general preference for sound decision making.

Do you agree with pleasant uncontentious thing, and that therefore we should do slightly controversial thing?

Essentially the problem is one of combining two statements together within an agree disagree statement, and therefore not giving people the chance to agree with one but not the other. For example “Children are the future, and therefore we should spend more on education” – people might well agree that children are the future (in fact, it’s relatively hard not to), but might not agree with the course of action that the question suggests is a natural consequence of this.

How likely are you to do the right thing? or Would you signify your displeasure at a nasty thing?

Our final duff question falls into the problem of social desirability bias. People are more likely to say they’ll do the right thing in a poll than they are to do it in real life. This is entirely as you’d expect. In real life the right thing is sometimes hard to do. It might involve giving money when you really don’t have much to spare, or donating your time to volunteer, or inconveniencing yourself by boycotting something convenient or cheap in favour of something ethical. Answering a poll isn’t like that, you just have to tick the box saying that you would probably do the right thing. Easy as pie. Any poll you see where you see loads of people saying they’d volunteer to do something worthwhile, boycott something else, or give money to something worthy, take with a pinch of salt.


243 Responses to “Questions that should be ignored”

1 2 3 5
  1. First!

  2. Another category of poll question which should be ignored is “Would you like there to be a referendum on [any policy under the sun]?”. This almost invariably produces a yes answer, because the choice offered to polled respondents is whether they ought to have a say, or whether the issue will instead be decided by a group of politicians whom we know most voters do not trust.

    I rack my brains to think of any poll asking whether people want a referendum which did not produce a majority in favour. There may have been one once. But it’s a gift of a question to any lobby group which thinks that politicians might possibly not decide an issue in the way they want them to.

  3. “Should the government stop and think before going ahead with a policy?”

    Actually, it might be an idea for them to just get on with things. If thinking is their (any government’s) forte, and few are happy, their thinking is not working.

    Perhaps “Should the government stop?” would be a better question, in that we can do with less government and more effective governance. Less meddling in other words.

    This is non-partisan and applies to the existing and previous governments, and also to current and past ones North of the border.

    Perhaps we need some sort of “if it’s going to be popular, block it” approach? Popularism has become the default position, and is not governance. If popularism was a good thing, directors would increase employee salaries excessively, and bankrupt the company.

  4. David – there is one! One sad, lonely example.

    Someone tracked it down last time I wrote about it. MORI once asked if people wanted a referendum on abolishing the monarchy and found marginally more people opposed than in support.

  5. John Major fuels he fire EM’s zeitgeist has set on energy matters.

    What a great series of illustrative questions from Anthony and qualitiative answers too!

  6. I’m sure there are plenty of people who would think that the government has done too much to tackle tax evasion. Not necessarily just those who are evading HMRC either.

    There is obviously a point where any additional money spent by government to tackle tax evasion collects less in tax than was spent to collect at. At that point you know the government has gone too far.

  7. Basically this relates to public perceptions of Governments

    The Current Government would do well to consider the proposal by Sir John Major, the former Conservative prime minister (and the Only Conservative Prime Minister to ever run away from the Circus to become an accountant), has said there is a case for an excess profits tax on the big six energy companies. He said the money could be used to help the poor as they struggle with massive energy bills this winter.

    He claimed many people this winter will face a choice between eating or staying warm.

    This would at least be an alternative to Labours price freeze ,effectively the result is the same as preventing increased prices to customers is fundamentally the same as paying them back when they are overcharged.

    I think most people would regard this as a “Good Thing”

  8. @Al

    I’m sure there are plenty of people who would think that the government has done too much to tackle tax evasion. Not necessarily just those who are evading HMRC either.

    I’d be very confident most of electorate would conclude entirely the opposite.

  9. Another Question that shouldn’t be asked is do you (as a particular subset of the population ) feel discriminated against in favour of another subset consequently we have the situation where people of all ethnicities ,religions, genders and sexual orientation will if prompted tell you that they are being discriminated against in favour of all the other people and vice versa.

  10. @ John Murphy

    “John Major fuels he fire EM’s zeitgeist has set on energy matters.”

    Yes, I’m looking forward to seeing posts about how positive it was that this closet Socialist was removed and it happened because we don’t leave in North Korea…

  11. @ Anthony Wells

    Nice (sad) collection. But I think those questions where one has to choose from two supposedly opposite statements when they are not opposite (would answer both with Agree or Disagree) is more dangerous.

  12. Apart from the obvious delight that John Major making an intervention means that the next issue of Private Eye will have a Secret Diary in it, he has delivered a series of very firm kickings to all sorts of people this afternoon.

    I rather think he’s given the phrase that will feature in IDS’ political obituary for a start – ‘his genius is not proven’. It has the smack of Widdecombes career-ender for Michael Howard.

    Then he got stuck into fuel prices. What fun! I would imagine some of the current Tory leadership are hopping mad, and I look forward to the hatchet jobs in the Mail and Telegraph. Some are suggesting that he’s waited 20 years for his revenge on IDS and co, and it sounds like he’s chosen his moment.

  13. @ AW

    You missed the Brasseye question- and when do you think the government should implement this policy?

    Answer: Immediately- 100%

  14. CHRIS RILEY

    Did John Major express any relief that Edwina Curry showed on Radio 5 Live that she has a very bad memory about which politician “really, really wanted” what?

  15. Chris Riley

    “I rather think he’s given the phrase that will feature in IDS’ political obituary for a start – ‘his genius is not proven’. It has the smack of Widdecombes career-ender for Michael Howard.”

    On the issue of IDS I can assure you that nobody in the party thinks JM has any influence on IDS career who is rather popular amongst grass root supporters for tackling the welfare budget head on with some success.

    I’m not sure JM had any effect on anything much even when he was PM, the only thing we remember him for is the crushing defeat of 97 , he is the last one to use the phrase ‘his genius is not proven’ it rather reminds us of his own efforts.

  16. Great stuff Anthony.

    Very amusing.

  17. Oh well, we can do away with elections altogether and all pollsters and sites like this will therefore be made redundant if we just let the Civil Service decide who should be in Government: http://www.bbc.co.uk/news/uk-politics-24625559

  18. Voting intention is overwhelmingly driven by big brush themes, party identification, perceptions of the party leaders, perceived competence on the big issues like the economy, health or immigration.

    Has immigration always been one of the ‘big issues’? My perception is that Thatcher more or less killed it, before it revived in the expanded-EU context. Perhaps this is one of the ways the Overton window shifts, in the changing salience of different potential ‘big issues’.

  19. NORBOLD

    Robust response from the Cabinet Office, though. :-)

  20. Turk

    Oh come on, major was the most popular thing about the major govt

  21. NORBOLD

    Only a former Civil Service Mandarin could have suggested that !

  22. @AW

    You forgot one (cf good job/bad job questions)

    “How willing are you to express a partisan opinion about a political party you won’t vote for?”

  23. Major’s 1992 victory was a remarkable triumph which the tory party seem very reluctant to give him credit for. Thatcher would almost certainly lost that one – and maybe thats what the problem is for them.

  24. Competition Time

    Write a question that includes all seven of Anthony’s points; The ultimate bad poll question.

    You know you want to!

    Peter.

  25. @REGGIESIDE

    Yes I think on a profound level you are right about JM and the Tory Party.

    This is the same impulse that genuinely made the ”Bennite” left ( as opposed to Militant) believe the policy product rejected in 1983 only needed a better salesman. These people hated Wilson for being a success electorally. The post Thatcher Conservative Party has been plagued with this zeal for ideological purity as well. The EU is to the Conservative right what Nationalisation was to the old Bennite left

    Interestingly on many levels post Brown the Labour Party (thus far) has not repeated these earlier follies of the 1950’s and 1980’s. They are looking seriously for different answers – even if many in the know – who may or may not know better – do not think they are looking in the right place.

    The Tea Party faction in USA acts the same way and believes ideologically in the same sort of thing….that if only they pursue purity more rigorously then all will be well…modern Media with its diverse constituencies of the like-minded tend to reinforce the zeal of the noisy few since they replicate their dissonance.

    It is a very human impulse of those who honestly believe in a cause, political or religious, to think we need to try their solution only harder and none of us are immune to this folly…

    But of course the peoples of Easter Island thought the same and acted in accordance with their belief to disastrous effect…

    Democracy is febrile and shallow but those very qualities restrain the well meaning foolishness of certain beliefs which ever rest upon uncertain argument.

  26. @Reggieside

    Major’s 1992 victory was a remarkable triumph which the tory party seem very reluctant to give him credit for.

    I am the furthest from a Conservative supporter that you could possibly get, yet I agree with you.

    John Major inherited a Conservative party hopelessly divided on Europe. He got 14 million votes, yet ended up with a tiny majority and a band of rebels (Bill Cash et la) that I would not wish on my own enemy.

    I respect John Major for one thing above all – the righting of the wrongs done to working class Harold Larwood, who took the blame for following his Captain’s instructions during the Bodyline Ashes series.

    It seems fitting that probably the most working class PM we have ever had, who suffered at the hands of those who thought he was above his station, understood precisely the class system that damned Larwood at the expense of his ‘Gentleman’ Captain Douglas Jardine.

  27. John Major was a conservative PM who was more popular than his party… Which appears to be the norm now. But he’s also had the benefit of time to mellow memories about him, and never really suffered the same slings and arrows that Thatcher did.

    Should the Conservative Party now repudiate him, it would not only be hanging the noose of high energy prices around their necks, but hanging a second noose of “well, that’s the kind of people poor John had to cope with sniping at him and now they run the party.”

  28. “It is a very human impulse of those who honestly believe in a cause, political or religious, to think we need to try their solution only harder and none of us are immune to this folly…”(John Murphy)

    That’s right, none of us wishes to be the person who cuts down the last tree on Easter Island. I think it is the falling membership of political parties, and the gap between the remaining members and the population at large that has left a rather dangerous vacuum.

    If anyone doubts the dissociation of many from politics, let them try to watch the TV show “Pointless” and discover that quite distinguished politicians have not been recalled in 100 seconds by any of a sample of the population (admittedly only 100 people). Tonight it was the LDs turn, but this been true of Labour and Conservatives as well.

    I have been told that it was WW2 that showed the better-off people of the time just how the other half lived, and this knowledge informed the compassionate Conservatism of politicians like Macmillan. However this genuine but rather condescending concern was dispelled by Margaret Thatcher and this generation has now passed away.

    I seem to remember also that it was quite common at one time to read of Labour MPs, quite often from Wales, dying in their fifties from overwork serving their communities. Is this still the case?

    Perhaps the fixed-term parliament is at least giving this cohort of politicians the chance, if they wish to take it, to consider new policies without the fear of a snap election.

  29. @Turk

    Michael Howard was sufficiently popular with the Tories to be elected leader, but he is remembered for Anne Widdecombe saying ‘There is something of the night about him’.

    Politics is not a fair business. IDS may be popular with Tory activists, but he isn’t with the country at large, and cruel political comment gets remembered if it resonates with the public. We shall see.

  30. “There is no point in telling people to get on their bikes, if there is nowhere to live when they get there.”

    Did John Major really say this in his speech today or is the Graun winding people up?

  31. Has major become a member of the “rational tendency”?

  32. ““There is no point in telling people to get on their bikes, if there is nowhere to live when they get there.””

    Blimey! Wot’s wrong with a bit of camping?

  33. O’ Donnell says: “The education system does not produce the skills that businesses need.”

    Last week Gove, at a Jeb Bush event in Boston, was saying kids were starting work unable to write a business letter.

    So what’s new?

    It took me a bit of time to get accustomed to the ‘with reference to your letter if the 14th Inst/be assured of our best attention/at your earliest convenience/enclosed herein’ way of doing things, it was all part of the induction. (In those days the typist made a world of difference – or could turn your letter writing into a world of pain.)

    Educational institutions reluctantly made space for Business Studies back 1970/80s, but some people are never satisfied.

    Businesses should really be spending a bit of time thinking about the kind of life expirience they will be enmeshing these young people into.

  34. Polling question – “Is the DWP barking mad?”

    http://www.westernmorningnews.co.uk/legged-man-accused-benefit-fraud-officials/story-19682497-detail/story.html

    A Department of Work and Pensions spokeswoman said:”We cannot comment on individual cases.” Then proceeds to do precisely that and make the Dept look even more asinine.

  35. Polling [Scotland] question – “Is INEOS barking mad?”

  36. I love the way Labour supporters are affectionate & nostalgic about Major & Heath.

    It is so sweet.

    I’s a bit akin to Conservative supporters admiring Blair , but without the awful guilt.

  37. Oldnat

    I’ve seen that story ages ago, but it seems there is a new case similar every other day, it seems they are a bunch of accountants that have taken up clowning

  38. Amber

    I have heard no mention of PetroChina’s view on closing the refinery – which they own 50% of.

    Have you heard anything?

  39. Chris Riley

    “Politics is not a fair business. IDS may be popular with Tory activists, but he isn’t with the country at large, and cruel political comment gets remembered if it resonates with the public. We shall see.”

    I would say he’s not popular with Labour activists but there’s no doubt his welfare reforms are popular in the country.

  40. @Catmanjeff

    “I respect John Major for one thing above all – the righting of the wrongs done to working class Harold Larwood, who took the blame for following his Captain’s instructions during the Bodyline Ashes series.
    It seems fitting that probably the most working class PM we have ever had, who suffered at the hands of those who thought he was above his station, understood precisely the class system that damned Larwood at the expense of his ‘Gentleman’ Captain Douglas Jardine.”

    If he could have confined his public endeavours to cricketing matters, then I might have fonder memories of the man, but, as politician, I thought he embodied just about everything that’s been mediocre and uninspired in British politics for the last 40 years or so. An accomplished greasy pole climber, his sole strength appeared to be his inoffensiveness and his ability to be somewhere else when the buck stopped. No political vision that I could discern, he led from the back and just about ran his party into political oblivion. Allegedly a peevish and petty-spirited man in private he is, in my view, nowhere near as nice as he looks.

    And who could ever, ever forgive him for Edwina Currie???

  41. Further to John Ruddy’s remarks about Bradwell, the Wikipedia page neglects to mention that Magnox was found guilty in Chelmsford Crown Court of allowing radioactive material to leak from a faulty sump for 14 years (1990-2004): “We have demonstrated that the company failed to appreciate, properly or at all, that the sump was part of the system for discharging relevant waste, and so failed to take the proper precautions.”

    They do, however, include the funny story about an employee took twenty uranium fuel rods off site in his van intending to sell them as scrap metal. Fortunately local police noticed that the van was not roadworthy.

  42. @Anthony W

    Of course another polling question that is more than a bit dodgy is the oft-asked; “Considering what a complete numpty he is, do you think there is any chance that the great British public might make the appalling error of electing Ed Miliband as Prime Minister? Pleas answer yes or no.

    lol

  43. “there’s no doubt his welfare reforms are popular in the country.”

    Yes there is. Unless what you really mean is that they are popular with a limited percentage of people, in the same way as they are also unpopular with another percentage.

    But that would hardly be worth posting as everything is isn’t it and you can hardly sensibly refer to your percentage as “the country” ‘cos that is about 60 million of us you’re trying to claim.

    Of course, the largest group will never have heard of him.

  44. @ Old Nat

    Unless something has changed, PetroChina own 50% of only the refinery; INEOS own 100% of the petrochemical operations.

  45. Interesting comment about Major’s working-class background. Makes you wonder if he would ever have made it in today’s Tory party.

    His support as party leader and PM initially stemmed from his being not-Thatcher, after 11 years of Mrs T’s abrasive, conviction-style politics, yet ironically it was Thatcher who gave him his chance, maybe because she was fundamentally anti-establishment and hated the hauteur of the ‘born-to-rule’ Tories, so Major’s humble roots would have appealed to her.

    AW: any chance we could have an update on the polling averages? Sorry to importune you, it’s just that there has been so much volatility lately, I wonder if the underlying averages have changed at all. Thanks.

  46. “I love the way Labour supporters are affectionate & nostalgic about Major & Heath.”

    Yes it is lovely isn’t it. In fact I’ve lost count of how many have been saying nice things about JM tonight – can’t remember if it was two or three.

    They’ll be the same about George once he’s out of office as well – the scoundrels.

  47. CROSSBAT11……………………..I suspect the majority of the great British public would answer yes to that question, they love a bit of irony. :-)

  48. Amber

    That was why I specifically referred to the refinery!

    The ethane feedstock for the petro-chemical complex is a different matter (I was watching Prof Kemp on Scotland Tonight :-) )

  49. @ Old Nat

    So you did. :-)

  50. “I love the way Labour supporters are affectionate & nostalgic about Major & Heath.”

    After what Labour supporters went through under Heath’s successor, it’s not surprising they wanted him back!

1 2 3 5