Or at least, questions that should be treated with a large dose of scepticism. My heart always falls when I see questions like those below in polls. I try not to put up posts here just saying “this poll is stupid, ignore it” but sometimes it’s hard. As ever, I’d much rather give people the tools to do it themselves, so here are some types of question that do crop up in polls that you really should handle with care.

I have deliberately not linked to any examples of these in polls, though they are sadly all too easy to find. It’s not an attempt to criticise any polling companies or polls in particular and only he who is without sin should cast the first stone, I’m sure you can find rather shonky questions from all companies and I’m sure I’ve written surveys myself that committed all of the sins below. Note that these are not necessarily misleading questions or biased questions, there is nothing unethical or wrong with asking them, they are just a bit rubbish, and often lead to simplistic or misleading interpretations – particularly when they are asked in isolation, rather than part of a longer poll that properly explores the issues. It’s the difference between a question that you’d downright refuse to run if a client asked for it, and a question that you’d advise a client could probably be asked much better. The point of this article, however (while I hope it will encourage clients not to ask rubbish questions) is to warn you, the reader of research, when a polling question really should be read with caution.

Is the government doing enough of a good thing?

A tricky question. This is obviously trying to gauge a very legitimate and real opinion – the public do often feel that the government hasn’t done enough to sort out a problem or issue. The question works perfectly well it is something where there are two sides to the question and the question is intrinsically one of balance, where you can ask if people think the government has not done enough, or gone too far, or got the balance about right. The problem is when the aim is not contentious, and it really is a question of doing enough – stopping tax evasion, or cutting crime, for example. Very few people are going to think that a government has done too much to tackle tax evasion, that they have pushed crime down too low (“Won’t someone think of the poor tax evaders?”, “A donation of just £2 could buy Fingers McStab a new cosh”), so the question is set up from the beginning to fail. The problems can be alleviated a bit with wording like “Should be doing more” vs “Has done all they reasonably can be expected to do”, but even then you should treat questions like this with some caution.

Would you like the government to give you a pony?*
(*Hat tip to Hopi Sen)

Doesn’t everyone like nice things? There is nothing particularly wrong with questions like this where the issue at question is something controversial and something that people might disagree with. It’s perfectly reasonable to ask whether the government should be spending money on introducing high speed rail, or shooting badgers, or whatever. The difficulty comes when you are asking about something that is generally seen as a universal good – what are essentially valence issues. Should the government spend extra on cutting crime, or educating children, or helping save puppies from drowning? Questions like this are meaningless unless the downside is there as well – “would you like the government to buy you a pony if it meant higher taxes?“, “would you like the government to buy you a pony or could the money be better spent elsewhere?

How important is worthy thing? Or How concerned are you about nasty thing?

Asking if people support or oppose policies, parties or developments is generally pretty straightforward. Measuring the salience of issues is much harder, because you run into problems of social desirability bias and taking issues out of context. For example, in practical terms people most don’t actually do much about third world poverty. Ask people if they care about children starving to death in Africa you’d need a heart of stone to say that you really don’t. The same applies to things that sound very important and worthy, people don’t want to sound ignorant and uninterested by saying they don’t really care much. If you ask about whether people care about, are concerned about or think an issue is important they will invariably say they do care, they are concerned and it is important. The rather more pertinent question that is not always asked is whether it is important when considered alongside all the other important issues of the day. The best way of measuring how important people think an issue is will always be to give them a list of issues and pick out those they consider most important (or better, just give them an empty text box and ask them what issues are important to them).

Will policy X make you more likely to vote for this party?

This is a very common question structure, and probably one I’ve written more rude things about on this site than any other type of question. There are contexts where it can work, so long as it is carefully interpreted and is asked about a factor that is widely acknowledged to be a major driver of voting intention. For example, it’s sometimes used to ask if people would be more or less likely to vote for a party if a different named politician was leader (though questions like that have their own issues).

It becomes much more problematic when it is used as a way of showing an issue is salient. Specifically, it is often used by campaigning groups to try and make out that whatever issue they are campaigning about will have an important impact on votes (and, therefore, MPs should take it seriously or their jobs may be at risk). This is almost always untrue. Voting intention is overwhelmingly driven by big brush themes, party identification, perceptions of the party leaders, perceived competence on the big issues like the economy, health or immigration. It is generally NOT driven by specific policy issues on low salience issues.
However, if you ask people directly about the impact of specific policy issue on a low salience issue, and whether it would make them more or less likely to vote for a party, they will normally claim it does. This is for a number of reasons. One is that you are taking that single issue and giving it false prominence, when it reality it would be overshadowed by big issues like the economy, health or education. The second is that people tend to just use the question as a way of signalling if they like a policy or not, regardless of whether it would actually change their vote. The third is that it takes little account of current voting behaviour – you’ll often find the people saying a policy makes them more likely to vote Tory is made up of people voting Tory anyway, people saying a policy makes them less likely to vote Tory are people who wouldn’t vote Tory if hell froze over.

There are ways to try and get round this problem – in the past YouGov used to offer “Makes no difference – I’d vote party X anyway” and “Makes no difference – I wouldn’t vote for party X” to try and get rid of all those committed voters whose opinion wouldn’t actually change. In some of Lord Ashcroft’s polling he’s given people the options of saying they support a policy and it might change their vote, or that they’d support it but it wouldn’t change their vote. The best way I’ve come up with doing it is to give people a long list of issues that might influence their vote, getting them to tick the top three or four, and only then asking people whether the issue would make them more or less likely to vote for a party (like we did here for gay marriage, for example). This tends to show that many issues have little or no effect on voting intention, which is rather the point.

Should the government stop and think before going ahead with a policy?

This is perhaps the closest I’ve seen to a “when did you stop beating your wife” question in recent polls. Obviously it carries the assumption that the government has not already stopped to consider the implications of policy. Matthew Parris once wrote about a rule of thumb on understanding political rhetoric, saying that if the opposite of a political statement was something that no one could possibly argue for, the statement itself was meaningless fluff. So a politician arguing for better schools is fluff, because no one would ever get up to the podium to argue for worse schools. These sort of questions fall into the same trap – no one would argue the opposite, that the best thing for the government to do is to pass laws willy-nilly without regard for consequences, so people will agree to the statement in regard of almost any subject. It does NOT necessarily indicate opposition to or doubt about the policy in question, just a general preference for sound decision making.

Do you agree with pleasant uncontentious thing, and that therefore we should do slightly controversial thing?

Essentially the problem is one of combining two statements together within an agree disagree statement, and therefore not giving people the chance to agree with one but not the other. For example “Children are the future, and therefore we should spend more on education” – people might well agree that children are the future (in fact, it’s relatively hard not to), but might not agree with the course of action that the question suggests is a natural consequence of this.

How likely are you to do the right thing? or Would you signify your displeasure at a nasty thing?

Our final duff question falls into the problem of social desirability bias. People are more likely to say they’ll do the right thing in a poll than they are to do it in real life. This is entirely as you’d expect. In real life the right thing is sometimes hard to do. It might involve giving money when you really don’t have much to spare, or donating your time to volunteer, or inconveniencing yourself by boycotting something convenient or cheap in favour of something ethical. Answering a poll isn’t like that, you just have to tick the box saying that you would probably do the right thing. Easy as pie. Any poll you see where you see loads of people saying they’d volunteer to do something worthwhile, boycott something else, or give money to something worthy, take with a pinch of salt.


243 Responses to “Questions that should be ignored”

1 2 3 4 5
  1. And I thought the most interesting bit was him saying that 95% of money is created by the commercial banks, yet another person saying straight out that money isn’t what we think it is

  2. But yes, the mega banks need to be broken up

  3. Amber

    Step 5 made me smile as well

  4. Rod Stewart recorded the song Sailing in Sheffield, Alabama, for his 1975 album Atlantic Crossing, and it was subsequently a number 1 hit in the UK in September 1975.

    Ah I remember the prime minister Harold Wilson sailing his yacht at the time.

  5. @ANN IN WALES

    Loved the underpants comment!

  6. @ RiN

    And I thought the most interesting bit was him saying that 95% of money is created by the commercial banks, yet another person saying straight out that money isn’t what we think it is.
    —————–
    Yes, indeed. His suggestion for how QE should work was scary in a UK context. He makes it clear that ‘his’ QE would simply have been a huge ‘bung’ to a few large banking corporations which would not have given loans or working capital funding to producers & makers in the way he envisaged QE supported banks would do.

  7. Grangemouth to close – that’s a really very serious blow to the Scottish economy, and very badly timed for the Yes campaign.

    I have no detailed knowledge of the refinery market, other than the fact that there is significant over capacity in Europe, so I’ve no idea whether a buyer can be found, but I expect that all manner of measures will be promised by Edinburgh to ensure the best chance of a buyer.

    Even if a buyer is found, I would imagine that this could lead to some significant concerns in the Yes camp about impressions of vulnerability on the world economic stage.

    ‘Yes’ is all about self confidence, as Salmond said, but within a week of saying this, if 2% of your economy is about to close because it isn’t making money, that has to dent that confidence.

  8. Alec

    Or it highlights the dangers of foreign ownership! Is there anything that an independent Scotland could do that can’t be done at the moment?

  9. @Alec

    I have looked at approval ratings for 2013.

    I plotted a CUSUM chart, as this identifies well the trends in data.

    The don’t knows were very stable and consistent.

    The approval percentage was consistently below the mean for 2013 until early July, when it has performed above the mean since.

    The disapproval shows the reverse at the same time.

    So approval has improved well since early July. If I get the chance I will post a link tonight.

  10. @ Alec

    It’s not the Grangemouth refinery which is under threat, it is the petrochemical plant. INEOS own that plant 100%; they only own 50% of the refinery.

  11. @Alec

    Talk is that there was poor management. This is not an industry that is currently doing badly, and I suspect that at least part of the story here is that substandard British management has struck yet again.

    It really is the issue nobody wants to talk about. We have terrible managers in this country, and that’s the top and bottom of an awful lot of troubles in the last few decades.

  12. I am skeptical that the closure of the PetroChemical part of grangemouth will have on the scottish economy – as opposed to the independence campaign.

    The root cause of the problem is that this plant is inefficient and suffering from competition from elsewhere. If that is the case, then the companies that currently use the output from Grangemouth should be able to switch suppliers fairly easily and possibly save money.

    There is the loss of jobs, of course, but is this any worse than other similar closure, eg the meat processing plant that closed a little while back?

  13. I’m reading that the plant has very high debts at high interest rates, are those debts from when it was bought or as a result of speculative investments which went wrong? Do any of our Scottish friends know?

  14. I dont think its something that anyone except those intimately involved with the management knows for certain Richard.

    Now Ineos is a new company that was specifically set up to buy out another plant. It has since expanded rapidly, taking on plants that existing owners were keen to dispose of (Grangemouth used to be a BP plant). They could quite easily have accumulated a lot of debts in order to finance these acquisitions.

    They are based in Switzerland.

  15. @Amber

    If the plant goes, the refinery probably goes with it.

    @RiN

    “Is there anything that an independent Scotland could do that can’t be done at the moment?”

    Voting in General Elections will produce the government that Scotland votes for. What else can it do? A list of ideas spring to mind, and it generally boils down to where the tax revenue and other monies from Scotland is spent (e.g. BBC license fees).

    @Alec

    From what I’ve been reading, most people are blaming Unite for not appreciating the seriousness of the other party’s resolve. It looks like an own goal for the unions, given that it’s a major ‘union v private corporation’. The latter has questions to answer over its decision to shift its accounting to Switzerland (tax avoidance).

    Either way, hundreds of people now face Christmas with no job, and the North of Britain faces a potential fuel crisis, if things get out of hand. Personally, I don’t see how the Scottish or UK governments could have influenced the two parties for the better.

  16. Never mind, he has and my first comment was moderated anyway.

  17. @Statgeek

    No the Oil refinery is perfectly fine as a stand alone plant. The Chinese have recently bought 50% of it, and it makes money.

    They’ve only closed the loss-making bit that they owned 100% of.

  18. @JR

    I’m getting conflicting reports:

    http://www.bbc.co.uk/news/uk-scotland-24631342

    “Ineos chairman and founder Jim Ratcliffe had said at the weekend that if the petrochemical plant closed it was likely the refinery would go as well.”

    h ttp://uk.reuters.com/article/2013/10/23/uk-scotland-refinery-idUKBRE99M08M20131023

    “Grangemouth refinery to reopen, chemical plant to close”

    Live Parliament coverage is talking about it as I type.

  19. I’m getting the impression that this is all about financial engineering and tax writeoffs

  20. @Statgeek

    The margins might be slim, but the oil refinery would find ready buyers. The petrochemical plant will not.

    Basically the Unions (and to a lesser extent the Edinburgh and London Governments) called the company’s bluff. The company showed they wernt bluffing.

  21. As I understand it the petro-chemical plant is viable if they switch to imported ehylene from USA-which was why they proposed the investment in the new port facility. But they wanted the payroll cost deal from the union .

    There is a pattern here on petro-chemical . BASF have recently been making noises about leaving GErmany because of energy costs there.

    The massive reduction in US energy costs is throwing ripples out across industries which are heavy energy users. Whilst EU countries pursue the green agenda & turn their face against nuclear & shale -we will see more of this.

    This report is symptomatic :-

    “French state-backed energy giant has thrown its weight behind Britain’s shale gas rush after agreeing to help fund a new drilling programme in Lancashire and Wales.
    GDF Suez has formed a joint venture with Dart Energy to drill up to 14 wells exploring for both coal bed methane (CBM) — gas strapped in underground coal seams — and for shale gas. It is due to start at the end of the year.”

    The Times.

    France has banned fracking-GDF is part owned by the French State !!

  22. Well DC has put the cat among the pigeons.

    Presumably determined to shoot Miliband’s fox & actually reduce energy prices, he has committed to reviewing “Green Taxes” in the Autumn Statement.

    No doubt the LDs will have something to say about this-NC was not nodding!

    Exciting times.

  23. THe ECB creating waves.It has outlined its planned health checks of eurozone banks ahead of taking over responsibility for supervising the top 130 EZ lenders in a year.


    As part of the ECB’s assessment, the central bank will conduct an asset quality review of bank balance sheets, leading to a recommendation in a year’s time that may involve demanding some banks bolster their capital reserves.
    “We expect that this assessment will strengthen private sector confidence in the soundness of the euro area and in the quality of bank balance sheets,” Mario Draghi, ECB president, said in a statement.

    The balance sheet tests will affect “all asset classes, including non-performing loans, restructured loans and sovereign exposures”, it said. The asset quality review, a risk assessment of each bank will feed into an EU-wide stress test to be conducted by the European Banking Authority. The exercise will be concluded by November next year, when the ECB will also publicly announce which banks need more capital”

    At the heart of this is an ECB imposed set of criteria for impaired assets which cuts across the myriad different accounting approaches of EZ Banks.

    Maybe we are approaching the point of some stability in the EU Banking sector.

    Bank shares across EZ fell this morning.

  24. Colin

    It looks like I was right in my report from the front line the other week […] that the Tories will be pressing ahead with cutting some of the green taxes from the energy mix, however I think it’s unlikely it will be something the LD’s will agree to, but stranger things have happened.

    As I said recently cutting part of the green part of the tax is a good counter to Labours freeze as it delivers a actual cut rather than freeze on existing prices and will get DC off the back foot over energy bills.

  25. ‘Nick Clegg will block Conservative plans to cut energy bills through big reductions in green taxes before the 2015 general election.

    The Deputy Prime Minister has agreed to a government review of the green energy measures the Tories blame for rising gas and electricity bills. But he will not allow them to cut subsidies to relieve fuel poverty, encourage householders to insulate their homes or boost renewable energy like wind power’

    and

    ‘Lib Dems believe their Tory coalition partners are raising “false hopes” that green taxes can be reduced’

    Independent Monday 21st October 2013

    But

    ‘You can do that through taxation or through bills but there is no free lunch’

    I think government will announce in the Autumn statement that they will subsidise the ECO directly and so remove them from energy bills.

  26. If they cut the green taxes and stuff, will the savings automatically get passed on to customers, or can energy companies just pocket it and keep prices the same?

  27. @Carfrew
    Theres nothing stopping them doing that – the current regulator doesnt have the power to stop them, which is why Labour’s price freeze includes replacement of the regulator, giving them much tougher powers.

  28. Request: can the polling average and latest polls list please be updated? The last update was about two weeks ago.

  29. “If they cut the green taxes and stuff, will the savings automatically get passed on to customers, or can energy companies just pocket it and keep prices the same?”

    I think [you all] should read and reread this contribution from @Carfrew.

    With the water industry, governments could lift or amend targets for investment, and simultaneously peg prices through OFWAT. There is no power whatsoever to do this with energy.

    You can switch green taxes to the consumer if you like, but there will be a strong element of profit taking within the industry and the taxpayer will end up paying twice for many of the green measures – once through energy bills, now going to the pockets of the energy companies, and once through taxation.

    Such a move in isolation would be stupid, […]

  30. If you could add just one topic to the “most important issues” in YG list, which one would you choose?

    I can think of several issues which I would have thought would be obvious ones to ask but which don’t appear for some reason. IMHO of course.

  31. @Turk
    Re: the politics of sailing
    Next time you are walking near a lake, you might notice that some boats have red and some blue sails. The red sailed ones are Mirrors – promoted in the early 60s by the Daily Mirror. The Blue ones are the Enterprise class – promoted by the News Chronicle in the late 50s (actually a British Liberal Party supporting paper which was subsequently bought by the Daily Mail in 1960 – according to Wikipedia).

    See both Wilson and Heath were men of the people.

  32. @Alec
    I was about to comment that that last sentence of yours might not last long, when I noticed that it hadn’t.

    Regarding the water industry, they do seem to offer the prospect of a second front. on energy prices. What is there to stop a government legislating to keep some limited targets for investment in place and still peg prices? Water company costs have suffered non of the squeeze applied to either the public or to a lesser extent private sectors and profits remain high, whilst bills have steadily increased by above inflation by tame regulators.

    To draw a parallel, councils have had to live with a freeze in council tax whilst their grant funding has been slashed. Were we still in the days of domestic rates, more questions might be being asked as to why one set of (domestic) rates had been frozen but the other set of (water) rates had not.

  33. @Colin

    “Exciting times.”

    Or you could say that the Leader of the Opposition is setting the political agenda. Miliband might be able to sit on his hands, watch the coalition partners fight like a couple of weasels in a sack and then claim, whatever fudge they eventually come up with, that they wouldn’t have moved without his initiative on price freezing. Then, when he’s enjoyed the luxury of watching the Government play their hand, he can then deal his next deck of cards. It’s one of the few delights of opposition!

    Just another way of looking at things.

  34. I see the Lib Dems are now accusing Cameron of a “panicky U-turn” following his suggestion at today’s PMQs that he’s going to review green energy taxes.

    It’s started already!

  35. Colin, I don’t see how this will shot ED’s fox he will just go ahead and freeze prices, It’s not green tax he attacking but the price rigging by the big 6.

  36. crossbat11

    I love the (weasels) bit very apt.

  37. @CB11

    “It’s started already!”

    What has? The reduction in prices? :))

  38. A recent BBC Poll carried a number of questions about whether the complaints procedure ‘met your expectations’ The problem with these questions is that they cannot possibly know what those original expectations are!

    If the respondent expects poor service and they are met, then then their expectations have been fulfilled ! In the same was as an expectation of good service is.

    Question writing is a science and it’s easy to get it wrong.

  39. This might not have been commented on before…but according to the Grauniad the “Government plans to use individual voter register for 2015 election…Plans to replace household voter register boosted by experiment suggesting most of electorate can be transferred automatically”

    Here’s the link but it might end up blocked..

    w ww.theguardian.com/politics/2013/oct/23/government-individual-voter-register-2015-election

  40. “Question writing is a science and it’s easy to get it wrong.”

    ———-

    It may suit them to get it wrong though…

  41. @ RogerRebel

    We’ve just fixed our energy until 2017, anyone can do it. I wonder if DC or EM or respective spouses have bothered, hope an interviewer asks the question.

  42. @John Ruddy and Alec

    It’s ok guys, everything’s gonna be OK, ‘cos apparently they’re gonna have… A competition test!! They are going to look into it, it’ll be great. Like a review or something. Clearly someone’s been watching “Yes, Minister”…

  43. Listening to the radio, and listeners testing in do not seem to be too impressed with the move on motoring costs. MoTs themselves aren’t that pricey, it’s the work that costs, and garages can just up prices on what needs fixing to compensate. Motorway petrol stations are usually a last resort, etc.

    One listener made the observation that the bigger costs – fuel and vehicle duty etc. – are in government control and they could just cut those…

    Keen to see the polling on all this…

  44. The Home Affairs Committee on the ‘plebgate’ scandal is very interesting. Police closing ranks. Politicians closing ranks.

    Us real plebs get so little entertainment these days.

  45. @CROSSBAT11

    “I see the Lib Dems are now accusing Cameron of a “panicky U-turn” following his suggestion at today’s PMQs that he’s going to review green energy taxes.”

    ——–

    This is a hidden benefit of Miliband delaying his policy announcements. If he’d announced his energy thing sooner, then government might have responded sooner, and claimed it was always a concern. As it is, they’ve left it for over three years, so not only does it seem like a u-turn, but it suggests that they weren’t really bothered in the first place.

    Pluswhich, as the LD response suggests, it’s a wedge issue of sorts, pitting coalition partners against each other. Then again, LDs may try the “Greece gambit” again and hope no one notices they’ve done a u-turn themselves…

  46. Laying

  47. John Ruddy

    The petrochemical plant will be under threat as there is a reason why the two are together. The economies of regenerating the waste from the refinery as ethylene and other petrochemicals has helped make this viable.

    The petroleum industry is run on very lean continuous processes that look at recycling, efficient energy use and waste minimisation to make it viable. By removing one of these then it makes the outlook shaky.

    A very poor show all round

  48. Human Nature?

    My wife heard something today in the Hairdressers which she, and when she told me, I found profoundly depressing.

    A customer was waxing lyrical about health tourists, benefit scroungers and the usual……The girl doin her hair countered timidly with “Yes, but isn’t it dreadful all these families having to use foodbanks because they can’t make ends meet?” The response from the customer was ” I don’t really believe it – 90% of them are probably just scrounging free food”

    I wonder which newspaper that lady reads?

    I’m sure her place in heaven is being made ready for her?

  49. TURK

    Thanks-yes your inside info is clearly spot on.

    Can’t wait to see what emerges.

  50. CB11

    @”Just another way of looking at things.”

    It is indeed-and I accept that EM has set this ball in motion.

    Let’s see who gets it into the net .

1 2 3 4 5