Or at least, questions that should be treated with a large dose of scepticism. My heart always falls when I see questions like those below in polls. I try not to put up posts here just saying “this poll is stupid, ignore it” but sometimes it’s hard. As ever, I’d much rather give people the tools to do it themselves, so here are some types of question that do crop up in polls that you really should handle with care.

I have deliberately not linked to any examples of these in polls, though they are sadly all too easy to find. It’s not an attempt to criticise any polling companies or polls in particular and only he who is without sin should cast the first stone, I’m sure you can find rather shonky questions from all companies and I’m sure I’ve written surveys myself that committed all of the sins below. Note that these are not necessarily misleading questions or biased questions, there is nothing unethical or wrong with asking them, they are just a bit rubbish, and often lead to simplistic or misleading interpretations – particularly when they are asked in isolation, rather than part of a longer poll that properly explores the issues. It’s the difference between a question that you’d downright refuse to run if a client asked for it, and a question that you’d advise a client could probably be asked much better. The point of this article, however (while I hope it will encourage clients not to ask rubbish questions) is to warn you, the reader of research, when a polling question really should be read with caution.

Is the government doing enough of a good thing?

A tricky question. This is obviously trying to gauge a very legitimate and real opinion – the public do often feel that the government hasn’t done enough to sort out a problem or issue. The question works perfectly well it is something where there are two sides to the question and the question is intrinsically one of balance, where you can ask if people think the government has not done enough, or gone too far, or got the balance about right. The problem is when the aim is not contentious, and it really is a question of doing enough – stopping tax evasion, or cutting crime, for example. Very few people are going to think that a government has done too much to tackle tax evasion, that they have pushed crime down too low (“Won’t someone think of the poor tax evaders?”, “A donation of just £2 could buy Fingers McStab a new cosh”), so the question is set up from the beginning to fail. The problems can be alleviated a bit with wording like “Should be doing more” vs “Has done all they reasonably can be expected to do”, but even then you should treat questions like this with some caution.

Would you like the government to give you a pony?*
(*Hat tip to Hopi Sen)

Doesn’t everyone like nice things? There is nothing particularly wrong with questions like this where the issue at question is something controversial and something that people might disagree with. It’s perfectly reasonable to ask whether the government should be spending money on introducing high speed rail, or shooting badgers, or whatever. The difficulty comes when you are asking about something that is generally seen as a universal good – what are essentially valence issues. Should the government spend extra on cutting crime, or educating children, or helping save puppies from drowning? Questions like this are meaningless unless the downside is there as well – “would you like the government to buy you a pony if it meant higher taxes?“, “would you like the government to buy you a pony or could the money be better spent elsewhere?

How important is worthy thing? Or How concerned are you about nasty thing?

Asking if people support or oppose policies, parties or developments is generally pretty straightforward. Measuring the salience of issues is much harder, because you run into problems of social desirability bias and taking issues out of context. For example, in practical terms people most don’t actually do much about third world poverty. Ask people if they care about children starving to death in Africa you’d need a heart of stone to say that you really don’t. The same applies to things that sound very important and worthy, people don’t want to sound ignorant and uninterested by saying they don’t really care much. If you ask about whether people care about, are concerned about or think an issue is important they will invariably say they do care, they are concerned and it is important. The rather more pertinent question that is not always asked is whether it is important when considered alongside all the other important issues of the day. The best way of measuring how important people think an issue is will always be to give them a list of issues and pick out those they consider most important (or better, just give them an empty text box and ask them what issues are important to them).

Will policy X make you more likely to vote for this party?

This is a very common question structure, and probably one I’ve written more rude things about on this site than any other type of question. There are contexts where it can work, so long as it is carefully interpreted and is asked about a factor that is widely acknowledged to be a major driver of voting intention. For example, it’s sometimes used to ask if people would be more or less likely to vote for a party if a different named politician was leader (though questions like that have their own issues).

It becomes much more problematic when it is used as a way of showing an issue is salient. Specifically, it is often used by campaigning groups to try and make out that whatever issue they are campaigning about will have an important impact on votes (and, therefore, MPs should take it seriously or their jobs may be at risk). This is almost always untrue. Voting intention is overwhelmingly driven by big brush themes, party identification, perceptions of the party leaders, perceived competence on the big issues like the economy, health or immigration. It is generally NOT driven by specific policy issues on low salience issues.
However, if you ask people directly about the impact of specific policy issue on a low salience issue, and whether it would make them more or less likely to vote for a party, they will normally claim it does. This is for a number of reasons. One is that you are taking that single issue and giving it false prominence, when it reality it would be overshadowed by big issues like the economy, health or education. The second is that people tend to just use the question as a way of signalling if they like a policy or not, regardless of whether it would actually change their vote. The third is that it takes little account of current voting behaviour – you’ll often find the people saying a policy makes them more likely to vote Tory is made up of people voting Tory anyway, people saying a policy makes them less likely to vote Tory are people who wouldn’t vote Tory if hell froze over.

There are ways to try and get round this problem – in the past YouGov used to offer “Makes no difference – I’d vote party X anyway” and “Makes no difference – I wouldn’t vote for party X” to try and get rid of all those committed voters whose opinion wouldn’t actually change. In some of Lord Ashcroft’s polling he’s given people the options of saying they support a policy and it might change their vote, or that they’d support it but it wouldn’t change their vote. The best way I’ve come up with doing it is to give people a long list of issues that might influence their vote, getting them to tick the top three or four, and only then asking people whether the issue would make them more or less likely to vote for a party (like we did here for gay marriage, for example). This tends to show that many issues have little or no effect on voting intention, which is rather the point.

Should the government stop and think before going ahead with a policy?

This is perhaps the closest I’ve seen to a “when did you stop beating your wife” question in recent polls. Obviously it carries the assumption that the government has not already stopped to consider the implications of policy. Matthew Parris once wrote about a rule of thumb on understanding political rhetoric, saying that if the opposite of a political statement was something that no one could possibly argue for, the statement itself was meaningless fluff. So a politician arguing for better schools is fluff, because no one would ever get up to the podium to argue for worse schools. These sort of questions fall into the same trap – no one would argue the opposite, that the best thing for the government to do is to pass laws willy-nilly without regard for consequences, so people will agree to the statement in regard of almost any subject. It does NOT necessarily indicate opposition to or doubt about the policy in question, just a general preference for sound decision making.

Do you agree with pleasant uncontentious thing, and that therefore we should do slightly controversial thing?

Essentially the problem is one of combining two statements together within an agree disagree statement, and therefore not giving people the chance to agree with one but not the other. For example “Children are the future, and therefore we should spend more on education” – people might well agree that children are the future (in fact, it’s relatively hard not to), but might not agree with the course of action that the question suggests is a natural consequence of this.

How likely are you to do the right thing? or Would you signify your displeasure at a nasty thing?

Our final duff question falls into the problem of social desirability bias. People are more likely to say they’ll do the right thing in a poll than they are to do it in real life. This is entirely as you’d expect. In real life the right thing is sometimes hard to do. It might involve giving money when you really don’t have much to spare, or donating your time to volunteer, or inconveniencing yourself by boycotting something convenient or cheap in favour of something ethical. Answering a poll isn’t like that, you just have to tick the box saying that you would probably do the right thing. Easy as pie. Any poll you see where you see loads of people saying they’d volunteer to do something worthwhile, boycott something else, or give money to something worthy, take with a pinch of salt.


243 Responses to “Questions that should be ignored”

1 2 3 4 5
  1. @Tony Dean

    Don’t get too down. YouGov had a poll out the other day showing 53% saying at least 4 out of 5 people in poverty are NOT there because of fecklessness. The number saying the opposite was tiny.

    Intellectual pondlife like the silly woman you describe are not typical, they just make the mistake of thinking they’re typical and as such they make the most noise. A bit like the archetypcal racist taxi drivers.

  2. Vaz is a seriously class act.

    This afternoon was riveting & he controlled his team & the agenda with consumate ease, wielding the scalpel with devastating effect.

    The three “Feds” were a revelation. Even David Winnick asked them if they thought they had destroyed their anti-cuts campaign by trying to “hook up ” * Mitchell to it.

    * A phrase which one of the three actually used !

  3. Further to what I just posted, the rebuttal to any idiot who says food ban users are just shnorrers after a free meal is to point out that people can’t just turn up at a food bank, they have to get a reference from a qualified professional who verifies that it’s an absolute last resort.

  4. Any chance of getting my 5.29pm comment out of automod? Thanks muchly…

  5. not a good day for Scottish chemicals – BASF is closing its pigments plant at Paisley. Perhaps ironic that the BBC had a news item, today that chemicals (together with refined petrol) was the fastest-growing part of the Scottish
    economy.

  6. Tony,My experience of things overheard in the hairdressers,is that empty vessels make the most sound.

  7. Thank you for demodding my post.

  8. A little late on the energy announcement earlier.

    However he’re a non-partisan article on how “green” taxes are comprised. http://www.theguardian.com/money/2013/oct/23/green-energy-levies-how-much

  9. “A little late on the energy announcement earlier”

    ———–

    Somehow I don’t think the energy thing is over yet. I mean, we haven’t had a polling question on the competition test yet. I’m quite looking forward to that…

  10. @BANTAMS
    We’ve just fixed our energy until 2017, anyone can do it. I wonder if DC or EM or respective spouses have bothered, hope an interviewer asks the question.
    ——————————–
    You didn’t wait for Ed?
    Your energy company must be con men! ;-)

  11. @Bantams

    I did the same yesterday. But that isn’t a price freeze, as the unit costs are about 10% higer than now. So it’s “pay about 10% more than you’re paying now for the next 4 years.” Of course after the, almost inevitable price increase later this week (prob 10%), and assuming further 10% rises for the following 3 years (on standard rate), then it will effectively be a price freeze for years 2-4.

  12. I’ve come late to this thread, but very good analysis by Anthony of the types of poor questions in polls and the difficulty of avoiding them.

  13. WOLF

    THat BASF move puts the INEOS plant closure in it’s right context:-


    BASF SE , the world’s largest chemical maker, will shutter a pigment production plant in Scotland while adding capacity in Asia as it seeks to boost profitability at its performance products division.
    The measures are part of plans to cut 650 jobs in the pigments unit globally by 2017, Ludwigshafen, Germany-based BASF said in a statement. The company will meanwhile invest 250 million euros ($344 million) in expanding capacity in Nanjing, China and Ulsan, South Korea, as well as in research.
    “The measures we are undertaking will make us more responsive to market and customer needs,” Markus Kramer, the head of BASF’s dispersions and pigments operations, said in the statement. “The future global production network will enable us to reliably and efficiently supply our partners from a competitive base.”
    The cuts add to the scrapping of 500 positions in the plastic-additives and pigment operations around Basel, Switzerland announced in April. BASF is joining rivals Saudi Basic Industries Corp. (SABIC) and Clariant AG (CLN) in making cutbacks as the advent of low-cost producers in Asia and a renaissance in the U.S.’s commodity chemical industry on the back of shale gas changes the dynamics of the marketplace.
    The closure of the pigments plant in Paisley, Scotland will see 143 employees lose their jobs, while a further 140 employees will leave the manufacturing site in Huningue, France. BASF may also sell or close its Maastricht factory in the Netherlands.”

    Reuters.

  14. If anyone wants to see the Approval ratings for 2013 polls, here is a link to the related CUSUM chart:

    https://docs.google.com/file/d/0BzTTW1ecy-NDbDJZNVExLU14Zzg/edit?usp=sharing

    The data shows that the 17th of July was when approval began to increase and disapproval decrease.

    Don’t knows haven’t really changed much.

  15. Colin

    You are comparing apples with pears….

  16. BCROMBIE

    Don’t think so.

    INEOS planned to convert the petro-chemicals site for US ethylene & build a new port facility.

    Same problem being addressed by BASF. If Germany isn’t mindful of their energy costs it is not beyond possibility that BASF will start moving from their too.

    US shale has altered the dynamics of global competitiveness for petro-chemical manufacture-and indeed other heavy energy using sectors.

    Good news for the US economy.

  17. COLIN

    Not a “new port facility”, but a storage facility.

    Not “ethylene” – that’s what is produced from the ethane they intended to import from the USA.

    Ethane importing is required because much of the gas from Forties is no longer ethane, but butane and propane.

    However, you are right that petro chemical manufacture and refining works in a global market place.

  18. Colin

    Far too many generalisations – the petrochemical and pigments industries, as well as pharma and fine chems are very different indeed.

    I find the joy with which people find moving technically advanced industries moving out of Europe very sad – where it will all end I do not know as corporations chase the lowest price.

    Moving from democratic countries with decent health,safety and environmental laws to undemocratic countries where lives and the environment are treated with a certain disdain is surely not to the benefit of us in the UK – we may have slightly cheaper plastic goods but not many industries

  19. Being reported that Unite has capitulated and “willing to recommend to members: give in to demands from Grangemouth owner Ineos, with no strike pledge”.

    Whether this is too late, or just Ineos seeing an opportunity to demand even more when they say that they are “sceptical” about the Unite offer.

  20. @ Old Nat

    Unite have been saying since last week that there needed to be negotiations & they would agree a ‘no strikes’ deal until the end of the year.

  21. @Tony Dean,

    I know a police office may not be the most compassionate of places, but the almost universal feeling where I work is that if you offer people free stuff they will take it.

    Most people put the increase in food bank uptake down not to increased poverty, but to increased awareness of their existence.

    Personally I think it’s a bit of both.

  22. Amber

    All reasonable people have been saying that there need to be negotiations.

    INEOS have not been reasonable, but it looks like Unite totally misunderstood their strength (or lack of it).
    However, Unite’s “last ditch” proposals seem to give INEOS all they wanted. My point was “will INEOS demand even more?”

    We are in the world that the neo-liberals created. Countries and Unions need to be much cleverer in responding to the power of what the site manager referred to as “the shareholder” – a term normally used in the plural!

  23. According to the Trussell Trust, every ‘client’ produces a foodbank voucher before receiving anything:

    “Care professionals such as doctors, health visitors, social workers, CAB and police identify people in crisis and issue them with a foodbank voucher.”

  24. Tomorrow’s prediction:

    Con 33%
    Lab 39%
    Lib 9%
    UKIP 12%

  25. No tweet from the S*n, so probably a status quo kind of result.

  26. Prediction with extra decimals:

    Con 33.5%
    Lab 39.2%
    LD 9.4%

  27. Thu October 24, 6 a.m. BST

    Latest YouGov / The Sun results 23rd October – Con 32%, Lab 39%, LD 9%, UKIP 11%; APP -30

  28. Five poll rolling average:

    Con 32.8
    Lab 39.2
    LD 9.4
    UKIP 11.8

  29. Are we seeing a trend developing

  30. Looks like Milliband has outsmarted Cameron/Crosby again – can clearly see from recent polls that Milliband has player a blinder on energy prices and Cameron just looks panicky and reactive.

  31. Approval continues to decline also

  32. OLDNAT

    Thank you for those corrections, to which I plead guilt with embarrassment.

    Excuse-more interest in the principle than the detail & superficial assimilation of news bulletins !

    BCROMBIE

    @”we may have slightly cheaper plastic goods but not many industries”

    But it is “we” who are choosing.

    These products go into intermediate products & so on down the production chain to something which an end user buys.

    And very few of the buyers in that chain will be looking beyond quality & price when they decide who to buy from.

    How often do you?

  33. On Food Banks – I suspect there is bound to be an element of push and pull for these. Wherever there are benefits systems, there will be people who play them – we need to decide if the wastage on these people is a price worth paying to help the genuinely in need. People with both viewpoints I think need to accept an element of truth in the contrary view.

    Every market intervention affects the markets to some extent. In my own field, I’ve seen how renewables subsidies affect the capital cost of renewables. Over the last eighteen months we’ve seen an absolute crash in the price of PV modules, as a response to the sharp cut in FiTs, so that they are now cost effective again. The implication of this is that the industry was overcharging because of subsidy.

    We see this in pensions, in farming – pretty much anywhere where there is a guaranteed state backed income.

    With food banks they are clearly trying to limit this with the deferrals system, so I tend to think the balance on these is more push than pull, but overall I think we should all ask ourselves why the food banks started in the first place. If there was no problem out there, this wouldn’t have happened, so regardless of how deserving some of the recipients are, I think we can conclude we have a problems, which is not good for the 6th richest nation on earth.

  34. Certainly looks as though Labour have stabilised, and to some extent reversed a downward drift in their lead.

  35. Colin
    “I accept that EM has set this ball in motion.
    Let’s see who gets it into the net .”
    We don’t often agree, Colin, but this is the most important part.

    If Cameron can abolish green taxes on energy bills (..and I think that the LibDems will fold on this, otherwise it makes a Labour majority more likely), I’m not sure many people, outside of those interested in politics, will remember who set the ball in motion. They’ll remember their energy bills falling by 9%.

    Of course, we’ll have to wait and see.

  36. TINGED

    Thanks

    Conversely if he produces a lemon , after raising expectation, he will lose all credibility on the issue.

  37. I think the UK has got a real problem with the cost of energy to consumers, but hey ho market forces will drive prices down…
    we were told it would… I am still waiting for that to happen

  38. TINGED

    “If Cameron can abolish green taxes on energy bills (..and I think that the LibDems will fold on this.”

    I can’t see the LibDems “folding” on this. That would just finish them off in the eyes of their left-leaning supporters and former supporters who have deserted to Labour. They would have no chance of retaining them or winning them back if they gave in on this. It would be a disaster for them.

  39. From what posters are suggesting, we have to assume people are prepared to go to a great deal of effort to pull the wool over care workers eyes.

    Having dishonestly obtained a food voucher, they then track down a foodbank, and walk away with three days of emergency food. This emergency food comprises a box of very basic in-date tinned or packet foodstuffs chosen by someone else, from a store cupboard in a garden shed or garage somewhere.

    I’ve had a look in a foodbank store cupboard at a friend’s house and there wasn’t anything in there that I would choose to eat… though if I hadn’t eaten properly for days or weeks I might feel differently about it.

    I’m suggesting anyone with as much as ten quid in their pocket will be straight round to Morrisons rather than going anywhere near a foodbank, but what do I know? Perhaps the Daily Mail will find evidence of people with food parcels doing dodgy deals in pub carparks all over the country.

  40. There are very few topics that are absolutely banned here – in fact, other than things where there is a risk of libellous comments or contempt of court, there are only two. One is whether the BNP and similar parties are left-wing or right-wing, as it only ever gets into a partisan back and forth. The other, for the same reason, is which party leader won PMQs (or had a good or bad PMQs or whatever). Most people here should know that (hell, I even had some comments yesterday that started “I know we’re not allowed to discuss this…” why waste your time typing it then?)

  41. Now the LibDems are backing Cameron on removing Green Taxes. What are they up to? Green is one of the main associations with the Lib Dems if they are ditching this as well I don’t know what to make of them anymore. I can’t imagine this will improve their poll position if anything it might be the final straw for those that are left. Why are they so complacent regarding their forthcoming electoral decimation?

    I think the problem of removing the Green Taxes is that it does not tackle the profiteering of the energy companies rather moves the burden to the tax payer or the poor who will no longer get grants to insulate their homes. So again DC can be potrayed as giving into the powerful while penalising the poor.

  42. Once again many reading perhaps to much into a few polls forgetting AW and he’s look at the overall trend mantra. When the polls shift by 1% 0r 2% it means very little, unless of course it stays like that which is unlikely, it’s more likely the consequence of almost daily polls moving round the country.

    Today Clegg seems to be offering some tentative support for a change in green tax levy, although he said he would not support scrapping green taxes, he did suggest they could be raised through general taxation.

    If DC plays this right he can make some political capital out of this with the warm winter allowance of £135 pounds for up to 2 million poorer energy users, and a general cut in energy bills for everybody by removing those uses of green taxes into general taxation, it could well make EM’s energy freeze look pretty small beer .

  43. COUPER2802

    @”the profiteering of the energy companies”

    I am interested in this assertion . It is clearly at the heart of EM’s stance.

    But assuming we accept that any company needs to earn enough profit to pay the cost of servicing it’s sources of capital, plus replacing it’s operating assets, plus investing in new technology; simply in order to stay in business; then we must define “profiteering”.

    Could you tell me what return on Capital would be acceptable to you as commercially prudent, but not rate as “profiteering” for an Electricity Generator/Retailer.

    And which of the Big Six do you consider to be exceeding your criterion.

    Thanks

  44. TURK

    A good package if he can pull it off.

  45. @COLIN

    That is why EM has been clever\lucky.

    If there was a polling question:
    “Do you think energy companies profiteer? ” I imagine that you would get about 80+% yes.

    John Major’s intervention regarding a windfall tax and DC response on green taxes have let that assertion go by default.

  46. @Turk

    I agree, I think they are going to remove the ECO, which helps pay for insulation for poorer families, from energy bills and they are going to pay for it through general taxation

    ‘…..insulating people’s homes, helping the fuel-poor, supporting our green economy for less, of course, I don’t want to see an extra penny going on people’s bills unless it’s absolutely necessary. That’s what we will do, as we always do in the coalition – whatever our differences, we resolve them.’

    Nick Clegg 24 Oct

    That will cut £47 average from bills, although i think bills are going up more than that, so it will be a cut in the increase.

    Also the money will come from the government, so it will increase government spending unless they will have to cut somewhere else, £1 billion +, I think.

    This change is now likely to happen, Lab will know about it, so how will they respond. Ed M is suppose to be quite good at these long term tactics. Let us see what he does.

  47. Colin

    The rate of return for a no risk business should be the same as no risk govt bond

  48. @Colin

    Something a small amount more than you are getting from savings i.e not much 2-4% or thereabouts

  49. @Floating Voter

    I am being to think EM is a bit like Jeeves (Jeeves and Wooster) in thinking many moves ahead and in common with many Labour voters (according to the polls) – I am getting confident he knows what he is doing and he knows how to win.

1 2 3 4 5