Or at least, questions that should be treated with a large dose of scepticism. My heart always falls when I see questions like those below in polls. I try not to put up posts here just saying “this poll is stupid, ignore it” but sometimes it’s hard. As ever, I’d much rather give people the tools to do it themselves, so here are some types of question that do crop up in polls that you really should handle with care.

I have deliberately not linked to any examples of these in polls, though they are sadly all too easy to find. It’s not an attempt to criticise any polling companies or polls in particular and only he who is without sin should cast the first stone, I’m sure you can find rather shonky questions from all companies and I’m sure I’ve written surveys myself that committed all of the sins below. Note that these are not necessarily misleading questions or biased questions, there is nothing unethical or wrong with asking them, they are just a bit rubbish, and often lead to simplistic or misleading interpretations – particularly when they are asked in isolation, rather than part of a longer poll that properly explores the issues. It’s the difference between a question that you’d downright refuse to run if a client asked for it, and a question that you’d advise a client could probably be asked much better. The point of this article, however (while I hope it will encourage clients not to ask rubbish questions) is to warn you, the reader of research, when a polling question really should be read with caution.

Is the government doing enough of a good thing?

A tricky question. This is obviously trying to gauge a very legitimate and real opinion – the public do often feel that the government hasn’t done enough to sort out a problem or issue. The question works perfectly well it is something where there are two sides to the question and the question is intrinsically one of balance, where you can ask if people think the government has not done enough, or gone too far, or got the balance about right. The problem is when the aim is not contentious, and it really is a question of doing enough – stopping tax evasion, or cutting crime, for example. Very few people are going to think that a government has done too much to tackle tax evasion, that they have pushed crime down too low (“Won’t someone think of the poor tax evaders?”, “A donation of just £2 could buy Fingers McStab a new cosh”), so the question is set up from the beginning to fail. The problems can be alleviated a bit with wording like “Should be doing more” vs “Has done all they reasonably can be expected to do”, but even then you should treat questions like this with some caution.

Would you like the government to give you a pony?*
(*Hat tip to Hopi Sen)

Doesn’t everyone like nice things? There is nothing particularly wrong with questions like this where the issue at question is something controversial and something that people might disagree with. It’s perfectly reasonable to ask whether the government should be spending money on introducing high speed rail, or shooting badgers, or whatever. The difficulty comes when you are asking about something that is generally seen as a universal good – what are essentially valence issues. Should the government spend extra on cutting crime, or educating children, or helping save puppies from drowning? Questions like this are meaningless unless the downside is there as well – “would you like the government to buy you a pony if it meant higher taxes?“, “would you like the government to buy you a pony or could the money be better spent elsewhere?

How important is worthy thing? Or How concerned are you about nasty thing?

Asking if people support or oppose policies, parties or developments is generally pretty straightforward. Measuring the salience of issues is much harder, because you run into problems of social desirability bias and taking issues out of context. For example, in practical terms people most don’t actually do much about third world poverty. Ask people if they care about children starving to death in Africa you’d need a heart of stone to say that you really don’t. The same applies to things that sound very important and worthy, people don’t want to sound ignorant and uninterested by saying they don’t really care much. If you ask about whether people care about, are concerned about or think an issue is important they will invariably say they do care, they are concerned and it is important. The rather more pertinent question that is not always asked is whether it is important when considered alongside all the other important issues of the day. The best way of measuring how important people think an issue is will always be to give them a list of issues and pick out those they consider most important (or better, just give them an empty text box and ask them what issues are important to them).

Will policy X make you more likely to vote for this party?

This is a very common question structure, and probably one I’ve written more rude things about on this site than any other type of question. There are contexts where it can work, so long as it is carefully interpreted and is asked about a factor that is widely acknowledged to be a major driver of voting intention. For example, it’s sometimes used to ask if people would be more or less likely to vote for a party if a different named politician was leader (though questions like that have their own issues).

It becomes much more problematic when it is used as a way of showing an issue is salient. Specifically, it is often used by campaigning groups to try and make out that whatever issue they are campaigning about will have an important impact on votes (and, therefore, MPs should take it seriously or their jobs may be at risk). This is almost always untrue. Voting intention is overwhelmingly driven by big brush themes, party identification, perceptions of the party leaders, perceived competence on the big issues like the economy, health or immigration. It is generally NOT driven by specific policy issues on low salience issues.
However, if you ask people directly about the impact of specific policy issue on a low salience issue, and whether it would make them more or less likely to vote for a party, they will normally claim it does. This is for a number of reasons. One is that you are taking that single issue and giving it false prominence, when it reality it would be overshadowed by big issues like the economy, health or education. The second is that people tend to just use the question as a way of signalling if they like a policy or not, regardless of whether it would actually change their vote. The third is that it takes little account of current voting behaviour – you’ll often find the people saying a policy makes them more likely to vote Tory is made up of people voting Tory anyway, people saying a policy makes them less likely to vote Tory are people who wouldn’t vote Tory if hell froze over.

There are ways to try and get round this problem – in the past YouGov used to offer “Makes no difference – I’d vote party X anyway” and “Makes no difference – I wouldn’t vote for party X” to try and get rid of all those committed voters whose opinion wouldn’t actually change. In some of Lord Ashcroft’s polling he’s given people the options of saying they support a policy and it might change their vote, or that they’d support it but it wouldn’t change their vote. The best way I’ve come up with doing it is to give people a long list of issues that might influence their vote, getting them to tick the top three or four, and only then asking people whether the issue would make them more or less likely to vote for a party (like we did here for gay marriage, for example). This tends to show that many issues have little or no effect on voting intention, which is rather the point.

Should the government stop and think before going ahead with a policy?

This is perhaps the closest I’ve seen to a “when did you stop beating your wife” question in recent polls. Obviously it carries the assumption that the government has not already stopped to consider the implications of policy. Matthew Parris once wrote about a rule of thumb on understanding political rhetoric, saying that if the opposite of a political statement was something that no one could possibly argue for, the statement itself was meaningless fluff. So a politician arguing for better schools is fluff, because no one would ever get up to the podium to argue for worse schools. These sort of questions fall into the same trap – no one would argue the opposite, that the best thing for the government to do is to pass laws willy-nilly without regard for consequences, so people will agree to the statement in regard of almost any subject. It does NOT necessarily indicate opposition to or doubt about the policy in question, just a general preference for sound decision making.

Do you agree with pleasant uncontentious thing, and that therefore we should do slightly controversial thing?

Essentially the problem is one of combining two statements together within an agree disagree statement, and therefore not giving people the chance to agree with one but not the other. For example “Children are the future, and therefore we should spend more on education” – people might well agree that children are the future (in fact, it’s relatively hard not to), but might not agree with the course of action that the question suggests is a natural consequence of this.

How likely are you to do the right thing? or Would you signify your displeasure at a nasty thing?

Our final duff question falls into the problem of social desirability bias. People are more likely to say they’ll do the right thing in a poll than they are to do it in real life. This is entirely as you’d expect. In real life the right thing is sometimes hard to do. It might involve giving money when you really don’t have much to spare, or donating your time to volunteer, or inconveniencing yourself by boycotting something convenient or cheap in favour of something ethical. Answering a poll isn’t like that, you just have to tick the box saying that you would probably do the right thing. Easy as pie. Any poll you see where you see loads of people saying they’d volunteer to do something worthwhile, boycott something else, or give money to something worthy, take with a pinch of salt.


243 Responses to “Questions that should be ignored”

1 3 4 5
  1. ‘In the coalition they will discuss this over the next few weeks.
    There is a sensible, middle-ground balance to be struck’

    N Clegg on LBC this morning

    So a deal has not been struck yet, but it will be in the coming weeks.

    Nick C sounded a reasonable fellow. He reminded me again why I voted for the LD’s in 2010.

    Although he was rude again about Ed M

    ‘Ed Miliband’s proposal is “an economically illiterate con, a 24-carat con’

    I can’t see how they will ever work together in a coalition.

  2. FPTP
    This is the key fact about the forthcoming election. It is a very blunt instrument. The Tories are happy with it, or rather would have been happy,if they hAd got the boundary changes through. The LDs scuppered those,they are not happy with FPTP,but they are stuck with it and as Howard,of this parish ,helpfully pointed out recently,on 9 or 10% they are going to get a max of 21 seats. The LDs are going to have to try appealing to lots of local circumstances if their % does not start creeping up soon. To do this you need lots of footsoldiers ,the indications are that their activist base has been hollowed out,anecdotally l have helped out in 4 or 5 council by~elections and found the LDs a shadow of their former selves. The LDs CAN still do major set~pieces,such as Eastleigh,and believe me they threw the kichen sink at that and the contents of the utility room,(I still marvel at the 8 page ‘Country Life ‘facsimiles that they produced there) however come a Country wide campaign,they are going to be stretched to get across the nuanced message they will need in different constituencies.

  3. Returning to the ‘Duff Questions’ comment, I think that ALL polling questions are a bit duff. One of the problems is that if you as a question, it causes people to think about the subject probably more than they otherwise would. Asking the question therefore guarantees that the answer is a little bit suspect.

    Publishing the result causes many more people to think about it and amplifies the effect.

    What seems to be forgotten on this site is that you cannot really see the future and the ‘trend’ which you hope to be able to predict is itself a fiction. There may be many trends operating at the same time, and the composite ‘trend’ is almost impossible to determine as a result. This is supported by the fact that different polling methodologies frequently differ by considerably more than their computed margins of error.

    The bald facts are that polling is an extremely rough guide to the actual voting intention, despite the care that goes into them. So rough, in my opinion, that it’s hardly worth taking very seriously.

    Politicians DO take them seriously, however, but I believe this may be to save them the odious task of talking to their constituents and assessing for themselves what really concerns the public. In many cases this is not a pleasant thing to do.

  4. @Colin

    This sort of things comes with the territory with privatised utilities. If you, as a company, have the advantage of supplying a commodity that is deemed essential rather than a luxury, then you have to deal with dangers to reputation stemming from everyone having an opinion, and this is exacerbated if, as is the case in energy, customers feel they have no genuine choice.

    And, even worse in this case, because wholesale prices are obfuscated, nobody actually knows what a fair price for energy actually is and have to trust the energy companies to set one. And they are not trusted to do that.

    This is a problem entirely of the energy company’s making, and when you start advancing the argument, as you are, that customers who don’t like you simply don’t have all the facts about how great you are, then you have lost that argument (and, of course, the same is true of politics).

    This one is a fearful bind for free marketeers and I think it’s best if they admit that something isn’t working here, because it transparently isn’t.

  5. Ed Miliband’s response will be to say that the prime minister is too weak to stand up for the consumer and that he always takes the side of the big six companies. Instead of dealing with them he is just moving the Green taxes from one place to another which doesn’t get to the root of the problem as outlined by And that even Sir John Major recognises that the job of government is to reform markets when they are failing and to protect people. This prime minister does not.

    OFGEM data for the period September 2011 to October 2013 shows that since September 2011 wholesale electricity price has gone up by just over 2%. NPower and British Gas have raised prices by nearly 30%; wholesale gas price has gone up by 8%. NPower and British Gas have raised prices by up to 35%.

  6. Sorry…”outlined by Sir John Major as even he…”

  7. COUPER2802

    OK-So I’m going into business with your criterion.

    I borrow £1000 from the Bank & get £1000 from a friend as permanent equity capital.

    The Bank wants 4% over BR ( the Manager is an acquaintance) -and my friend is happy to accept 1% pa dividend-looking for growth in the business & the value of his shares.

    So-I invest £1500 in operating assets & need £500 to cover working capital-stock & debtors etc.
    Away we go with £2000 capital employed.

    I need to earn :-
    4.75% on £1000 , and 1% on £1000 to cover my cost of funding-that’s £57.50.
    Then I need to set funds aside to replace my equipment-it will wear out over 5years-so that’s 20% X £1500=£300.

    And I need to pay tax on any profits at 20% -so what does my profit & loss account need to look like?

    I need £432.50 before the items I have mentioned so I finish up with :-

    £432.50
    Less cost of funding £57.5=£375 ; less Tax at 20%=£300 =enough to replace my kit.

    And you will only allow me to make 4% max on my Capital Employed of £2000……….which is £80.

    So…………I won’t bother……….can’t be done.

  8. I’m sure Ed will say well that’s a start now lets tackle the profiteering

  9. @Colin

    You shouldn’t be borrowing the money to invest – that is what caused the problems in the banks – speculation.

    It should be money you have earned doing something productive that you then invest to help others do something productive and get a better return than leaving your cash in the bank.

  10. @ Colin

    Haven’t you just assumed that the people who kindly responded to you meant ~4% earnings after interest & tax? They may have meant 4% EBIT.

  11. Any forecasts for the Dunfermline MSP election?

  12. @Colin

    Of course if you are talking about setting up your own business risk and hard work etc. But that is not the situation we are talking about here we are discussing a no risk investment in a essential industry which is dominated by a cartel.

  13. COUPER

    @”It should be money you have earned doing something productive that you then invest to help others do something productive and get a better return than leaving your cash in the bank.”

    Yep-my business was going to be productive-solar panel installation & manufacture.
    I was going to invest out of profits-as my business plan showed you.

    But you wouldn’t let me make enough return to invest.

    So I’m off to the Job CEntre.

  14. AMBER

    I haven’t assumed anything-I am waiting for a definition from those who wish to define “profiteering”.

    Yes -of course-definition is everything-which you understand unlike some others.

  15. COUPER2802

    There were no risk premiums built into my model-simply the bare minimum to pay providers of capital , tax & replace operating assets.

    If my business has major risks inherant , that will increase the return I need.

  16. @Floating Voter

    “Although he (Nick Clegg) was rude again about Ed M”

    I’m reminded of Denis Healey’s description of what it felt like to be attacked by Geoffrey Howe. He likened it to being savaged by a dead sheep and I would have thought Miliband is currently filled with the same amount of fear and dread!

  17. @Colin

    When inflation is 2.7% and wage growth 0.7% it is not credible that you need a yearly hike of 10+% to get by.

    When your costs have gone up 2-8% but your prices are up 35%

    You are in a Cartel in an essential industry that people cannot do without.

    Then I think it is fair to assume you are profiteering.

  18. @ Colin

    Your 10:39am is comedy gold. Energy company CEOs forced to sign on the dole after Ed’s energy price freeze. We are ROFLOL at a comment which implies that would happen! :-)

  19. It not just the declared profits either, it’s the excess pay, the CEO’s of these energy companies get multi million pay packets with bonuses for what is basically a glorified civil servant job, there is no reason why they should get paid more than the prime minister

  20. COUPER 2802

    Here are Centrica’s last Accounts.

    http://www.centrica.com/files/reports/2012ar/files/pdf/centrica_annual_report_2012.pdf

    Here’s a rough guide to the numbers you need in order to assess whether they are profiteering :-

    Turnover ( continuing business) £ 23,942 m
    OPerating Profit £ 2562 m
    less Interest £183m
    Profit after interest £ 2379m
    Less Taxation £ 1029m
    Profit after Tax £ 1350m
    Less Dividend £ 816m
    REtained Profit £ 534.

    Fixed Asset Investment -£1957m
    ( of which £6897m provided out of operating profit by depreciation charge)

    Capital Employed :-
    Net Assets £ 5927 m
    plus DEbt £ 5943m
    = £ 11870m

    Is this “profiteering” -and if so why?

    I am genuinely interested in the question-having just switched supply from them.

  21. AMBER

    @” Energy company CEOs”

    CEO would have been a stretch-there was only me & my mate in the business.

    But it didn’t last long as you could see-think I will move to the US. lol

  22. Colin

    You are arguing against your own declared free market principles. According to the textbooks these companies should be making very little over the normal bank rate because the risk is practically non existent and competition should according to the textbooks reduce profits to a bare minimum and executive pay with it

  23. RiN

    @”glorified civil servant job,”

    That’s the way they ( it !) used to be run.

    I remember it well-service levels were great lol

  24. Colin

    That’s just silly, no sane person would expect a start up taking huge risks to be content with such a small return on investment. The problem is that when you can get reasonable returns from a risk free business why will anyone take any risks

  25. RiN

    You are talking nonsense.

    You don’t seem to understand the basic principles-but I invite you to critique Centrica’s Financial Results.

  26. Colin

    Your example shows a much better return than govt bonds for the same amount of risk

  27. COUPER2802

    @”( of which £6897m provided out of operating profit by depreciation charge)”

    Sorry-typo

    …of which £897m…….

  28. @ Colin

    Centrica 2012 Performance Highlights:

    Earnings per share growth of 25%
    Dividend per share growth of 28%
    Total shareholder return of 36%

    Does that make it clear to you? Total Shareholder return of 36%

  29. @ Colin

    That’s why your comment is so silly. You asked what people thought the return should be on a massive, captive market for the dominant ‘players’.

    You then applied that to a tiny start-up. I cannot believe that an ex-FD does not know the difference. Therefore I assumed you were going for LOLs!

    If you were serious, you need to go back to accountancy school!

  30. RiN

    @”Your example shows a much better return than govt bonds for the same amount of risk”

    If you really are equating the risks of constructing & running a multi-million pond power generation & distribution business with owning a government bond…..there is nothing more that we can usefully say to one another.

    Lets see what the voters think eh?

  31. AMBER

    @”Total Shareholder return of 36%”

    Which , as the Accounts show, includes both dividend growth & share price growth.

    ……..So you want to control the change in share prices too?

    How will that work?

  32. @ Colin

    If you really are equating the risks of constructing & running a multi-million pond power generation & distribution business with owning a government bond…
    ——————–
    You ought to know that the benchmark is US Bonds plus 4% to 8% depending on the risk factor. Large power corporations were viewed as very low risk indeed prior to Enron’s shenanigans.

  33. @ Colin

    The growth in share price reflects the current & forecast profitability of the future business.
    It takes current + the prospect of high returns to generate a 36% total shareholder return.
    Your use of strange assumptions & odd interpretations of financial data to fit an agenda is not pleasant for a fellow professional to see.
    I am all for you having an opinion e.g. in your view 36% total shareholder return is acceptable but please think before you seek to imply that Centrica is struggling & the CEO/ Shareholders will be signing on the dole any time soon!

  34. Colin

    It’s risk free, that’s the point, just as risk free as govt bonds in fact it might even be safer. But even if it wasn’t risk free, the economic theory that you believe in says that profits should come down to a level that is only slightly above what you would get from govt bonds, either the theory is incorrect or the market is not free. You defend that theory when it makes people unemployed or reduces their wages or sends their jobs overseas but invent excuses when it’s shown to be either wrong or ineffective. If you want to believe in free markets that’s great but don’t say that free market consequences are alright for ordinary people but don’t apply to the elites, which is exactly what you are arguing even if you are oblivious to it

  35. Hi there. I’m looking for the UKPR site please. The one with the polling comments.

  36. AMBER

    @” in your view 36% total shareholder return is acceptable”

    Did I say that it was?

    @” you seek to imply that Centrica is struggling & the CEO/ Shareholders will be signing on the dole any time soon!”

    Did I say they are? ( what on earth does “struggling” mean -I asked for a definition of “profiteering” from those who think CEntrica is._ haven’t had one.-
    Did I mention the CEO?

    You really must stop doing that Amber-it doesn’t aid an argument .

  37. Colin
    “So I’m off to the Job CEntre.”

    Take heart then that in the event of your benefit claim being delayed, they will give you a voucher for the food bank (that may or may not be anywhere close to where you live).

    There you will get 3 days worth of food – the result of somebody else’s generous act.

    Disclaimer: I have arrived very late to this thread & haven’t read all the comments yet so if somebody else has made this point then I apologise for repeating it, although the facts around the utter desperation of those requiring food banks can never be repeated often enough imo.

  38. Statgeek
    “Hi there. I’m looking for the UKPR site please. The one with the polling comments.”

    I think it’s been hacked by a free school looking for some economics teachers.

  39. New thread!

    And Colin, from a customer’s perspective, it takes excessive profits to generate a 36% shareholder return.

  40. @ Colin

    And you move the goalposts faster, further & more often than badgers do. Because, despite your feigned bewilderment, all with eyes can see that I (& those who provided % returns) were responding directly to your challenges, whereas you were responding by changing the underly!ng points of the debate. :-)

  41. @”( of which £6897m provided out of operating profit by depreciation charge)”

    Sorry-typo

    …of which £897m…….”

    I wouldn’t worry: can’t imagine that anyone was following the details of this ludicrous, theoretical debate.

    Get a life springs to mind.

  42. ROSIEANDDAISIE

    This is our life!! Lol

  43. Colin

    The energy business you were setting up (fpt) will no doubt be delighted to hear that, according to OFGEM, duel fuel profit margins across all suppliers has tripled over the last 18 months. Now around £90, it was around £25 in Spring 2012.

    And they wonder why customers are disgusted at blatant profiteering…..

1 3 4 5