I’m regularly sniffy about questions asked in the format of “do you agree or disagree with this statement”, a construction much beloved of the Independent. Here is why, using the December ComRes phone poll as an example.

ComRes found 74% of people agreed with the statement that “The Government should not increase public borrowing any further and its top priority should be to pay off the nation’s deficit as soon as possible” with only 18% disagreeing. Fairly clear result there surely, people agree with the government’s policy?

Except, ComRes also asked whether people agreed with the statement “The Government should borrow more in the short term to increase economic growth as much as possible even if it means reducing the deficit more slowly”, roughly the Labour policy at the time. On that one 49% of people agreed, with 40% of people disagreeing. In other words, at least 23% of people agreed with both.

Now, in this particular poll we could compare the two results and make reasonable conclusions about public opinion overall. However, imagine that only one of the statements had been asked. If a poll had only asked the first statement, one might reasonably have concluded that people overwhelmingly supported the government’s policy. If a poll had only asked the second statement, one might reasonably have concluded that people preferred Labour’s policy.

Another good example was this poll for UKIP from back in 2009. It found 55% of people agreed with the statement “Britain should remain a full member of the European Union”… but also found that 55% agreed with the statement “Britain should leave the European Union but maintain close trading links”. Depending on which statement you asked people to agree with the results were polar opposites.

Again, in this case you could see the contradition because it was asked both ways. But what if the poll had only asked it one way? Well, the BBC did just that the same year – leading to a headline saying “Poll: Brits want to leave EU”. Of course, if they’d asked people to agree or disagree that Britain should stay in the European Union rather than agree or disagree that Britain should leave they’d have found the exact opposite.

Thus is the problem with “agree or disagree with this statement questions” – the statement itself goes in one direction and often gives some justification for it, hence people are more likely to agree… but if the statement had been in the opposite direction they’d have been more likely to agree with that one. They carry a heavy risk of bias in the direction of the statement.

Agree or disagree statements do have a place in testing out various messages that can’t really be unbiased and comparing them against one another, and they are acceptable as trackers where the story is the change in them rather than the absolute value, but they should be interpreted with extreme caution, especially when the poll doesn’t test opposing views in the same way.


23 Responses to “The problem with “Do you agree or disagree with this statement” questions”

  1. Anthony

    You are a pollster. Referendums aren’t opinion polls.

    An elected Government makes a proposal, and asks the voters whether they agree with that proposal or not. There is no way round that.

    Whether the question uses the Scottish Government’s wording or makes a statement followed by “I agree” or “I disagree”, is a matter of field testing and analysis.

    Of course, the question wording can make a difference. Hence the importance of which Government asks the question.

    Doubtless, a UK Government would want to ask a question as close as possible to “Do you want Scotland to be wrenched out of the United Kingdom?”

    Report comment

  2. Ahem – this isn’t a thread about Scottish independence (I pondered whether to put a comment at the bottom of the post pointing out that I wasn’t making some comment about it, as I thought it might be misconstrued. Looks like I should have done)

    I’ve deliberately not said anything about the referendum wording yet (I did give one quote to a journalist, but I suspect it wasn’t really what they wanted, I certainly didn’t see it in any papers)

    Report comment

  3. Anthony

    I do respect your ability as a pollster. Fortunately, you didn’t go into politics.

    Raising this topic, with that title, at this time, in that context, might have been interpreted as expressing your opinion on the referendum question.

    I am reassured that it is a total co-incidence, and that you are expressing no opinion at all on referendum questions.

    Report comment

  4. It seems bizarre to me that pollsters should try to avoid getting contradictory answers to questions. Surely if completely different answers can be obtained merely by wording a question in a different or slightly leading way then the best conclusion to be drawn from this is that the public doesn’t have a strong opinion either way. But then of course that might undermine the pollsters claim to holding the crystal ball to the public’s mind that their careers depend on. Perhaps the more honest thing to do would be to ask the question in several different ways and only claim to know the public’s view if the result varied little under these conditions.

    Report comment

  5. I’m not a big fan of any of these opinion type questions “what do you think”. Decision type questions “what would you do” seem much more useful.

    One of the reasons is the ability to read lots of different things into any statement, not least because people will be coming from very different backgrounds and will therefore interpret words and phrases very differently (you might guess I’m not a fan of Myers/Briggs type questionnaires either). My best example is the word “normal” to me the first thing that spring to mind is a probability distribution. Whilst that may be sad it does show how variable interpretation can be.

    Report comment

  6. Actually a thought: ask the question both ways and you have a much better error measure. In the first case a margin of at least 23% (and you could argue that this isn’t anywhere near the 95% confidence level)

    Report comment

  7. @Eric A Blair

    Welcome – haven’t seen you here before.

    Are you Eric Arthur Blair by any chance?

    Report comment

  8. I suppose we (voters and pollsters) can notice the polling wuestions over time, and many will conclude which polling companies are ‘at it’ when it comes to rigging questions to generate the answer their clients want.

    Ultimately the polling companies are paid by their masters and if instructed to pop in a question with specific wording, who cares? It’s only a poll.

    I’m inclined to favour quick fire, yes/no questions, such as,

    “Are you in favour of:

    The EU –
    Capital punishment –
    Public Sector Cuts –
    Jelly Babies -”

    Questions such as:

    “Are you in favour of the UK being a member of the EU?”

    …don’t actually tell you if people are in favour of the EU or not. Some might not be in favour of it, but prefer the UK to be in it, if it has to exist.

    Report comment

  9. Hi Phil
    Yes the A is for Arthur.

    Report comment

  10. Very Interesting Mr Wells.

    The (recently) much quoted Professor Cialdini (Arizona), doesn’t like “Do you agree” questions – describing them as loaded.

    Cialdini says “Do you agree” questions send people down a particular cognitive shute designed to locate agreement with a preferred option that is assumed in the question.

    Yet… Anthony is suggesting that offering an agree/disagree alternative still does not get round the problem… “the statement itself goes in one direction and often gives some justification for it, hence people are more likely to agree.”

    Any stage hypnotist is well aware of the percentages – a small proportion of the population cannot be hypnotised, the majority are moderately suggestible – but there is a significant propotion of the population who just are way too suggestible for their own good… they will always comply with any suggestion that is put to them in an authoritative way.

    Report comment

  11. Statgeek –

    Hell, I care and so do most other pollsters – we do have some pride in our jobs! I’ve told clients who won’t accept unbiased wording that, actually, we can’t proceed like that and they’ve had to take their business elsewhere. So do other companies I’m sure.

    Bob Worcester, for example, famously refused to work for James Goldsmith because Goldsmith wouldn’t accept MORI framing the questions fairly.

    Report comment

  12. Billy Bob –

    Slightly different things. Just saying “Do you agree” is a basic error in a polling question – it should always be balanced: “Do you agree or disagree”

    The words “Do you agree or disagree” are a good formulation for the start of a question “Do you agree or disagree with taking away Fred Goodwin’s knighthood?”, “Do you agree or disagree with the government’s policy on fish?”. That sort of thing is fine.

    The problem comes when you ask people to agree or disagree with a statement in a particular direction, because while the agree/disagree bit is balanced, the statement isn’t.

    I don’t think I’ve explained it very well! Perhaps an example is better:

    “Do you agree or disagree with the policy of cutting VAT?”

    is fine (I wouldn’t ask it like that, but it’s fine)

    “Do you agree or disagree with the following statement? “Cutting VAT is the right thing to do to help the economy”

    would be skewed. it’s not the agree/disagree formulation that is bad, it’s the agree or disagree with a non-neutral statement formulation that can be dangerous if not balanced with agree/disagree statements in the other direction.

    Report comment

  13. YouGov/Sun – CON 39%, LAB 40%, LD 9% App -23%

    Report comment

  14. @AW

    Would it be better to offer two statements:

    “The Government should not increase public borrowing… etc”

    “The Government should borrow more in the short term… etc”

    then invite people to chose one?

    Report comment

  15. Billy Bob – yes, that’s often a question structure I use (though you need to give a neither, and even then, to be cautious not to create a false dichotomy).

    Report comment

  16. @AW

    Thanks for your response.

    Report comment

  17. @AW

    “Hell, I care and so do most other pollsters – we do have some pride in our jobs! I’ve told clients who won’t accept unbiased wording that, actually, we can’t proceed like that and they’ve had to take their business elsewhere. So do other companies I’m sure. ”

    I was thinking “who cares” from one poll versus thousands of polls, rather than the habit or the practice of ‘who cares’. It’s good to know you care. AW junior (2?) can sleep easy. ;)

    Report comment

  18. “Hi Phil
    Yes the A is for Arthur.”

    ———————————————————

    Phew. I thought it might have been for Anthony for a moment.

    There was a distinct whiff of incense coming from somewhere down Hampshire way in premature celebration of the Second Coming.

    Report comment

  19. Eric

    Is your Aspidistra flying?

    Report comment

  20. @ Old Nat

    Fortunately, you [Anthony] didn’t go into politics.
    ——————————–
    I thought Anthony is/was a Conservative Councillor – i.e. he did go into politics.

    FWIW, I thought your comments were partisan & impolite. It may be hard for you to believe but Scotland’s not the center of everyone’s universe, Old Nat.
    8-)

    Report comment

  21. @ Anthony

    Hi, long time reader, first time poster here, and a big fan of your site’s great neutrality.

    “Do you agree or disagree with the policy of cutting VAT?”

    This question akin to “would you like a tax cut?” to which I’m sure everyone would say “yes!”

    You put this question up earlier and suggested this was an “ok” non-leading question… but surely this is an extremely leading question.

    Since you’ve only given 1/2 of a (theoretical) policy, you’ve not mentioned where the money comes from. Something like “do you agree or disagree with the policy of cutting vat and increasing the top rate of income tax to pay for it?”

    I’m sure plenty of people would still say yes, but I think it gives a stronger foundation to make their decision

    Report comment

  22. Planky – that’s why I said I wouldn’t ask it like that, there are certainly better ways to do it. I just plucked something out of the air to put in an agree/disagree question, and an agree/disagree with this statement question.

    Report comment

  23. Hi, first off, just to say how much I always enjoy UK Polling Report……now, I don’t know whether this is a daft comment but –

    “The Government should not increase public borrowing any further and its top priority should be to pay off the nation’s deficit as soon as possible”

    doesn’t that also highlight the unhelpfulness of this kind of question because actually if the Government “didn’t increase it’s borrowing any further” from now on that would be a deficit of *0%* for 2012-13 and an expenditure cut of numerous % across the board – something far more extreme than even the Coalition’s policies? and I doubt many would agree to the almost-certain consequences of that option!

    Although this seems to be confused somewhat by the term “pay off the nation’s deficit as soon as possible” – that would seem to make much more sense read as debt, not deficit. Or alternately, even the would-be “dinosaurs” of today’s politics would agree to reducing the deficit *as soon as possible*….possible could mean all kinds of things in this context.

    Though maybe that says more about my pedantry, or one sloppy question than a general methodology……

    Report comment