I’m regularly sniffy about questions asked in the format of “do you agree or disagree with this statement”, a construction much beloved of the Independent. Here is why, using the December ComRes phone poll as an example.
ComRes found 74% of people agreed with the statement that “The Government should not increase public borrowing any further and its top priority should be to pay off the nation’s deficit as soon as possible” with only 18% disagreeing. Fairly clear result there surely, people agree with the government’s policy?
Except, ComRes also asked whether people agreed with the statement “The Government should borrow more in the short term to increase economic growth as much as possible even if it means reducing the deficit more slowly”, roughly the Labour policy at the time. On that one 49% of people agreed, with 40% of people disagreeing. In other words, at least 23% of people agreed with both.
Now, in this particular poll we could compare the two results and make reasonable conclusions about public opinion overall. However, imagine that only one of the statements had been asked. If a poll had only asked the first statement, one might reasonably have concluded that people overwhelmingly supported the government’s policy. If a poll had only asked the second statement, one might reasonably have concluded that people preferred Labour’s policy.
Another good example was this poll for UKIP from back in 2009. It found 55% of people agreed with the statement “Britain should remain a full member of the European Union”… but also found that 55% agreed with the statement “Britain should leave the European Union but maintain close trading links”. Depending on which statement you asked people to agree with the results were polar opposites.
Again, in this case you could see the contradition because it was asked both ways. But what if the poll had only asked it one way? Well, the BBC did just that the same year – leading to a headline saying “Poll: Brits want to leave EU”. Of course, if they’d asked people to agree or disagree that Britain should stay in the European Union rather than agree or disagree that Britain should leave they’d have found the exact opposite.
Thus is the problem with “agree or disagree with this statement questions” – the statement itself goes in one direction and often gives some justification for it, hence people are more likely to agree… but if the statement had been in the opposite direction they’d have been more likely to agree with that one. They carry a heavy risk of bias in the direction of the statement.
Agree or disagree statements do have a place in testing out various messages that can’t really be unbiased and comparing them against one another, and they are acceptable as trackers where the story is the change in them rather than the absolute value, but they should be interpreted with extreme caution, especially when the poll doesn’t test opposing views in the same way.
