Opinion polling on Brexit has not necessarily been the best. Highly politically contentious issues do tend to attract polling that is sub-optimal, and Brexit has followed that trend. I’ve seen several Brexit polls coming up with surprising findings based on agree/disagree statements – that is, questions asked in the form:

Do you agree with the following statement? I think Brexit is great
Agree
Disagree
Don’t know

This is a very common way of asking questions, but one that has a lot of problems. One of the basic rules in writing fair and balanced survey questions is that you should try to given equal prominence to both sides of the argument. Rather than ask “Do you support X?”, a survey should ask “Do you support or oppose X?”. In practice agree-disagree statements break that basic rule – they ask people whether they agree/disagree with one side of the argument, without mentioning the other side of the argument.

In some cases the opposite side of the argument is implicit. If the statement is “Theresa May is doing a good job”, then it is obvious to most respondents that the alternative view is that May is doing a bad job (or perhaps an average job). Even when it’s as obvious as this it still sometimes to make a difference – for whatever reason, decades of academic research into questionnaire design suggest people are more likely to agree with statements than to disagree with them, regardless of what the statement is (generally referred to as “acquiescence bias”).

There is a substantial body of academic evidence exploring this phenomenon (see, for example Schuman & Presser in the 1980s, or the recent work of Jon Krosnick) it tends to find around 10%-20% of people will agree with both a statement and its opposite, if it is asked in both directions. Various explanations have been put forward for this in academic studies – that it’s a result of personality type, or that it is satisficing (people just trying to get through a survey with minimal effort). The point is that it exists.

This is not just a theoretical issue that turns up in artificial academic experiments – they are plenty of real life examples in published polls. My favourite remains this ComRes poll for UKIP back in 2009. It asked if people agreed or disagreed with a number of statements including “Britain should remain a full member of the EU” and “Britain should leave the European Union but maintain close trading links”. 55% of people agreed that Britain should remain a full member of the EU. 55% of people also agreed that Britain should leave the EU. In other words, at least 10% of the same respondents agreed both that Britain should remain AND leave.

There is another good real life example in this poll. 42% agreed with a statement saying that “divorce should not be made too easy, so as to encourage couples to stay together”. However, 69% of the same sample also agreed that divorce should be “as quick and easy as possible”. At least 11% of the sample agreed both that divorce should be as easy as possible AND that it should not be too easy.

Examples like this of polls that asked both sides of the argument and produced contradictory findings are interesting quirks – but since they asked the statement in both directions they don’t mislead. However, it is easy to imagine how they would risk being misleading if they had asked the statement in only one direction. If that poll had only asked the pro-Brexit statement, then it would have looked as if a majority supported leaving. If the poll had only asked the anti-Leave statement, then it would have looked as if a majority supported staying. With agree-disagree statements, if you don’t ask both sides, you risk getting a very skewed picture.

In practice, I fear the problem is often far more serious in published political polls. The academic studies tend to use quite neutrally worded, simple, straightforward statements. In the sort of political polling for pressure groups and campaigning groups that you see in real life the statements are often far more forcefully worded, and are often statements that justify or promote an opinion – below are some examples I’ve seen asked as agree-disagree statements in polls:

“The Brexit process has gone on long enough so MPs should back the Prime Minister’s deal and get it done”
“The result of the 2016 Referendum should be respected and there should be no second referendum”
“The government must enforce the minimum wage so we have a level playing field and employers can’t squeeze out British workers by employing immigrants on the cheap”

I don’t pick these because they are particularly bad (I’ve seen much worse), only to illustrate the difference. These are statements that are making an active argument in favour of an opinion, where the argument in the opposite direction is not being made. They do not give a reason why MPs may not want to back the Prime Minister’s deal, why a second referendum might be a good idea, why enforcing the minimum wage might be bad. It is easy to imagine that respondents might find these statements convincing… but that they might have found the opposite opinion just as convincing if they’d been presented with that. I would expect questions like this to produce a much larger bias in the direction of the statement if asked as an agree-disagree statement.

With a few exceptions I normally try to avoid running agree-disagree statements, but we ran some specially to illustrate the problems, splitting the sample so that one group of respondents were asked if they agreed or disagreed with a statement, and a second group where asked if they agreed-disagreed with a contrasting statement. As expected, it produces varied results.

For simple questions, like whether Theresa May is doing a good job, the difference is small (people disagreed with the statement that “Theresa May is doing a good job by 57% to 15% and agreed with the statement that “Theresa May is doing a bad job” by 52% to 18%. Almost a mirror image. On some of the other questions, the differences were stark:

  • If you asked if people agree that “The NHS needs reform more than it needs extra money” then people agree by 43% to 23%. However, if you ask if people agree with the opposite statement, that “The NHS needs extra money more than it needs reform”, then people also agree, by 53% to 20%.
  • If you ask if people agree or disagree that “NHS services should be tailored to the needs of populations in local areas, even if this means that there are differences across the country as a whole” than people agree by 43% to 18%. However, if you ask if they agree or disagree with a statement putting the opposite opinion – “NHS services should be the same across the country” – then people agree by 88% to 2%!
  • By 67% to 12% people agree with the statement that “Brexit is the most important issue facing the government and should be its top priority”. However, by 44% to 26% they also agree with the statement “There are more important issues that the government should be dealing with than Brexit”

I could go on – there are more results here (summary, full tabs) – but I hope the point is made. Agree/disagree statements appear to produce a consistent bias in favour of the statement, and while this can be minor in questions asking simple statements of opinion, if the statements amount to political arguments the scale of the bias can be huge.

A common suggested solution to this issue is to make sure that the statements in a survey are balanced, with an equal amount of statements in each direction. So, for example, if you were doing a survey about attitudes towards higher taxes, rather than asking people if they agreed or disagreed with ten statements in favour of high taxes, you’d ask if people agreed or disagreed with five statements in favour of higher taxes and five statements in favour of lower taxes.

This is certainly an improvement, but is still less than ideal. First it can produce contradictory results like the examples above. Secondly, in practice it can often result in some rather artificial and clunky sounding questions and double-negatives. Finally, in practice it is often difficult to make sure statements really are balanced (too often I have seen surveys that attempt a balanced statement grid, but where the statements in one direction are hard-hitting and compelling, and in the other direction are deliberately soft-balled or unappetising).

The better solution is not to ask them as agree-disagree statements at all. Change them into questions with specific answers – instead of asking if people agree that “Theresa May is going a good job”, ask if May is doing a good or bad job. Instead of asking if people agree that “The NHS needs reform more than it needs more money”, ask what people think the NHS needs more – reform or more money? Questions like the examples I gave above can easily be made better by pairing the contrasting statements, and asking which better reflects respondents views:

  • Asked to pick between the two statements on NHS reform or funding, 41% of people think it needs reform more, 43% think it needs extra money more.
  • Asked to pick between the two statements on NHS services, 36% think they should be tailored to local areas, 52% would prefer them to be the same across the whole country.
  • Asked to pick between the two statements on the importance of Brexit, 58% think it is the most important issue facing the government, 27% think there are more important issues the government should be dealing with instead.

So what does this mean when it comes to interpreting real polls?

The sad truth is that, despite the known problems with agree-disagree statements, they are far from uncommon. They are quick to ask, require almost no effort at all to script and are very easy for clients after a quick headline to interpret. And I fear there are some clients to whom the problems with bias are an advantage, not a obstacle; you often see them in polls commissioned by campaigning groups and pressure groups with a clear interest in getting a particular result.

Whenever judging a poll (and this goes to observers reading them, and journalists choosing whether to report them) my advice has always been to go to polling companies websites and look at the data tables – look at the actual numbers and the actual question wording. If the questions behind the headlines have been asked using agree-disagree statements, you should be sceptical. It’s a structure that does have an inherent bias, and does result in more people agreeing than if the question had been asked a different way.

Consider how the results may have been very different if the statement had been asked in the opposite direction. If it’s a good poll, you shouldn’t have to imagine that – the company should have made the effort to balance the poll by asking some of the statements in the opposite direction. If they haven’t made that effort, well, to me that rings some alarm bells.

If you get a poll that’s largely made up of agree-disagree statements, that are all worded in the direction that the client wants the respondent to answer rather than some in each direction, that use emotive and persuasive phrasing rather than bland and neutral wording? You would be right to be cautious.


1,954 Responses to “Why you should be wary of agree/disagree statements”

1 2 3 40
  1. Do you agree that Brexit is a good idea?

  2. New poll out from ComRes. Page 39 onwards is about Brexit. Seems No Deal is now the most popular leave option across both leavers and remainers combined, and that the majority think Parliament is delibarately thwarting the referendum.

    https://www.comresglobal.com/wp-content/uploads/2019/03/LML-Mar19-Final-Tables.pdf

  3. Another great article from Anthony. It’s a real pity so few people read these summaries, particularly journalists whose job it is to pass findings on to the general public.

  4. @ADW

    The ComRes questions seem designed to prove Anthony’s point about agree/disagree questions, being very tendentially worded, even if one of them they try to ask both ways.

  5. @adw

    “New poll out from ComRes”

    I’m guessing that the irony of posting that and reading the results from it as a given on a blog post all about the deficiencies of agree/disagree polling is going to be lost on you.

  6. UK Cabinet will soon have been meeting for 8 hours which will make it one of the the longest on record apparently.

  7. Comres has surprisingly come up with some interesting stats:

    Q.2 Do you agree or disagree with each of the following statements?Remain-supporting MPs and other Establishment figures trying to stop Brexit have damaged the UK’s negotiating positionBase:

    All respondents
    54% agree
    24% disagree
    22% DK

    Q.2 Reversed: If Brexit is thwarted by Remain-supporting MPs and other Establishment figures, it will be for the good of the countryBase:

    All respondent
    26% agree
    46% disagree
    27% DK

    42% to 37% think if TM’s deal is voted down again we should leave on March 29th! Too late.

    If Theresa May’s deal is voted down again, the UK should just remain in the EU. Big surprise here:

    38% agree
    45% disagree
    18% DK

    Too many stats to copy onto here but the other big one is 55% to 16% saying having a second referendum would be betraying those who voted to leave in 2016, 13% said neither and 17% DK.

  8. This does make me wonder how many of those who post on here don’t even bother reading Anthony Wells’ editorial.

  9. @ James E

    It’s where the same question is asked with different slants and the result is reversed that you need to be more cautious.

    On the ComRes poll it’s quite clear with this particular set of stats that the public are realising Arch remainers are now just as culpable as the ERG for holding everything up.

  10. Robert Peston saying the rumour is a long extension being requested from the EU to better prepare for a no deal!

  11. “On the ComRes poll it’s quite clear with this particular set of stats that the public are realising Arch remainers are now just as culpable as the ERG for holding everything up.”

    No, there is no question in this poll (sponsored by Leave means leave) about the ERG, and therefore no evidence for making that comparison.

  12. Thanks Anthony

    Lots to digest a’s usual

    It would be great if people could be tested to make sure they had read and understood the summary before they were allowed post on the thread!

  13. YG Live poll on MPs compromising on Brexit

    Stick to principles on Brexit: 18
    Be more willing to compromise : 54
    Neither: 13
    DK: 15

    Fairly consistent in the x-breaks but you can see why Norm Lamb (LDEM) is a bit upset with his colleagues as LDEM VI have the highest % for compromise.

    No x-break for TIG or SNP but FWIW Scotland is bang on the UK average. Age breaks interesting though.

    https://yougov.co.uk/topics/politics/survey-results

  14. MPs announce a third round of indicative voting for Monday.

    An utterly bizarre thing to do being as the EU were explicit this morning that if we wish to seek an extension, then by Monday we must produce a fully detailed plan for how long, to achieve what, acceptable to the entire EU27 and it must be provably achievable, in order that they have enough time to consider a response and deliver it on wednesday.

    So does the glaringly obvious incompatability mean that the Remain MPs think the EU are liars?

  15. @ BANTAMS – The tricky issue with long extension to “No Deal” is the EP elections.

    One way to “fudge” that is to pay for said extension but legally have left the EU on 12April.

    Transition is “agreed in principle” and full BrINO anyway but EC-EU27 are v.v.v.unlikely to “lose face” and agree to a “slimmed down” WA at this late stage (as it would mean no “backstop” and throwing RoI under the bus)

    The other “fudge” some folks mention is to keep current MEPs on for extra 9mths or something similar. That might involve less lose of face for EC-EU27 but is pretty dodgy on the legal side of things.

    Macron+co. don’t want the blame for “No Deal” but finding a “fudge” around the EP election issue is not going to be easy.

    We might find out more about UK side when cabinet meeting ends and we see if any resignations or at least see what they’re planning next.

  16. Anthony

    This is really informative an analytical post.

    Thank you.

  17. Well, I didn’t see that coming. Will Corbyn play ball?

  18. Bit late. Attempt to share the blame – a customs arrangement & confirmatory vote (and General Election?) wrapped in with her deal will do nicely

  19. Classic Mayb0t can kick – she can’t make a decision so back to asking Corbyn and EC-EU27 for “help” (ie Groundhog May v17)

    I bet Corbyn and Macron+co loved that :-) :-)

    I had to check it wasn’t still April 1st ;)

  20. May playing party politics in order to keep the Tory party together.

  21. The short extension is a trap since once we get to 22nd May no further extension is possible. I do hope Corbyn does not agree to this.

  22. Now Hireton you know it is only Labour, SNP, DUP etc who claim party politics. By definition, the Tories must always act in the National Interest.

  23. Yes, Nickp, quite right. Someone trying to be too clever by half. Eight hours trying to avoid making a cabinet-splitting decision.

    I would add: “Very sweet of you, Theresa, to try to involve/implicate us. On second thoughts, I’d also like to add in (to what I-agree-with-Nick said) EEA membership. I hope that’s ok. If not, I’m awfully afraid we can’t help you any more. Ever, Jeremy”

  24. Labour’s wants are well known – customs union, closer alignment, people’s vote, GE not necessarily in that order

  25. Sorry, should have read play party politics.

  26. Great summary Anthony – many thanks. Very pertinent as well.

    Movement again today on Brexit it seems. May wants ‘as short an extension as possible’. It’s a further signal, if ever people needed one, that this government and this Parliament will not want and not allow a no deal exit. She hasn’t actually asked for the 22nd, but only said that a deal needs to be agreed before then so we don’t need the EP elections.

    The question is the response from the EU. May 22nd was previously ruled out unless a deal had been passed, due to fears about the EP elections, so it’s a struggle to think that they would revert to the 22nd. My guess is that May will ask for the 22nd but be offered October (or whenever) but that we must have the EP election issue sorted. Or maybe a flexible one to end when the deal is passed.

    A GE wasn’t wanted, and also it seems clear that May is finally abandoning the appeasement of her right wing and is now ‘reaching out’ to Labour.

    Lots of water still to flow under numerous bridges, but the no deal concept does seem to be slowly subsiding.

  27. Thinking about it – might be a double trap – spread the blame AND run down the clock while Parliament is distracted.

    Maybe JC should say we need a longer extension.

  28. Very interesting and informative article Anthony. Thanks.

  29. I believe that Jeremy Corbyn should go to talk with TM, and demand that she asks for a long extension. He should also ask to join TM in the approach to the EU for this extension, and he should argue for the SNP (Blackford or Sturgeon) to be included.

    This cross-party attempt at a resolution and would be more impressive to the EU than May going alone. And solving the impasse could involve the Scottish government doing different things to rUK especially on immigration.

  30. @valerie

    Sorry, I should have remembered that. :)

  31. Bantams,
    I really think you should have a little think about what you have written above
    “Q2 reversed” is no such thing. It uses the extremely pejorative word “thwarted”

    It is very hard to think of a proper reversal of such a loaded question, but I would go for:
    “Arch brexiteers are pursuing an extreme agenda trying to force a damaging no-deal Brexit. The only way to break the deadlock in Parliament is to let the People decide via a referendum”

  32. @ HAL – I agree it is a “trap” but the date is 12April and more like 8th, max 10th. Happy for Remainers to think it is 22May though ;)

    May knew Cooper+co had a motion set for tomorrow so she’s having another go at “trust me”. It really is tough not to laugh at how obvious her “cunning plans” are becoming.

    Corbyn had no choice but to accept (as he has). Maybe we get a miracle and they agree on something that cabinet and then HoC also accept all in the time left but more likely they’ll just try to another round of “blame game” and which party split the most.

    ERG(+DUP) obviously need to visibly furious but the Essex Research Group seem pretty happy with the latest “cunning plan” to tick down the clock – mildly disappointed Tusk didn’t respond as hoped though.

  33. trevor warne: Corbyn had no choice but to accept (as he has).

    If he had not walked out of the meeting with May, where Chuka Umunna was present, I reckon he could have turned down the present meeting, called May’s latest plan demented and said that a new PM is required.

  34. Jeremy Corbyn has given a serious interview to the media. He put emphasis on the Customs Union making sure that jobs are protected and supporting the Good Friday Agreement. He emphasised the need for certainty and security about the future, and a jobs-first emphasis.

    He said that whether or not the UK takes part in European Elections is not important.

    He said it was important to find a solution that works not just for those who voted leave but also those who voted remain, and to make sure that the UK did not crash out.

    He only found out about Mrs May’s invitation by watching it on TV.

  35. He also said it was important to have a strong relationship with the Customs Union.

  36. Sorry I meant to say that he also said it was important to have a strong relationship with the Single Market.

  37. @Trevors – “Classic Mayb0t can kick – she can’t make a decision so back to asking Corbyn and EC-EU27 for “help” (ie Groundhog May v17)”

    and ” I agree it is a “trap” but the date is 12April and more like 8th, max 10th. Happy for Remainers to think it is 22May though ;)”

    I’m not sure you’ve understood what has just happened. May has just abandoned the ERG and DUP in effect, and is pivoting to a soft Brexit. Her cabinet was split on a long extension, hence her non asking for the 22nd, which has already been rejected. I fully expect the extension to be much longer than the 22nd, with a get out clause if we agree the WA in the meantime, but with a requirement to hold EP elections, which May will accept. The flexibility in the extension will enable May to tell her cabinet members who voted against a long extension that it is only a long one if we can’t agree something.

    Corbyn, meanwhile, has stepped up and played this well. His statement lays out clearly that Labour don’t see participating in the EP elections as a big worry (code for ‘let’s have a long extension’) and has made clear his outright opposition to a crash out Brexit. May has nowhere left to turn, so a Brexit on Labour terms – maybe even with a confirming vote – now seems pretty much nailed on.

    The ERG are furious. Last night they were cock a hoop, telling journalists that parliaments failure to agree to anything meant a hard Brexit on the 12th. They failed to understand (as I think you also do) that this won’t happen with May and this government.

    You really need to understand this. The fundamental point is that if a no deal exit was remotely plausible, it would have already happened. May is seeking a lifeline – any lifeline – even a compromise with Corbyn – to avoid no deal, to the extent that she has all but given up on the majority group in her own party.

  38. @Nick P

    “Thinking about it – might be a double trap – spread the blame AND run down the clock while Parliament is distracted.

    Maybe JC should say we need a longer extension.”

    From his interview he is definitely heading that way.

    I don’t think it’s a trap. May has simply run out of cards to play. As a result she had to reach out to Labour. There was no other choice unless you believe she wants no deal. Of course she wants to share the blame but it’s hardly a masterstroke to follow the only viable course left.

    TM has made it clear that if she and JC can’t agree (and they won’t agree – that is certain) then the matter will be decided (or not) by indicative votes. This process will take a while and certainly overshoot 10th April, and so the UK will have to participate in the Euro Elections and the EU will grant a long extension.

    Once that long extension has been secured, we’ll end up in an unusual place where the ERG will demand a referendum to prevent a soft Brexit!

  39. POLL ALERT: YG

    “What do the public think might break the Brexit deadlock?”

    Excellent write-up with link to tabs

    https://yougov.co.uk/topics/politics/articles-reports/2019/04/02/what-do-public-think-might-break-brexit-deadlock

    Note the difference between what folks “want” and what folks think as “likely” – hence the key is so be to seen to have tried and get the “blame vectors” ready.

    PS Final graphic has “No Deal” ahead of “Remain” (similar to Opinium’s tracker). Whilst we’re pretending to try and for a “national unity govt” please make sure to check the x-breaks in the tabs for how polarised the two main parties are.

  40. Winston Churchill Quote from November 1942:

    Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.

  41. PS: My apologies to Nick Soames if he has already said what I have just posted.

  42. FWIW, I am currently in Narbonne where the few remaining members of the legitimate Spanish government escaped to after the fall of Catalonia.

    It took nearly 50 years for the Spanish government to evolve into a set of human beings. Would any UK subject expect the current UK government to evolve that quickly?

  43. PPS: May hasn’t played Danny’s script to the letter but I hope others will at least recognise that what May has been trying to achieve has proved to be pretty close to it.

    If it wasn’t obvious to her fellow Cons then is it unsurprising to many who post here?

  44. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/791594/CDL_letter_to_the_Electoral_Comission.pdf

    I am able to confirm that Cabinet Office will reimburse reasonable spending by Returning Officers on contingency preparations for European Parliament Elections. We are today issuing the following advice to Returning Officers:

    Following Friday’s vote in the House of Commons, the opportunity to guarantee that theUK would not participate in EP elections has been removed.

    Cabinet Office is therefore confirming that Returning Officers will be reimbursed in the usual way for any expenditure on activity that is necessarily undertaken, at this stage and in the coming weeks, to prepare for the possibility of European Parliamentary elections on Thursday 23 May 2019. An Order setting the Appointed Day of Poll would be required under the European Parliamentary Elections Act 2002 to hold elections.

  45. @Joe James B

    “Dreadful.. Meetings.. Meetings. no decisions or ideas. This really needs to be knocked on the head now. The country wants to be inspired.”

    I’m afraid inspiration left the room some time ago. Even dyed-in-the wool Tories like Anne Widdicombe are now saying that May is an atrocious Prime Minister and the worst in living memory. Coming from someone who can barely spit out Tony Blair’s name, such is her visceral loathing of him, Widdicombe’s withering condemnation is extraordinary. The Cabinet meeting today, leading to another empty, self-serving and insincere statement from May, delivered in the usual chilling and charmless way of hers, encapsulates all that is so wanting in our current government and the leadership of the Prime Minister. There she was this evening, after all that has gone before, and at a time of acute national crisis, batting for the Tory Party again. 8 hours Cabinet discussion on Brexit and the outcome? A wheeze to trip up Corbyn and get Labour’s fingerprints on the unfolding disaster. That’s what you got from the country’s leadership today. A last chance to rise to the occasion and, as you say, inspire and lead, and we get some Tory smart-alec strategist stunt to keep the Tory Party hanging together for a few more days.

    For that’s what this is. The civil servants have already leaked that all the recent Cabinet discussions on Brexit have prioritised the interests of the Conservative Party over those of the nation, and I would imagine today’s was have been no different. Set traps for Corbyn, keep the ERG on side and, somehow, hold the tawdry show on the road for a few more days.

    This is the very opposite of leadership and statesmanship. It’s disgraceful, in fact and fast becoming one of the great scandals in post war British politics.

  46. CROSSBAT11

    I agree with all you say, but that begs the question of why May was the only candidate for PM.

  47. Splits both parties?

  48. @Barbazenzero

    “I agree with all you say, but that begs the question of why May was the only candidate for PM.”

    Weren’t the runners and riders in 2016, after Cameron’s resignation, May, Leadson, Gove, Crabb and Fox, with all but Leadson either being eliminated in the first ballot or dropping out? It then became an anointment after Leadson withdrew from the run-off before it went to a vote. Doesn’t that say it all, when you look at the stature and ability of those who half-hardheartedly contested May’s ascension to the vacant throne? The Party didn’t do the necessary due diligence by testing May’s leadership abilities in a proper contest against plausible alternative candidates. Instead, the rival candidates were all implausible for a variety of reasons and the hustings and campaign non-existent. Hence May’s elevation to leader and PM.Without breaking sweat.

    In a healthy political party with a diverse membership and a reservoir of talented politicians of stature to call upon, someone like May would never have got anywhere near to being a leader, let alone a PM. The fact that the Tory Party lacks both of the qualities I’ve mentioned is the reason that we now have May as the leader of our country.

    It’s a tragedy that we do.

  49. May will leave fairly soon and could well be replaced by a Brexiteer. If May does move to a softer Brexit the PD will be changed to reflect this.My memory of the PD is vague but it will give a good steer to the final outcome and will not make it easy / possible to change the direction of travel

  50. So -Boris & Rees Mogg completely pi88ed of . ERG reps all over TV news saying this is betrayal.

    Yep-May puts Party before Country again.

1 2 3 40