A month ago I wrote a guide to How Not to Report Opinion Polls. I have a history of starting regular features and then failing miserably to deliver them, but I am at least going to try to come back to it at the end of every month and highlight particularly poor reporting in the weeks just gone by.

For July I’m going to start with this report from the Independent, claiming that a ComRes poll shows “Dramatic change as two-thirds now support GM crop testing” and that “Public opinion appears to be shifting in favour of the development of genetically-modified crops, according to a ComRes survey for The Independent”.

ComRes found 64% of people agreed with a statement that “Experiments to develop genetically-modified crops should be encouraged by the Government so that farmers can reduce the amount of pesticides they use”. However, the article doesn’t mention any past results that it can be compared to in order to justify the claim that there has been a dramatic turnaround in support for GM crops.

Historical polling on the issue by MORI here does indeed suggest a much lower level of support for GM crops, but these differences could easily be explained by the question wording – MORI was asking things like “How strongly, if at all, would you say you support or oppose genetically modified food?” while ComRes’s statement specifically links the development of GM crops to a positive outcome of reducing the use of pesticides.

To see the impact asking different questions could make, look at this more extensive polling on the issue by Populus. Asked a generic question on whether or not GM food should be encouraged 27% of people say yes, 30% no. However, if you look down the survey to page 38 it asks specifically about whether people are supportive of using GM wheat to repel aphids and reduce the need for pesticides and finds 58% of people are supportive of this specific use. It seems plausible that the reason ComRes found such high support is not because of some great shift in support, but because their question specifically mentioned a popular potential outcome from GM.

If we want to see whether or not public support for GM actually is growing we need to have a question that has been asked consistently over time. This is surprisingly difficult to find – MORI don’t seem to have asked the question above again since 2004. The best I can track down is the Eurobarometer polling here, which every 3-5 years has asked if people agree that GM food should be encouraged. As you can see from the table on the first page, there is no obvious trend in the UK’s answers, support for encouraging GM food has moved by 45% to 25% to 35% over the years. Certainly the picture it shows is not one of a strong trend towards people supporting GM food.

(The Populus poll, incidentally, asked a similar question to the Eurobarometer question, but they can’t be directly compared either, not least because 43% of people told Populus they “neither agree nor disagree”, an option that the Eurobarometer did not offer. This, in turn, was misreported by the Daily Mail back in March.)

Once again, the lesson is to look at the polls in the round, not to take a single finding out of context, especially when it that question is one that is likely to put an issue in a particularly good or bad light. If you are looking at trends over time, you should always compare apples with apples. If two significantly different questions give different results it is as likely to be down to different wording as it is to a change in opinion, especially in cases like this.


50 Responses to “GM food and rubbish poll reporting”

  1. AW —

    Yes this is indeed Rubbish Polling Reporting. As you say, we need to look at trends over time. Anthony, isit true that all the major pollsters seem to have homed in now on about a ten point labour lead, with Libdem on 10% and is the ‘rogue’ ICM?

    When you get a minute, could you update your moving average, I’m sure I’m not the only one here who finds it
    really useful, to look at trends, and compare pollsters. –especially the graphs that are linked to it.

    GB first gold medal — watch out for a polling bost for the incumbents !!!

  2. Andyo – the ICM poll isn’t a “rogue”. ICM and Populus tend to show lower Labour leads for methodological reasons, not because of random sample error.

    (Specifically the reallocation of people who say don’t know tends to help parties that have lost support since the previous election and hinder parties that have gained support since the previous election).

    I shall try to update the averages soon. I don’t do it that regularly because I can’t access the database from work, so I need to remember to do it from home!

  3. AW — thanks Anthony.

  4. Anthony (FPT – Sorry)
    “With the Olympics totally dominating all news coverage the next couple of weeks will have significantly fewer of all those things, so you can imagine how the absence of bad news may have an effect.”

    I thought of these wise words of yours when listening to this morning’s news headlines. Is there not more going on in the world than a few shuttlecocks being hit in the wrong direction?

  5. As for the ComRes political poll, the Indy’s reporting of it wasn’t up to much either. The poll didn’t seem to generate the obviously desired “Osborne must go” response. OK, 22% of Conservatives would prefer Cable to Osborne, but that’s hardly worth a headline (given that 59% wouldn’t).

    By contrast, the interesting but unreported finding that didn’t fit the narrative was that more than twice as many current Lib Dem voters would prefer Osborne to Cable (54% to 23%). Mind, it has to be said (even in Chris Lane’s absence on holiday) that it’s a small sample.

  6. Desperately bad PMI manufacturing data for July. The June figure has been revised down to 48.4 with the July figure even worse at 45.4. This suggests that in this sector, Q2 was worse than the first ONS estimate (June is the month with the least certain data when the figures were released, so this could lead to a further downgrade, rather than upward revision) while the worse figures in July mean that Q3 is getting off to a very shaky start.

    Within the detail, if anything these stats are worse than the headline. Output slumped, which could be partly explained away by weather, Olympics etc, but the decline in exports is accelerating and new orders and the backlog of work also contracted sharply, showing their fastest declines since Feb/Mar 2009.

    Oddly enough, there was a marginal rise in employment which was mainly to complete exiting contracts and to meet planned company expansions. My reading of this is that many companies seem to have expected a pick up and were preparing for this. If these trends continue, presumably the employment pick up will reverse.

    Also further worry in the detail is that although the survey points to a slight rise in output in the consumer good sector, output fell heavily in the intermediate and investment sectors – again suggesting limited expansion in overall economic capacity in the next quarter or so.

    While only a relatively small sector of the overall economy, these numbers are dire. Those who believe the Q2 GDP numbers were wrong have even more explaining to do this morning.

  7. @AW
    Would not sending a copy of your article to the Indy editor not perhaps produce improvements?

  8. The detailed tables for the ComRes survey are here:

    http://www.comres.co.uk/polls/Independent_Results_GM_Crops_3rd_July12.pdf

    Of course the question (only one) that was asked:

    Do you agree or disagree with each of the following statements … Experiments to develop genetically-modified crops should be encouraged by the Government so that farmers can reduce the amount of pesticides they use

    is not only one of Anthony’s loathed ‘agree or disagree’ questions, it’s also open to a wide range of interpretations. For example for some people ‘experiments’ might mean lab-based experiments only; other people might take the question to mean to that current experiments are not aimed at pesticide reduction and want this to change.

    If you genuinely want to know what the public thinks you have to ask a lot of different questions to unravel the complexities and maybe contradictions involved. Unfortunately a lot of current polling seemed more directed to getting the right answer that suits the required narrative – and you keep on asking slightly different questions till you get the answer you want.

  9. I think that the “so that” makes the question both misleading and leading.

    It might mean people saying “yes” when they mean they ONLY approve of developing GM crops for the purpose of reducing pesticide use. So growing a hardy potato in the desert without much water wouldn’t fit that definition.

    The “so that” shoves an acceptable justification in for something has a number of different aims. Begging a yes.

    What if you just asked;

    Do you agree or disagree with the following statement … ?
    Experiments to develop genetically-modified crops should be encouraged by the Government

  10. Gold and Bronze from Wiggo and Froome.

    Ere Wiggo ere wiggo

    etc etc

  11. You guys are having a big day thus far at the Olympics. You won your first gold medal today and just now, Bradley Wiggins won another gold for Great Britain in his individual cycling event (another Brit, Chris Froome, won bronze). Congrats!

  12. @NickP

    I thin the problem is that the phrase “genetically modified” is both inaccurate (selective breeding is also genetic modification) and, unfortunately a phrase that is already loaded with emotion.

    And what about a why? If it’s to breed triffids, then no. If it’s to stop hunger, then yes.

  13. What’s wrong with triffids?

  14. @ Nick P

    You beat me to it! Last week, I was watching the Colbert Report and Colbert had on the British Ambassador to the U.S. to talk about the Olympics. Colbert asked him what sports the Brits were good at. He proceeded to list the various sports and then wryly remarked “we’re good at anything that requires sitting.”

  15. Rowing (2), cycling (3), gymnastics, swimming & something to do with horses and royalty.

  16. NickP – “What’s wrong with triffids?”

    Exactly! Triffid oil is the perfect solution to the world’s dwindling fuel supplies. As long as we aren’t all struck blind by some sort of evil green soviet meteorite storm they can’t possibly pose a threat…

    oh…

  17. Come on, AW.

    What are the chances of that? Triffids are a perfectly fun plant to have, as long as you keep them penned in.

    Health & Safety gone mad.

  18. The ‘sit down sport’ jibe originated from the Australians at the last Olympics.

    So the ambassador either does not possess original thoughts or is using self deprecating humour. Naturally, i hope the latter.

  19. UK/GB can play darts and snooker too. Anything played in pubs. Crown Green Bowling. Not olympic sports, like that synchronised swimming.

    Might still get a medal or two at track & field, you never know.

  20. Should there be a sport based upon luzzing breezeblock at Riot Police?

  21. NYT reports thus, very clear gap at this stage

    h ttp://elections.nytimes.com/2012/swing-state-polls

  22. Anthony: “GM food and rubbish”

    Shockingly biased lumping them together like that- shouldn’t be allowed.

  23. Awsome Wiggo !

    Congratulations Helen Glover & Heather Stanning -well done Oggy Land (and the Army ).

    How about officially changing those 15 Gold Bezants to 16 in commemoration ?

    :-)

  24. New YouGov Scotland poll over on PB.

    “30% are in favour on independence. This represents a decline of 1% since March and a decline of 3% since January.

    54% are opposed to independence. This represents no change since March and an increase of 1% since January.

    However it is over two years until the actual referendum, there is a lot of time for the polling to move.

    Both sides should remember that a year before the AV referendum, an ICM poll had 56% voting for AV and 35% for FPTP.

    There was also some Holyrood/Westminster VI polling

    The poll also reinforces the gap in voting intention between Westminster and Holyrood, with Labour now ahead of the SNP on Westminster voting intentions to 14 points, up from just five points compared to a YouGov poll in May.

    Labour is on 43 per cent, up three from May. The SNP is now on 29 per cent, down six. The Tories are up one point to 15 per cent, while the Lib Dems are up two points to 7 per cent.

    The SNP and Labour’s roles are reversed for voting at Holyrood, where the latest polls show the SNP is now some 15 points ahead of Labour, and has easily held the gains it won at last year’s Holyrood election.”

    Good news there for both centre-left parties, and bad news for the FM… although there is two years to go of course.

  25. ALEX HARVEY

    I’m not sure whether you didn’t read Anthony’s header to this thread – or if you were quoting the PB article as a good example?

  26. @Oldnat – is there an issue with the poll quoted by @Alex Harvey?

    If there is something amiss with the poll, could you specify your apparent criticism of him? I’m genuinely interested in the stated poll findings, but don’t know if this is a misleading summary.

  27. There’s not an issue with the YG poll for the Fabian Society as far as I am aware.

    The issue is with the reporting of it (which is the subject of this thread).

    “an increase of 1% since January.” Is that supposed to have any meaning whatsoever in a poll with a 3% moe?

    I’m interested in the findings too, but presumably (since this is a YG poll) Anthony will be flying the saltire over a different thread on it.

  28. Alec

    Further points

    Was the “March” question the same one as the Jan & July questions? (I may have missed one, but the one I found in the YG March archive is different).

    As guru Anthony pointed out to us ” If you are looking at trends over time, you should always compare apples with apples.”

    (Of course, if AW can’t be @rsed to provide a Scottish thread, we’ll have to discuss it here! :-) )

  29. I apologise if I breached etiquette, I thought that each new thread was to be used as continuation of the previous conversation!

  30. ALEX HARVEY

    I meant no disrespect to you.

    I’d forgotten that you are relatively new on here, and it’s a wee while since Anthony posted a reminder that he prefers each thread to be unique – at least to start with before the inevitable happens! :-)

    However, as AW notes, he has a record of ” failing miserably to deliver”, so things may have changed and its just pedants like me who haven’t realised that. :-)

  31. Roger Mexico

    “Unfortunately a lot of current polling seemed more directed to getting the right answer that suits the required narrative – and you keep on asking slightly different questions till you get the answer you want.”

    That is very much the case with the unmentionable Macbethian question. We know that questions created by partisan organisations designed to support their stance can produce the answers that they want, it’s also possible for partisan organisations to select a question that has been asked previously which they think will help their cause.

    If anyone imagines that the Fabians are neutral on the issue, that view should be dispelled by their comment to the media on the question that they paid YG to ask.

    “It looks like the SNP has blown it. A clear majority of Scots now oppose separation.”

  32. OLDNAT:

    No need to apologise!

  33. The latest wheeze from Boris to showcase London to international investors
    http://m.apnews.com/ap/db_284370/contentdetail.htm?contentguid=MDCnXOU4

  34. Alex, I posted this question to you 2 threads ago, but one minute later AW started a new thread. Perhaps you could reply now that you’ve had a chance to read it:

    Alex said: – “…opposed to independence but super pro-devo… “

    Did you see this piece in today’s Scotsman Alex?

    ‘No third option may encourage Labour supporters to vote Yes’

    “… Labour supporters being driven into the arms of the pro-independence cause if voters are denied the chance of a second question on more powers in the referendum and simply asked to chose between the status quo and a break with the UK.
    There’s a significant body of opinion within Scottish Labour that wants Holyrood to be handed enhanced economic powers to allow the pursuit of policies linked to the redistribution of wealth and greater social justice.
    As things stand it’s hard to see how the current devolution settlement would allow the pursuit of such an agenda.
    A particularly vocal supporter of this stance has been the Labour MSP Neil Findlay – one of the most impressive of the party’s new intake at the last election.
    Mr Findlay is not likely to come out for the Labour for independence campaign, but former Scottish Labour chairman Bob Thomson has said that if he was “pushed into a corner” in a straight yes-no question on whether Scotland should remain in the UK, he would back independence.”

    http://www.scotsman.com/the-scotsman/opinion/comment/andrew-whitaker-no-third-option-may-encourage-labour-supporters-to-vote-yes-1-2441162

    How about yourself Alex? If there is no third “Devo Max” question (and I don’t think there will be), how will you vote? Yes or No?

  35. NickP: “breezeblock/police”

    I’ve passed your copmment on to the relevant authorities: you can expect a visit around 5.00 am

  36. Footy/The “union”

    Having gradually moved to feeling only “English” I’ve been surprised by how much I’ve loved the idea of a GB team playing with such fantastic Welsh players and seeing how well they have gelled together as a team. Shows the England team up in my view.

    I particulary liked Giggs laughing at Bellamy moaning.

    Just a shame ole Ally McCoist couldn’t have got a game.

    pAul

  37. CON 32%, LAB 43%, LD 10%, UKIP 8%; APP -38

  38. Team GB wining Golds, Boris doing what Boris does best, and yet still no boost in the Tories VI

  39. PAULCROFT

    Perhaps a more interesting question might be “Why should sports have anything to do with a particular level of political governance?”

    It doesn’t currently apply in a number of cases.

    I rather like elements of Alfred Heineken’s ideas.

    http://euobserver.com/7/117123

  40. @ CLAD

    I’m guessing Danny Boyle effect hasn’t worn off yet. :-)

  41. OldNat:

    Well, of course it doesn’t, its nothing to do with politics. I prefer SNP policies to the English tories but that wouldn’t make me want Scotland to beat England at tiddlywinks, even if George Osborne was the English captain..

    : Barnard Castle, England, the Uk, Europe are simply places I identify with in different ways – and yet I care far more about a multi-national footy team run by a brilliant Frenchman than any of them.

    And yet….. and yet I warm to the idea of players like Craig Bellamy and Ryan Giggs being given a stage for their talents at international level within a GB team.

    Perhaps I’m soppy but, as Queen Elizabeth I said “I careth not as long as the lads keep winning.”

  42. PAULCROFT

    “its nothing to do with politics.”

    Ah! Your naivety is so refreshing in this cynical world! :-)

    Were your Queen Lizzie I alive in the current era, she might have said something like that – but would probably have been less sexist.

  43. Less sexist than what?

  44. PAULCROFT

    There are more than “lads” competing. There are significant differences between lads and lasses – but this probably isn’t the place to explain them to you.

  45. OldNat: If you and I end up in the same “care” home I shall gnaw my legs off.

    “lads” – a term commonly used to describe the MEN playing MEN’s foorball which Giggsy and Bellamy are part of – being MEN] are participating in.

    I am unconvinced that is is sexist to refer to the lads as lads but, as always am sure you know best.

    I also enjoyed watching highlights of the women/ladies/girls/females beating Brazil but that – as you correctly point out – was probably due also to my political naivety. {Unless your just a grumpy ole curmudgeon and I don’t believe that to be true ‘cos you come over as a very light-hearted ole curmudgeon.]

    pAul

  46. PAULCROFT

    Congratulations on still having legs. :-)

  47. Thanks. Mostly used for lying down whilst supporting the lads.

  48. PAULCROFT

    Oooh! I’m so tempted to respond with a post full of sexual innuendo – but I resist!

  49. Remind me what that is ??

  50. @Stuart Dickson

    How about yourself Alex? If there is no third “Devo Max” question (and I don’t think there will be), how will you vote? Yes or No?

    ————————

    I’m not Scottish, so I can’t vote for anything! Flattered you thought I was though ;)