Over on the British Polling Council’s website there is a guide for journalists writing about opinion polls, written by YouGov’s Peter Kellner. The key point on it in number 13, advising journalists on what to look for in a poll – was it conducted by a reputable agency? What was the sample size? Was it properly sampled? Who was it conducted FOR? Peter writes:

In either event, watch out for loaded questions and selective findings, designed to bolster the view of the client, rather than report public opinion fully and objectively.

Now, a common criticism of polls is that they get the answers the company commissioning them want. I consistently argue against this accusation – it is the duty of a professional pollster to measure and report public opinion as it actually is, not as we would like it to be. It is our duty to craft questions that are fair and unbiased and accurately reflect public opinion.

Of course, most people could come with examples of questions that could have been better – I certainly could. I would contend these are cock-ups rather than conspiracy, simply because of pollsters’ professional dignity. A Bob Worcester, a Peter Kellner, an Andrew Cooper or a Martin Boon simply wouldn’t sign off a question they thought was biased (and, I hasten to add, neither would an Anthony Wells)

So why does the person who commissioned the poll still matter? Well, because they choose what the questions are asked about. The pollster should ensure whatever questions asked are unbiased, but it’s the client who choses what areas to ask about, and few clients commission polls they expect to damage their case. Hence, a client campaigning for tougher sentencing might commission a poll asking if people want to see longer prison sentences (since they do). A client who supports sentencing reform however might commission a poll asking if people thought prison was effective at reforming criminals (as they don’t). The broader picture is that public support long sentences, despite not thinking prison is particularly good at reforming or rehabilitating criminals. They like it for retributive reasons. However, if you saw only the questions commissioned by the imaginary pro-prison client, or only the questions commissioned by the imaginary anti-prison client, you wouldn’t know that, you’d only get one side of the story.

There are two lessons to take away. One, look at polls in the round, not in isolation. Don’t cherry pick those that tell you want you want to hear and ignore the others, the differences between them tell a story. Two, be careful with polls commissioned by partisan campaigns – if the pollsters are doing their job properly the questions will be fair and balanced, but they won’t necessarily look at the issue from all sides. Ask yourself what questions were not asked – if a poll is on the subject of a policy, proposal or suggestion and doesn’t ask whether or not people actually support that policy, ask yourself why they didn’t commission that question. Sometimes it might just be because there are a million existing polls on the subject. Sometimes it might be because they didn’t think they’d like the answer…


136 Responses to “The questions NOT asked…”

1 2 3
  1. First?
    Sorry.

  2. An excellent article. Very relevant to some discussions on LibDemVoice at the moment where I find myself wishing “If only they’d asked such-and-such a question.”

  3. Well it’s always the dogs that don’t bark in the night that are significant.

    There’s an old statisticians’ remark about someone using statistics “like a drunk uses a lamppost – for support rather than illumination” and an awful lot of polling seems to be requested on this basis nowadays. I suspect a lot of it has to do with the background of the media-political classes. Most come from a PPE or legal background plus youth Party or NUS politics, the Oxford Union and so on. So they tend to see their job as advocating or defending a particular position rather than finding out about a situation and taking action accordingly – something a background in science or the social sciences might encourage.

    Hence the sort of questions that get requested and the results that get used. And of course it also leads to the obsession with spin and appearance. The true reality is deemed to be what is reported, not what actually happens.

  4. “A Bob Worcester, a Peter Kellner, an Andrew Cooper or a Martin Boon simply wouldn’t sign off a question they thought was biased (and, I hasten to add, neither would an Anthony Wells)”
    _______

    In the case of Andrew Cooper of Populus, the evidence suggests otherwise. Here’s the link to a January 2011 poll commissioned by Asda from Populus at a time when the Scottish supermarkets were opposing plans for a “Tesco tax” in Scotland. I challenge anyone to come up with a more leading, and therefore more biased series of questions than these.

    http://www.populus.co.uk/Poll/SNP-Retail-Tax-Poll/

    Incidently, shortly afterward this poll, he left Populus to take up a post as senior political aide to one David Cameron.

  5. I’m not going to go into individual polls – I’m sure every pollster has carried out, shall we say, sub-optimal polls if you look through enough, and I’ll let he who is without sin cast the first stone.

    However, the CEO of the company doesn’t necessarily personally sign off every poll. Andrew is one of the best.

  6. Anthony,

    Why not just paste the Link to the “Yes Minister” Clip again.

    It sums it up perfectly.

    Peter.

  7. @AW

    “Ask yourself what questions were not asked”

    Excellent advice, and if I may pop my toe in the water of a topical subject (but will refrain from trying to hijack the thread), how about an additional question in a Northerly region’s 2014 political future:

    “Do you feel you have been given enough facts to make a decision one way or another?”

    While it’s not going to happen I would bet that such a question would get the highest percentage of all. Another general question I would like polling companies to ask:

    “Which parties have you voted for in the past?”

    Which might give a better idea of how willing the people polled are to change their vote. If a high percentage never change their vote, there’s a case for parties to ignore siad poll (i.e. it is a valid poll, but not one that can measure the likelihood of the nation to change their vote).

  8. Much more important than all that, at Euro 2012 if Germany win their group and Greece come second in theis, we could see a real grudge match in the quarters.

  9. Peter – that’s somewhat different. That’s choosing what to ask in order to influence responses to later questions, and is the sort of thing pollsters would (hopefully) refuse to do. Were Sir Humphrey to come to me with those questions, the first thing I’d do would be to put his final question first…

    Statgeek – “which parties have you voted for the in the past” is something that definitely has been asked. Let me see if I can track down an example.

  10. Of course, it’s all well and good telling journalists to take this approach, but we know full well that many in our mass media do not actually want to provide balanced and neutral reporting, and will happily use faulty data to reinforce their personal/organisational prejudices or ideology.

  11. A good article, Anthony.

    From 1985 until 2003 I worked for three different polling companies. Each was, on the whole, highly reputable.

    Whilst I don’t think any of the questions that got asked were biased, I did come across two incidents in which there was what could be interpreted as selective reporting or suppression of results in order to favour the ideological view of the client.

    Perhaps I can tell you about the most memorable of these without giving the name of the company (and by the way, I have previously mentioned working at MORI – well it WASN’T them, so hopefully you won’t have to edit this for legal purposes):-

    A client who headed a professional organisation had commissioned us to do a survey about their members’ attitudes towards joining the Euro. The questions were entirely balanced and for the most part were of the agree/disagree strongly/slightly variety.

    When the results of the pilot survey started to come through it emerged that a plurality of this organisation’s members were ‘slightly’ in favour of joining the single currency (and, when added to those who were ‘strongly’ in favour, the total number of ‘agrees’ out-numbered the total number of ‘disagrees’).

    The reaction of the client was to storm out of the research centre shouting “Cancel the survey, we might as well hand the whole place over to the Germans”, and so the survey was duly cancelled and no results were ever published. (By the way, that person continues to be a public figure, allbeit in a different capacity).

    As a pro-European, I think that’s a disgrace.

    So I would contend that there is more to this issue than simply asking unbiased questions. It proves that there are other ways in which clients or publishers with an ideological bent can bias the flow of information about public opinion.

  12. NICKP

    @”we could see a real grudge match in the quarters.”

    Germany will win on fiscal penalties. The Greek team will vote to reject the result……..but to stay in the competition.

    UEFA will convene a meeting in Rio-fixed for January 2013-both teams to stay on the pitch until then.

  13. @ROBIN HOOD

    “The reaction of the client was to storm out of the research centre shouting “Cancel the survey, we might as well hand the whole place over to the Germans”, and so the survey was duly cancelled and no results were ever published.”

    Hence why we, the UK electorate do not get a referendum on the EU. The politicians don’t want to ask the question. They neither want to have their powers taken from Westminster, nor want to be more seperated from Europe.

  14. @COLIN

    Could we see several instances of extra time, as the decision makers in UEFA struggle to find a solution?

  15. STATGEEK

    Undoubtedly :-)

  16. @Anthony Wells

    Fine, given your comment I’m happy to redirect the focus of my criticism above against Populus as a company rather than against any one individual. The point being that, in the recent past, Populus were willing to take the money to ask what appear to me to be a very biased series of questions. A month later the supermarkets got their way when the proposals of the Scottish Government were voted down. For acting as they did in such a sensitive political context, Populus deserve to be called out for that poll.

  17. Of course it’s not just what you ask, but who you ask as well. There’s an interesting piece on the Guardian datablog about a poll just out on LGB attitudes to marriage and civil partnerships:

    http://www.guardian.co.uk/politics/reality-check-with-polly-curtis/2012/jun/08/gay-marriage-reality-check

    The poll was done by ComRes for a group called Catholic Voices (you may remember that they have form on this topic). Of course the subsequent presentation of the results by Catholic Voices, the Mail and the Telegraph is exactly what you would expect and the Guardian article and comments pick up a lot of the methodological problems and presentational sleight of hand. But I’d like to pick up problems of who was included/excluded in the questioning.

    Firstly as well as people self-identifying as (male) gay(194), lesbian(55), or bisexual(219) they also included 61 people who ticked the ‘other’ box. Now is no reason to assume that these would have similar responses to LGB respondents – even assuming that they didn’t just tick other by mistake or see what would happen.

    More important no further questions were asked of those 9,348 identifying as “Heterosexual or Straight” (and let’s not even go near the whole “men who have sex with men” problem). But this meant that there were no comparisons to show if the LGB attitudes were unusual or not and if so by how much.

    Now I’m unsure whether this was done for cost reasons (but surely only a sample needed to be used) or because Catholic Voices were scared of the answers they would get. Imagine if a majority of the population agreed with True marriage equality would mean that same-sex couples could marry in places of worship as well as in civil locations or even Faith groups should be forced to allow gay weddings in places of worship.

    Incidentally there’s a new horror to add to Anthony’s hatred of ComRes’s Agreed/Disagree questions. The options for a lot of the questions in this survey were instead Applies/ Does not apply/Don’t know. But this meant that the negative response to for example “I am not in a Civil Partnership but would seriously consider it” included those already in a CP as well as those opposed to the idea[1]. This a whole new level of incompetence and must stop now.

    [1] I’m not even beginning to consider how those LGB people such as Brian Paddick who would consider themselves already married (to a national of a country that offers it) should have answered.

  18. I think the Germans will insist on choosing the Greek manager, and Captain, and also insist that Greece aren’t allowed a goalkeeper on the grounds that the last one they had made too many saves and had overused his allocation.

    The referee will of course be a neutral Austrian or perhaps Turkish.

  19. @Nick P

    In which case it would be perfectly understandable for Greek fans to respond to such adversity by spreading their arms and humming the “Dambusters” march. Harmony and respect between European nations, as delivered by the Euro.

  20. The Germans will win because they have a better team, stronger more skilled more disciplined and able to stand the pace including extra time and the tension of penalties.

    The Greeks want that kind of team and used to think they had, but they don’t.

    They might have some day but it will take a lot of hard work and pain and it will take time.

    For now they just have to try their hardest and learn how it’s done even if it means getting a bit of a kicking, as the alternative is to drop out of the competition and play in the lower league where they can go back to pretending they are great!

    Peter.

  21. perhaps if Germany scores too many goals, they will lend some to Greece.

  22. @Gwuk
    “Of course, it’s all well and good telling journalists to take this approach, but we know full well that many in our mass media do not actually want to provide balanced and neutral reporting, and will happily use faulty data to reinforce their personal/organisational prejudices or ideology.”
    ________

    See for example evidence of Richard Peppiatt, former freelance reporter to the ‘Daily Star’ who on 29 Nov 2011 told Lord Leveson:

    “And so whatever a story may be, you must try and adhere to their ideological perspective. Say there is a government report giving out statistics. Well, you know, any statistics which don’t fit within that framework you ignore or sort of decontextualize and pick maybe the one statistic which does. If there’s something that comes out saying crime has gone down, you then go and look for the statistic which says knife crime has gone up 20 per cent…”

  23. Anthony, Thank you, again for your wonderful work.

    On the football.

    Do people seriously think that a reserve player for a mid table club, Kelly from Liverpool, and a full back as well, was a better selection that Rio Ferdinand as cover for the centre half place for England?

  24. @ Peter Cairns

    The Germans will win because they have a better team, stronger more skilled more disciplined and able to stand the pace including extra time and the tension of penalties.
    ——————————————————-
    If this is an analogy for the eurozone crisis, IMO, you are mistaken. The Germans will not make it into ‘extra time & the tension of penalties’ because their economy is faltering & there are rumours that some of their major banks could be in trouble.

    Politically speaking, in the home league Merkel’s team is already a goal down, completely out-classed in midfield & Merkel is screaming at the manager. She appears to be hoping for a break-away to pull them back to level pegging.
    8-)

  25. I expect the German team is vulnerable to a concerted attack on the left wing.

  26. Amber,

    If the German economy can be described as Faltering, ours must be on life support!

    Peter.

  27. @Peter,

    Germany = Faltering

    UK = Faltered

    Greece = De-Fa(u)lted

  28. It will be interesting to see how the various “piggies” with bankrupt heavily indebted economies perform in Euro 2012 – by “piggies” I mean Portugal, Italy, Greece, Ireland, England and Spain. If one of them wins, will this help its economy? I include England/UK, as it has a poorly performing economy and a large per capita debt (greater than the other 5), but its situation is different as it isn’t in the other Euro.

  29. @ Peter

    I was thinking of another F word regarding the UK but this is a polite forum, so I will let people decide for themselves what it is.
    8-)

  30. The problem is that, no matter how good the intention of the pollsters are, the majority of the public don’t bother looking at the detailed polls. They only notice what’s stuck in front of their noses and I suspect only a minority bother to consider whether they are looking at cherry-picked information.

    This is where the real power of the commissioning bodies lies. It’s hard to sneak a loaded question past a polling company, but easy to selectively interpret the results afterwards to prove whatever it was you wanted to prove. The papers aren’t much better, sticking the sensational results on the first page and burying the boring ones (probably reflecting reality better) on page 17.

    In principle, I would rather polling companies steered clear of anyone with any vested interest at all. If only reality made that possible.

  31. Faultless?
    Flawless?
    Fool proof?
    Fastidious?
    Felicitous?
    Fairly good?
    Flippin great?
    Fantastic?
    Fine?
    Fabulous?
    Favourable?
    Feasible?
    Fecund?
    Funky?
    Fit?
    Flourishing?
    Fructiferous?
    Fulgent?

  32. Roger Mexico

    “There’s an old statisticians’ remark about someone using statistics “like a drunk uses a lamppost – for support rather than illumination” ”

    George Bernard Shaw, I think.

  33. Mark Twain?

  34. Andrew Lang

  35. Here’s a stonking great tome on the outlook for Sunday’s French parliamentary elections, curiously entitled a “mini-guide”. In terms of length, think Virgilio and multiply by a factor of about 50.

    http://welections.wordpress.com/2012/06/07/france-legislative-elections-2012-guide/

  36. While we are on the subject of footie- were there any polls during 1970?

    There has always been the Alf Garnett joke about Wilson getting the team to play in red so people would think Labour had won the World Cup in 1966, but semi seriously, unlike the discussion about the jubillee, I could actually see a feel bad factor 4 days after being knocked out by Germany in 1970!

  37. More bad news for the government tomorrow http://yfrog.com/odd7jngj

    Will this have an effect on tomorrow’s VI or has the fieldwork for that already been done?

  38. Liz,interesting,but not sure exactly why they are investing this.is osborne being
    Called to appear at leveson..sorry about lack of capital letters.still not used to
    I pad.

  39. Today Obama talked about European unemployment rates hits 20 or 25%, using the phrase ‘too much austerity too quickly’.

    His intervention today was remarkable – the equivalent of picking up a megaphone and shouting very loudly at the European leaders to get on with a solution. It’s also remarkable as it sounds suspiciously like ‘too far too fast’.

    Ed Balls is rapidly becoming the new economic orthodoxy.

  40. @ Colin

    Fool proof?
    ————————
    If only… :-)

  41. Keeping strictly to the thread topic, I don’t think the question of who commissions a poll is particularly important as long as the questioning and sampling methodology is sound and the commissioning organisation is prepared to report the poll’s findings fairly, even though they may dislike what it is telling them. Of course, what media outlets with their own particular agendas and political slants do with polling data is entirely another matter and there have been numerous examples in the past when a poll has been reported only partially and presented in a tendentious way. In other words, we should be more suspicious of those who report polls rather than those who commission them.

    I suppose an individual poll only ever represents a snapshot of opinion within normal sampling variation and statistical margin of error parameters. It’s quite possible that we all read too much into them on occasions, especially when it’s not unusual to see wide differences between different pollsters on something so basic as voting intention. Angus Reid and ICM are, no pun intended, currently poles apart, yet they are basically asking the same question.

  42. I wonder whether the Police investigation into the Tories cash for action, will actually go anywhere. What law would have been broken and would the CPS want to consider any prosecutions. Is lobbying and paying money to have dinner with senior politicians a crime ?

    I suspect that because a complaint has been made, the Police have reluctantly agreed to investigate.

    Whatever the rationale for this, it could hit the Tories polling for a short period, if coverage in the media becomes more widespread.

  43. Doh, I meant cash for access and not cash for action.

    Cash for action would definitely be illegal.

  44. Colin: Your original footie jest was excellent.

    My weedy thought is that England should head home with the ball when they realise Europe’s involved, and face an honourable disqualification. Its our flippin’ game after all.

  45. PAULCROFT

    You can only take the ball and go home with it if it’s your ball.

    Even then, you will be universally seen as being dishonourable (more appropriate terms may be used).

    Re the current footie competition, it seems unlikely that any of our neighbours will proceed beyond the group stages, so I’ll support our neighbours as appropriate in each group.

    A – No neighbouring nations
    B – Denmark
    C – Ireland
    D – England [1]

    [1] That assumes that the TV commentators for what are supposed to be UK channels don’t refer to England as “we” or “us”. Should they be so disrespectful, I’ll need to transfer my support to Sweden – not that it’ll make any difference – neither will proceed past the group stage.

  46. Good Grief Ole Nat, I’ve listened to many Scottish commentaries [thanks Stew Lee] and not only found the “we” totally inoffensive but, back in the days when gallant, plucky, unlucky Scotland got through to tournament finals found myself self shouting [OOT LOOD}

    “Aye, c’MON Archie !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!”

    as he dribbled past fifteen Dutch defenders and slotted the ball past the goalie

  47. @ Anthony

    “Now, a common criticism of polls is that they get the answers the company commissioning them want. I consistently argue against this accusation – it is the duty of a professional pollster to measure and report public opinion as it actually is, not as we would like it to be. It is our duty to craft questions that are fair and unbiased and accurately reflect public opinion.”

    Don’t tell that to Scott Rasmussen! :)

  48. YouGov has today a very interesting VI poll on tomorrow’s French GE. Whereas it does not differ fundamentally from the polls published by the French institutes, it has nevertheless some slight, but interested, if confirmed by actual voting results, divergences. The numbers are the following:
    Socialist+Green 39 (French institutes: 36.5-39)
    Left Front 6 (FI: 7-8.5)
    TOTAL PARLIAMENTARY LEFT 45 (FI: 44.5-46)
    UMP+Various right 32 (FI: 33-35)
    National Front 18 (FI: 14-15.5)
    Center-MD: 3 (FI: 2-3)
    Extraparl. Far Left 1 (FI: 0.5-1.5).
    The main divergence (albeit within MoE) is that YG gives higher scores to the governing alliance of Socialists, Green and various left and to the NF, in detriment of the Left Front and the UMP respectively. Yet this difference might signify 10-30 seats more for the Left, and this might be lead it from a relatively narrow to a very comfortable OM in June 17. Note also that 39 and 18 are all-time highs for Socialist+Green and FN respectively, whereas 32 is all-time low for the Right. 6% for the LF is rather disappointing, compared to Melenchon’s 11% in PE, the Communist Party had 4.3 in 2007 and 4.5 in 2002, but these were its all-time lows, its average percentage between 1986 and 1997 was 9.5% and before 1986 it had double-digit scores (all-time high: 28.2 in 1946). It seems that 35-40% of Melenchon voters now intend to vote for a Socialist-Green candidate, whereas the voters of Bayrou (9%) are split in 3 equal parts between his centrist MD party, a Left vote and a Right vote. Note also that if the FN actually polls 18%, it means it can maintain its candidates in as many as 100 constituencies out of 577 for the runoff (a candidate that has more than 12.5 of registered voters at 1st round can remain in the runoff even if he/is not first or second), thus provoking the so-called “triangulars” that are usually fatal for the UMP candidates (it is to a great extent because of them that the Left got OM in 1997 GE)

  49. @ Paul Croft

    “My weedy thought is that England should head home with the ball when they realise Europe’s involved, and face an honourable disqualification. Its our flippin’ game after all.”

    Is this why Canadians riot whenever one of their teams loses the Stanley Cup?

    The LA Kings are up 3-1 iin the Stanley Cup Final right now (they were up 3-0). Tommorow they play a Game 5 that could prove decisive, if they win they win the championship, something they’ve never done before. Actually until this series, they had only been to the Cup once and I think they were swept during the series. I don’t really care at all about hockey but I’m of course rooting for them to win out of civic duty and civic pride. To see them win the Cup would bring a great deal of pride and joy to the city.

    Re the current footie competition, it seems unlikely that any of our neighbours will proceed beyond the group stages, so I’ll support our neighbours as appropriate in each group.

    “A – No neighbouring nations
    B – Denmark
    C – Ireland
    D – England [1]

    [1] That assumes that the TV commentators for what are supposed to be UK channels don’t refer to England as “we” or “us”. Should they be so disrespectful, I’ll need to transfer my support to Sweden – not that it’ll make any difference – neither will proceed past the group stage.”

    I’ll root for England first (unless there’s a Scottish team in the mix).

    On your footnote……that reminds me of how angry I sometimes get when I hear legal information misreported in the media. Not much I can do about it. Lol.

  50. @ Virgilio

    We in CA had a major Lionel Jospin moment this past Tuesday, the pain of which you would appreciate.

    The state of California has adopted a new open primary system where basically, everyone runs on the same ballot and the top two finishers advance to the general election, regardless of parties. So there were quite a lot of wild races throughout the state where multiple candidates from the same party were running for the same seat. A full on electoral free for all. Especially coupled with extremely low turnout.

    The boundaries of the state’s Congressional seats have been redrawn as well further adding to the electoral free for all. Three seats are virtually open and are great pickup opportunities for the Democrats: CA-31, CA-41, and CA-47. Or they were prime pick up opportunities.

    That’s because the top Democratic recruit in CA-31, Pete Aguilar, pulled a Jospin and finished in third place last Tuesday. The top two finishers are Republicans. So a district that voted for Barack Obama by 15% and will likely vote to reelect the President by an even larger margin is now also guaranteed to elect a Republican to Congress.

    Thought you’d appreciate (or commiserate).

1 2 3