A couple of years ago I wrote a piece giving advice on how to report opinion polls, or rather, how not to. Look specifically at the third point on being careful of extremely small sample sizes in cross-breaks.

There was a cracking example of the media failing on this front on BBC Look East this week, which has done the rounds on Twitter. The clip is here, the offending portion starts at the three minute mark. It claims to show the results of a poll of the Eastern region that put UKIP on 44% of the vote.

The figures come from page 36 of this ComRes poll. It wasn’t a bespoke, properly weighted poll of the Eastern region. It’s a crossbreak on a normal national poll. The figures are based upon only only 58 respondents, giving a margin of error of plus or minus 13 points. The figures are not even accurately quoted, the Lib Dems are actually on 7%. The were no caveats about sample size offered (the youtube clip from UKIP cuts out suddenly, but at the moment the full programme is on iplayer). This is truly appalling reporting of polls – they is no way that such a tiny cross-break should be reported out of context as if it were a representative poll.


93 Responses to “NOT a proper poll of European Election VI in the East”

1 2
  1. I suppose it is a combination of journalists wanting a headline and genuine ignorance. You have us all well trained….

  2. That means within MOE, UKIP might be as high as 57% in the region? Perhaps they should aim for independence for East Anglia?

  3. you can give timed links to programs on the iPlayer like this:
    http://www.bbc.co.uk/i/b03zhpds/?t=15m23s

    same on youtube.

  4. I’m doing Statistics for Journalists next year. I’m hoping they staple your rules to our foreheads for the whole semester.

    Oh, and your response to that rubbish EA poll quoted on The Escapist is used in my podcast this week. Uploading tomorrow!

  5. Someone mentioned a while back that support for independence in Wales is only 5%. I reckon if you polled every region you’d get about that. For some like Cornwall and Yorkshire you’d probably beat it.

  6. The news that the MoE for UKIP has a floor of 31% in Eastern is interesting though.

    @MrNameless – I replied to you about EA before AW did!

  7. The clip in question had no cuts at the end as you suggest.
    Who gets to decide what ‘proper weighting’ is?

  8. Howard and Pressman
    Re Tory Central Office wanting to portray Cameron as more like a Sun reader,
    if they could get Samantha to feature in ‘Readers’ Wives’, then they might get somewhere…..

  9. That wasn’t my suggestion – the opposite in fact. The video seems to me to end abruptly, but if you check the full version of iplayer the BBC did not offer any caveats afterwards, your clip shows the full coverage of what the BBC broadcast.

    ComRes will confirm that the margin of error on such a small sample is c. +/- 13 points. This is not something unique to a particular polling company, it is purely a function of sample size.

  10. ” no way that such a tiny cross-break should be reported out of context”
    But then, shouldn’t you report the change in the trend which is produced by the latest poll, rather than the poll itself? You tell us, rightly, to discount single polls and look at the trends. Though if you did, most of the discussions would be dead in the water.

  11. In Carfrew’s ideal government, journos would be required to attain Chartered Statistician status, and membership would be revoked for crimes against polling. (There would also be tax breaks for members of the public acquiring stats ability. Agree/disagree questions might be outlawed. Inappropriate use of crossbreaks would result in an on-the-spot fine, delivered by an army of… Stats Wardens!!… Ok, I might be getting a bit carried away…)

  12. Complaint to the BBC submitted. Let’s see what joys it produces…

  13. @AW – “Complaint to the BBC submitted”

    Ha! I hope you sent it from your official work email address.

  14. Ukipwebmaster

    The polling companies decide what the best factors are to weight to. This forms part of their “expertise” in reducing the sampling bias of each poll. Differences in methodology will lead to different biases in reports and who is more right or wrong is probably looked at closely after each election as it’s a big part of credibility for a polling companies to get election predictions right. It’s their shop window and if they are bad at forecasting elections they are likely to be seen as bad at everything else.

  15. “Complaint to the BBC submitted. Let’s see what joys it produces…”

    ———-

    Now, see, under my regime, they’d already be getting an on-the-spot penalty…

  16. @Ewen Lightfoot

    “..if they could get Samantha to feature in ‘Readers’ Wives’, then they might get somewhere…..”

    You remind of that rather surreal interview Paxman conducted with Blair many years ago. In typical Paxo style he asked Blair whether he was happy to accept the political support of Richard Desmond, the Express proprieitor, considering the other publications and media outlets that Desmond owned at the time. He reeled off an exotic list of soft porn magazines and cable TV channels and asked Blair if he had ever read or watched them. Blair responded that he hadn’t heard of any of them, although this was before Jacqui Smith’s husband had attained his notoriety!

    More topically, and talking of our old MP Jacqui (who I had a lot of time for, by the way), I see Charlie Brooks was talking about his attempts to avoid his own “Jacqui Smith” moment when giving evidence in the Hackgate trial yesterday. Disposed DVDs and all that.

    Well, there you are, Jacqui, you’ve gone in to folklore now and become a byword for a particular type of political embarrassment, although I suspect you probably craved something a little more noble in terms of posterity.

  17. So when are yougov going to start prompting for the party that has consistently been in third place for over a year now? Surely that year would be an appropriate point at which to stop regarding their support as a blip?

    I realise that shifting methodology has its downsides and makes comparisons less valid, but it’s getting a bit ridiculous.

    And that isn’t partisan, I had a lively exchange with a Ukip supporter on here that was putting out the ‘pollster conspiracy against Ukip’ line. I disagree with him regarding it more as institutional inertia.

    Like how it’s a drag to redesign website layout to make room for four boxes. But if you want an easier life you could always make the third box for the third placed party.

  18. When we think it would lead to more, not less, accurate results.

  19. Carfrew

    I think it is important to correct a mistake made earlier. You said that people (voters?) said that the cuts were too deep and being done too quickly.

    Looking at the last YouGov polling on this (23/24th March) the figures were actually

    Too deep 38
    Too shallow or about right 46

    Too quickly 38
    Too slowly or about right 46

    Quite the reverse of what you suggested. You have to add in those like me who think they are not being done quickly enough or deeply enough to get the real picture of how the cuts are perceived. Agreed they are seen as being done unfairly by 53 to 30.

  20. No way is this an innocent misreporting of a poll by BBC East. The journalist concerned will have seen the tables and considered that a wholly misleading unqualified half truth would be a good way to add some meat to a vacuous puff piece about UKIP’s prospects. We’ve ended up with a BBC report that is so biased that UKIP posted it verbatim on You Tube.

  21. Succinctly put AW.

    How do decisions get made regarding shifts in methodology? Is it a case of locking 20 statisticians in a room until they reach a decision?

    Might be more entertaining than Celebrity Big Brother.

  22. @ToH

    You are lumping “about right” and “too shallow” together!!

    Squirrels abound…

  23. @Chris,

    True. But “Chris from the internet” carries less weight than “Anthony Wells, actual pollster”!

  24. Here are the “unlumped” figures…
    36% too deep 33% too shallow 12% about right
    38% too quick 13% too slow 33% about right

    So too deep, and too quick, are the most popular of the three options.
    And only 12% think the amount of cuts is “about right”, and only 33% think the pace is about right.

  25. @Mr Beeswax & @Alan – There are pros and cons to prompting for UKIP.

    If we were sure that prompted support for UKIP in a poll will translate to support for UKIP on the ballot paper, every pollster would prompt for UKIP.
    If prompting or not prompting for UKIP made no difference, then we wouldn’t be having the discussion.

    I.e., we’re looking at people who would say UKIP if prompted but wouldn’t say UKIP if not prompted.

    So far there is insufficient evidence to suggest that those people will actually vote UKIP rather than voting for a mainstream party or staying at home. It’s one thing to claim to be a UKIP voter online; it’s another thing to turn up to the polling station, actually put your X in the box, and thereby potentially waste your vote or even let in a less-favoured candidate.

    There are certain seats where UKIP is strongly-favoured, and in constituency-level polling there it would definitely be appropriate to prompt for them. But at the national level the narrative supporting prompting is a lot less persuasive.

  26. Finally we get the truth!

    The real reason the BBC aren’t allowed to do Political Polling;

    ..None of them can Count!

    Peter.

  27. @MrNameless – I’m an actual pollster too, just being less “onymous”, for obvious reasons…

  28. Talking of UKIP and their electoral prospects, inflated or otherwise, I was fascinated by the Channel 4 documentary about Farage that screened last night. It was quasi-hagiographic, I thought, and finished with something approaching a personal eulogy from the narrator. It was produced, and I think narrated, by Martin Durkin, commonly referred to as a “controversial TV producer”. It would appear that he is a former member of the Revolutionary Communist Party who, rather like Peter Hitchens, experienced a Damascene type conversion, and now makes programmes for Channel 4 that are highly critical of both the NHS and the theory of man-made climate change. He was also incensed by Danny Boyle’s Olympic Opening Ceremony, apparently. I think we get the picture.

    If ever Farage wanted a one hour party political broadcast on prime time and mainstream TV, then he got one last night. Extraordinary stuff, but highly entertaining nonetheless!

  29. Carfrew the figures are

    Too quickly 38, too slowly 13, about right 33, so too slowly or about right 46.

    Too deep 36, too shallow 12, about right 33, so too shallow or about right 45.

    You Gov Archive 23/24th March 2014.

  30. Chris

    I was more wondering about how a decision on methodology was arrived at. Is it a case of “ooh we got UKIP a bit low for this election, lets start prompting and see how that goes” or do they make “unreported” polls where they do prompt in the run-up to an election to compare prompting vs. non-prompting and make a change if there is clear evidence to change.

    I’m sure there will be alternatives to prompting by tweaking how “likelihood to vote” or “past voting recall” are factored to correct perceived sampling bias. At some point someone has to make a decision as to which set of methodologies to use.

    I can imagine different methodologies regarding UKIP might be appropriate in European, Local and General elections but there is a risk of “overfitting” if you tune your approach to each election.

    I guess it raises the question “At what point do you stop prompting for Lib Dems?” :P

  31. @ToH

    Sure if you wanna lump them together.

    But it won’t change the fact that more people think they are too deep than about right, or too fast than about right.

  32. CB
    Yes l noticed Mr Rebekah had pulled that one out of the hat, will it become known as ‘The Smith Defence’ ?

  33. @ToH

    Nor will that particular lumping together change my original point: that those who think the cuts are about right are only about a third, which may explain why despite a majority accepting the cuts, VI struggles.

    Although perceived fairness, lack of, may also play a part.

  34. ToH – I assume you voted yes in the AV referendum.

    Sorry couldn’t resist.

  35. @Alan – My experience is that it’s unusual to change horses mid-stream. The changes are normally made in the aftermath of a major election – in the UK that is most commonly the general. That event will largely answer the question of what the people in that UKIP prompting gap will really do on the day.

  36. Also worth noting that a majority of Tories say that the cuts are not having an impact on their life, which sheds some light on the stoicism thing. For the remaining LibDems, it is exactly 50% who say no impact, vs 39% no impact.

  37. In terms of people being worried about having enough to live comfortably…

    Tories not worried = 54%
    Lab not worried = 26%
    LibDems not worried = 52%
    Ukip not worried = 21%

    So a majority of Tories and also LDs are not worried, which may help with the stoicism thing.

    Age breakdown is salient too…

    %age not worried
    18-24 = 29%
    25-39 = 32%
    40-59 = 37%
    60+ = 41%

    So the older you are, the less need to be stoical.

  38. Probably be more salutary if Comres wrote to BBC and complained – it was their poll that was misused. Good of Anthony to take up the cudgels however, very collegiate.

  39. typo-removal…
    (For the remaining LibDems, it is exactly 50% who say no impact, vs 39% impact).

  40. UKIP on at least 31% in the Eastern region according to this poll.

  41. Chris

    I agree noone will change methods over a by-election that produced a surprising result.

    It’s quite plausible that the European elections will be a point of which people check whether they have captured the UKIP share of the vote or if they have a bias for/against them. Although you would be making assumptions about whether that bias appears in GE polls.

    Prompting creates a bias, the trick is introducing the right biases to reproduce what actually happens at elections. The problem I see about prompting vs. not prompting a particular party is the size of the bias isn’t know until after a change (unless you try parallel polling to create a “test set” of data in the name of improving your methodology).

    There is a risk that by creating more and more complex biases (for example only prompting for UKIP in the southeast) is that you might fit the result for a particular set of elections but in reality you aren’t capturing real information and end up fitting to the noise.

  42. “Probably be more salutary if Comres wrote to BBC and complained – it was their poll that was misused. Good of Anthony to take up the cudgels however, very collegiate.”

    ———

    I’m quite taken with the idea of AW modding the Beeb…

  43. Andy JS

    Only to a 95% confidence level.

    I’m not sure on this, are the re-weightings for each region even appropriate or is it only re-weighted to get to appropriate weights at a national level?

    This was one of the parts that made my head hurt when trying to think about weighting.

    Are each of the weighting categories re-weighted in isolation on a macroscopic
    level or is it the combination of categories that are used to weight at an individual level? I can see the latter being perhaps more accurate but requiring more information than is available.

  44. @Alan – No, we already know that UKIP perform broadly as expected in European elections. They’ve been doing that for a decade or so. But up to 2015, they’ve not performed at general elections – even while they did perform at the Europeans. 2015 will be the test of whether that has changed yet.

  45. Alan – polls are only weighted at the national level. Individual regions are not weighted to be representative in their own right.

    Sometimes a pollster will interlock more than weighting variable (e.g. YouGov weight by age & gender interlocked as one of their weighting variables) but beyond that companies use rim weighting at the national level.

  46. What do you expect of the shoddy excuse for BBC journalism these days? Just look at the dreadful/biased political and economic coverage they dish out to us, not to mention the appalling Andrews Neil and Marr we have to suffer. ( And don’t get me started on Paxo!)
    The sad fact is I used to be a staunch supporter of the BBC ( and there are still many good things about it.)

  47. East Anglia is full of kippers?

    We’re Moving!

  48. AW:

    Thanks for clarifying that point.

    In other words trying to infer anything about a particular region based on a national poll will have the wrong weightings for that particular region?

    In the poll illustrated the assumption reported is the Eastern region would breakdown identically to the national average with respect to the weighting variables, which is almost certainly without merit?

    It’s not just the MOE on the tiny subgroup that is a problem, it’s that the 58 people sampled are weighted to be representative of the national average and not the Eastern region?

  49. @Alan – Polls are almost never interlocked for government region because it’s such a pain to get the right sample for each. Missing that twenty-something Northern Irish man for your quota? Tough luck, you have to get him or you can’t close the study. And then the solitary twenty-something Northern Irish man on your entire panel ends up answering almost every survey, which is unrepresentative for the study and fatiguing for him.

    (Young Northern Irish men are the worst to find. Lots of private companies politely ignore Northern Ireland when doing UK samples, and political pollsters have it easy because Northern Irish politics is so separate. But governmental bodies have to be seen to be visibly representing the whole country, so they have to get that representative Northern Irish sample – the process can get time-consuming and expensive. My version of Carfrew’s statistical dictatorship would compel everyone in Northern Ireland to answer at least five surveys every day.)

  50. @Alan – Actually the problem is even worse than that – it could be that everyone in Eastern was a female pensioner, and everyone in South East was a male youngster, everyone in South West was male and middle-aged, etc, so long as overall the figures are 52:48 for gender, and so on. When you don’t interlock the quotas, you can end up with such grotesque figures in the subgroups fairly easily. But interlocking the quotas is expensive and time-consuming.

1 2