A couple of years ago I wrote a piece giving advice on how to report opinion polls, or rather, how not to. Look specifically at the third point on being careful of extremely small sample sizes in cross-breaks.

There was a cracking example of the media failing on this front on BBC Look East this week, which has done the rounds on Twitter. The clip is here, the offending portion starts at the three minute mark. It claims to show the results of a poll of the Eastern region that put UKIP on 44% of the vote.

The figures come from page 36 of this ComRes poll. It wasn’t a bespoke, properly weighted poll of the Eastern region. It’s a crossbreak on a normal national poll. The figures are based upon only only 58 respondents, giving a margin of error of plus or minus 13 points. The figures are not even accurately quoted, the Lib Dems are actually on 7%. The were no caveats about sample size offered (the youtube clip from UKIP cuts out suddenly, but at the moment the full programme is on iplayer). This is truly appalling reporting of polls – they is no way that such a tiny cross-break should be reported out of context as if it were a representative poll.


93 Responses to “NOT a proper poll of European Election VI in the East”

1 2
  1. Anthony

    I predict the response you will get will be akin to their standard response to complaints:

    I understand you were unhappy with xxx

    I can assure you this is simply not the case.

    I’d also like to assure you I’ve registered your complaint on our audience log. This is an internal report of audience feedback which we compile daily and is available for viewing by all our staff. This includes all programme makers and presenters, along with our senior management. It ensures that your points, along with all other comments we receive, are considered across the BBC.

  2. Carfrew

    I suspect we will never agree. However the main point is that like it or not the cuts are here to stay for the forseeable future regardless of which Government gets into power in 2015. Britain’s economic situation makes that certain. The alternative high tax and spend approach has been clearly demonstrated to fail as can be seen in France.

  3. So UKIP were happy to let the misrepresntation stand? Quell Surprise Rodney.

  4. Bramley –

    We shall see. I managed to provoke this out of the Observer last time I decided to go all complaintsy, so let’s see what the Beeb manage…

    http://www.theguardian.com/theobserver/2012/mar/18/observer-readers-editor-on-polls

  5. ‘East Anglia is full of kippers?’ — Normal For Norfolk

  6. Carfrew

    It will be interesting to follow the detailed polling on cuts as we move towards the election. Looking at the trends over the last six months on all questions the answers seem to be moving in a favourable position for the Gopvernment

  7. @HookesLaw – I don’t think that that’s fair – I imagine that UKIP simply trusted the report.

  8. Chris

    Yes, I see what you mean, all that has been done is the Eastern region has had it’s numbers re-weighted so the appropriate numbers of people from the Eastern region appear. Nothing has been said about how representative they are of the Eastern region or even nationally, only that the sums of all the regions (and other variables) end up at a national level.

    Just be glad you don’t have to look for a young northern Irishman, in the ABC1 bracket, who reads xxx and voted yyy at the last election!

    A simpler solution would be to poll everybody, (daily) and eliminate sampling bias and error completely.

  9. @THE OTHER HOWARD

    “I suspect we will never agree. However the main point is that like it or not the cuts are here to stay for the forseeable future regardless of which Government gets into power in 2015. Britain’s economic situation makes that certain. The alternative high tax and spend approach has been clearly demonstrated to fail as can be seen in France.”

    ————-

    Well, it will be quite difficult to reach agreement if I am considering simply the VI implications of Colin’s stoical idea, saying nothing about whether cuts are right or wrong or not, and you interpret that as an opportunity to disagree over cuts.

    We were always going to have cuts. Whether they will be the same cuts, or something less injurious to growth or a balanced economy and recovery we shall see.

    France doesn’t prove anything. It’s a zombie argument, that keeps rising from the dead. As pointed out before… they were constrained by the Euro, and it depends how you do it anyway. Moreover, we already know, in our situation it was possible to get back to growth quickly. Because we did. And while only spending 30Bn more than coalition. Which is how much a year’s growth is worth in terms of GDP. Pays for itself…

  10. @THE OTHER HOWARD

    “It will be interesting to follow the detailed polling on cuts as we move towards the election. Looking at the trends over the last six months on all questions the answers seem to be moving in a favourable position for the Gopvernment”

    ———-

    Haven’t checked across all questions but it wouldn’t surprise if true. I wonder if Labour might not have gained a bit on cost of living after the energy gambit though…

  11. Anyone else getting “The Case for Fracking – how you could profit from the UK fracking ‘story’ with two shrewd and timely investments” ads? Kinda funny given my interest in Thorium…

  12. @Alan – “Just be glad you don’t have to look for a young northern Irishman, in the ABC1 bracket, who reads xxx and voted yyy at the last election!”

    Believe me, there are always clients who ask for that kind of thing.

  13. @Carfrew – I use Chrome, so I don’t see adverts.

  14. @ Carfrew

    I think its linked to what you search on the internet, For example if you were reading up about thorium it figures you are interested in energy.

    That’s what I believe happens anyway.

  15. @CHRIS

    “@Carfrew – I use Chrome, so I don’t see adverts.”

    ————–

    Ah, well apparently you are missing out on the fracking story then…

  16. @Bluebob

    yeah, I’ve been mostly getting camera ads of late. I get the Maritime thing from time to time, ISA ads… never anything about reduced storage rates though…

  17. We’ll. I just go an add suggesting I stock up for the Tamil new year.

    So I think how this system works is………badly!

    Peter.

  18. Apparently, HMS Tireless has joined the hunt in the Southern Ocean for the missing Malaysian Airliner.

    Well we found the Belgrano!

    Peter.

  19. Ukip were on 19.6% in East of England with 2 MEPs in 2009
    (Con 31.2%, LD 13.8%, Lab 10.5%, Green 8.8%… 3:1:1:0).

    BNP and Eng Dems are both on the ballot paper again this year (6.1% and 2% in 2009), but will likely lose out to Ukip.

    A further eight smaller parties accounted for 8% (UK First + Libertas totalled 3% and NO2EU + Socialist Labour totalled 1.7%), but none have candidates in 2014.

    For Ukip to get near 40% they would need to mop up most of the others, and drive Conservatives down into the high teens/low twenties.

  20. @Peter Cairns

    Quite right too. It’s just silly for ships to have tyres.

  21. People curious about how Ukip is doing in certain areas would be well advised to look at Statgeek’s regional tracking graphs. They can’t give VI figures (since they’re still based on unbalanced crossbreaks) but they should give an accurate picture of trends.

  22. @RAF

    How else can they keep the ship’s wheel afloat?

  23. @Statgeek

    A good point. Well made.

  24. I managed once to get an apology out of the BBC in 2010 after Nick Robinson had decided that it was more important for listeners to hear him waffle on about nothing in particular even though this drowned out the results of the 2nd round of the Labour leadership contest while they were being announced live.

    But the apology only came after I had complained again after first being on the receiving end of a clearly prewritten vacuous standard e-mail that bore no relation to the substance of the complaint.

    Anyway I passed the e-mail on to Mike Smithson which led to this:

    http://www1.politicalbetting.com/index.php/archives/2010/10/25/bbc-apologises-for-the-edm-results-cock-up/

  25. Phil Haines,
    Good for you!

  26. Those of you who still think that Putin is the victim in l’affaire Ukraine may wish to note that Gazprom just raised[1] the price of gas to Ukraine from $268 to $385 per 1,000 cubic metres, and may raise it further to $480.

    You may also consider this analysis[2] of why he’s doing it.

    [1] h ttp://www.telegraph.co.uk/finance/newsbysector/energy/oilandgas/10737892/Gazprom-uses-gas-to-tighten-noose-on-Ukraine.html
    [2] h ttp://www.washingtonpost.com/opinions/russia-is-remaking-itself-as-the-leader-of-the-anti-western-world/2014/03/30/8461f548-b681-11e3-8cc3-d4bf596577eb_story.html

  27. What criteria do polling organisations use to alter their methodology? Presumably it is more than a hunch? Or is it always retrospective after general elections? Populus recently altered theirs not in the light of a general election. Are by elections ever used?

    And as the uniform national swing is used to predict seats yet clearly there are regional variations why is this not factored in to estimating likely seats won?

  28. @Bramley

    The bulk of your response is quite correct but I’m afraid you missed a very important element where the BBC mirrors the very sincere response of Virgin Media et al….

    “Your call is very important to us”

  29. @ Chris

    How are you managing to avoid adverts with Chrome? I’ve opted out of interest-based ads, but I’m still getting scantily-clad Filipinas alternating with Esther Rantzen offering me injury compensation. I can’t decide which I find more annoying.

  30. Sun Politics [email protected]_Politics 8s

    YouGov/Sun poll tonight – Labour lead up one to four points: CON 33%, LAB 37%, LD 10%, UKIP 12%

  31. Well it is starting to look like the budget lift for the Tories was short lived. Afer a couple of polls of 36, they are back down to 33. Labour have been pretty consistent throughout the period.

  32. “Filipinas alternating with Esther Rantzen offering me injury compensation. I can’t decide which I find more annoying.”

    Would it be less annoying if it were a scantily clad Esther Rantzen alternating with offers for Fillipino injury compensation?

  33. Is the “latest voting intention” table on the right ever going to be updated?

  34. @ Adam

    It is only updated once or twice a month and Anthony has already exceeded his limit!

    Plus judging by tonight’s poll nothing to update.

  35. @ Muddy Waters

    No, I don’t believe it would :)
    (I can’t actually tell what Esther’s wearing since all that appears is her head – complete with teeth.)

  36. Evening All.
    United better.

    CATMANUEFF.
    10% HIGH, but 4% not bad, not good, for Cons or Labs

  37. @ R Huckle,

    I’m not so sure. Sunday’s 40% nonwithstanding, it looks to me like Labour are still down a point and the Tories up a point.

    Sine Nomine’s dreams of crossover do seem to be retreating into the distance, though. (Of course, now that I’ve said it it’s bound to happen in tomorrow night’s poll…)

  38. R HUCKLE it still seems barring one outlier that the Labour lead has gone from over 5% to under 5%. Whether this will last and we are seeing the start of a a slow drift back or not remains to be seen.

    The last Tory lead in any main national poll was ICM on18th March 2012, over two years ago and that was 3% and not enough to give them a majority. So despite this possible Tory budget bounce the next government still looks to be Labour with size of majority the only thing in doubt.

  39. It does appear the budget bounce (if that is what it was) has nearly worn off.

    One trend is curious for me – there is about 10% of 2010 LDs (2.3% total VI) who flips from poll to poll, from LD to Lab, in recent times. This subset seems very volatile when compared to others in the poll.

    I will post tomorrow when the tables are available.

  40. @Spearmint @RHuckle

    I concur that Labour has lost a touch, including some 2010 Labour voters.

    I early March, Labour had about 87 % of 2010 Lab voters.

    That is now about 82 – 83 %

    3 – 4% = about 1 % total VI.

    I think we will see the Conservatives possibly taking 3 steps forward, and 2 back until 2015.

    I think Ed needs to make more noise and give people a reason to vote Labour, a unique selling point.

    Is Labour tacking close to the Government on welfare and spending slightly demotivating it’s core vote?

  41. Labour won’t have wanted to give too much away too soon, lest it be nicked, or countered, or the effect fade too much etc., so it is mostly about containment at the moment, hence tracking on negatives like welfare and absence of positives… with occasional moves if necessary to shore things up like on energy…

  42. Polls settling back to pre budget range

  43. carfrew

    “Moreover, we already know, in our situation it was possible to get back to growth quickly. Because we did. And while only spending 30Bn more than coalition. Which is how much a year’s growth is worth in terms of GDP. Pays for itself…”

    LOL

    The voters don’t agree with your view.Last Sunday Times Poll.

    Who do you think is most to blame for the economic problems that britain has experienced in the last few years.
    The last Labour Government 41.
    The Coalition 12.
    Very negative for Labour.

1 2