There was a time when I used to spend an inordinate amount of time telling people not to spend too much time looking at regional cross breaks in voting intention polls – they have small samples, large margins of error, and are not internally weighted. Needless to say, it goes double or triple for London boroughs! There is a post on LabourList casting doubt on the ComRes poll because it shows wacky figures in some of the Borough crossbreaks and I’ve seen similar comments on Twitter.

There are only around 20-40 people in the London sample in each borough and often less than 20 who have given a voting intention, so the margin of error is about plus or minus 22 points, even if you did have a representative group, which really isn’t plausible with that few people. In other words, if a sample that small showed a borough as 50/50 Ken and Boris, all you could say would be that Ken was between 28% and 72%, and the same for Boris.

Polls are supposed to be representative at the level they are conducted at – a London poll will be representative of London, a GB poll will be representative of GB. They don’t claim that they are necessarily representative of sub-samples of that whole, and if you go down to small enough sub-samples, you will find absurd results. It doesn’t invalidate the total results.


95 Responses to “Regional crossbreaks again”

1 2
  1. My contention though, is that each borough in London has its own political micro-climate, and that polling should seek to take that into account…

  2. One could say the time about GE polls (and considering they have local Parliamentary candidates for people to vote for, with greater justification)… they still generally get it right.

    If a poll has too many Labour respondents in Chelsea, too few in Barnet, it evens out at the London level once the poll is properly weighted.

    Sure, 32 separate polls of each London borough aggregated together might be better. It would be extremely expensive though… and it wouldn’t be very much better. Bigger sample sizes have sharply diminishing returns.

  3. Correct, as usual, Anthony.

    The average punter doesn’t understand the scientific power of statistical sampling and probability theory.

  4. I agree it would be curiously interesting to see how the 32 boroughs are – but then you get into complications of turnout too….

    All I can say is that at times when you see the party politics of polls – it’s little wonder politicians get a reputation for being meretricious….

  5. Speaking of regional crossbreaks (which region, I hear you ask), I spotted this article on turnout in the Scottish local elections next week.

    http://www.scotsman.com/news/ian-swanson-vital-vote-lacking-polling-power-among-electorate-1-2257591

    “City Labour group leader Andrew Burns says: “Turnout is a concern because this is the first standalone council elections since 1995.”

    He says it is “not realistic” to expect turnout to equal the levels seen when the Scottish Parliament elections were on the same day in 1999, 2003 and 2007.

    “It was 44 per cent in 1995 and turnout generally has gone down since then, so I’m expecting it to be around the 40 per cent mark,” he says.”

  6. Anthony I apologise for highlighting some wacky borough figures. I should have known better!

    However, can you comment on the male/female split. This breakdown should be much more accurate than the borough breakdown. Is it credible given other evidence (recent polls, acutally election results) to say that Boris is ahead of Ken amongst men by an even larger margin than amongst women? I seem to remember previous polls where Boris had a big lead amongst women but Ken was doing better amongst men. It seems odd to me that he’s had a big bounce amongst men!

  7. Big D – you get the same volatility from sample size, some of those previous articles you read should perhaps have had greater caveats (and generally speaking, the Conservatives fell back amongst women since around January 2011, so I wouldn’t take the Conservative female advantage as being set in stone)

  8. So roughly how much would the volatility be around male/female breakdown. Presumably more than the +-3% of the headline voting intention but no where near the borough volatility. Is that a fair assumption?

  9. These “the polls can’t be real!” comments almost make me want to start resarch into patterns of behaviour of these True Believers pre-election and post-election – especially when the polls turn out to be realistic and cognitive dissonance begins to raise its ugly head… ;-)

  10. Despite the ComRes poll, I am getting off the fence & saying that I think Ken will win. So the YG poll, which is expected soon, will be very interesting.
    8-)

  11. The Ken vs Boris battle heats up:

    http://www.bbc.co.uk/news/uk-england-london-17816869

    “The RMT has started legal action against Boris Johnson over a poster allegedly portraying leader Bob Crow as “corrupt, venal and scandalous”.”

    “The mayor’s spokesman described Mr Crow as a “delicate flower”.”

    Love it.

  12. So, to speak in polling terms, I think the Boris +8 is an outlier; maybe even a ‘rogue’. But we’ll see, when YG comes out.
    8-)

  13. I’m not saying it is wrong it just doesn’t fit the wider picture. You can get a rogue poll every once in a while!!! YouGov poll should tell us whether it is rouge poll or whether there has been a shift to Boris.

  14. For those who have been anxiously awaiting them, the ICM tabs are now up here:

    http://www.icmresearch.com/guardian-poll-april-2012

    (and as some speculated, without topline adjustment for don’t knows, it would have been a 12 point Labour lead)

  15. I agree that the cross-breaks are almost meaningless, but why then are they published? Wouldn’t it make more sense to only publish cross-breaks if the MOE is less than a certain figure – say 5%?

  16. @Peter J

    “These “the polls can’t be real!” comments almost make me want to start resarch into patterns of behaviour of these True Believers pre-election and post-election – especially when the polls turn out to be realistic and cognitive dissonance begins to raise its ugly head… ”

    There was a good deal of discussion about the validity and accuracy of polls before the last election, as well as the tendency for people to want to disbelieve the accuracy of polls that run against their particular party political preferences or instinctive “gut feels”. We had some great ones before the last GE. “I can’t believe Labour are on 30% because I haven’t met anybody who doesn’t think Brown should be hung drawn and quartered” or “I’ve never been asked to participate in an opinion poll, or met anyone who has, so God knows who they’re talking to blah de blah”. Anthony dealt with these anguished, and wholly erroneous, protestations at the time.

    Now, whilst I share some, but not much, of the frustration and disappointment at the likely outcome of the London Mayoral election, I have a rather horrible feeling that this ComRes poll might be pretty accurate and that the actual result will be something fairly close to the one it predicts.

  17. Ed Milliband receives a ringing endorsement. Only 18% think he’s good in a crisis, which is half the level Cameron got and LOWER than Milliband secured in Dec (21% agreed then). He does fare slightly better than Cameron on understanding people but at 35%, he’s lower than he himself performed in Dec (37% agreed then). These are hardly figures that underpin a positive movement toward Labour. These bad polling numbers are clearly the result of frustration with the Tories, rather than a philosophical move to Labour. This lead may be short lived.

    That is the bit from the Grauniad link on Ed. They probably meant to say NOT a ringing endorsement.

  18. Prediction :

    Boris + 6 in YouGov poll

  19. Got a week of Levenson, Hunt, recession and rain and then we’ll see what Ken can do.

  20. Pete B – ComRes didn’t actually publish them as crossbreaks, only as one of the demographic questions they asked in order to weight their sample by inner/outer London. People have worked out the VI in borough because that was a crossbreak for all the questions.

    ComRes have been admirable in publishing all the questions they asked, including all the dull demographic stuff.

  21. @ Blue Bob

    Prediction :
    Boris + 6 in YouGov poll
    ————————
    Prediction:
    Ken +6 in the actual election. :-)

  22. I will make a prediction about the mayoral election….

    The fires are lit; the altar swept but the three chickens and the lamb are giving me a wary look….

  23. statgeek interesting how boris can dish it out with respect to bob crow but went ballistic when ken suggested his financial affairs weren’t all they seemed.

  24. I have no idea how some posters are convinced Livingstone will win, despite every recent poll showing Johnson in the lead. There might be a late swing (but remember that many postal votes will already have been sent), or it could be that all these polls err in the same direction (but that’s not very likely).

    Johnson is given a fair bit of leeway by Conservative Central Office, and this has enabled him to remain largely unaffected by the government’s recent difficulties.

    In the absence of any shocking scandals emerging before polling day, I expect Johnson to win, narrowly, while Labour make a gain or two in the Assembly.

  25. We shall see Amber we shall see.

    May 3rd prediction :

    Boris wins by a good margin.

  26. Have just posted my vote for London elections after hours of deliberation.

    No sooner had I posted it and got back in, found a Ken leaflet on my door, too late sunshine.

  27. The way I see it, Boris’s lead comes from Labour supporters saying they’ll vote Boris, right? Because on GE VI, Labour are well ahead. I think many Labourers will sit it out & not vote; some may even change their minds & vote for Ken at the 11th hour.

    We’ll see what YG shows. Will Ken have 2 polls to ‘buck’ or just one?
    8-)

  28. “The way I see it, Boris’s lead comes from Labour supporters saying they’ll vote Boris, right?”

    That’s half true. I believe polls show about 10-15% of people who usually vote Labour saying they will vote Boris, but I think that’s not enough to close the gap, what closes the gap, is an even larger proportion of Labour party voters who just won’t vote at all, they can’t bring themselves to vote for a Tory, but there not going to vote for Ken either so will stay at home.

  29. Are you a betting woman Amber, if you ill offer you 5/2 on Ken :p

  30. @Joe

    Trying to lay some of your exposure on Boris at 1/4? :)

  31. oops, got that backwards

  32. @ Joe

    My betting system goes like this: If there are good odds on the candidate I DON’T want to win, I might make a bet. Then I have a little consolation prize to myself, if my ‘guy’ loses.

    The betting odds for Boris have always been unattractive for the casual punter who isn’t betting a huge amount, so I decided not to bother.
    8-)

  33. Political spinners are either too dense or too dishonest to benefit from instruction of this sort, Anthony

  34. It’s actually not true to say that small-sample crossbreaks are useless. Although they have large confidence limits, they are still most likely to be close to where the sample says it is – but it’s difficult to tell which ones are completely loopy. It just requires a much more sophisticated level of analysis than casual punters are capable of.

    For instance, from the crossbreaks we can construct our own more meaningful crossbreaks, such as inner/outer, east/west, north/south, council composition… These might be valuable in discerning what underlies any change.

  35. The Boris v Ken, is one reason why I would not vote for an elected Mayor elsewhere.

    Don’t we have enough politicians ?

    As for polling crossbreaks I can’t see that such small samples can be seen as a standard that any professional polling organisation should be conducting. Why not conducts fewer polls, but have a higher sample when you do them ?

  36. R Huckle: “Why not conducts fewer polls, but have a higher sample when you do them ?”

    Because they would require a lot more time, effort and money to conduct for very little statistical difference.

  37. R Huckle wrote :

    As for polling crossbreaks I can’t see that such small samples can be seen as a standard that any professional polling organisation should be conducting. Why not conducts fewer polls, but have a higher sample when you do them ?

    ________

    I would prefer that also but would it make much difference in the actual polling accuracy Anthony ?

  38. @AW

    Interesting. There’s also a Mayor vs Councillors question showing over 60% prefer the latter to a 30% Mayor – notable for going in completely the opposite direction to a few weeks back Sunday Times Yougov question (perhaps they were asking different questions – I can’t really remember much beyond groaning at what looked like support for mayors being rolled out).

  39. I agree with R Huckle and the reason I have just voted “no” – although the actual question on referendum ballot was so obscure I ahd to read it threre times! – in my postal vote in referendum here. I find it ironic that the number of MP’s in this area being reduced because we seemingly can’t afford 50 MP’s but we apparnetly need loads of elected mayors – plus offices and staff – which will cost a lot more. Given the Ken and Boris show and the woefule xperience of Doncaster, Stoke etc, this is the last thing we need! Ken and Boris – what a pair. At least I think Livinstone’s heart seeems to be in the right place. Fancy having a buffon such as Johnson as your elected mayor. Makes you despair of democracy.

  40. What Ken needs is:

    1 Cameron to come out strongly in support of Boris

    2 Somebody to bring up Boris’s very close relationship with Brooks and the Murdoch outfit

    3 Labour’s lead to keep growing nationwide

    I suspect all those will happen.

  41. Crossbath, yes, I’ve been there and done that, saying “these polls can’t be real” when they showed the Conservatives getting a hammering in the 1993-2003 period.

    Sadly, the polls are almost always broadly correct.

    Unless Boris commits some major gaffe between now and polling day, he is likely to be re-elected, given the consistency of his lead.

  42. A bit disappointed by the London poll,especially as I am probably going to make a fool of myself having said that Ken is going to win…But as they say, a week is a long time in politics

  43. NICKP………What Ken needs is:

    1 To stop being seen with Ed M, you are judged by the company you keep.

    2 For Boris to withdraw from the contest.

    3 To ask Gok Wan to help him to stop looking like an ageing Columbo look-alike, on a bad hair day, he wants to represent the worlds greatest Capital city, not some rundown backwater.

    Oh, and by the way, ComRes are definitely not gold standard, more platinum, IMO.

    :-)

  44. OK, I’m getting a real pasting today for casting into question the reliability of this poll.

    Here’sthe scientific part of my reasoning. Look at the difference in samples between this one and the YG a few days ago.

    1. You Gov

    Unweighted: Boris: 592 Ken 596
    Weighted: Boris: 587 Ken 614

    2. Comres:

    Unweighted: Boris 441 Ken 307
    Weighted: Boris 397 Ken 355

    The two pollsters samples are too different to make a comparison. And it’s not just that Comres has a more pro-Boris sample, but that they also weight in favour of Boris whereas You Gov weight in favour of Ken. I’m not saying one is wrong and one right, we don’t know that yet. But they are too different. You can’t compare Monday’s YG with today’s Comres.

    In any event Comres have had problems in the past with their national VI polls, due to their sameples being more Conservative than other polsters (don’t know why). And this seems to be replicated here.

    And no matter how Comres adjust their unweighted sample, the numbers are just too stark a contrast to allow for reliability. If they are looking for a wighting of +40 Boris to Ken, then starting with a sample of +130 Boris/Ken is not helpful.

  45. Erm, from what you’ve got up there both companies had to weight their samples to be more Labour. ComRes would appear to have needed slightly more weighting, but that’s not a huge problem (degree of weighting just decreases effective sample size a bit) – it is more important what pollsters weight to, and the samples are very similar on that front.

  46. @ Smukesh

    Nothing wrong with having the bearings to come out and say what you think will happen even though the polls say otherwise.

    Like i think we are in for a Cons largest party in 2015 if the economy has only small growth still and in for a majority of 18 if its starting to motor.

  47. Robin: Another point that looking at tiny cross breaks is dangerous is they are unweighted (or at least they are misweighted, weighted to the global sum rather than to the subregion of interest). Weighting is the most important part of converting a set of responses into a poll.

    What effect can a misweighting have on a sample? Practically anything. Not only will the error bounds be huge, the middle point of those error bounds might be WAY off.

    Even averaging over a number of polls (even a large number) is risky as each poll individually will be misweighted and in effect you are hoping that increasing the sample size will result in an “correct weighting” once averaged.

  48. That’s not really nice Colin… way below the belt…

  49. I hope that Ken can pull it back, but I think at the very least, getting a licking from Boris whilst the Cons are down in the polls and Labour are miles ahead might make him think it’s time to leave politics.

    I think Ken would make a better Mayor, but I think he was clearly the wrong candidate to field in this contest, and Labour didn’t have much of a choice as otherwise Ken would run regardless, splitting the Labour vote.

    He needs to move on, he’s dragging the Labour party down, at the moment where a more electable candidate might be hugely ahead at this point. If you can’t win in these conditions Ken, get out of the ring please and let someone else have a go.

  50. Are french pollsters not allowed to poll anymore? We don’t seem to be having any polls lately about it. Last one we got was on Sunday Hollande 54 v 46 Sarkozy

1 2