There may yet be a final London mayoral election poll from Ipsos MORI and I understand there will be one from YouGov, so we may yet get polls that show closer figures, but on the figures we’ve got at the moment there is an almost unbridgable gulf between them. MORI show Ken Livingstone ahead on the first round by 3 points and winning by 4 on the second round. YouGov show Boris Johnson with a commanding 11 point lead on the first round, and ahead by 10 points on the second round. Add to the mixture mruk who show Livingstone leading by one point. Why the big difference, and who is really ahead?

Unless there are final polls to come, and those final polls are much closer to one another, someone is going to have their reputation seriously damaged. Polls have got it wrong before, in the 1997 and 2001 election some pollsters were still showing Labour leads far greater than they actually achieved – the reason it received little or not attention compared to 1992, where the polls were famously wrong, is because they got the winner right. In 2001 the polls showed another Labour landslide, another Labour landslide is what happened – few cared that a couple of pollsters had shown a Labour lead almost double what they actually got. People notice when pollsters get the winner wrong, and it looks as if someone is going to do that. Another reason is that attention has been focused on the difference by the Livingstone campaign criticising the methodology of polls that show results they don’t like – and people are already focusing on the MORI vs YouGov angle of the race. From one side or the other, there will be a lot of crowing come Friday.

So what are the reasons for the difference? Firstly there are some long standing differences that apply not just in the mayoral race. Alone amongst the main pollsters Ipsos MORI do not use any political weighting, relying upon demographic weighting alone. This is a long standing debate, based on things like the degree of false recall and how volatile it is. The bottom line however is that most phone pollsters like ICM and Populus believe that without specific political weighting phone samples will contain a higher proportion of Labour supporters than in the population as a whole. To correct it ICM, Populus, ComRes and mruk all weight by how people voted at the last general election (adjusted by their estimate for false recall – the way people aren’t actually very good at reporting how they voted), because MORI are concerned that levels of false recall can fluctuate they don’t use it to weight by, so their samples are more Labour than other pollsters – this will explain some of the difference.

What of mruk though? They do weight by past vote, but they too show Livingstone marginally ahead. Here we should turn to the issue of turnout. Turnout is one of the hardest things for pollsters to get right since people tend to grossly overestimate their likelihood to vote. This is for two reasons. Firstly taking part in elections is seen as the socially responsible thing for an involved citizen to do. You should vote. Therefore people are rather embarrassed to admit to pollsters that they can’t be bothered. Secondly people just aren’t very good at predicting – they think they will vote, but after a long day once they’ve sat down on the sofa in front of Eastenders they just don’t get round to it in the end. An additional problem is the way the question is asked – are people who say 10/10 actually more likely to vote than 9/10s, or are 9/10s just people who are being a bit more honest and recognise that they might get stuck late at work?

Identifying voters at general elections isn’t quite so bad, because the proportion of people who say they are certain to vote would be a vaguely credible figures for actual turnout. With low turnout elections like London it becomes impossible – MORI found 61% of people 10/10 certain to vote. Mruk’s previous vote found 57% of people 10/10 certain to vote (though they included the 73% who were 8+/10 likely to vote). At the last mayoral election the actual turnout was 36%. We know that when polls are filtered to remove those people less likely to vote it tends to favour the Conservatives. For example, if mruk had included only those 10/10 certain to vote in their final poll it would have shown Boris ahead by 1 point. The problem for mruk and MORI is that if 60% or so of people say they are certain to vote, there is no way for them to further filter out the most certain of them to get down to a figure in the 30s or 40s.

One solution is that used by ICM – to skew the wording of the question to make respondents feel it is socially acceptable to admit they might not vote. In their last poll earlier in the campaign they asked the question “Many people we have spoken to have said they will NOT vote while others have said they WILL vote. Can you tell me how certain it is that you will vote?”. They found only 38% of people said they were 10/10 certain to vote, which sounds far more feasible. I’m expecting ICM to have a London poll in the next day or two, and it will be interesting to see if their approach to measuring likely turnout produces better figures for Boris – their earlier poll in the campaign didn’t, but who knows?

And what about YouGov? Uniquely amongst pollsters they take no account of likelihood to vote at all, excluding only the 17% of respondents who say don’t know or won’t vote. Everyone else is assumed to be equally likely to vote. In practice this should produce figures that are far more favourable to Labour, but in practice it doesn’t. Equally surprisingly, when YouGov have experimented with filtering or weighting by likelihood to vote it has made their figures less accurate. In the 2004 mayoral election their filtered figures were far too favourable to the Tories, but their unfiltered figures taking no account of turnout were almost spot on. It seems counter-intuitive, but ignoring turnout seems to work for them. Perhaps YouGov’s panel is just more representative of the type of people who actually vote.

A third issue is young people. Here there is a very specific difference between mruk and MORI and YouGov. All the pollsters have tended to show Boris leading amongst elderly voters and Ken doing comparatively better amongst middle aged voters. The contrast is with young voters – YouGov have consistently shown Boris leading amongst young people, MORI and mruk show Ken leading amongst young people, often by large margins (though it’s closed a bit in MORI’s last poll). This could be a major factor in the difference between the pollsters, and my guess is that it’s down to the sampling. Young people aren’t easy to poll, they are the most likely to rely on only mobile phones and not have landlines, they have the busiest social lives if you try to ring them – one can easily imagine that it would be difficult for phone pollsters to get a good sample of them. At the same time, one could also imagine that YouGov has young people who are too studenty and online.

Finally, not an issue that affects any particular side of the argument, but a possibility for error amongst all the pollsters. The Other share still seems very low. When polls for the campaign began I said I expected the “Other” score to gradually climb. It didn’t. It’s still incredibly low compared to how minor parties faired in 2004. This may not necessarily be wrong – many doubted the Scottish election polls for the same reason, but it turned out minor parties had lost support. Equally, while the electoral system means voters can give their 1st vote to a minor party without missing out on the Ken v Boris race, that doesn’t mean they realise they can and minor parties may be being squeezed by the closeness of the race.

So who is right? Personally I think, on their track record of polling in low turnout elections, that YouGov are more likely to be correct (though 11 points seems one hell of a lot, I’d be less surprised by 7 or 8 points), but then, I work for YouGov, so try as I might I am hardly an unbiased observer here. I’m sure if you asked MORI, mruk or ICM’s employees they’d think they were right – no pollster is ever deliberately wrong. We’ll know on Friday.


37 Responses to “So, is Ken or Boris ahead?”

  1. As ever, a most compelling and balanced case.

    However, surely missing from your list of reasons for the discrepancies is mode effect. Both in terms of sampling coverage (i.e. issue of online penetration vs landline penetration) as well as the issue of how ‘honest’ respondents are when speaking to a person (face to face or over the phone) against being sat in front of a PC. The former would, to my mind, count against YouGov where as the latter would be in YouGov’s favour, from what I’ve read about this. Would you agree?

    Another point on weighting – I understand YouGov do not make use of propensity score matching in attempting to control for offline or light online respondents. Could this not compromise the validity of their results?

    Report comment

  2. It will be interesting to see if the result, whichever way it goes, will lead to any changes in methodology – as someone must be wrong here. I guess many of us will be trying to reverse engineer the results back into the general election/national polls. I can see a long couple of days comng up

    Report comment

  3. it will be terribly boring if the final polls converge with a 5 point Boris lead!

    Report comment

  4. One thing you don’t mention is the potential for a Boris equivalent of the ‘Bradley effect’, where voters say they will vote for the fun candidate but chicken out in the polling booth. This could be a big factor for Boris – which might mean that, although YouGov was right in its measure of voting intentions, the final outcome incorrectly makes them seem wrong all along…

    Report comment

  5. Anthony:

    “The bottom line however is that most phone pollsters like ICM and Populus believe that without specific political weighting phone samples will contain a higher proportion of Labour supporters than in the population as a whole.”

    Why is this? Are labour supporters more likely to answer the phone to pollsters? Where does the belief spring from?

    Report comment

  6. Cyberkarst – other than political weighting, because it would probably be a factor in differences between mruk/ICM and MORI, I’ve tried to concentrate on differences that are specific to the mayoral race, since we’ve obviously got a far greater difference in this race than the normal difference between the pollsters.

    YouGov don’t use any propensity weighting – they rely only on demographic, political and attitudinal (in the form of newspaper readership) weighting. If there is a genuine difference beyond that which can be controlled for by demographic or attitudinal weighting between people who are online and people who are offline then obvious online polling faces a severe problem in those areas, I’m uncertain to what degree any weighting could could get round that. The question is whether there is any (beyond obvious areas like tech take-up), since where things are testable – as in voting intention – online polling seems to work, and where it isn’t – attitudes to taxation for example – it’s hard to prove if the difference is based on the sample or the mode.

    Turning to specifics, I suspect the difference in young people’s attitudes is down to very different groups of young people being accessible online and using landlines, so that probably is a factor here (though whether it is an important one, or who is doing better, is a different question). Mode effect’s biggest impact here would potentially be on the level of BNP support recorded in the polls, but since I haven’t seen any polls of London Assembly list voting, where the BNP vote is likely to be highest, we can’t really contrast.

    Report comment

  7. Jon H – I think ICM came to the belief because without the weighting their polls overestimated Labour support!

    I don’t think the reasons are known for sure, though its possible to come up with plenty of hypotheses about affluent people being more likely to be out or more likely to have answerphones. Perhaps Tory voters are less likely to give up 20 minutes of their time to answer a coldcaller’s questions for free ;)

    Report comment

  8. Freda – actually it could be a factor in the poll difference, but working the other way round. If there are people out there who support Boris but are a bit embarrassed to admit voting for the “clown candidate”, then his support might show up more in impersonal online polls than polls with live interviewers.

    Report comment

  9. I agree that turnout could well be a factor in the differences between the pollsters . It is a puzzle though why Yougov should be more accurate with a sample which forecasts 80% or so turnout to a telephone poll which forecasts a lower but still unlikely 60% turnout . It could be that the Yougov panel whilst not representative of the population as a whole are representative of that portion of the electorate who actually do vote . The result will be interesting to see who is correct in the pollster’s contest .

    Report comment

  10. Anthony. However determined, it is impossible for anyone to say they are 100% certain to vote(except postal voters).I don’t know the weightings- would treating 10/10 as 9/10 have any significance?

    Report comment

  11. The BBC should be disbanded, and resources should be sunk up north! A waste of valuable space!

    [P.S.: Kudos the font! Works in Firefox!]

    Report comment

  12. Very interesting Anthony-and very honest.

    Given the huge disparity in Polls which you highlight, people are bound to draw conclusions about reliability as a result of the outcome.

    Could you say-before we know the actual result-would it be reasonable/logical to assume that the Pollster getting closest to the truth will be the most reliable indicator for the GE-and the Pollster(s) furthest from the truth will be the least reliable for the GE.

    If the answer is No, could you explain why.

    Report comment

  13. Colin – No, not necessarily. If the reason for the error is down to the difficulty in polling a low turnout election then it won’t necessarily be a problem in an election with much higher turnout.

    Report comment

  14. Folks – Ipsos MORI have not been commissioned to undertake another survey before the great day. Our last survey was on the 23 and 24th April – it is conceivable that attitudes could change in the last week of the campaign. I do agree with Anthony that Boris walking it with a double digit lead sounds unlikely – our last survey of last week is basically too close to call given margins of error involved – but lets see what the final polls this week show; they are those that ought to be closest to the final figures, as there is less time for late swing to explain discrepancies.

    Report comment

  15. Many thanks Anthony.

    Regarding propensity weighting – I am probably a little bit more ‘optimistic’ about it’s use to correct for a wider range of questions than you.

    But overall I take your other points.

    Report comment

  16. Anthony,

    In your learned opinion can we judge the pollsters (that existed then) on their performance last time around or have they altered methodologies too much?

    Report comment

  17. Nice to see some new commentators. So close to an election too!

    One understands Ralph has his own blog. Does Jon H or Mark Senior (or your gracious self, Mr Anthony Wells) recognise the others..?

    Any ho, welcome! And – once again – thanks for correcting the font within the [my bad] CSS.

    Report comment

  18. remember a monkey was elected mayor of hartlepool

    Report comment

  19. Fluffy,

    Well I recognise Mr Page certainly – thanks for the update Ben – the only other new poster here is “Freda” who is actually a friend of mine, so do stop being so paranoid ;)

    Report comment

  20. Although a long-time reader of this blog, this is my first post. I should first of all say how much I enjoy reading the posts and comments on here. Great job, Anthony!

    I’m a young voter and also a member of the YouGov panel. I read a comment the other day to the effect that online voters are becoming more anti-Labour than the population at large. My own anecdotal experience strongly confirms this. Much is said about the so-called ‘digital divide’ and I feel that we might be reaching a point whereby internet and online penetration may have gone about as far as it’s ever likely to. I think it’s a fair assumption that those on the wrong side of the digital divide will vote Labour in greater numbers.

    I’m also relatively familiar with the fringes of the young London professional class and, for some, a vote for Boris has the mark of an apolitical act, though it is often combined with the general moans one often hears about how ‘awful’ The Tube is in rush-hour and how extortionate council tax is when ‘we don’t use any of the council services anyway’.

    I’m not doubting that the vote will be close, but, although I may be proved wrong, my money is on YouGov being the furthest away from the actual vote shares on May 1.

    Report comment

  21. Anthony – “Perhaps Tory voters are less likely to give up 20 minutes of their time to answer a coldcaller’s questions for free.”

    In which case wouldn’t YouGov be biased towards Tories who are only interested in the money… :wink:

    Report comment

  22. I am a YouGov contributor. I’m over 60 and 99.99% likely to vote : and vote Conservative.
    I refuse point balnk to answer ANY telephone polls and am a member of TPS so get very few.
    (I’ve been with YouGov for 4 years).

    I hope this helps the debate.

    Report comment

  23. Anthony – regarding turnout (at a GE), do any of the polling organisations do anything to try and predict turnout and perhaps increase the margin of error on their polls as a result?

    Fluffy – I know nothing and no-one. I just like the (misleading) certainty the nice numbers give me… You might be confusing me with John H who might know people.

    Report comment

  24. Anthony-thanks….you weren’t a solicitor in a previous life by any chance were you?.

    Just watched the brilliant Sky debate as a result of which I really do believe Boris can win.

    For the sake of Londoners, the Country-and YouGov I hope he does.

    Trust that’s not too partial!

    Report comment

  25. Anthony just to say that this was a really intelligent and thoughtful analysis which I found extremly interesting. Thanks very much.

    Report comment

  26. Who’s right will come down to which system is best at identifying how those who will vote, will vote.

    Yougov have a “active” standing panel who seem far more likely to actually vote than the general public so therefore don’t filter like phone pollsters.

    In this case they issue becomes how representative of the full electorate are the panel and particularly in this case the panel in London. In addition we should also ask due to the nature of life in the capital: work patterns, mobile use, ethnicity, does phone polling give the same accuracy that it would nationally.

    In addition we have the issue of a low poll election where perhaps only two in five will turn out.

    If Yougov are right then at the moment Labour voters aren’t motivated to go out and vote for ken but Tories will vote Boris and Boris will be mayor because of a feeling of time for change and apathy.

    It would be an irony if the man New Labour didn’t want because he could damage their election chances lost because of the damage done by New Labour.

    This is however the Ahmadinejad/Chavez factor where the media’s had both as outsiders and was surprised when they both achieved victory because we were listening to the middle classes and the “Blogosphere” and not to the man in the street.

    So who do I think has got it right?

    I’ll plump for YouGov, as I think the panel is long enough established large enough and has been refined to the extent that it will accurately predict how those who actually go out and vote are going to split. I think the actual result will end up with Boris between 54-56% to Ken’s 44-46%.

    Peter.

    Report comment

  27. Are the full tables from the second mruk poll available? – their website doesn’t seem particularly helpful.

    Report comment

  28. Can Kate Hoey bring Boris the Brixton vote? Here is hoping!

    Report comment

  29. Doesn’t MORI’s lack of a poll on the eve of the election leave them a ready made excuse if they come out wrong here with their polling?

    Late news like Kate Hoey coming on the Boris team is a plus to his campaign in my book and they miss these late developments.

    So their polling (indeed any firms) is rather pointless unless all pollsters come up with their own results on the eve of poll surely?

    Report comment

  30. Peter,

    Not sure whether we can compare Ken too closely with Hugo (despite their political pally-ness) – or are you suggesting that he will cheat as much as Chavez does?

    Report comment

  31. simon,

    Chavez isn’t my favourite politician, but in a straight vote he would still win. What caught the media out was that they were listening to the wrong people. Chavez’s popularity will wane as his policies aren’t that well thought through, but given that Opec is talking about $200 a barrel I think he is in for a while yet.

    For all his many flaws if he had never won, the oil money that has gone to the millions of poor since he was elected would have never got near them, so by and large he’s probably done more good than harm.

    Peter.

    Report comment

  32. Isn’t this a classic/crass ‘play the (wo)man and not the ball’ by Ken.

    Speaking on LBC radio, Mr Livingstone described Ms Hoey as “eccentric”.

    “She was one of the few Labour MPs to vote against banning fox-hunting,” he told LBC Radio.

    “But I’m surprised he’s going to take her advice on sport because I think the reason Tony Blair sacked her at the end of his first term, was because she’d been involved in all the fiasco over Wembley.

    “But I suppose she knows more about it than Boris does.”

    Kate Hoey, MP for the south London constituency of Vauxhall, has so far made no comment on the announcement.

    Report comment

  33. Incidentally, when I first put the new design up, lots people said the font was horrid and unreadable in firefox. I’ve changed it from the lovely Candara font to horrible old Verdana so you can all read it, but I was sure that vasn’t the cause, since I designed the Theme in Firefox and it looked very nice in there.

    Found the reason now: if the text looked horrible it’s because you haven’t got ClearType switched on in Windows. In IE7 it’s on automatically, Firefox & IE6 I assume just take the Windows setting.

    Go to Display Properties. Click the appearance tab, the effects button and where it asks how to smooth the edges of screen fonts choose ClearType instead of Standard. The seats guides, where I’ve still got Candara as the font, will now look nice :)

    I still like it, so as more people get IE7 or Vista I’ll switch back the blog too.

    Report comment

  34. I am a new contributor, having just found this site.

    I find the comments and analysis most helpful and informative.

    I am interested in the debate on variation in poll results, particularly as I on the Yougov panel. My wife and I are 100% likely to vote as we sent off our postal ballot forms more than a week ago. I always felt that Yougov would naturally have a bias to the Conservatives for several reasons, such as
    - more likelihood of broadband internet users being better off
    - people who ore older and/or have a more flexible lifestyle (Non executive positions, semi-retired etc.) are more likely to be willing to spend the time responding to an on-line survey
    - the political leanings of people who are more willing to do boring things like filling in voting intention surveys are more likely to be Conservative than those who are not. I apologise if that sounds very pejorative, but I find it difficult to explain clearly.

    These and other reasons give an unrepresentative Conservative bias, which however is then counteracted by a significantly higher propensity to vote by Yougov contributors.

    Adding these two variations together means that Yougov end up near the bullseye. It also means though that if they get one of the countervailing forces seriously wrong they won’t even hit the double ring!

    Does this make sense, or am I talking rubbish?

    Report comment

  35. Please add to the end of my third last paragraph “and people like them.”

    Regards

    Report comment

  36. Pingback: test » Blog Archive » So, is Ken or Boris ahead?

  37. Pingback: My new WordPress MU Site » Blog Archive » So, is Ken or Boris ahead?