Only one day to go till London and the local elections, but let’s have a quick break to discuss Crewe and Nantwich, the by-election for which is to be called on the 22nd May.

On paper Crewe and Nantwich is quite a distant target for the Conservatives. On its existing boundaries it requires a swing of just over 8% for the Conservatives to gain it, that’s the equivalent of a thirteen point lead in the national polls, so around the level some of the pollsters are indeed showing the Tories at in recent polls.

Historically we’ve seen bigger swings in by-elections than one sees at general elections – governments normally do badly in by-elections and pull back when it gets to actual elections. On paper, therefore, this should be a Conservative gain, no problem. In practice there are a couple of hurdles in the way of that.

While the regional breaks in individual polls aren’t reliable enough to tell whether a party is doing better or worse in a particular region, pollsters do occassionally publish aggregated results which give us a firmer picture of how the parties are performing in different regions and they’ve consistently shown the Conservatives performingly most strongly in the South and somewhat less so in the North. The pattern is similar to some degree in local elections – though there is it Yorkshire where the Conservatives are manifestly not performing as well as elsewhere, there are plenty of places in the North-west where they have produced some strong showings. Either way, if the Conservative’s aren’t doing as well in the North as in the South Crewe and Nantwich looks a somewhat harder task. On the other hand, Gwyneth Dunwoody was a particularly respected MP, so past elections probably include a strong personal vote for her which will now be up for grabs.

Secondly there is the strong Conservative track record at being hopeless at by-elections. To some degree that is because they haven’t had many good opportunities, very few by-elections have occured in seats where the Conservatives have a realistic chance of winning, but even in by-elections in their own seats they have shown themselves inept (Bromley and Chislehurst anyone?). In contrast the Liberal Democrat machine is finely honed. In by-elections in government held seats where the electorate wants to give the government a kicking the campaign is often about the parties positioning themselves as the best party to deliver it. Since the Lib Dems have a reputation for pulling off spectacular by-election gains, and the Conservatives – to put it kindly – don’t, this is somewhat easier for the Lib Dems.

In this case the Liberal Democrats are in a distant third place and while the Conservatives look almost certain to keep their existing candidate Ed Timpson, the Lib Dems previously selected candidate has stood down (as I understand is the norm for Lib Dem candidates in seats where a by-election occurs) and – since he lived in Derbyshire giving opponents an open goal to shoot at – I expect they’ll pick someone new to carry the flag. This time the Conservatives do at least have a head start in making sure they are recipients of anti-government votes, but we shouldn’t underestimate the Liberal Democrat’s skill at by-elections and the Conservatives’ weakness.

Who does win this by-election is important. One party or the other will already have their tails up from taking or retaining the London mayoralty. If the Conservatives win it will be a huge boost to them, give them great momentum and will be painted as a groundbreaking achievement after 26 years without a by-election gain. If they lose their opponents will say that it proves they aren’t doing well enough to win a majority at the next election, they aren’t doing well enough outside the south and their support in the polls fades away when people get to the ballot box. This one will undoubtedly be built up as a big test for Cameron, so it he fails it it will dent his momentum.

For Labour, if they hold the London mayoralty and then hold Crewe and Nantwich it will be building into a nice perception of fighting back. If they loose both they are going to really look as though they are on the ropes. Finally for the Lib Dems, unless they chose to build them up themselves there shouldn’t be any expectations for them to live up to – they are in a distant third place and needn’t risk their reputation on this one. The problem is that by-elections are the life blood of the Liberal Democrats, the elixir that gives them publicity, momentum and show that Lib Dems can indeed “win here”. Everyone that they don’t win is a missed opportunity.

UPDATE: The Lib Dems have announced their candidate, she (and other candidates as they turn up) is on the Crewe and Nantwich page here. Another thought, the writ for the by-election was issued on Wednesday and, 15 days afterwards ignoring bank holidays and weekends, polling day on the 22nd May is the earlier day possible to hold it. What this means is practice is that the Conservatives won’t be able to call the Henley by-election on the same day – the soonest it could be done is the following week.

Despite the report in today’s Telegraph, which managed to publish ICM’s polls from a couple of week’s back as a brand new one, there will not after all be another ICM poll on the mayoral elections. That means YouGov’s “morning-of-poll” effort is the only one left to go before the results on Friday.

There may yet be a final London mayoral election poll from Ipsos MORI and I understand there will be one from YouGov, so we may yet get polls that show closer figures, but on the figures we’ve got at the moment there is an almost unbridgable gulf between them. MORI show Ken Livingstone ahead on the first round by 3 points and winning by 4 on the second round. YouGov show Boris Johnson with a commanding 11 point lead on the first round, and ahead by 10 points on the second round. Add to the mixture mruk who show Livingstone leading by one point. Why the big difference, and who is really ahead?

Unless there are final polls to come, and those final polls are much closer to one another, someone is going to have their reputation seriously damaged. Polls have got it wrong before, in the 1997 and 2001 election some pollsters were still showing Labour leads far greater than they actually achieved – the reason it received little or not attention compared to 1992, where the polls were famously wrong, is because they got the winner right. In 2001 the polls showed another Labour landslide, another Labour landslide is what happened – few cared that a couple of pollsters had shown a Labour lead almost double what they actually got. People notice when pollsters get the winner wrong, and it looks as if someone is going to do that. Another reason is that attention has been focused on the difference by the Livingstone campaign criticising the methodology of polls that show results they don’t like – and people are already focusing on the MORI vs YouGov angle of the race. From one side or the other, there will be a lot of crowing come Friday.

So what are the reasons for the difference? Firstly there are some long standing differences that apply not just in the mayoral race. Alone amongst the main pollsters Ipsos MORI do not use any political weighting, relying upon demographic weighting alone. This is a long standing debate, based on things like the degree of false recall and how volatile it is. The bottom line however is that most phone pollsters like ICM and Populus believe that without specific political weighting phone samples will contain a higher proportion of Labour supporters than in the population as a whole. To correct it ICM, Populus, ComRes and mruk all weight by how people voted at the last general election (adjusted by their estimate for false recall – the way people aren’t actually very good at reporting how they voted), because MORI are concerned that levels of false recall can fluctuate they don’t use it to weight by, so their samples are more Labour than other pollsters – this will explain some of the difference.

What of mruk though? They do weight by past vote, but they too show Livingstone marginally ahead. Here we should turn to the issue of turnout. Turnout is one of the hardest things for pollsters to get right since people tend to grossly overestimate their likelihood to vote. This is for two reasons. Firstly taking part in elections is seen as the socially responsible thing for an involved citizen to do. You should vote. Therefore people are rather embarrassed to admit to pollsters that they can’t be bothered. Secondly people just aren’t very good at predicting – they think they will vote, but after a long day once they’ve sat down on the sofa in front of Eastenders they just don’t get round to it in the end. An additional problem is the way the question is asked – are people who say 10/10 actually more likely to vote than 9/10s, or are 9/10s just people who are being a bit more honest and recognise that they might get stuck late at work?

Identifying voters at general elections isn’t quite so bad, because the proportion of people who say they are certain to vote would be a vaguely credible figures for actual turnout. With low turnout elections like London it becomes impossible – MORI found 61% of people 10/10 certain to vote. Mruk’s previous vote found 57% of people 10/10 certain to vote (though they included the 73% who were 8+/10 likely to vote). At the last mayoral election the actual turnout was 36%. We know that when polls are filtered to remove those people less likely to vote it tends to favour the Conservatives. For example, if mruk had included only those 10/10 certain to vote in their final poll it would have shown Boris ahead by 1 point. The problem for mruk and MORI is that if 60% or so of people say they are certain to vote, there is no way for them to further filter out the most certain of them to get down to a figure in the 30s or 40s.

One solution is that used by ICM – to skew the wording of the question to make respondents feel it is socially acceptable to admit they might not vote. In their last poll earlier in the campaign they asked the question “Many people we have spoken to have said they will NOT vote while others have said they WILL vote. Can you tell me how certain it is that you will vote?”. They found only 38% of people said they were 10/10 certain to vote, which sounds far more feasible. I’m expecting ICM to have a London poll in the next day or two, and it will be interesting to see if their approach to measuring likely turnout produces better figures for Boris – their earlier poll in the campaign didn’t, but who knows?

And what about YouGov? Uniquely amongst pollsters they take no account of likelihood to vote at all, excluding only the 17% of respondents who say don’t know or won’t vote. Everyone else is assumed to be equally likely to vote. In practice this should produce figures that are far more favourable to Labour, but in practice it doesn’t. Equally surprisingly, when YouGov have experimented with filtering or weighting by likelihood to vote it has made their figures less accurate. In the 2004 mayoral election their filtered figures were far too favourable to the Tories, but their unfiltered figures taking no account of turnout were almost spot on. It seems counter-intuitive, but ignoring turnout seems to work for them. Perhaps YouGov’s panel is just more representative of the type of people who actually vote.

A third issue is young people. Here there is a very specific difference between mruk and MORI and YouGov. All the pollsters have tended to show Boris leading amongst elderly voters and Ken doing comparatively better amongst middle aged voters. The contrast is with young voters – YouGov have consistently shown Boris leading amongst young people, MORI and mruk show Ken leading amongst young people, often by large margins (though it’s closed a bit in MORI’s last poll). This could be a major factor in the difference between the pollsters, and my guess is that it’s down to the sampling. Young people aren’t easy to poll, they are the most likely to rely on only mobile phones and not have landlines, they have the busiest social lives if you try to ring them – one can easily imagine that it would be difficult for phone pollsters to get a good sample of them. At the same time, one could also imagine that YouGov has young people who are too studenty and online.

Finally, not an issue that affects any particular side of the argument, but a possibility for error amongst all the pollsters. The Other share still seems very low. When polls for the campaign began I said I expected the “Other” score to gradually climb. It didn’t. It’s still incredibly low compared to how minor parties faired in 2004. This may not necessarily be wrong – many doubted the Scottish election polls for the same reason, but it turned out minor parties had lost support. Equally, while the electoral system means voters can give their 1st vote to a minor party without missing out on the Ken v Boris race, that doesn’t mean they realise they can and minor parties may be being squeezed by the closeness of the race.

So who is right? Personally I think, on their track record of polling in low turnout elections, that YouGov are more likely to be correct (though 11 points seems one hell of a lot, I’d be less surprised by 7 or 8 points), but then, I work for YouGov, so try as I might I am hardly an unbiased observer here. I’m sure if you asked MORI, mruk or ICM’s employees they’d think they were right – no pollster is ever deliberately wrong. We’ll know on Friday.