In the last week we’ve had one poll putting Labour at their lowest level since 1987, and another poll giving Labour their largest lead since April. Earlier in the week David Cameron was being asked on the radio why his party was 2 points behind, now a poll shows him 10 points ahead? That’s a 12 point difference in the lead between polls just a week apart. So, whats the real picture?
There are three basic reasons why different polls produce different figures. Firstly the actually level of party support can change, the actual picture might have changed between them. Secondly there are the methodological differences between the pollsters, the way they draw their samples, the way they weight them, the way they take account of turnout and so on. Thirdly there is sample error – the so called “margin of error”.
Underlying party support in Britain is pretty stable – if you graph opinion polls over the last couple of years it’s clear that the only events that have actually shifted political support have been David Cameron’s election as Conservative leader, which moved the Labour and Conservative parties pretty much neck and neck, and the week prior to the local elections when Labour was hit by both John Prescott’s affair and the foreign prisoner release scandal, since when the Conservatives have had a consistent lead.
However, while the underlying picture is quite stable, political events do have a short term effect. For example, Tony Blair’s conference address this year saw his personal ratings shoot up and this was reflected by several voting intention polls that showed a dramatic drop in the Conservative lead.
These short term shocks can explain part of the difference – MORI’s poll was conducted in the middle of October, and was perhaps affected by the tail end of Labour’s fading conference boost. ICM’s poll was conducted when the Conservatives were getting a lot of publicity over the report of their policy commission on taxation, which in theory could have given them a boost.
The second factor is the pollsters’ “house bias”. By that I don’t mean that any of the pollsters actually have a partisan bias, but that the differences in their methodology, the differences in the way they draw their samples, the way they weight them, the different approaches to turnout and so on inevitably have a partisan impact on the figures they produce. If you take an average of all the comparable polls since the last election (i.e. excluding polls outside the regular monthly tracker polls, and those from months when not all four companies produced polls) you get the following figures:
YouGov CON 35.8%, LAB 36.5%, LDEM 17.8% (average Labour lead 0.7%)
Ipsos-MORI CON 35.1%, LAB 36.3%, LDEM 20.8% (average Labour lead 1.2%)
Populus CON 34.5%, LAB 36.2%, LDEM 19.5% (average Labour lead 1.7%)
ICM CON 35.2%, LAB 35.5%, LDEM 20.8% (average Labour lead 0.3%)
Despite all the major differences in methodology, on average there isn’t a great difference in the results the different pollsters produce. The main difference in the reported level of Lib Dem vote, with on average 3 points difference between the lowest (YouGov) and the highest (ICM and MORI), with Populus somewhere inbetween. In terms of Conservative and Labour support, there really isn’t a vast difference between the companies.
While MORI’s last two polls have reported Labour leads in amongst polls by other companies showing Conservative leads, looking at these average figures it’s clear that MORI’s methodology does not produce figures that consistently favour Labour (or at least, hasn’t in the past. There is no guarantee that different methodologies won’t react differently to changed situations). On average ICM produce polls that show a Labour lead around 1 point lower than MORI’s polls, so while 1 point of that 12 point difference might be down to MORI’s methodology being more favour to Labour than ICM’s, it’s only a small part of the picture.
That leaves us with sample error. Opinion polls are generally quoted with a margin of error of 3% or thereabouts. In reality this a polite fiction. The formula used to calculate the margin of error is based on a genuine random sample, in reality samples are a long way from truly random. RDD phone samples used by ICM and Populus exclude people who don’t have a landline, and the 5/6 people who don’t answer or refuse to take part. MORI and YouGov polls don’t use random sampling at all, instead using variations on quota sampling. Add to that the effect of weighting, adjustments and so on. The reality of opinion polling doesn’t bear much resemblence to the mathematical theory.
In reality the margins of error are probably somewhat larger than the statistical formulas suggest*, and for whatever reason, MORI’s polls are slightly more variable than their rivals. Two polls with a margin of error of 3% are almost enough to cover the 8 point difference between ICM’s Labour score and MORI’s Labour score anyway, once you’ve considered that MORI’s real margin of error will be larger than that, potentially sample error alone could explain the whole thing.
So, looking back at the recent polls, ICM have Labour at 29%, MORI have them at 37% – a difference of 8%. Take into account the “house bias” of the two pollsters and the difference comes down to 7%. The fading of Labour’s conference boost and the Conservative publicity boost from their tax proposals could easily have shifted support by a couple of points, leaving us with 5 percentage points or so to explain using sample error. The actual levels of party support are therefore probably somewhere inbetween the two polls, but given that ICM is less volatile than MORI, they are probably somewhat closer to ICM than to MORI. Over the next few days we will have the monthly polls from both YouGov and Populus, which – touch wood – will give us a better idea how the ground really lies.
(*Can we estimate the actual margin of error based on the observed variability of the polls, rather then the theoretical formulas? We can’t take the actual deviation from the average, because polls are taken weeks apart and the actual levels of support that the polls are deviating from is moving as political support shifts. I am not a mathematician, but the best estimate I can come up with is if we adjust the figures to remove the house biases, and then measure the deviation from a rolling average of the adjusted polls (I used a rolling average of the last 8 polls for these calculations). These give a standard deviation of 1.35 for YouGov, 1.62 for ICM, 1.67 for Populus and 2.44 for MORI meaning that, assuming a normal distribution, the actual “margins of error” (the 95% confidence intervals) for the polls are about 2.7% for YouGov, 3.2% for ICM, 3.3% for Populus and 4.9% for MORI.)