Rob Hayward, the former Tory MP turned psephologist, gave a presentation at ComRes on Monday which has stirred up some comment about whether the polls are underestimating Conservative support.

Historically the polls have tended to underestimate Conservative support and/or overestimate Labour support. It was most notable in 1992, but was a fairly consistent historical pattern anyway. Since the disaster of 1992 this bias has steadily reduced as pollsters have gradually switched methods and adopted some form of political control or weighting on their samples. In 2010 – at last! – the problem seemed to have been eliminated. I hope that the polling industry has now tackled and defeated the problem of Labour bias in voting intention polls, but it would be hubris to assume that because we’ve got it right once the problem has necessarily gone away and we don’t need to worry about it anymore.

In his presentation Rob compared polls last year with actual elections – the polls for the European elections, for the by-elections and for the local elections.

I looked at how the polls for the European election did here and have the same figures as Rob. Of the six pollsters who produced figures within a week or so of the election five underestimated Conservative support. The average level of Tory support across those polls was 22.2%, the Tories actually got 23.9%. The average for Labour was 27%, when they actually got 25.4%.

Looking at by-elections, Rob has taken ten by-election polls from 2014 and compared them to results. Personally I’d be more wary. By-election campaigns can move fast, and some of those polls were taken a long time before the actual campaign – the Clacton polls, for example, were conducted a month before the actual by-election took place, so any difference between the results and the polling could just as likely be a genuine change in public opinion. Taking those polls done within a week or so of the actual by-elections shows the same pattern though – Conservatives tend to be underestimated (except in Heywood and Middleton), Labour tends to be overestimated.

Finally in Rob’s presentation he has a figure for polls at the local elections in 2014. I think he’s comparing the average of national Westminster polls at the time with Rallings and Thrasher’s NEQ, which I certainly wouldn’t recommend – the Lib Dems for example always do better in local election NEQ than in national polls, but it’s because they are different types of election, not because the national polls are wrong). As it happens there was at least one actual local election poll from Survation.

Survation local election: CON 24%, LAB 36%, LDEM 13%, UKIP 18%, Others 10%
R&T local election vote: CON 26%, LAB 36%, LDEM 11%, UKIP 16%, Others 12%

Comparing it to the actual result (that is, the actual total votes cast at the local election, which is what Survation were measuring, NOT the National Equivalent Vote) these figures were actually pretty good, especially given the sample size was only 312 and that it will be skewed in unknown ways by multi-member wards. That said, the pattern is the same- it’s the Conservatives who are a couple of points too low, Labour spot on.

So, Rob is right to say that polls in 2014 that could be compared to actual results tended to show a skew away from the Conservatives and towards Labour. Would it be right to take a step on from that and conclude that the national Westminster polls are showing a similar pattern? Well, let me throw out a couple of caveats. To take the by-election polls first, these are conducted solely by two companies – Lord Ashcroft and Survation… and in the case of Survation they are done using a completely different method to Survation’s national polling, so cannot reasonably be taken as an indication of how accurate their national polling is. ICM is a similar case, their European polling was done online while all their GB Westminster polling is done by telephone. None of these examples includes any polling from MORI, Populus or ComRes’s telephone polling – in fact, given that there were no telephone based European polls, the comparison doesn’t include any GB phone polls at all, and looking at the house effects of different pollsters, online polls tend to produce more Labour-friendly figures than telephone polls do.

So what can we conclude? Well, looking at the figures by-election polls do seem to produce figures that are a bit too Laboury, but I’d be wary of assuming that the same pattern necessarily holds in national polls (especially given Survation use completely different methods for their constituency polling). At the European elections the polls also seemed to be a bit Laboury… but the pollsters who produced figures for that election included those pollsters that tend to produce the more Laboury figures anyway, and didn’t include any telephone pollsters. It would be arrogant of me to rule out the possibility that the old problems of pro-Labour bias may return, but for the time being consider me unconvinced by the argument.

UPDATE: Meanwhile the Guardian have published their monthly ICM poll, with topline figures of CON 30%(+2), LAB 33%(nc), LDEM 11%(-3), UKIP 11%(-3), GRN 9%(+4) – another pollster showing a significant advance for the Green party.


152 Responses to “Are the polls underestimating the Tories?”

1 2 3 4
  1. First?

    Laboury !! Yes it all seems too Laboury.

  2. But is this the same as “swingback”? Or a different thing entirely?

  3. File this in the “Reasons why the polls are wrong, and the Conservatives will still win!” cabinet.

  4. ICM (The unnameable paper has details)

    Lab 33% (nc) : Con 30% (+2) : LD 11% (-3) : UKIP 11% (-3) : Grn 9% (+4) : Others (“most importantly SNP”) 7% (+1)

  5. Extraordinary Green vote from ICM given their weighting methods.

  6. I guess the ‘shy Tories’ question will never go away until we have a general election where the Tories are actually the incumbents. The last time this happened was obviously back in 1992, though obviously polling methodologies have improved significantly since then.

    Nice to see the Greens are continuing to advance.

  7. Correction: 1997!

  8. I’m still investigating this polling accuracy thing!

    One thing I would highlight re 2010 is that the polls did underestimate the Conservatives, as they have at every election since 1983, but Labour were underestimated as well, with the Lib Dems overestimated. The assumption has been that this was Cleggmania exaggerating the Lib Dem vote share at the (roughly equal) expense of Labour and the Tories. That makes sense, but Anthony is right not to be complacent.

    I’ve read Hayward’s presentation and yes there is some evidence of a shy Tory factor, and reasons for pollsters to be nervous, but considering Anthony’s caveats, I don’t see any smoking gun.

    However – there is one piece of evidence (the thing I thought he’d looked at) that it seems he hasn’t used… So I’m taking a look at the moment

  9. ICM’s SNP share was 5% across GB. For those that must look at crossbreaks of about 100 people (multiple caveats), it was 52% in Scotland

  10. Very interesting. Now how about investigating why the Greens poll ratings are currently being hyped up whilst UKIP’s poll ratings are being under-played?

  11. @KeithP

    Depends how you measure swingback – if it’s vs the actual election result, then yes it’s a component of swingback. Some people (including Peter Kellner) adjust for the error in the final polls – in those cases the two things are entirely separate

  12. Even if the polls are a bit too Laboury it’s all still within the 3% margin of error is it not?

    “I looked at how the polls for the European election did here and have the same figures as Rob. Of the six pollsters who produced figures within a week or so of the election five underestimated Conservative support. The average level of Tory support across those polls was 22.2%, the Tories actually got 23.9%. The average for Labour was 27%, when they actually got 25.4%”
    __

    The only thing that strikes me as being too Laboury is the amount of extraordinary media coverage Jim Murphy is enjoying in Scotland right now.

  13. @MrNameless

    Hollande has received a major boost in his personal ratings since the terrorist attacks. Admittedly from an historically low starting base, his 20% positive leap is the biggest recorded by French pollsters for a sitting President since polling on such matters began, and he now has the net approval levels enjoyed by the likes of Mitterand and Chirac at similar stages in their presidencies.

    Of course, as the article in the paper suggests, it’s what he does from here on in that is key, but Hollande now has a precious opportunity to redefine his Presidency and recast his reputation in the eyes and minds of the French people. Unusually for a politician, he has been granted a fresh start and I think there is a general view, even amongst his political opponents, that such has been the dignified and unifying leadership that he has displayed in the wake of the tragedy, that he deserves it.

    The transformation from partisan politician to national leader is a difficult and elusive one and, sadly, it sometimes requires a national trauma to provide the opportunity for politicians to become true statesmen, able to transcend the image of being a mere leader of a certain political faction or creed. Not all manage to rise to the occasion, but it would appear Francois Hollande has managed to do so.

    I’m glad for both him and his country. A confident and resurgent French leader is just what Europe needs, particularly as a counterweight to German dominance. On the subject of Franco-German relations, I thought Merkel and Hollande looked as if a certain, hitherto absent, personal chemistry was developing between the two of them at the recent Paris solidarity marches. There’s a classic photograph of them getting particularly close. Quite touching and unusual in the sense that Hollande isn’t wearing a motorcycle helmet! :-)

    @Anthony W

    I think you’re right to be sceptical about Mr Hayward’s hypothesis. It’s a bit like a cornered and beleaguered football manager saying “we didn’t get beaten 6-1, it was 6-2 and that will make all the difference in the second leg.”

  14. Allan –

    No, that’s not the way it works. If the actual figure is 25% then just through normal random sample error you would expect to get figures between 22% and 28%. However, they should be normally distributed, either side of the actual figure.

    If all the figures you actually got were between 22% and 25%, while they would all be within the normal margin, it would suggest some sort of systemic bias. It was the same with the Scottish polls – companies were mostly or all within the margin of error… but all their small errors were in the same direction.

  15. Quite recently, 18 Jan 11.57 p.m. or 23.57, Miserable Old Git posted some musical thoughts.

    This was further to a thoughtful post about how the situation for young people today is different from the favourable conditions that middle-aged people faced when they were younger. Middle-Aged Git gives examples of the differences between now and then.

    His musical example are the words of the song The End of the Rainbow by Richard Thompson.

    I should like to give my example of some of the best, most thoughtful words that I know in a pop song – words to guide people, young people or older people, maybe?

    Here are the words. It is quite brief.

    “Here’s a little piece of advice.
    You’re quite welcome. It is free.
    Don’t do nothing that is cut-price,
    Because you know what that will make you be.
    They will try their tricky devices.
    Trap you with the ordinary.
    Get your teeth into a small slice.
    The cake of liberty.”

    Do you recognise the song? It is Sex and Drugs and Rock n’ Roll by Ian Dury and the Blockheads.

  16. Most of these polls for months now have been ‘within the margin of error’. Lets go back to fortnightly or monthly polls and all calm down!

    :)

  17. Who will sus it before may 7 ?

    Ex tories voting ukip who will let in labour

    Ex labour voting snp who will let in tories

    Ex libdems voting green who will let in tories or labour

    Not much sign of any sussing yet

  18. @OLDNAT

    Are the SNP really at 7% in the ICM Poll, or is that figure for all Others including PC. The figures already add up to 101% ?

  19. BP
    Small point but SDLP I think take the Labour whip. Will Sinn Fein come to town? At some point they will.

  20. Toonie

    The SNP at 5% and PC at 1% according to Tom Clark’s twitter:

    https://twitter.com/guardian_clark

    OldNat only said “most importantly”, 5 out of 7 obviously is.

  21. @ROGER MEXICO

    Thanks for the link.

  22. PETE

    “Most of these polls for months now have been ‘within the margin of error’. Lets go back to fortnightly or monthly polls and all calm down!”

    The daily polls are extremely valuable in allowing rolling averages to be done over a fairly short timescale. Whilst this would not address systematic bias, it does allow random statistical noise to be ironed out.

  23. Barney Crockett,

    Yes, the SDLP effectively have a permanent tacit confidence-and-supply arrangement with Labour, in that provided Labour stays within an acceptable range of policy positions, the SDLP take the Labour whip. As far as I know, this has not significantly undermined the peace process. (I don’t think it’s fair to blame the SDLP for the deterioration of the peace process in the latter day Mowlam years, for instance.) And there’s already a history of governments using the DUP for controversial votes; Labour had no problems dealing with them in 2005-2010 to keep down backbench rebellions.

  24. Statgeek – fpt – re:Scotland

    “Very small sample in that poll’s CB, so not much to go on. In addition, about 40% of the Scottish sample is made up of WNV, DK, Refused.”

    You finish with the word ‘Beware’. I agree. It looks as though much of the Scottish electorate is playing its cards close to its chest. Much may depend on how things look south of the border, as Scots try to decide which way to vote. After all, we’ve got used to ‘horses for courses’ voting – marking the ballots in different ways depending on whether its Euros, Westminster, Holyrood or District.

    I think that if Labour do not enthuse the ‘don’t knows’, then the SNP will gain considerably in May. That’s not the end of Labour in Scotland, of course. They’ll be back at some point – because that’s the nature of a democracy.

    But with three and a half months to go, I find little to suggest that either the SNP are certain to be victors overall, or that Labour will hold on to thirty five plus seats.

    Or has someone else a clearer view of matters……?

    As for flag waving, I see a fair number of Saltire flags being flown from very middle class garden flag polls every day. Scotland is not England… apparently…..

  25. 07052015 – 3.36 p.m.

    Ex tories voting ukip who will let in labour
    Ex labour voting snp who will let in tories

    Wrong – or at least not so simple! The difference between the two lines above is obvious.
    An ex Tory voting UKIP may let in a Labour MP.
    No ex Labourite who votes SNP will let in a Tory except, possibly, in Dumfries and Galloway.
    Nor will any additional SNP MPs give the Tories any help at Labour’s expense whatsoever, for the Tories will still be short of an OM whichever of the two wins.
    Most Scots realise this, of course. If they vote Labour it won’t be because that is the only way to defeat the Tories.

  26. I see Smithson is getting in on the idiotic extrapolation from by-election-polling action. Oh God, this is embarrassing to watch.

    @ Barbazenzero,

    Strike-though is (pointy brackets) [strike] [/strike].

    Works for me, anyway.

  27. Sinn Fein on principle don’t take their seats but it does occur to me that in a very very tight vote (ie a budget or VONC) they could threaten to come in and defeat the government unless they got major concessions.

  28. @ Davemoon –

    UKIP have been more or less stagnant, fluctuation up and down but no clear trend, for months, now. The Green rise is much more recent, and much faster than any recent UKIP trend. The latter is newsworthy, the former is no more news worthy than the stagnating Tory and Labour vote.

    I find it funny, though, that any Kipper would have the gall to try to complain about not having enough media coverage.

  29. John B 07052015 – 3.36 p.m.

    Ex tories voting ukip who will let in labour
    Ex labour voting snp who will let in tories

    Wrong – or at least not so simple! The difference between the two lines above is obvious.
    An ex Tory voting UKIP may let in a Labour MP.

    ————————

    I agree but you are right to caveat that with “or not so simple”. SNP may be willing to do a deal with Tories after the election, Labour will not. Also it would help them be the largest party and so get first go at forming a Govt.

  30. With all this green hype we are going to see a lot of disappointed 18-24 year old middle class faces come May…

  31. Paul

    Where do you get the idea that the SNP might do a deal with the Tories? Do you not think the lesson of what happened to the LDs has been learned? It would take the SNP a generation to recover from such a move (IMO)

  32. Mr N

    Re SF

    That’s a point I’ve made before.

    What I haven’t been able to discover is whether there is any time limit after the election for MPs to take the oath.

    Obviously, the description on the HoC site of what usually happens with MPs who stood for election in order to get into HoC isn’t relevant to this scenario.

  33. Skippy,

    So many predicting a red green coalition and getting really excited about it on my social media. It is impossible.

  34. John B

    I suspect that much of the speculation about how NI and Scottish parties might be involved in government formation is based on very little knowledge of political dynamics in these countries.

  35. Would the SNP risk dealing with the Tories simply because one reason Labour has declined in the polls in Scotland is because they shared a platform with the Tories during the ndy ref. If the SNP actually were seen governing with the Tories even informally then they most likely will suffer a similar fate perhaps worse as they will be governing with them.
    Anyway the SNP and the Tories don’t have that much policy in common and certainly much less then Labour/SNP have in common. I cant see the Snp voting for change to the Human RI

  36. Mr N

    “It is impossible.”

    I’m surprised to see a Lab activist so certain that his party would be so opposed to Green principles that they wouldn’t work with the Green parties.

    Still, you probably know best.

  37. I cant see the SNP voting for changes to the Human Rights Act or the EU Referendum or even abstain in a vote. They would face Lib Dem style reaction in Scotland.

  38. And further: a Labour party led by EM might not want to do a deal with moderate Tories, (if such animals exist!), but some in Labour would happily do so in order to make sure we stay in Europe and avoided ‘left wing policies’ (see Mandelson today).

    Confidence and Supply may be offered to Labour by the SNP; it would not be offered to the Tories.

    A couple of months back I flew a kite regarding a Lab-Tory coalition as the only way of avoiding the SNP dictating everything. Few thought it likely. Judging by last Sunday’s polls, however, folk south of the border have no stomach for further constitutional changes. The SNP will not enter any deal which does not offer such changes – and which does not get rid of Trident. Perhaps we shall see the English (both Tory and Labour) joining to form a government……..

    NO, I don’t think it likely either, but it has to be one of the (many) possible outcomes of the GE!

  39. Old Nat

    There is no time limit, otherwise when the time was reached there would have to be a by-election and there haven’t been anyway.

    However, I can’t see SF MPs taking the oath whatever the circumstances. How could they “solemnly, sincerely and truly declare and affirm that I will be faithful and bear true allegiance to Her Majesty Queen Elizabeth, her heirs and successors according to law”?

  40. @OldNat “I’m surprised to see a Lab activist so certain that his party would be so opposed to Green principles that they wouldn’t work with the Green parties.”

    Nameless is right. Bennett has already all but ruled out being part of a coalition, and even if she hadn’t, the Greens won’t get close enough seats, even if their vote continues to rise, to become kingmakers because of First Past the Post.

  41. There haven’t been any not anyway!

  42. ON

    “I suspect that much of the speculation about how NI and Scottish parties might be involved in government formation is based on very little knowledge of political dynamics in these countries.”

    Good to be back in conversation with you! I agree. It worries me how so many on this site want to impose English assumptions (better – assumptions based on English political realities) on everyone else. You would have thought that on a site like this the contributors would realise that there are other political games going on, other than the English one of Tory versus Labour with additional entertainment provided by LibDems, Greens and UKIP.

    Now, I’ll be castigated for that…. sorry all!

  43. A potential SNP and Tory agreement would be around constitutional reform – EVEL, Devolution plus in Scotland. I don’t believe a coalition is likely, but in return for further autonomy, the SNP might support them in the lobbies.

  44. Norbold

    I suspect that, in the situation envisaged, a Sinn Fein member might well swear ‘true allegiance’ on the basis that such allegiance does not (in his/her opinion) exist. It would fulfill legal requirements, whilst in no way altering the MP’s view on what constituted truth.

  45. @ John B,

    Perhaps we shall see the English (both Tory and Labour) joining to form a government…

    It’s as likely as a Tory-SNP coalition, for precisely the same reason. None of the parties are so dedicated to the national interest that they’re willing to destroy themselves to provide stable government (except possibly the Lib Dems, but I suspect that was an accident and if they knew in 2010 what they know now they would never have gone in to coalition).

    What we might see is a Labour minority government in which Labour pass most of their legislation with the Nats and the Greens but rely on Tory votes for, eg. Trident renewal.

    Even that would play pretty badly in Scotland, but OTOH allowing a few Scottish MPs to override the overwhelming will of Parliament on a critical issue of national defence also seems like a strategy error, and more to the point one which would play badly in England where general elections are lost or won. So they might go for it if they couldn’t find a better alternative.

  46. ANTHONY WELLS
    Allan –
    “No, that’s not the way it works. If the actual figure is 25% then just through normal random sample error you would expect to get figures between 22% and 28%. However, they should be normally distributed, either side of the actual figure.
    If all the figures you actually got were between 22% and 25%, while they would all be within the normal margin, it would suggest some sort of systemic bias. It was the same with the Scottish polls – companies were mostly or all within the margin of error… but all their small errors were in the same direction”
    _____

    Thanks AW for explaining this.

  47. Paul

    In theory, yes. But political reality says ‘no’ for two reasons:
    1. the obvious one is the destruction of the left of centre SNP base;
    2. the other is the need for the SNP to demonstrate the ability to work with the powers already on their way – Smith commission plus…. ‘So far, so good’ seems to be the verdict, but Scots are wary of any major jump ‘forwards’, in case it turns out to be a jump ‘backwards’.

    EVEL is something which must be sorted out south of the border. Just as Scots did not want English interference in constitutional change here, so, too, we may assume that the SNP will not want to interfere in how the English govern themselves….. yourselves….

  48. Spearmint

    Yes, I see your point. And the SNP were happy to garner support from anyone who would give it to them when they were a minority at Holyrood, so could hardly complain if Labour did the same in Westminster.

    My concern, however, is how long a minority government could last at Westminster. And who would get the blame for its collapse?

  49. Norbold

    Good point about no by-elections. Thanks.

    As to the oath, SF MPs would take the oath as Irish citizens. I don’t know what Irish law says about the duty of its citizens to bear true allegiance to the British monarchy.

  50. Spearmint

    But your last phrase (5.38) would indicate you think a Lab-Tory coalition at least possible – or did I misread you?

1 2 3 4