Time for a post-mortem of the European election polling. I’m not a fan of the sort of horse-race approach to these things – just because they are the final poll of the race, they still have normal margins of error, so if one pollster is a fraction more accurate than another it is often just the luck of the draw. Realistically the best a pollster can ever hope to do is get all the results within the margin of error. Better than that is luck. However away from the public “who won” stuff, comparing poll predictions to actual election results is an absolutely critical tool for pollsters – it’s our chance to compare our figures with reality, to improve and finesse our methods.

The final polls from each company are here. Note that I haven’t included Populus – they did conduct one European poll, but it was a fortnight before the election when just a week is a long time in politics! While I’ve included it in the comparison, one should allow some leeway for ICM for the same reason; their poll’s fieldwork finished a week before the actual election.

CON LAB LD UKIP GRN Average Error
ACTUAL RESULT 23.9 25.4 6.9 27.5 7.9
YouGov 22 26 9 27 10 1.4
(-1.9) (+0.6) (+2.1) (-0.5) (+2.1)
ICM 26 29 7 25 6 2.0
(+2.1) (+3.6) (+0.1) (-2.5) (-1.9)
Opinium 21 25 6 32 6 2.1
(-2.9) (-0.4) (-0.9) (+4.5) (-1.9)
TNS 21 28 7 31 6 2.2
(-2.9) (+2.4) (+0.1) (+3.5) (-1.9)
ComRes 20 27 7 33 6 2.6
(-3.9) (+1.4) (+0.1) (+5.5) (-1.9)
Survation 23 27 9 32 4 2.6
(-0.9) (+1.4) (+2.1) (+4.5) (-3.9)

The most obvious current difference between Westminster polls is the reported levels of UKIP – there is a big gulf between the levels of UKIP support report recorded by companies like ICM, MORI, YouGov and ComRes’s phone polls and polls from newer companies like Opinium, Survation and ComRes’s online polls. We don’t know what the reasons for this are – there are a couple of things like prompting and re-allocating don’t knows that we can account for, but mostly the difference is not easily explained. It may be something to do with interviewer effect, or the representativeness of different companies samples. We can’t tell.

The European elections were obviously an opportunity to check figures against reality. I half expected the polls to all converge together in the run up to the election, as they have a tendency to do before general elections, but in reality we got the same sort of contrast as we do in Westminster polls. Higher figures for UKIP amongst newer online companies, lower figures from YouGov, lowest from ICM… and when the votes were counted the YouGov figure was the closest.

Of course, European elections aren’t general elections. On the issue of prompting, for example, every company prompted for UKIP in their European polling, whereas only Survation do it for general elections. There were no telephone polls for the European election, so it can tell us nothing of them. European elections are low turnout elections, so some of the errors may have been down to too strict turnout filters (ComRes used a very strict turnout filter for Euros and would probably have been better if they’d used the method they use for general election polling. There was the issue of the Independence from Europe spoiler party on the ballot paper and so on. At a purely personal level though, getting UKIP right at the next election is the biggest challenge currently facing pollsters, so I’m relieved that in the first real proper national test we got it right. Phew!


440 Responses to “European polling post-mortem”

1 2 3 4 5 9
  1. @Chris

    That worked. Use (omit the hyphens):

    (Upper) &-Chi-;
    (Lower) &-chi-;

  2. @StatGeek – That is sad. I was using what should have been the Unicode chi.

  3. Oh, okay – some sort of HTML?

  4. Χ
    χ

    &Chi
    &chi

  5. *sighs*
    I worked up a nice, detailed essay on the Euro elections…and I think linking to the article that RogerH linked to yesterday got it caught in the moderation filter. I /really/ wish I knew what triggered moderation.

  6. @Anthony Wells – As you will know, that “average error” is a rather blunt instrument, and really you should be using a chi-squared test. Also, you missed out the Other category, where YouGov’s performance was worse than other pollsters’.

    A chi-squared test, including a category for Others, suggests that YouGov actually only had the fourth-best final prediction (χ²=2.1), behind ICM (χ²=1.6), TNS (χ²=1.8) and Opinium (χ²=2.0). ComRes (χ²=2.5) and Survation (χ²=4.8!) still performed worst.

    (Reposted with chi symbols!)

  7. Gray

    The most likely culprit is the verb/noun l-i-e and its derivatives. Anthony’s automod thinks it’s John Bercow and any occurrence of these words get immediately censored.

    The Sc*nthorpe effect means that any other word that has the same letters in that order (such as underl*ing) also gets caught, so you’ll just have to refer to the underterminologicalinexactitudeing reason for things instead. Or find a more elegant way to put it.

  8. AW Thanks for clarifying.
    Re you and other pollsters looking to improve methodology:
    Average as calculated assumes source of error is random (fluctuation due to sample size?) I think you need to distinguish this from a set of various systematic errors (which may actin different directions and so look like random errors) But then if you find one acting to throw your results high, and eliminate it, then an undiscovered one may cease to balanceit out and now throws your results low, so that you appear to be (are in fact by eliminating some good luck) worse off than before.

    I speak from bitter experience, having struggled for three years eliminating sources of error from a method of measuring thermal conductivity, only to find at the end of it that removing various influences at times pushing the results up 10% in one case and down 8% in another etc, I was left with a 5% random? variation and could hardly better the initial results leaving the systematic variations out of account! However, it did reveal some circumstances in which a systematic error did throw the results way out (for example it was unwise to use very thin samples)
    In polling this might mean that some systematic error which has little overall effect on the polled VI for a major party (eg throws it say 1% high always) is more significant for a small party whose VI is in single figures.
    You might get at this by varying sample sizes, or conducting a duplicate poll on the same day with a different set of respondents. Obviously I have no idea about the practicality and expense of doing this, especially if the end result either shows nothing or indicates a small effect which is almost negligible in most cases.
    Having said that, some possible sources of systematic errors are discussed in your last paragraph, but do not apply to the EU poll.

    Finally, is there another Dave, or was it just some other quirk of the system?

  9. @Roger:
    Thanks for lightening my mood. The “L-Word” might be a culprit, and that can be worked around, though I didn’t use it in this essay. Trying to work around the “F-Word” (which is another plausible culprit) is a bit harder, though, in context. Any suggestions there?

  10. RogerH provided a link to an article on the EU elections in the previous thread that I’ve been working up a commentary on for a bit. I would provide the link, but I think the filter on here was causing me a fit with it.

    I really think Mr. Mason hit it on the head. He put down in words what I think I have tried to explain to people, but I’ll try to explain my thoughts:

    (1) The EU is, at the moment, a bit of a mess.

    What presently exists was assembled in substantial part due to processes which either avoided public engagement or which came across as heavy-handed. The Irish situation comes to mind, but some of what I’ve read about the EU Constitution campaign (with rather over-the-top campaigning by the “Yes” side) also highlights this. Ultimately, there are two underlying problems:

    -First, the institutions were not designed to be either straightforward or directly democratically responsible. The first point comes down to the fact that if you ask people who is able to do what in the EU, most couldn’t tell you. The second is that the Parliament can’t pass legislation without the Commission being on board, in spite of the fact that the Commission is at best an indirectly appointed body.

    -Second, there is a pretty clear attitude among the “elites” in Europe that amounts to “Get people in any way possible and they won’t be able to back out later”. Emblematic of this is how the Treaty of Lisbon was handled: Where the Constitution was going to be handled by referendum in a number of countries, referenda were painstakingly avoided with the Lisbon Treaty except in Ireland (where the court required them to amend their constitution)…and there, it went down to defeat the first time. Another example has been the handling of Greece (which probably ought to have been thrown off the Euro somehow, since a devaluation would have served them well). Basically there’s a hook-or-crook mentality to pushing the EU forwards and not letting anyone back out.

    (2) Consensus is a problem.

    This dovetails with the “elites” issues raised in point one, but it is a problem in and of itself. There is a cross-party consensus at high levels in favor of certain issues (relatively open immigration, for example, certain aspects of open markets, and a trend towards privatization) which are not also reflected in a popular consensus. When this happens, people are going to park their votes somewhere else. This may happen once or twice as a protest, or it may turn into a sustained defection.

    I would point out that by and large, the major parties in various countries by and large no longer contain seriously “socialist” parties, partly as a side-effect of EU legislation (for example, on the railways).

    The other problem is that there has been a tendency to deride anyone “not on board” with the consensus as being a fascist or something similarly toxic, and to subject them to withering attacks. This is probably a side-effect of the “cordon sanitaire” in Europe: There are positions that will get a party ostracized and locked out of power by other parties, so the “main parties” in a given country will not take up those even if a great many people support those positions.

    The other problem is, of course, that invariably governments make mistakes and lose support. When a government is by a broad, multi-party consensus then inevitably votes will migrate to “other” parties in some fashion…particularly if the result of the consensus government has been to force parties to abandon too much integrity in the interest of supporting the government. To put it another way, the parties inevitably get stuck supporting policies they don’t support. An excellent example of this problem can be found in both Germany in 2009 (where every significant party not in the grand coalition increased their vote share while the governing parties lost vote share). Another example would be in Austria, where the joint SPO/OVP government has slid from holding about 70% of the vote in 2006 to a shade over 50% in 2013.

    (3) The decline of anti-fascism: An interlude.

    One of the problems with the present consensus is that it assumes that an in-built knee-jerk reaction against fascists, dating from after the Second World War, would persist. An occasional extremist party might win a seat here or there, but such incidents would be rare and fleeting, often down to some scandal or other localized problem. Generations would be able to hand down an understanding that such parties were Very Bad Things and had to be stopped without fail.

    The problem is that this reaction was a result of a specific incident (World War Two). Unfortunately, V-E Day was over 69 years ago. While this reaction was built in at one time, it has faded with time, particularly as the WW2 generation has withdrawn from public life and then passed away. For someone to have been politically aware in the early 40s, you are basically down to people well over 80.

    A second problem is the over-application of the “fascist” label, among others. I will note that when someone was referring to a bunch of fascist parties entering the European Parliament, I was unsure as to whether they were lumping UKIP in there. I have heard this label (and similar toxic labels such as “racist”) applied rather interchangably. The incorrect over-application of a label will invariably dilute the power of that label…and when everyone who, say, wants to reduce immigration gets slapped with that label you’re going to find it applying to a lot more than just skinheads in a street fight.

    Finally, heretofore it has been expected that new parties on the “far right” (I use quotes here because assigning a part of the spectrum to these parties is not always the easiest thing in the world) would implode on their own. They would have their fifteen minutes of fame and promptly collapse. This did, in fact, tend to happen as the parties would “act up” repeatedly. Two things have changed in the last few years, however:
    -Some of the parties have gotten much better at presenting themselves. In some cases, this is a generational shift: Marine Le Pen is far more presentable than Jean-Marie Le Pen, and lacks the baggage of her father’s earlier career. In other cases, it may be the sheer amount of time “on the stage” a party has had…if a party or movement doesn’t die off quickly, it has a chance to learn from mistakes.
    -In other cases, such as Greece (or probably Belgium), a good number of voters have simply stopped caring.

    (4) Where this leaves us.

    Over the last decade or so, there has been a substantial migration of votes to parties opposed to the political consensus of the 1990s (which I discussed above). However, with the “main” parties in most countries effectively signing on to most or all elements of that consensus, voters have been stuck turning elsewhere. And this is where Europe has gotten into a spot.

    As Mr. Mason said, if the only option people have to vote against something they oppose is a “toxic” vote, a certain number of people will cast that vote anyway. If the only parties opposing the consensus on immigration are extremist parties, some people will end up voting for those extremists. If the only parties fighting the bailout in Greece are on the extreme wings, those extreme wings may win.

    Moreover, there is a palpable dissatisfaction with how the European project has played out. One example is that voters in the UK have been promised a referendum of some kind for quite a while (I think we’re looking at a decade or so now) and have not gotten it. In other cases, such as France and the Netherlands, voters cast a “No” vote and effectively got “Yes” anyway. When you add to this the fact that the general economy hasn’t exactly been great for about six years, this is not a formula for happy people. If anything it creates a “perfect storm” of disenfranchisement, both political and economic, which sends people looking for “something different”. When they cannot find that by switching parties within the mainstream, they will search elsewhere.

    (5) In closing.

    What we saw in the European election results was not just “protest voting” but widespread frustration at the direction people feel they have been dragged or misled over the last few decades with the EU. The opposition comes from multiple directions (left and right alike), in no small part because the consensus of 15-20 years ago is no longer nearly so broad. If anything, it would seem that there are an increasing number of people who consider it to be discredited.

    Moreover, with the rise of parties opposed to the consensus that can put forward eloquent leaders and which are not dragged down by a membership inclined to behaving badly and acquiring criminal convictions (such as the BNP), one can expect that this will not be a mere “flash in the pan”. It is true that some parties will see 2014 as their high-water mark. However, some others will likely keep growing.

    This begs a question: Can this be headed off? I don’t know. The EU structure is a bureaucratic mess that nobody seems to be happy with but that nobody can agree on what needs to be done. Somehow, I do not see the Left Party in Germany and UKIP agreeing on a package of changes. But absent a change of either structure or attitudes at the European level, we have likely not seen the end of this trend.

  11. @Dave – Anyone can pick any name they choose when entering a comment. I posted a few times as “Chris” before realising there was another.

  12. Do we not see here a factor we regularly see in British polling, whereby the Tories are marked down in polls, but get higher in the voting booth?

    Across the board almost we’re seeing higher polling for Labour LD’s and UKIP, and lower polling for the Tories.

    What is this phenomenon?

  13. Ok, it isn’t “Nazi” that caught it. I attempted a modified repost to clear up the issue (I copied RogerH’s link from yesterday in the first posting) and it still got caught.

  14. @Gray – A recent post of mine was automodded for referencing (in a non-judicial context) a former journalist who went on to work as an adviser to Cameron and is now in the dock. I think you simply can’t predict what will be modded.

  15. GORDONING

    All the results by district are on the BBC Vote 2014 web site.

  16. There’s speculation that the Deputy PM of Ireland is going to resign soon, after his party’s poor performance.

    Also the Spanish leader of the opposition has resigned, after PSOE hit a historical low of 23% (-15.8) and going from 23 MEPs to 14 (-9).

    PP suffered worse losses, in terms of % drop (-16) but only lost 8 MEPs (dropping to 16).

  17. dave

    Finally, is there another Dave, or was it just some other quirk of the system?

    Wasn’t the previous one a David? But as Chris Green said you can change the how your name appears to make it clearer.

    On the other hand, with that name have you thought of getting a job in the Cabinet?

  18. @Chris Green:
    Ugh. I do wish there was some way to avoid being auto-modded. This is particularly frustrating since I think I’ve been active here on and off for…I do not know how many years. I do not believe that I have ever had a post ultimately denied for content.

  19. I thought I’d apply the Nick Robinson “without London, where they always do well, this would have been a disaster for Labour” methodology in a slightly different way. Remove the South East, and the results look like this:

    Lab – 27.26%
    UKIP- 26.69%
    Con – 22.72%
    Green – 7.66%
    LD – 6.67%

    “Without the South East, where Labour never do well, these elections would have been an absolute triumph for Labour, beating even the UKIP earthquake. Since very few of their target seats are in the South East, they are clearly well-set for a resounding victory next year.”

  20. Breaking news – the Deputy Prime Minister and leader of the junior Coalition Party has just resigned because of the poor showing of his Party in the local elections and the European elections.

    Except that this is in Ireland and it’s the Labour Party (just heard it live so no link yet).

  21. *cough*

    Someone beat YouGov actually Anthony.

    http://www.peter-ould.net/2014/05/26/euro-elections-the-results/

    *cough*

  22. TF
    I don’t see why party leaders resign following poor results; after all, it’s the voters’ fault.

  23. Robin,
    Could you please provide the vote figures for those numbers?

    I’m not implying that I doubt you, it would just put my mind at rest..

  24. Congrats to YouGov!

    I thought I’d throw in a postmortem of the numbers backed out of the Ashcroft table(s) plus a GRN estimate. Here is a recap of the numbers, with the errors in brackets:

    Using table 2 only

    CON 23.8 (-0.1)
    LAB 23.3 (-2.1)
    LIB 5.8 (-1.1)
    UKIP 29.3 (1.8)
    GRN 11.5 (3.6)

    Ave error 1.7

    Using all tables

    CON 23.8 (-0.1)
    LAB 21.4 (-4.0)
    LIB 7.9 (1.0)
    UKIP 29.0 (1.5)
    GRN 11.5 (3.6)

    Ave error 2.0

    The tables still haven’t been published (and maybe they never will be!) but given that they seem to be very much in the ballpark, I’m glad I did this.

    Once again, thanks to Martyn for his input.

    Interesting that the LAB-CON spread was consistently 3-7 points in all polls but only 1.5 on the day.

    Also, the good Lord just tweeted that his weekly poll for this week will be out tomorrow due to the bank holiday.

  25. Robin

    :-) :-)

    You gat it girl. (I have been watching the Indy 500).

  26. @TingedFringe

    I took the votes total from the BBC website, and subtracted the SE votes total from the BBC website. Simples!

  27. @Howard

    I’m male! Although I *was* named after my mother!

  28. Presumably due to lack of public interest, politicians can make outrageously stupid statements in the aftermath of elections and expect to be believed. Miliband’s “Labour are advancing” and the Tory ” the UKIP voters will all come back to us at the GE”. These daft lies come across to those of us who do take an interest in national politics, as an insult to our intelligence. A straight forward admittance, like “we fcuked up” would be much better.

  29. Ireland’s deputy prime minister Eamon Gilmore has resigned after his party’s disastrous performance in European and local government elections.

    At a press conference in Dublin today, Gilmore said he had to “act on the clear message we received last Friday” in the elections.

  30. I have a feeling I have asked this before, but when calculating turnout, is it good / bad form to include rejected votes as part of the official turnout for calculating the percentage against the electorate?

    I tend to go with:

    Valid votes: 12345
    Rejected: 50
    Turnout: 12395

    Going on the logic that people ‘turned out’ and spoiled their ballots.

  31. @Roland Haines

    But….er…Labour did advance. By 10%. From a painfully low base admittedly, but they certainly advanced significantly.

  32. This been posted. Helps explain the shambles in Tower Hamlets last night. It’s from Tory leader. –

    “I was, for obvious reasons, at both omnishambles. We were asked to attend the Mile End Leisure centre at 2pm, Sunday for the council counts with the European count commencing at 4pm. We were let in at 3pm and then the chaos started.

    Any checks on people attending soon collapsed, particularly when the Supreme Ruler, Mayor Lutfur, was instructing his fan club from across London to walk in and enjoy the panto.

    However, ex Councillor Kabir Ahmed (Tower Hamlets First, Weavers ward) was barred from entering the count as he had revealed details of the first figures on his facebook page. Presumably he stayed at home in Gants Hill, Redbridge.

    There were arguments, threats, and chaos at the counting tables. Tower Hamlets First supporters were challenging vote after vote, forcing supporters of other candidates away from the tables. They often made their points ( excuse the pun) with pencils and pens, against ballot papers.The supreme ruler smiled, whilst checking town hall staff were not stopping his supporters from doing exactly what they wanted.

    Former Cllr Mohammed Shahid Ali (defeated Mile End) was bawling in Bangla down a mobile phone at a counting table. He was asked by a (female) officer to stop and he shouted that she ( emphasise she) had no right to tell him to stop doing anything and that she (emphasise she) should go away.I drew this to the attention of the returning officer and Shahid Ali then needed to be restrained from attacking me.

    Tower Hamlets has interesting rules on the media at counts. Mainstream journalists can only be on the counting floor if they are escorted by a member of the town hall staff at all times.

    The special media, that supports the supreme ruler, is excluded from this, so, far example everywhere I went I was stalked by a weird old trot who kept taking flash photographs of me in my face and then grinning. He declined to say what organ he reported for.

    He was not afraid of expressing his views as he shouted the short version of “see you next Tuesday”, during one of the declarations.

    Despite everything we had been told, the count was a shocker. There was a a 21% discrepancy in the votes in Island Gardens between the first two counts.

    Issues regarding our count process that I raised as far back as 2009 were ignored and there was no change at all; despite the fine words of the electoral commission.

    I will be writing to them on Tuesday, with more details and will make my letter public.”

  33. @Statgeek

    Turnout includes all that voted, not just those who registered a valid preference for a particular candidate or party.

  34. Ed -that is hilarious-unless you live there I suppose.

  35. ……….not quite as hilarious as that UKIP MEP for Scotland though :-)

  36. @Peter Ould.

    And when can we expect your GE prediction?

  37. @RAF
    Considering where they were in the opinion polls on this site 6 months ago and their results last night, I cannot see “advance” as quite the word to use. Presumably they wish to win the GE next year, at that rate their position at present, given the circumstances, is weak.

  38. May I remind all that the Mayor was re-elected with an increased majority. His policies are very popular.

    His party however are another matter.

  39. @Roland Haines

    Much of the drop has been due to the EP elections period – an area of policy where Labour are not particularly popular.

    I expect Labour’s lead to increase again in the next few weeks.

  40. @RAF
    Well I expect David Cameron to be chosen as King of the World with Boris Johnson as his Queen, but it ain’t necessarily gonna happen.

  41. raf

    I expect Labour’s lead to increase again in the next few weeks.”

    It will interesting to see if it does. What seems to happen is polling moves in line with media narrative and then moves back when the narrative unravels.

  42. Any evidence (apart from anecdotal) that there was a very late swing from ukip to conservative of 1 to 1.5%? Meaning the polls were much more accurate than we think given their time constrained remit.

  43. @ spearmint

    Very nice of you to do the prediction tables when I came out 3rd!

    I should really be disqualified though as I did it by the “school bully on the bus” method by going “oi wells give us your stats” and then changed a couple so the teacher would not notice…

  44. @Roland Haines

    At least my predictions are possible :)

  45. Not sure where this leaves the ukip vote for next year- technically Ashcroft thinks 50% may stay with them but this is on nearly half the turnout of a GE so it doesn’t stop them still only getting 7% and pollsters being right

  46. Deborah

    Any evidence (apart from anecdotal) that there was a very late swing from ukip to conservative of 1 to 1.5%? Meaning the polls were much more accurate than we think given their time constrained remit.

    Well it would have to have been very late – the YouGov poll was only finished the day before polling for instance. What may have happened however is that Conservative voters turned out to vote a little more enthusiastically than they said they would be.

    One of the surprising thing about the polls in the run up to 22 May was that those saying they would vote Conservative for Westminster were actually slightly less likely to say they would vote in the Euros than supporters of other Parties were (with UKIP the keenest). Normally Tories are always the most likely to vote and it may be these reluctant voters reverted to type and got out anyway. This would explain why the pollsters who used a strict ‘certain to vote’ filter were the ones who underestimated the Conservative vote the most.

  47. Robin
    You are the second lady I have apologised to recently, only in this case you aren’t.

    Ah well, vive la difference, except on UKPR.

    It was a very good post though. Congratulations.

  48. @RAF

    I’ve done one or two already – last one was here – http://www.peter-ould.net/2014/05/07/general-election-forecast-update/ – and I’ll be popping it onto it’s own site soon.

  49. NC: ‘it’s not the time to resign” – and nobody asked him what time (what circumstances) would be right to resign. But it’s not my business.

    An interesting promise from the MEP of the Party in Germany. He will resign in 2 months and somebody else will take his place in this way they will have 30 people benefiting from the 0.67% votes for a party whose slogans included”with Europe, without Europe). Well, it sounds better than UKIP MEP who’s said he won’t attend. He will sign though the registry to get his allowance above his salary.

  50. If we feed Friday’s projected shares of the national vote into Chris’s model, we get these predictions of voting in the 2015 general election:

    Conservative: 35.9 per cent (plus or minus 1.3)
    Labour: 30.9 per cent (± 3.4)
    Lib Dem: 16 per cent (± 2.29)
    UKIP 11.6 per cent (± 3.9)

1 2 3 4 5 9