If I were TNS or Opinium I would be rather annoyed today. Looking through social media, twitter and so forth there are lots of comments about the polls all being wrong and it being a terrible night for the pollsters, etc, etc. Both TNS and Opinium had final call figures of REMAIN 49%, LEAVE 51% – within a point of the actual result. Far from being a terrible night, they got it pretty much spot on, and should be getting congratulated.

The last general election was a disaster for all the pollsters. Last night wasn’t the same at all, it was a very bad result for some pollsters, but other companies did very well. Below is a chart of the Leave lead in the final results of all the pollsters who did a poll in the last week or so


The polls in blue were conducted online, the polls in orange were conducted by telephone. Note that ORB and Survation’s fieldwork both finished a few days before the referendum, so one cannot rule out a change in support in the days between their fieldwork and the vote itself. Disappointingly for me personally YouGov’s final poll had Remain ahead, albeit, only by two points. Unlike in May 2015 though I’ve a good idea of what went wrong (the turnout model we used for the poll weighted down people who didn’t vote at the last general election, when in reality turnout ended up being higher than the last general election), which is something that can be worked on.

During much of the campaign discussion of polls focused on the gap between telephone and online polls. The division is, as ever, really not as simple as that – Populus showed the largest Remain lead and it was conducted online, until they stopped polling a few weeks from the referendum ICM’s telephone polls were showing figures as Leave as their online polls. However, the general trend was clear – online polls tended to show a closer race than telephone polls; online polls tended to show it neck-and-neck, telephone polls tended to show Remain clearly ahead.

Many media commentators bought into the view that phone polls were “better” in some way, and should carry more weight than online polls (a debate I sought to avoid as much as possible, as there really wasn’t good evidence either way). I suspect this has played into the perception that the polls as a whole were wrong. If you’ve spent the last few months focusing on the polls showing a solid leave lead, and playing down the polls showing a neck-and-neck race, then you’d have been very surprised by last night.

The gap between online and phone narrowed during the campaign, and that was largely due to changes in online polls. The debate about the gap between phone and online polls has focused largely on potential differences in sampling – studies like that of Matt Singh and Populus found that people gave different answers to questions on things like immigration and national identity in online and telephone polls, that people in online sample seemed to be less socially liberal than people in telephone samples. In response several online pollsters adopted things like attitudinal weights to make their samples more like phone polls… perhaps, in hindsight, it should have been the other way around.

Since the error in the polls in 2015 I’ve said that the problems won’t be solved overnight. Pollsters are experimenting with different methods. Some of those things will work, some will not – it is a learning process. The record of polls conducted online is getting more promising – the performance of the mostly online polls at the May elections was mostly good, and most of the online polls for the EU referendum were either good, or at least only a few points out. While the problems of 2015 are probably not entirely cured yet, online companies are showing clear progress, for some phone polls there is clearly still work to be done.

2,711 Responses to “EU referendum post-mortem”

1 2 3 55
  1. Independence Refs for Scotland and London; start polling.

  2. hmmm

    don’t remember it being a triumph for the pollsters actually

  3. Thanks AW for an interesting comment on the polls in general. As I posted earlier i thought three online polls were within normal error limits , so quite good. Two were very close. The other looks very wrong Populus have some work to do.

  4. NickP

    Most of the polls were showing a statistical tie. I thought before that anything closer than 53-47 either way would be a pretty good result allowing for MoE

  5. With a 12 point spread you can fit pretty well any outcome within the range, but it’s another problem that you have such a large spread and then only just got the right answer within that spread.

  6. So reassure me here guys. Scotland very likely to hold a second referendum and quite likely to vote for independence, Northen Ireland discussing uniting with the South (particularly worrying for me as I have a brother over there who’s just gotten married and taken out a mortgage) and financial turmoil (albeit potentially short term).
    I voted for Remain, am 30 trying to save up for a house myself, work for the MOD and am trying to see the bright side here.
    At least we won’t have to hear about David and George’s long term economic plan again (unless that was part of the plan!)

  7. Pollers messed up big time, the weights about turnout from classes C D AND E maybe wrong? Certainly the old vote held up and voted out. Obviously, I got it very wrong and the as some said last night, the vote didn’t move to the status quo, it moved the opposite way on polling day, which is really unusual.

  8. @AW
    “…If you’ve spent the last few months focusing on the polls showing a solid leave lead..”

    Should that have been “…remain lead”?

  9. DSL,

    Scotland certainly. In fact it’s a symbiotic process. Sturgeon won’t push for indyref2 unless there are lots of polls that put support at 55% plus.

    London is just silly. Anyone who imagines that London could survive as an independent state, detached from the Home Counties that surround it (and where so many of its employees live) is delusional. It’s not all about the size of population or the size of an economy. It’s about the hinterland. London is not Singapore or Hong Kong, it is intrinsically linked to the counties it pinched it’s land from (and some “Londoners” refuse to lose that link, considering themselves to be in Surrey, Essex, Herts etc).

    Can you imagine border controls on virtually every tube line and overground line into the Capital? Literally millions of checks on documents every day?

    And how would London cope with the 200k immigrants it would receive every year without ever being able to expand its borders?

    It’s just cosmopolitan London venting frustration at a result it mostly didn’t want.

    Scotland and Northern Ireland is different. My personal belief is that neither will leave the UK. But there’s a significant chance that Scotland will, and a small chance that Northern Ireland will. In Northern Ireland I think you have two groups of “Remain” voters with different agendas. I suspect the Unionist remainers will coalesce with other Unionists rather than with Nationalists, as the Unionist part of their makeup is much stronger than the Remainer part.

    In Scotland, a lot will depend on Ruth Davidson. I expect she will enjoy even more prominence, and if she continues to perform as well as she has being, that should help the “No” vote. However, she was pretty passionate about “Remain” and may not be comfortable with the direction of the party in the years leading up to an Indyref2.

  10. Neil A

    Go to Geneva. It could happen.

    Probably not though, but that is also true of quite a lot of other things that have ended up happening in the last ten years.

  11. Just caught up with the Boris speech, apparently all is well, were just as European as ever, it wont affect anything ya ya ya ya…..

    One day the gall of the man will catch him up good and proper.

  12. PEEBEE

    Soon rather than later I suspect.

  13. @Neil A

    Very little will depend on Ruth Davidson. The key issue is the SNP and Green axis and the views of individual SLab and SLD MSPs who now have liberty to make their own minds up.

  14. I think Scotland is quite complex Support for independence does not directly correspond with support for Scotland in the EU

    I’m not sure how many previously voting NO for independence would vote yes instead to stay in a Europe without England?

    Go to Geneva. It could happen.

    Quite so, and it was the case for many years before Switzerland joined Schengen.

  16. Re Northern Ireland:
    I think the simple unionist v republican view is too simplistic when it comes to Remaining in Europe. There would be nothing to prevent Ireland setting up a federal structure with Northern Ireland retaining its assembly within a structure in Europe that protects minority rights. My wife is Northern Irish and in my experience of visiting Belfast what is surprising is that the need to be British is balanced by the need to be devolved.
    I think we are in for a period of uncertainty and I for one cannot predict but would not rule out any feasible outcomes.

  17. Owen Jones has it about right –

    “In a matter of months, he [Cameron] went from suggesting he could support British withdrawal from the EU to warning of economic Armageddon if the country did so. It looked preposterous. He spent years suggesting immigration was a huge problem that needed to be massively reduced, and failed to do so, breeding further contempt and fury.”

  18. While we know from last night that markets know nothing, a skim through the exchanges tells an interesting story.

    The FTSE is down 4.4%, with a modest recovery and now stabilization from the early big falls, whereas the DAX is down over 7%, the CAC down 8.6%, and the IBEX down by over 12%.

    It couldn’t be that markets think European economies face worse consequences than the UK, could it?

  19. In spite of Anthony’s post, this morning there was a hell of a lot of head hunting for polling companies in LinkedIn – in particular theoretical (methodological) statisticians. It is actually not a joke.

  20. Even the currency moves might not be as dramatic as though. The £ is at $1.37, just a touch below the $1.41 it hit on June 14th. Similar story for the Euro.

    Instant market judgements are poor, so we’ll have to wait and see, but I suspect the headline falls have as much to do with markets misreading the polls and artificially rising in the last ten days of the campaign as with the perceived impacts of the actual results.

  21. According to John Redwood, all will be well. We’ll just take back our money and keep all the trade bits that we like. No reason to go with the timetable in the Lisbon treaty at all. Johnny foreigner will just have to do as he is told. That’s ok then!

  22. I suppose it is a given now that, not only will the UK leave the EU, but also Scotland will leave the UK. In other words, it is the end of the UK as we know it, with England going back to its original borders. After a prolonged period of economic turmoil, this new “Little England” might eventually stabilize as relatively prosperous, small open economy, but it will never recover the status of a global or even regional power, which is quite sad really, but was the choice of the English people.

    The ironic part is that many people in the UK may have voted under the misguided pretense that a vote to leave was not really “a vote to leave”, but rather a way to put pressure on the EU to renegotiate UK membership in more favorable terms than Cameron had secured before. Those misguided voters, who believed in a “second confirmation referendum” and other similar implausible scenarios, are now waking up to the tough reality of France, the European Commission and even Germany playing hardball and calling for a “quick” negotiation leading to ta ” clear and unequivocal” breakup with the UK under the very strict terms of the EU treaties.

  23. I think the big issue for the SNP will be currency.

    Their biggest pitch in 2014 was keeping sterling. rUK will still be there, and will still be the biggest market, so maintaining a shared currency would presumably still be the preferred option for the SNP, for all the reasons cited in 2014.

    However, this dredges up all the unresolved arguments about an agreed shared currency arrangement, which the rUK may not yield on, along with the added complexity now of whether iScotland would need to adopt the Euro. If this was required as part of the EU membership deal, which seems highly likely, then the SNP could not argue the independence case based on retaining seamless trade with their biggest market.

    Even putting aside oil prices, which will bounce back one day, those assuming Scottish independence is a foregone conclusion need to understand that the economic case has become infinitely more complicated to make now.

  24. @Alec

    “The FTSE is down 4.4%, with a modest recovery and now stabilization from the early big falls, whereas the DAX is down over 7%, the CAC down 8.6%, and the IBEX down by over 12%.”

    The FTSE 100 is a highly internationalised index – there are a number of major companies on there who are listed in London, but the bulk of whose business is actually elsewhere (often truly global).

    A better measure of how markets view prospects for ‘UK’ companies might be the FTSE 250. After sharp initial falls, it rallied, then fell back and has flat lined at 8.4% down on the day. This is much more comparable with other major European markets.

  25. @TOH
    ” i thought three online polls were within normal error limits , so quite good. Two were very close”
    If the polls have an error of about +/-2 or 3%, they are bound to be nearly right if the result is within 2-3% of 50-50.
    In other words you can’t be sure of measuring to within 1cm with a ruler marked in inches.

  26. Well congrats to some of the pollsters and a boot up the arse to some others..

    Something should be said of the wee specky guy over at Political betting.com. All day and all evening he was going ape with his tweets …81% chance for remain, 90% chance for remain, 85% chance for remain and on he went ..tweet after tweet after tweet…I’m not sure who his sources were but I wouldn’t be surprised if they came out of a Christmas cracker.

    He was even tweeting gibberish about private exit polls, I mean the guy must be wired to the moon.

  27. ALEC: Surely the $1.41 was a low point when LEAVE caught up with REMAIN in the polls. It was $1.47 in May, before the jitters set in, and $1.50 when it was thought REMAIN had won. $1.37 is quite a drop.

  28. I think London could be a very rich Monaco of the north.

    Redwood is an idiot and now we see why he was chopped out. We can just do it slowly and put the pieces in place and live with humungous uncertainty. Yeah, business will love that. All those sparkies, retail and service staff, chippies, brickies, hair-dressers who believe it won’t affect them because they don’t export – you’ve gonna see some very bad effects pretty quickly and you will be hurt the worst. The rich can protect themselves, the poor can’t.

  29. “ALEC: It couldn’t be that markets think European economies face worse consequences than the UK, could it?”

    It is more likely the markets are worried about the breakup of the entire EU, whether there will be other referendums.

  30. Investment hits zero. No investment means less jobs. No company is going to invest in ‘we haven’t got a clue how it will pan out over the next 2.5 years’.

  31. Alec

    I think a lot of it is the shocked reaction to expectations being wrong – expectations which as the OP says were actually not entirely rational unless perhaps people were fixated on other signs like the crazy betting odds and the populous poll.

  32. There was a bloc of risk-averse voters who said ‘No’ to Scottish independence partly out of concern that the change would be sudden, but who would have supported a devo-max option.

    Those voters voted Remain yesterday and would be likely to support moves towards Scottish independence if that meant continuing as part of the EU. For them this result may have made independence the continuity option, the less risky option.

    I don’t know how many of them there are, perhaps it’s time to start wading through historical data…

    My hunch is that there are quite a lot of voters in Scotland and the rest of the UK who have come to feel that whether they like it or not the direction of travel is firmly towards Scottish independence and it’s a question of when, rather than if. The narrowness of the Scottish result crystallised a growing sense that the UK is no longer what somebody referred to the other day as a ‘natural demos’. The result of the EU referendum reinforces that.

  33. @Bardin1

    Some of us wonder why Scotland would leave one union ( the UK) to stay in another ( the EU). If Scotland wanted to join the Euro it would make more sense.

  34. ALEC

    “Even putting aside oil prices, which will bounce back one day, those assuming Scottish independence is a foregone conclusion need to understand that the economic case has become infinitely more complicated to make now.”

    I watched part of Sturgeon’s speech on BBC24 and she did not mince her words and looked and sounded very angry at Scotland being dragged out of the EU and you have to acknowledge the remain vote in Scotland was overwhelming.

    I agree the economic case for independence is now more complicated because effectively we are talking about a rUK EU border post Scottish independence.

    On the back of the Scottish Gov and UK govs being on the same side for the EU vote Westminster might offer Scotland some big concessions, ie devo max with even control over immigration in return for the Scots not to call a indy ref 2.

    Scotland appears to be very angry with the EU vote but if I was NS I would hammer for a better devolution settlement with Westminster and I’m talking federal stuff here.

  35. “Even putting aside oil prices, which will bounce back one day…”


    Well there are arguments that they may never recover to what they were…

  36. First minister of Wales wants the treasury (England really) to compensate for the lost EU subsidy for Wales …

    Really crazy times …

  37. so now its over how many of the dire warnings we have heard will prove accurate?
    if not another good reason for cameron and osborne to go or look really stupid
    seriously the imf obr treasury etc were either wrong and should be scrutinised as to why they got it so wrong=or right and its going to be a total disaster

  38. @Alec “It couldn’t be that markets think European economies face worse consequences than the UK, could it?”
    It might well be. They sell more value to us than we do to them, and a weaker pound makes their goods more expensive in Britain (unless they cut their prices in home currency) and our exports cheaper.
    But the other factor is how quickly can the EU react to rapid market fluctuations, and indeed take any momentous decisions?
    I see that Tusk is saying “EU law will apply in UK for next 2 years”
    (I thought we already made our own laws?)
    Others are saying we must leave as soon as possible; others that negotiations will drag on for 7 years,
    UK has not yet used Article 50; Cameron’s timetable is ‘not while he is PM’ EU voices want it now, but it is not up to them. When France voted against the EU Constitution, they were not thrown out, but nor did they seek to leave.

    We shall soon find out how a European ‘superstate’ is managed. Let’s hope they don’t need to declare war on Russia.

  39. Actually yougovs final poll had Remain winning 52 to 48 percent therefore Leave should be -4 not -2 in the chart.

  40. DAVE

    “We shall soon find out how a European ‘superstate’ is managed. Let’s hope they don’t need to declare war on Russia”

    According to some in NATO Russia could easily over run NATO in a matter of days if a major conflicted was to break out and by the time US enforcement arrive Russia would have probably achieved its objectives, pushing back NATO from its near abroad.

    I also don’t understand why Putin is moaning about a few NATO battalions holding war games near his borders.? What threat can they pose to the Worlds biggest Nuke power? It’s all political bluster from both sides…Putin needs to be seen as a strong man in Russia and NATO need a reason to exist, both need each other.

  41. @McClane

    It’s possible there will be a reason that one doesn’t count. Roger can prolly explain…

  42. The critical issue is actually how the EU will now react to Scottish independence. Previously, the EU was very hostile to it and to a renegotiation of Scottish membership, but that might change now. Some countries like Spain may still oppose it because of the Catalan problem, but the European Commission and countries like France or Germany might support an independent Scotland even in the Euro zone.

  43. Lord Ashcroft’s article and associated tables are now available:


  44. It’d be a boost for the smuggling industry if Scotland leaves the UK. The gift that keeps on giving!

  45. Cricket’ll be starting shortly for any needing respite.

    (Last game was quite the thriller, game tied after Plunkett struck a six off the very last ball. Closer than the polling!!…)


    Actually yougovs final poll had Remain winning 52 to 48 percent therefore Leave should be -4 not -2 in the chart.

    You’re thinking of the poll that came out at 10pm on polling day which asked how people had voted. Anthony is referring to the last one before polling day (the ‘Final Call’)[1]:


    which had Remain at 51% (it was actually even closer at 45% each and Remain was only boosted by a ‘squeeze’ question).

    Obviously the fact that when YouGov asked about how people had voted, it was wrong suggests their sampling and adjustment wasn’t completely accurate – though not far out. It also suggests a small swing to Remain on the day.

    [1] Though this would have include about 30% of people who had voted already by post.

  47. MBRUNO

    It certainly puts the Scots in a stronger position but before talk of a Scots bolt kicks off we need to see evidence if Scots are really disgruntled by leaving the EU or not. Is the EU such a big issue for Scots that they would vote to leave the UK to join it?

  48. The split in the Labour Party is gathering force. While Corbyn is probably safe if he is on the ballot the divide in the party is now perhaps worse than it’s been since his election I wouldn’t be shocked if an SDP style split happens at some point this year if Corbyn or Mcdonell is still leader on 31st December.

  49. @ Anthony Wells,
    With respect Anthony, why should the pollsters be left out of the imbecile ravings on social media. Just like any other topic which attracts the attention of social media, any semblance of truth or accuracy is of no consequence.

    “It’s all political bluster from both sides…” Exactly. Let’s hope it’s never real. That NATO analysis is probably all too accurate. In past times decades ago, the question was never how a USSR advance could be stopped – always how soon tactical nucklear weapons might have to be deployed.
    “The critical issue is actually how the EU will now react to Scottish independence. ”
    There is ‘now’ no Scottish independence; no Scottish independence referendum; not even a UK Article 50 notice of withdrawal. The EU has tougher decisions to take than bothering about a hypothetical independent Scotland.
    “Angela Merkel, expressed “great regret” at Britain’s decision, but said the EU should not draw “quick and simple conclusions” that might create new and deeper divisions.” Note the wording of the Guardian link:

1 2 3 55