The Scottish polls at the end of last week and the weekend were broadly clustered around a small No lead. There are a couple of days campaigning left, but perhaps a more likely route to a YES victory is if the polls are underestimating the level of YES support for some reason. Over the last couple of days I’ve seen several blogs or articles pondering whether the polls could be wrong, could they be underestimating YES or NO?

It would be hubris to suggest the polls couldn’t be wrong. Obviously they can. At most elections there are polls that perform better or worse than their peers, some of that is better methodology. When the polls are close most is probably just normal sample variation. That’s a matter for another time though, here I’m pondering more about the possibly that all the polls are wrong, the potential for a systemic bias with everyone a bit too yes or a bit too no. This is possible too – think of the way all polls overestimated Lib Dem support in 2010, or most famously how all the polls overestimated Labour support in 1992. How likely is that?

The Scottish referendum is a bigger challenge for pollsters than an election would be because it’s a one-off. In designing methodology for voting intention the experience of what worked or didn’t work at previous elections weighs heavy, and most companies’ weighting schemes rely heavily upon the previous election – if not directly through weighting by recalled vote, in using the data from the previous election in designing and testing other weighting targets. For a referendum you can’t take that direct approach, pollsters needed to rely more on modelling what they think is an accurate picture of the Scottish electorate and hoping it reflects the Scottish people well enough that it will also reflect their referendum voting intentions – it’s complicated because Scotland has a complicated electorate. Scottish voters have two Holyrood votes and a Westminster vote, and they use them all in different ways with different political loyalties. Within the space of a year Scotland managed to be a Labour stronghold at Westminster and to produce a SNP landside at Holyrood – using either election alone for weighting gives a rather different picture of what the Scottish electorate are like, even though you are trying to model the same population. Different companies have arrived at different methods of political weighting to deal with the issue – Survation, ICM and TNS weight by Holyrood recalled voted alone, YouGov weight by Holyrood recalled vote with a nod towards 2011 Holyrood voters who backed Labour in 2010, Opinium weight by Holyrood and Westminster recalled vote, Panelbase weight by Holyrood and European recalled vote, Ipsos MORI don’t use political weighting at all. Despite the variance they have all converged to produce the same sort of result, and that gives me some confidence – if there was a particular skew from being online or from using Holyrood recalled vote we would expect to see different results.

Most speculation about whether the polls might be wrong has – rightly in my view – concentrated on two particular issues. Very high turnout and differential response rate.

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election. This poses a problem. Any professionally run opinion poll will make every effort to get as representative a sample as possible, but in practice there are limitations. People on the very fringes of society, people struggling in absolute poverty, those utterly detached from mainstream politics and civic society – people on the extreme edge are probably underrepresented in opinion polls of all sorts. In most voting intention polls this doesn’t matter, as people on the very fringes are also extremely unlikely to vote… but if the Scottish referendum does manage to engage some who were previously totally detached and, crucially, those people vote in a substantially different way to other people of similarly marginal demographics, then it could be a source of error.

The second potential pitfall is differential response. Much of the media discussion around this has called it “shy Noes” – people who want to vote no but are reluctant to admit it to pollsters. That’s possible, but it should be much less of a problem with online polls when people are giving their opinion to an impersonal computer screen. I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother? Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.

So, how confident am I about the polling in the Scottish referendum? Well, I suppose I’m fairly confident – if there was anything I thought we were doing critically wrong we’d have corrected it. If I had to put money on the result, I’d certainly back the polls, but the potentials for error are there. We’ll know on Friday if they’ve been avoided.


199 Responses to “Will the polls get the Scottish referendum right?”

1 2 3 4
  1. Good luck with your final Yougov polling!

    I’m predicting a result of No 56 – Yes 44 as I think there will be a small but significant Shy No / Spiral of silence effect in the polling.

  2. Very detailed & good analysis Anthony. Are any further polls expected today?

  3. Barnaby – Opinium tonight and *possibly* TNS (I had heard they weren’t doing a final call, but since then I’ve heard suggestions they will, so I’m unsure!)

  4. @AW
    Best of luck.

    @NS
    My very first prediction was No to win by 5%. I then upped that to 10%. I now think that the polls are likely to be right and that we are heading for the tightest finish imaginable – it could come down to a few thousand votes. Technically then, Yes 49, No 50. Actually Yes 49.4, No 49.6

  5. An interesting article, certainly a full one. It will be interesting to see result.

  6. Oh Anthony, get off the fence…Yes or No?

    Peter.

  7. With all the various methodologies producing increasing similar results (bar one!), I would merrily conclude that the polling agencies have it pretty much nailed.

    I’d say the only variable left unconsidered would be when the voter gets into the booth and picks up the pen. Will ‘greater autonomy with a safety net’ appeal more then ‘stepping into the unknown’ at that fateful moment?

  8. Ps In his latest blog Mike Smithson highlights Ipsos/MORI poll as one to look fir due to accuracy if their poll in Scottish election of 2011. Is that a fairly accurate assessment AW?

  9. Excellent article AW . I’m still going for a 55% -45% for the winner but who will that be?

    If the polls (so far) are correct then I agree with RAF “Actually Yes 49.4, No 49.6”

    Again we have those in the margins and turnout could be a factor…The truth is..we all don’t know..

  10. It’s worth noting that Bing Predicts is showing 52.5% No and 47.5% Yes. (Just type in “scottish referendum” into Bing’s search engine to get the prediction).

    They arn’t opinion polling – they are using algo analysis of searches from Scottish IP addresses and social media.

    Be interesting to see if they get closer than the opinion pollsters. FWIW they showed a narrowing of the race before the pollsters did.

  11. Excellent analysis, AW.

    Of course, if the differential responses and high turnout both apply to shift the poll in opposite directions, you could end up back in the middle, with a 50/50 split.

  12. On the marginalised groups side of things I thought I’d have a quick look at the strong SSP Holyrood performance in 2003 which surprised a few people with the number of MSPs they had elected.

    My logic being that 2003 SSP voters might have some similarities with the sort of RIC Yes voter that might be hard to reach in polling.

    SSP got 6.7% of the regional vote (and 6 MSPS) in 2003.

    In the final 2/3 polls before the election:
    TNS had them on 6, 10 and 7
    Yougov had them on 8 and 10

    So it seems that potentially marginalised Holyrood SSP voters were measured fairly accurately at that point.

    Of course the sort of people RIC are targeting may never have voted at Holyrood and be even more marginal than 2003 SSP voters but I thought it might be of interest.

  13. There is one thing that surely benefits the NO vote, and those are the postal votes. When those went out, NO had a bigger lead, so they accumulated some advantage beforehand. To counter that, the YES votes should have a strong (3%+ lead) on polling day to turn it around.
    So my guess is a very close (52-48) NO victory with a high (80%+) turnout.

  14. @Adams

    Postal votes went out in the immediate aftermath of the 2nd debate which may have been related to the rise in Yes polling numbers. (I’m not so sure of the correlation personally)

    So you can make an argument that the postal vote timing will have helped Yes rather than No.

  15. @ Anthony

    Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.
    ——————-
    The Labour voters are not the best example, as I see it. The issue is the responses from the 50% of registered voters who didn’t vote in the 2011 elections. It’s whether or not the responses from this group are too Yes-y because of over-enthusiasm or too No-y because of the ‘marginalised’ voters not being included or whether the two balance each other out.

  16. Ooh i will have a go at this! 56 percent no ,44 percent yes and an 82.4 percent turnout.

    May i also point out that it has turned from a reasoned debatish type issue into total farce.I will be glad when it is all over and our country can be united again as one people not factions.

  17. @Northumbrainscot

    Oh, I thought the postal votes went out just before the 2nd debate. Oh, well… then it’s going to be a tough one, down to the wire.

  18. ICM’s Martin Boon seems to have an identical view to Anthony – which wouldn’t be surprising.

    http://www.heraldscotland.com/politics/referendum-news/icm-director-boon-admits-a-polling-waterloo-is-possible.25336542

    Though I prefer Anthony’s more measured language! :-)

  19. @ NorthumbrianScot

    I know of quite a few Labour voters (not me) who voted 2003 for Labour constituency MSP & SSP regional MSP. Anecdote isn’t data, of course – but I’m inclined to think that the 2003 SSP vote didn’t necessarily come from the ‘marginalised’ voters.

  20. I’m still going to stick with a 60/40 No on a 85% turnout.

    What can I say, despite aaaaaall the ‘grievances’ I have heard from the nats, I still think Scots are overwhelmingly an intelligence and rational bunch.

    Post-referendum, there is going to be a lot of work needed to dump all the bile and venom into a skip and start rebuilding a cohesive and mutually supportive society.

  21. @ Steve2

    I’m still going to stick with a 60/40 No on a 85% turnout.
    —————
    Me too.

  22. @ Amber

    Quite possibly not, I was just trying to think of a proxy for a campaign that appealed to the marginalised in recent Scottish electoral history and that was the best I could think of.

    Weighting by recalled vote SNP 2011, Labour 2010, SSP 2003 might be informative but how accurate an 11 year old recall might be is probably up for debate!

  23. “…intelligence and rational bunch”

    *facepalm*

  24. Anthony,
    I recall the polls being very accurate for the 1975 EEC Referendum – and indeed for the later Referenda in Scotland and Wales in 1979. Why has such polling become more problematic over the intervening years?

  25. My guess is Yes 47 No 53 with a 82.6% turnout

  26. Northumbrianscot

    I don’t think there can be any proxy for the marginalised voter phenomenon. Even going back to the record high voting figure of 81.5% in the Scottish 1951 election, it was in a very different society.

    In 2003 both the parties and the electorate were “experimenting” with how they wanted to use the Additional Member System.

  27. Either way I think Salmons on a winner, he may well win and get full independance and if he loses he gets far more powers and cash for Scotland. Not only that the cost to the English tax payer if he loses will split the Westminster parties assunder as the cry for an English parlement goes viral. As I see it its heads Salmon wins and tails England loses

  28. Graham

    The turnout in the 1997 Scottish referendum was 60%.

  29. @Adams

    It may vary by LA the date they go out but the first Social Media mentions I saw of people getting their votes were the days after the second debate.

    The debate was 25 August and just checking the websites of the two biggest Local Authorities Edinburgh gives 27 August as the date their postal votes went out and Glasgow 28 August.

  30. Speaking of experimenting with AMS does anyone know if Scottish Labour have changed their prohibition on standing for Constituency and List in Holyrood AMS elections?

    On the one hand I think it helped bring in some fresh faces in 2011 but going forward I think they have to adopt the SNP approach to ensure more long term consistency amongst their MSP intake.

    Possibly 2016 election is still too far away for a policy to have been adopted but seem to recall it was one of the issues Johann Lamont was going to look at after being elected leader.

  31. I predict a victory for democracy, either way, and a very, very long and dirty election leading up to May 2015, either way.

  32. close to 60/40 or my name is not Angus Caber-Tosser McKilt.

    [Pour NON – absolutement]

  33. Interestingly, if we do vote Naw, then half of the countries in the UK will have voted to be part of the UK. Contrast this with the USA, which fought a long and bloody war when roughly half of their states tried to leave.

  34. Excellent as ever, AW. I, too, feel that it is possible that the Yes vote is being overestimated on the basis of enthusiasm. I would like to think otherwise, of course, but even I have to be realistic sometimes!

    Anyone hear Christopher Chope on The World at One? Doubtless speaking not only for himself but for many backbenchers (and not only Tory ones!) he as much as said that ‘The Vow’ from the three party leaders isn’t worth the paper it’s written on. (Not that I’m surprised, of course – if they were serious about it they would already have got a Bill through parliament to implement the Brown ‘proposals’ as of September 19th, as I wrote on the previous thread).

    Now, I wonder if AS is thinking of offering Chope Honorary Citizenship of Scotland after the result?

    I, though, am still expecting a No vote, though not by much 53-47, perhaps.

  35. @RoyKite – I reckon they’ll lose the Scottish Parliament next time (regardless of result). A lot of people I know (non-enthusiastic yesses) are very hacked off with the yes campaign and Salmond in particular, I could forsee an anyone but the nats line develop in peoples thinking.

  36. Graham:

    It’s not that polling is more problematic but that some polling circumstances introduce more potential for systematic bias than others.

    Without listing all possible biases the pertinent one here is that some polls refer to more socially charged issues than others.

    And since these are dependant on human psychology they are very difficult for pollsters to predict.

  37. Northumbrianscot

    Amber will know more exactly, but Labour have certainly removed the total ban on standing in constituency and list.

  38. Chope is clearly eyeing something as he is so regularly quoted – would have thought the whip would have been after him already.

  39. iirc Dewar was #1 on the Glasgow list in 99.

  40. Referendum expected declaration and expected result:

    http://vis.oobrien.com/indyref/

  41. Should have posted Twitter feed, as comments are interesting:

    https://twitter.com/MSmithsonPB/status/511789176325701632

  42. And I would go along with Roykite (1.58). The renowned poker player may not have got everything he wanted, but IF ‘The Vow’ goes ahead he’s got a lot more than he started off with. Gradualists, such as myself, delighted!

    But see my previous contribution, re Christopher Chope. DC may well be in huge trouble with his backbenchers. EM probably less so with his, but he has been weakened by being seen to go along with big business, bankers and so on (as well, of course, as the Tories). Whichever way it goes, AS seems to have proved himself to be rather a good player of the game….. more so that I thought at certain points along the way.

    But let’s wait until the result comes out. I’ve said all I want to say on the subject so will keep stumm until Friday. I wish you all a very happy Referendum!

  43. Difficult to poll the vast ‘hidden electorate – homeless, itinerant and never before engaged – probably Yes, I agree about hidden Nos – previous referendums have seen big last minute swing to yes in Scotland.

  44. Black financial Wednesday and Sillar’s comments for me was the moment Yes campaign got derailed and has been since then.

    Better Together also made an effective counter attack on the NHS front and today’s NHS revelations will be difficult to ignore for the electorate. We have also heard today from Spanish government that Scotland will have to wait at least 5 years and not use the pound in order to be admitted to the EU. Again, a massive blow to Yes campaign.

    In the highly probable event of a No win, I expect Cameron’s position to still be completely weakened. If anyone, it will be Gordon Brown, Ruth Davidson and even George Galloway who should be credited with saving the Union.

  45. Anthony – I note what you say about taking 2010 Westminster vote into account when weighting… Does YouGov publish the yes/no breakdown by 2010 Westminster vote anywhere? This would be very interesting to see – ICM had:

    CON 10/90
    LAB 42/58
    LIB 41/59
    SNP 91/9
    OTH 57/43
    DNV 60/40

    but I can’t find it in the YouGov tabs.

    Thanks!
    NC

  46. Oldnat,
    Turnout in the 1975 EEC Referendum was 65/66% – Scotland’s 1979 Referendum produced circa 63%.

  47. @RAF

    My very first prediction was No to win by 5%. I then upped that to 10%. I now think that the polls are likely to be right and that we are heading for the tightest finish imaginable – it could come down to a few thousand votes. Technically then, Yes 49, No 50. Actually Yes 49.4, No 49.6

    ———————

    99%?

  48. Yes campaign complaining about Dad’s Army episode shown on the BBC.. I thought Amber and Old Nat had cleared Dad’s Army a few week’s back.

    Either way come Friday morning I guess half the Scots population will be saying “we’re doomed I tell ya”….

  49. “there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.”
    Which, being interpreted, means “We adjust the raw data to get the result we think we should, but we don’t know if we get that right.”

  50. SHEVII

    @”Yes campaign complaining about Dad’s Army episode shown on the BBC..”

    Jeez-they have more chips on their shoulders than on their Mars Bars.

    An Independent Scotland will make Iran seem like a bastion of Free Speech & Liberal Values.

1 2 3 4