The Scottish polls at the end of last week and the weekend were broadly clustered around a small No lead. There are a couple of days campaigning left, but perhaps a more likely route to a YES victory is if the polls are underestimating the level of YES support for some reason. Over the last couple of days I’ve seen several blogs or articles pondering whether the polls could be wrong, could they be underestimating YES or NO?

It would be hubris to suggest the polls couldn’t be wrong. Obviously they can. At most elections there are polls that perform better or worse than their peers, some of that is better methodology. When the polls are close most is probably just normal sample variation. That’s a matter for another time though, here I’m pondering more about the possibly that all the polls are wrong, the potential for a systemic bias with everyone a bit too yes or a bit too no. This is possible too – think of the way all polls overestimated Lib Dem support in 2010, or most famously how all the polls overestimated Labour support in 1992. How likely is that?

The Scottish referendum is a bigger challenge for pollsters than an election would be because it’s a one-off. In designing methodology for voting intention the experience of what worked or didn’t work at previous elections weighs heavy, and most companies’ weighting schemes rely heavily upon the previous election – if not directly through weighting by recalled vote, in using the data from the previous election in designing and testing other weighting targets. For a referendum you can’t take that direct approach, pollsters needed to rely more on modelling what they think is an accurate picture of the Scottish electorate and hoping it reflects the Scottish people well enough that it will also reflect their referendum voting intentions – it’s complicated because Scotland has a complicated electorate. Scottish voters have two Holyrood votes and a Westminster vote, and they use them all in different ways with different political loyalties. Within the space of a year Scotland managed to be a Labour stronghold at Westminster and to produce a SNP landside at Holyrood – using either election alone for weighting gives a rather different picture of what the Scottish electorate are like, even though you are trying to model the same population. Different companies have arrived at different methods of political weighting to deal with the issue – Survation, ICM and TNS weight by Holyrood recalled voted alone, YouGov weight by Holyrood recalled vote with a nod towards 2011 Holyrood voters who backed Labour in 2010, Opinium weight by Holyrood and Westminster recalled vote, Panelbase weight by Holyrood and European recalled vote, Ipsos MORI don’t use political weighting at all. Despite the variance they have all converged to produce the same sort of result, and that gives me some confidence – if there was a particular skew from being online or from using Holyrood recalled vote we would expect to see different results.

Most speculation about whether the polls might be wrong has – rightly in my view – concentrated on two particular issues. Very high turnout and differential response rate.

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election. This poses a problem. Any professionally run opinion poll will make every effort to get as representative a sample as possible, but in practice there are limitations. People on the very fringes of society, people struggling in absolute poverty, those utterly detached from mainstream politics and civic society – people on the extreme edge are probably underrepresented in opinion polls of all sorts. In most voting intention polls this doesn’t matter, as people on the very fringes are also extremely unlikely to vote… but if the Scottish referendum does manage to engage some who were previously totally detached and, crucially, those people vote in a substantially different way to other people of similarly marginal demographics, then it could be a source of error.

The second potential pitfall is differential response. Much of the media discussion around this has called it “shy Noes” – people who want to vote no but are reluctant to admit it to pollsters. That’s possible, but it should be much less of a problem with online polls when people are giving their opinion to an impersonal computer screen. I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother? Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.

So, how confident am I about the polling in the Scottish referendum? Well, I suppose I’m fairly confident – if there was anything I thought we were doing critically wrong we’d have corrected it. If I had to put money on the result, I’d certainly back the polls, but the potentials for error are there. We’ll know on Friday if they’ve been avoided.


199 Responses to “Will the polls get the Scottish referendum right?”

1 2 3 4
  1. Graham

    Scotland’s 1979 Referendum turnout –

    Question 1 – 60.43%
    Question 2 – 60.45%

    I’ve often wondered about why 734 more people voted on Q2 than Q1.

  2. Graham

    Sorry 1997.

  3. @Dave – I must say AW’s post seemed to bend over backwards to assure us there will be a No vote, listing all the reasons that No might be under-represented in the polls. Of course the same entusiasm detected for the Yes voters could produce a bigger turnout of their supporters.

  4. Did Yes ever respond the topic of tuition fees for rUK students in iScottish uni’s?

    I know supporters said that they should still be required to pay tuition fees (chip on shoulder alert!) but was there any ‘official’ response?

  5. Oldnat – “I’ve often wondered about why 734 more people voted on Q2 than Q1.”

    There was some sort of cock up at the count in Fife. The total number of votes counted there in the second vote was more than the number of ballot papers issued, but it was certified as the “official” result so it’s the “official” result. Even though it’s wrong.

  6. @ Shaun:

    “May i also point out that it has turned from a reasoned debatish type issue into total farce.I will be glad when it is all over and our country can be united again as one people not factions.”

    Has the general election been cancelled then?

    More seriously, I think the result will show that you are 100% wrong – and the result that I’m talking about is the turnout. The referendum has has engaged more people in Scotland than any other political event, which in my view shows that politics is about deciding between real choices (and not pseudo-choices carefully crafted around a single ‘united’ position).

  7. I go for 55% YES 45% NO.

    Well, probably not, but at least it’s as logical as the 60/40 Noes

  8. Anthony

    Thanks. That’s one mystery solved!

  9. colin

    Iran = Scotland

    Silliness alert.

  10. I think we sometimes lose sight of the fact that this is a referendum for independance from the United Kingdom, It is not simply a vote for Alex Salmon and the SNP. Salmon may well be the media focus but the issue of independance crosses the entire political spectrum, hence its unpredictability. However Salmon will gain political kudos regardless of the result as I pointed out earlier.

  11. Far as I can tell so far here’s what we are expecting for Scottish polls:

    TNS – 5pm Tues evening
    Survation – 10.30pm Tues night, Daily Mail
    Opinium – Tues night, Telegraph
    ICM (online) – Tues/Wed?, Scotsman
    Panelbase – ?? unconfirmed but expected
    YouGov – Wed night, Times/Sun
    MORI – Wed night, STV
    MORI – Thurs morning, Evening Standard

    UPDATE: Actually not sure about that TNS one, they’ve got a GB one about Scotland out at 5pm, and I think people might be thinking of that.

  12. @Paul

    Possibly.

    But for all that we worry about another 1992, polls don’t get it all *that* wrong. A couple of polls have shown a Yes lead, but the vast majority have shown a No of varying degrees of magnitude. So there would have to be an awful lot of shy Yesses to scrape over 50%. And one this Yes is not guilty of is sneaking around quietly.

    I’m going for 55-45 No wins.

    And then I’m going to worry about how the almost 50-50 divisions in Scotland are going to play out. That genie isn’t going back in the bottle anytime soon. Wouldn’t it be ironic if Salmond’s legacy isn’t independence but a Scotland more thoroughly divided than at any recent time.

  13. SHEVVI, not sure where that comes from but probably better you keep it.
    Here’s one the pollsters can’t allow for, the Scottish tendancy when undecided to say f*** it and take the course they deem most destructive. There’s a lot of pent up frustrations heading for those polling booths.

  14. I think most of the recent polls have it spot on, a close run contest I reckon. Whoever wins will win by a very small margin sadly, which in my eyes is the nightmare scenario. The polls tonight and tomorrow will all likely show the closeness once again, bring them on.

  15. TNS-BMRB: 5pm Tuesday (today)
    ICM: Tuesday night (tonight)
    Ipsos Mori: Wednesday night for STV News and Thursday morning for the Evening Standard
    YouGov – Wednesday night
    Survation – Wednesday night
    Opinium – Wednesday night
    Panelbase – Wednesday night

    Skippy predicts (And he will be wrong!)

    TNS: Small yes lead
    ICM: Small No lead
    Ipos: Small no lead
    Yougov: 6% No lead
    Survation: 6% No lead
    Opinium: 6% no lead
    Panelbase: 2% no lead

  16. I think the most interesting is that the polls are reacting to local and online campaigning. It is also telling that BT are not questioning the tight polling despite losing a comfy lead.

    I think a yes but just. Amazing that I thought 45 was likely and 48 if lucky.

  17. @Paul

    ‘99%?’

    Perhaps 1% are going to vote for becoming the 51st state (or a French Overseas Territory)

  18. If it’s really close what is the hanging chad equivalent? People writing “Thanks” under the “No” and having them counted as spoiled?

  19. Thanks to everyone that posted feedback on my previous analysis, which examined what a Scotland-wide 50-50 tie would look like in each area if the regional dispersion were the same as in the 1997 devolution referendum.

    The point that came up repeatedly was “Area X is a strong/weak SNP area, so the yes vote should be higher/lower there” (and sometimes other parties were mentioned too). Also, some areas (solid Lib Dem territory was one example given) might have been very pro-devolution but very “No” on the matter of independence. These are, of course, very valid concerns. So with that in mind, I’ve come up with the following alternative approach:

    -Take the 2010 Westminster ward-level vote estimates (as per Electoral Calculus)
    -Aggregate them, not to constituencies, but to local authority totals
    -Take the referendum VI by 2010 vote from opinion polling. To weight the 2010 non-voters, I had to make an assumption about turnout, so I used 85% with an evenly-spread increase over 2010, but using 75% or 95% doesn’t make much difference
    -Combine the two to get an implied referendum vote for each local authority
    -Normalise the overal result to a national tie

    This solves the main problem with the original model, but it does create some new ones:

    -Except where local authorities corespond exactly to Westminster constituencies, ward level (and hence council area) figures for 2010 are estimates. The Electoral Calculus methodology is sound (it’s the same methodology you would use to calculate notional results after boundary changes) but they are still estimates
    -Using polling data rather than purely real votes means sampling error. Because only ICM seems to have published a breakdown by 2010 vote the samples are small and hence the MoEs are rather wide (though this is arguably still preferable to using the results of another referendum 17 years ago and on an entirely different question)
    -It also means we have to trust opinion polling, although a biased sample would also have to be biased geographically to matter in this case
    -Voters for each party aren’t necessarily homogenous, and this might have a regional element to it. New approach, similar problem…
    -The turnout concern still applies

    So with all of those caveats noted, here is the revised version:

    http://numbercruncheruk.blogspot.com/2014/09/how-should-we-interpret-early-results.html

    We can see that Aberdeenshire, Angus, Perth and Moray are all much higher than the original model had them, as are the islands. Glasgow is quite a bit lower than before.

    In relation to feedback I’ve received, Dundee is also higher but not by all that much. The big 3 cities collectively are lower (though not by a huge amount.)

    Incidentally, I was concerned that the difference between referendum and Westminster turnout patterns (ie safe seats vs marginals) could affect the regional pattern by ‘misweighting the national average. So I tried reweighting the average using the 1997 turnouts, and it made almost no difference whatsoever.

    I’ll update this on Wednesday or Thursday once the final call polls are in, hopefully some more of them will release a breakdown by 2010 vote. Ideally I’d be able to do this by 2011 Holyrood vote, but as far as I know, the local breakdowns aren’t readily available for Holyrood Elections.

    NC

  20. Now what happens if Scotland votes “NO” and the VOW made by the three main parties gets vetoed or greatly diluted in the HOC as angry MP see Scotland getting more Power and cash from the rest of us?. Will Scotland revolt or claim the referendum was a fix? Will the masses who wind up paying for this last ditch desperate measure by the three main parties abandon their political loyalties and decamp to UKIP at the GE.
    I think we are heading for a major political upheaval the like of which is unpresedented.

  21. @Roykite – we get another referendum if the vow isn’t upheld. They talked about this last night on ‘Scotland tonight’ (David Torrance and another lady from the herald). They both felt that the SG would legitmately be able to call another referendum either if the vow wasn’t upheld or in leaving the EU.

  22. “MORI – Thurs morning, Evening Standard”

    One other poll:

    Scotland – Thursday, all day, official

    :))

  23. @Robin

    If England & Wales stops charging for tuition fees, there wouldn’t be a problem, as far as I know.

  24. Can’t we just avoid all these polls and ask Robin Hood what’s going to happen? :-)

    Or maybe not if we’re going to be serious and vaguely scientific about it all. Probably no longer an appropriate time for comedy now we’re getting nearer the crux.

    I’ve been surprised by the late narrowing after such a consistent, stable, significant and relatively durable No lead, but I’m still convinced No has a structural majority that will manifest itself in the vote on Thursday. At one time I thought that it might be 60:40 but it could well be closer now although I have the very vaguest of hunches that we might wake up on Friday and wonder what all the fuss and bother was about. Opinion polls can walk us up some unusual garden paths sometimes.

    With that in mind, I’m going Yes 44 No 56 and for a slightly lower turnout than some are predicting. 70-75% is my guess. That’s still impressive by recent UK standards, but I fear a sizeable “can’t be bothered” element will still stubbornly sit it all out.

    I’m quite prepared to be wiping egg off my face on Friday, by the way!

    :-)

  25. @ Martin – the “leaving the EU” part is interesting, because nationalists (I assume) would typically vote to stay in the EU in a referendum, but if they lose the independence referendum, maybe they would vote to leave in order to get another bite of the cherry? Could matter if the EU referendum were close…

  26. ROSIEANDDAISIE

    Only just read you’re much earlier post re electing Dog-Catchers – surely you should be in favour of that so you could vote for the least competent of the candidates!

  27. @Roykite
    I agree, however it is equally possible that if a stare into the abyss makes Scotland think twice, a similar sobering moment may make rUK voters do the same regarding the EU. Most of the arguments in favour of Scotland staying in the UK, apply to the UK staying in the EU – OK maybe to a lesser extent but the principles are similar in my view.

    Let’s face it, the fact that Scotland would immediately demand another indyref in the event of a UK exit from the EU would be a massive disincentive to going down that route. After all, what is now the point of UKIP if UK independence means the certain end of the UK?

  28. Without a doubt no future UK government would ever grant another referendum over its break up after this divisive and extremely damaging campaign.

    I think the country as a whole is now so deeply wounded and scarred that it won’t contemplate an EU referendum either.

    UK will need quite some time to reflect on itself and in doing so it requires stability.

  29. paul a

    they wouldn’t like the tension.

  30. Amber Star

    The issue is the responses from the 50% of registered voters who didn’t vote in the 2011 elections. It’s whether or not the responses from this group are too Yes-y because of over-enthusiasm or too No-y because of the ‘marginalised’ voters not being included or whether the two balance each other out.

    The answer is both of course. There’s evidence that those who didn’t vote in 2011 but did in 2010 are more likely to lean No. The ICM-Guardian poll[1] asked about both recalled vote and of the difference between the two (about 8% of the sample, which seems plausible[2]), over 80% were No voters. This makes sense in part because those who feel Holyrood are less important are going to be No voters and not even bother to participate.

    But if you look at those who didn’t vote in 2010 (and presumably not 2011 either) they split 35% Yes, 30% No, 35% DK. This suggests that these more detached voters lean towards Yes, though possibly not heavily enough to counteract the overwhelming No among the eighth of the electorate who comprised the ‘2010 but not 2011’.

    But these are the ‘known unknowns’ and there ‘unknown unknowns’ as well. In particular we don’t know who all the extra people on the register are, especially the most recent additions[3]. Some may indeed be the most marginalised and demographics suggest they could be Yes voters – you only have to look at the difference between the most and least affluent quintiles in MORI’s usual figures). But others may be temporary exiles like your mother making an effort[4], and expats, even temporary will tend to be more towards No. Non-UK origin voters may also have been under-registered and these tend to lean towards No, though not as much as UK non-Scots.[5] Turnout among such groups could also be important

    And of course there are still the Don’t Knows. Normally at this stage in a campaign you would say these are really non-voters, but 65% in ICM said they were certain to vote (compared to 95% for Yes and 93% for No). You suspect many may only make up their mind in the polling booth.

    I’m inclined to think that the 2003 SSP vote didn’t necessarily come from the ‘marginalised’ voters

    Turnout was only 49.4%, nearly ten points down on 1999, which hardly suggests the usually disenfranchised flooding out to vote. Presumably a lot of the SSP (and Green) vote was in reaction to the Iraq war, though the gains they made were at the expense of the SNP as well as Labour – indeed more so. Also the increase in Greens and SSP looked dramatic because their increase pushed them over the limit to get regional seats in many places.

    [1] Phone polls are more likely to get non-voters than other methods and so the best to look at when discussing this.

    [2] Those who said they didn’t vote or didn’t know how they did (which probably means they didn’t). I haven’t included the Refusers though some of those wouldn’t have voted either – certainly the majority of them say they don’t know how they will vote on Thursday.

    [3] Some of the earlier extras may be down to better registration of 18 year olds (plus the one-off 16- and 17-year olds of course) and of EU voters, in part caused by the Euros.

    [4] I know your mother hasn’t double -registered, but it’s worth pointing out that if the property she owns isn’t let out (or is occupied by family members) she could be legally down there as a holiday home. Of course she couldn’t vote in both constituencies in the General Election, but she would have the choice.

    [5] Previous online ICM’s haven’t weighted for country of birth which we know makes a considerable difference. In the phone poll they don’t say they have weighted by it, but the breakdown looks about right (previously it was too non-Scot) so they either did weight, quota sampled for it or struck lucky.

  31. Betfair has already started paying out for a ‘No’ win…

    http://www.telegraph.co.uk/finance/newsbysector/retailandconsumer/leisure/11098848/Betfair-pays-out-early-on-Scottish-independence-No-vote.html

    “Betfair said this morning that gambling patterns indicate a 79pc likelihood of a “No” vote.

    Despite the odds on “Yes” shortening last week, they have lengthened significantly in the last few days.”

    Probably a Westminster plot.

  32. I was joking but I have just found this in the Comments section for that article:

    “This is being done as an attempt to affect the outcome of the referendum.

    I am still voting yes”

    It has 27 ‘agrees’.

  33. @Statgeek

    “If England & Wales stops charging for tuition fees, there wouldn’t be a problem, as far as I know.”

    What the UK does has no bearing on whether UK students have to pay in an independent Scotland.

  34. I have a friend who is convinced the outcome of Thursdays referendum has been decided by those in control of us a long time ago. Only on this basis would the Goverment have been allowed to agree to a referendum. But they cannot tell me which way the vote will go!!

  35. @MacTavish

    If you want violence try Barcelona. Last time they had two civil wars running at the same time.

  36. In Canada the experience was of having a decent number of “shy no” voters. The last 3 polls prior to the referendum had leads of 6%, 6% and 4% for the Yes (Oui) side. If your guys’ polls skew the same way you’ll be fine.

    Of course, Quebec’s nationalism was a bit more negative towards Canadian nationalism than Scotland’s seems to be with respect to Britain, so maybe there won’t be the same skew at all.

  37. Prediction:

    YES 49.7%
    NO 50.3%

    It’s a nailbiter and nae mistake.

    I honestly dread the consequences of a YES vote – far from scaremongering, I think they’ve been understated.

    That said, NO have been foolish not to concurrently promote the Positives for staying together throughout the campaign, of which there are many – both economic and emotional. If they lose it will be seen, rightly, to be because of this.

    They would have got away with it with a leader less wily than Salmond. In spite of not one answer to any of the big questions, he’s literally convinced enough undecideds not to worry and to ride the crest of the nationalist wave to take him to the brink of victory.

  38. 5pm… where is my TNS poll

  39. i’m liking the yes campaign, but am still undecided!

    help please!!

  40. Off twitter…

    Cicero Elections [email protected] 24s

    ICM poll out at 21:00 tonight with Survation results at 22:30.

  41. I expect No 55%, Yes 45%, just a guess as I have not been following the story in any great detail.

  42. It appears the TNS poll was for England and Wales only!

    D’oh!

  43. Good Afternoon All.

    The Labour Party is ahead by 4% according to latest YG poll for the UK GE.

  44. Betfair are rarely wrong, so am assuming definite No vote if they are paying out.

  45. @Rich

    Yep. Us kippers ignored the polls and trusted the bookies last May. It worked out for us.

  46. IF we look at the timeline for results announcements by region and their “Yes rating” then Yes % should peak with West Lothian (the early regions are more inclined to Yes) and start to revert once the more No biased regions announce, culminating with Edinburgh / Glasgow and Aberdeen. So expect to see a Yes lead (up to 60% vs 40%) at around 3:30am then start to come in sharply. If Yes aren’t comfortably ahead by 3:30am then it should be a clear No victory. All assuming the rough “Yes rating” estimates are a reasonable guide ….

  47. GUYMOND
    “(or a French Overseas Territory)”

    Well, that solves one problem: Scotland can join the CFA franc.

  48. Graun reporting Miliband had to abandon his walkabout due to intimidation. Tom Bradby pulling his punches but finds this ‘unusual’ in an advanced democracy. Amazing how these Yes mobs spontaneously appear. Good job no one is organising them or anything.

  49. Seriously, though, this kind of intimidation will turn off DKs and undecideds. And will help No.

  50. Katy

    You have to help yourself. There is no correct answer!

    My advice –

    Decide what things are most important to you.

    Weigh up the risks to how you think the kind of future you want to see will be under both scenarios.

    Distrust politicians.

    Vote anyway! :-)

1 2 3 4