The Scottish polls at the end of last week and the weekend were broadly clustered around a small No lead. There are a couple of days campaigning left, but perhaps a more likely route to a YES victory is if the polls are underestimating the level of YES support for some reason. Over the last couple of days I’ve seen several blogs or articles pondering whether the polls could be wrong, could they be underestimating YES or NO?

It would be hubris to suggest the polls couldn’t be wrong. Obviously they can. At most elections there are polls that perform better or worse than their peers, some of that is better methodology. When the polls are close most is probably just normal sample variation. That’s a matter for another time though, here I’m pondering more about the possibly that all the polls are wrong, the potential for a systemic bias with everyone a bit too yes or a bit too no. This is possible too – think of the way all polls overestimated Lib Dem support in 2010, or most famously how all the polls overestimated Labour support in 1992. How likely is that?

The Scottish referendum is a bigger challenge for pollsters than an election would be because it’s a one-off. In designing methodology for voting intention the experience of what worked or didn’t work at previous elections weighs heavy, and most companies’ weighting schemes rely heavily upon the previous election – if not directly through weighting by recalled vote, in using the data from the previous election in designing and testing other weighting targets. For a referendum you can’t take that direct approach, pollsters needed to rely more on modelling what they think is an accurate picture of the Scottish electorate and hoping it reflects the Scottish people well enough that it will also reflect their referendum voting intentions – it’s complicated because Scotland has a complicated electorate. Scottish voters have two Holyrood votes and a Westminster vote, and they use them all in different ways with different political loyalties. Within the space of a year Scotland managed to be a Labour stronghold at Westminster and to produce a SNP landside at Holyrood – using either election alone for weighting gives a rather different picture of what the Scottish electorate are like, even though you are trying to model the same population. Different companies have arrived at different methods of political weighting to deal with the issue – Survation, ICM and TNS weight by Holyrood recalled voted alone, YouGov weight by Holyrood recalled vote with a nod towards 2011 Holyrood voters who backed Labour in 2010, Opinium weight by Holyrood and Westminster recalled vote, Panelbase weight by Holyrood and European recalled vote, Ipsos MORI don’t use political weighting at all. Despite the variance they have all converged to produce the same sort of result, and that gives me some confidence – if there was a particular skew from being online or from using Holyrood recalled vote we would expect to see different results.

Most speculation about whether the polls might be wrong has – rightly in my view – concentrated on two particular issues. Very high turnout and differential response rate.

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election. This poses a problem. Any professionally run opinion poll will make every effort to get as representative a sample as possible, but in practice there are limitations. People on the very fringes of society, people struggling in absolute poverty, those utterly detached from mainstream politics and civic society – people on the extreme edge are probably underrepresented in opinion polls of all sorts. In most voting intention polls this doesn’t matter, as people on the very fringes are also extremely unlikely to vote… but if the Scottish referendum does manage to engage some who were previously totally detached and, crucially, those people vote in a substantially different way to other people of similarly marginal demographics, then it could be a source of error.

The second potential pitfall is differential response. Much of the media discussion around this has called it “shy Noes” – people who want to vote no but are reluctant to admit it to pollsters. That’s possible, but it should be much less of a problem with online polls when people are giving their opinion to an impersonal computer screen. I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother? Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.

So, how confident am I about the polling in the Scottish referendum? Well, I suppose I’m fairly confident – if there was anything I thought we were doing critically wrong we’d have corrected it. If I had to put money on the result, I’d certainly back the polls, but the potentials for error are there. We’ll know on Friday if they’ve been avoided.

199 Responses to “Will the polls get the Scottish referendum right?”

1 2 3 4
  1. @INTERESTED – yep your absolutely right about the assumption.


    East Renfrewshire amongst the 3 lowest no votes?

    I don’t think so..I agree it will probably be no but you have the Barrhead and Neilston factor with a combined population of around 30,000 out of around 90,000 for the whole council area.They are working class areas.

    I know many moons ago it used to be a Tory strong hold but the council is a Labour/SNP run council. I will go 45% Yes and 55% naw in the council area but I agree D&G and the Borders will be low points for YES,

  3. Wow, tough gig for Miliband today!

  4. PETE B

    “Anyway, I hope the referendum vote is Yes, but it will be very close. I wonder how big an effect postal vote fraud will have, or is that more an English problem?”

    Yes I mind the last mess postal voting had. No I’m pretty sure it wont be a factor due to the nature of this vote. Police Scotland I think will had been prepared for any interfering…..I hope.

  5. @OLDNAT

    Re: street by street – its done by taking your previous support, canvasing a sample of it and seeing how much of its stayed with you or turned.

  6. @Pete B

    Not in terns of MPs they don’t. The SNP have six seats.

  7. My prediction:

    If the polls tonight and tomorrow show a yes lead the yes supporters on here will declare them uniquely accurate finally capturing the mood out inthe real world.

    If they remain close to 50/50 they will declare this good news, the evil westminsters propaganda campaign has failed to dent their support proving there will be a yes victory.

    If they show an increased lead for No, they will declare this excellent news as its just the boost necessary to ensure fewer nos turn out and more yeses make that extra push for victory.

    and in all 3 circumstances the reason for this portent of victory is the failings/offence caused by (insert BT/westminster figure or policy of your choice.)

  8. @ Katy,

    Just decide what sort of country you want to live in and how you’re most likely to get it.

    An independent Scotland will be:
    – an independent country, if that’s emotionally appealing
    – freed from the the conservative voting block of southern England
    – likely to rejoin the EU, with the corresponding commitment to workers’ rights
    – likely to rejoin an EU already signed up to TTIP without an exemption, which may put the NHS (SHS?) at risk
    – likely to have a greatly reduced financial sector, giving bankers much less influence over politics
    – facing serious financial problems during the transition period which will probably lead to austerity and cuts in public services, at least in the short term
    – at risk of Dutch Disease, which will make revitalising the manufacturing sector very difficult
    – able to distribute oil revenues among a smaller population
    – able to reduce business taxes relative to England as the SNP have promised
    – a smaller country which is more vulnerable to the lobbying of large corporations
    – likely to have a large portion of the media bought up by Alex Salmond’s good friend Rupert Murdoch

    A unionist Scotland will be:
    – Part of the United Kingdom, if that’s emotionally appealing
    – able to pool pension commitments with England to take advantage of England’s younger population
    – able to pool revenue with England to buffer volatile oil prices
    – in a currency union with rUK, preventing Dutch Disease, facilitating trade with England and guaranteeing Scotland a stable currency for the future
    – bound by Westminster fiscal policy, which is usually run for the benefit of the London financial sector
    – at risk of occasional Tory government from Westminster, with the attendant cuts to the block grant and public spending
    – at risk of being dragged out of the EU by England
    – if the UK remains in the EU, bound by whatever happens with TTIP in Westminster: if the Tories don’t secure an exemption, the Scottish NHS will be vulnerable; if Labour gets one, the Scottish NHS will be protected

    Whatever your politics, either choice carries benefits and risks for creating the sort of Scotland you want to see. You’ll have to decide what seems like the best bet.

  9. @Oldnat

    It seems you are right. They didn’t canvass. Oops.

    “Labour MPs are privately criticising the No campaign for failing to canvass voters ahead of Thursday’s vote. “We cannot have a proper ‘get the vote out’ campaign because we don’t know who our voters are,” one senior MP told the Standard. “The Yes campaign have a much better machine.”

    I am guessing that Yes has canvassed? Surely those results would have leaked to the Govt?

  10. With a turnout expected in the high 80’s I’m not sure getting the vote out is an issue- just the persuasion that occasionally comes from a one to one conversation.

  11. Some news (rumours) on the Yes canvass results

    “On this, I bring further hopeful news from someone who has seen the Yes campaign’s canvass returns up to Thursday last week: that Yes does not have a lead in any of its target areas. Despite throwing everything into the campaign, the Yes campaign itself reckons that it is well behind and that to get anywhere near the figures suggested by opinion polls would be a remarkable achievement.”

  12. @Richard They didn’t canvass. Oops

    Oops indeed!

  13. @Allan Christie

    East Renfrewshire will be no more than 35-40% Yes

    Barrhead and Neilston are traditional Labour areas but not neglected or run down. They are tight knit communities with a strong Labour presence. At the 2012 council elections the SNP couldn’t field a candidate who lived in the Barrhead ward.

    Elsewhere in East Ren the Labour vote is largely a middle class professional ABT vote that will break more strongly for No than the national Labour average.

    The Conservatives may not be part of the ruling coalition but they came a strong second at the council elections in 2012 with 30% of the vote. I believe Newton Mearns South is the only 3 member ward in Scotland with 2 Conservative Councillors. (Unless South Ayrshire maybe has 1 too?).

    SNP poll 20% across the Local Authority and Yes may get some decent returns in Thornliebank but I guarantee East Renfrewshire will be one of the highest No votes in Scotland.

  14. @Katy

    People will tell you that this is the most important decision you will ever make in your life. People say that a lot. About many things. And it’s almost invariably untrue as no single decision is irreversible – even major decisions.

    So don’t pressure yourself. Think about the decision and cast your vote. And remember, whatever you decide, the sun will still come up on Friday morning, and your day to day life won’t really change that much.

  15. Richard:

    There`s been no canvassing to offer transport as far as I am aware on Deeside.

    But about 15 deliveries from each side of leaflets through letter boxes.

    I can report my count of posters on a 100-mile round trip today up to the Glen Shee ski lift:

    6 Yes, 12 No, of which 5 were trashed but still recognisable as No Thanks.

    The Yes posters were intact, but we have had in our local paper what seems a valid report of a single Yes sign being removed near Ballater.

    And the nearest poster to Balmoral that I spotted was a Yes, though I don`t think HMQ is either responsible or likely to take a walking stick to it.

  16. Martin

    The canvass returns for the 1707 campaign have, unfortunately, been lost.

    Anyone who thinks this campaign is the same as an election, really have no understanding of what is happening.

  17. Well it seems Yes has canvassed

    “Yes Scotland has been feeding information from an “unprecedented” 25,000 canvass returns a day into its “Yesmo” data collection and management system.

    Chief executive Blair Jenkins insisted he was not taking victory for granted, but said: “On everything I know, I think we’ve got a Yes vote.”

    “The information we have would tell us that there is a Yes vote there. We have to make sure we get it out, we have to make sure we get every last vote out.”

    He added: “I think we will win if everyone who has told us they are voting Yes votes Yes.”

    So someone somewhere does have some idea of how close this race and the opinion polls really are….

  18. The crux of it is there was a huge shift of Labour and LD voters to the SNP in 2011 and it looks like they’ve retained them as Yes voters and gained even more . I expect Glasgow , West Renfrewshire, Ayrshire and Central Scotland to deliver big Yes votes . Word is that Argyll and the Highlands is a cert for Yes .

  19. @ChasGlas

    I thought there were a lot of Tory types in Argyll?

  20. @chasglas

    The polls don’t show that at all.

  21. Who really believes canvass data?

    Any tweeted information a the canvass returns of either side I would treat with a huge dose of scepticism.

    Additionally, both sides will be keen to big their own side this close to the poll regardless.

  22. Richard: i’d rather assume that the ‘Yes” side has the votes in their target areas and all ‘No’ supporters have to show up massively at the polling stations to offset that. There is no room for complacency when the survival of your nation is at stake.

  23. Richard

    “I think we will win if everyone who has told us they are voting Yes votes Yes.”

    Of course, people lie to canvassers – like the guy who insisted to me that he was a Yes, when I could see the huge “Ulster Scot” flag in his back garden. Might have been true, I suppose – but I put him down as a definite No.

  24. Canvassing is a pretty poor way to get information anyway.

    If a large group of YES voters were walking around with clipboards, badges and flags and were in my face demanding to know how I will vote, I’d just say yes to avoid the stares and tuts when i say no.

  25. Self evidently both sides are going to claim that their canvass returns show that they are just ahead…to encourage their own voters whilst not wanting to encourage any sense of complacency…

  26. Mbruno

    “when the survival of your nation is at stake”

    That’s probably not a terribly useful bit of terminology in the context of this referendum.

    There are those who are passionately “British”, and for them that is their nation, just as their are those who are passionately Scottish.

    I’d expect both groups to register a huge turnout.

    The decision will be made by people who don’t even think in those terms.

  27. MBruno

    Richard: i’d rather assume that the ‘Yes” side has the votes in their target areas and all ‘No’ supporters have to show up massively at the polling stations to offset that. There is no room for complacency when the survival of your nation is at stake.

    Isn’t survival of your nation a bit OTT?

    Whether Scotland is independent or in a union, it still exists and will mostly be the same.

  28. Raf,

    There are Tory types , as you put it in Argyll , but not significant numbers of them . Most of their vote comes from the Helensburgh end of the seat , a dormitory seaside town .

  29. Yes, maybe “survival of the nation” may be a bit dramatic. The economic consequences of independence will be severe though, at least in the first decade or so. In the long run, the economy may stabilize, after all the casualties are counted.

  30. ICM

    Latest ICM poll: 52% (-2) Yes, 48% (+2) No.

    Despite the slide towards NO, will probably be hailed as a victory for Yes anyway.

  31. Chasglas,

    Actually, the only council seat in Argyll where the Tories gott first place at the 2007 local elections was South Kintyre. It’s true that they also have some strength in Helensburgh.

  32. Scottish Independence Poll (ICM/Scotsman):
    YES – 48%
    NO – 52%

  33. Please ignore my post. Seems some people messed it up on Twitter and posted YES ahead.

    Either that or I’m blind.

    Going from my past posts, it’s probably the latter. Shoot me now!

  34. It looks like the ICM poll is out: No 45 %, Yes 41 %, DK 14 %,

    Excluding DKs, it is No 52 % and Yes 48 %.

  35. @Skippy
    As long as the returning officer doesn’t do the same!

  36. Bill Patrick ,

    Kintyre is an area I know well but I wouldn’t read too much into its council elections .

  37. Same for Opinium
    52/48 for No

  38. So the last ICM was rogue then.

  39. Opinium/Telegraph

    No 52
    Yes 48

  40. Chasglas,

    I have a lot of ancestry in Kintyre, but I’ve not actually been there yet, so I can’t really comment much about it!

  41. @Martin

    I think the ICM/Scotsman poll shows Yes up 3 and No down 2 compared to their August poll?

  42. If only the polls would give a regional breakdown , it means a lot in this election . We really only know what’s going in areas we know .

    Unless the polling panels are geographically representative and reflect the 25-30 % returning to vote after years of disinterest , they are only a very rough guide .

  43. With Dks at 14 %, 52/48 excluding DKs doesn’t mean that much for either side. ‘No’ is probably slightly closer to winning, but the final outcome is still uncertain.

  44. This was the ICM I was expecting (on Sunday). The Scotsman is reporting it compared to ICM’s last (online) poll for them (11-14 Aug):

    Yes 41% (+3)

    No 45% (-2)

    DK 14% (n/c)

    (Rounding on Aug poll before you ask)

    What I find astonishing is the lack of change in the DKs, given that there has been a month between the two (f/w 12-16 Sep). The ICM poll in the ST had 10% DKs.

  45. Just listening to Gordon Brown.

    Why didn’t he lead the ‘No’ Campaign?

    I find him more convincing and substantial than Mr Darling.

  46. Roger Mexico

    “What I find astonishing is the lack of change in the DKs”

    Can I suggest a wee change to that? “the lack of change in the overall number of DKs”.

    We shouldn’t assume that the same individuals are DKs. There remains a significant churn between DKs, soft Yes, and soft No.

    I’d suggest that the combined total of those three is much bigger.

  47. Well, as the egg spurts have said it looks like a clear yes – ph…….apart from the polling figures and the D/Ks also to break towards no come that moment of serious intent with no one else around.

    And also nicely close enough [in theory] to obviate any “soddit, I’ll tick yes just to make a point” votes.

  48. New thread

  49. The registration of votes in Scotland stands at 97%, apparently.

    If the turnout is going to be a factor, what turnout are the pollsters weighting for?

1 2 3 4