The Scottish polls at the end of last week and the weekend were broadly clustered around a small No lead. There are a couple of days campaigning left, but perhaps a more likely route to a YES victory is if the polls are underestimating the level of YES support for some reason. Over the last couple of days I’ve seen several blogs or articles pondering whether the polls could be wrong, could they be underestimating YES or NO?

It would be hubris to suggest the polls couldn’t be wrong. Obviously they can. At most elections there are polls that perform better or worse than their peers, some of that is better methodology. When the polls are close most is probably just normal sample variation. That’s a matter for another time though, here I’m pondering more about the possibly that all the polls are wrong, the potential for a systemic bias with everyone a bit too yes or a bit too no. This is possible too – think of the way all polls overestimated Lib Dem support in 2010, or most famously how all the polls overestimated Labour support in 1992. How likely is that?

The Scottish referendum is a bigger challenge for pollsters than an election would be because it’s a one-off. In designing methodology for voting intention the experience of what worked or didn’t work at previous elections weighs heavy, and most companies’ weighting schemes rely heavily upon the previous election – if not directly through weighting by recalled vote, in using the data from the previous election in designing and testing other weighting targets. For a referendum you can’t take that direct approach, pollsters needed to rely more on modelling what they think is an accurate picture of the Scottish electorate and hoping it reflects the Scottish people well enough that it will also reflect their referendum voting intentions – it’s complicated because Scotland has a complicated electorate. Scottish voters have two Holyrood votes and a Westminster vote, and they use them all in different ways with different political loyalties. Within the space of a year Scotland managed to be a Labour stronghold at Westminster and to produce a SNP landside at Holyrood – using either election alone for weighting gives a rather different picture of what the Scottish electorate are like, even though you are trying to model the same population. Different companies have arrived at different methods of political weighting to deal with the issue – Survation, ICM and TNS weight by Holyrood recalled voted alone, YouGov weight by Holyrood recalled vote with a nod towards 2011 Holyrood voters who backed Labour in 2010, Opinium weight by Holyrood and Westminster recalled vote, Panelbase weight by Holyrood and European recalled vote, Ipsos MORI don’t use political weighting at all. Despite the variance they have all converged to produce the same sort of result, and that gives me some confidence – if there was a particular skew from being online or from using Holyrood recalled vote we would expect to see different results.

Most speculation about whether the polls might be wrong has – rightly in my view – concentrated on two particular issues. Very high turnout and differential response rate.

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election. This poses a problem. Any professionally run opinion poll will make every effort to get as representative a sample as possible, but in practice there are limitations. People on the very fringes of society, people struggling in absolute poverty, those utterly detached from mainstream politics and civic society – people on the extreme edge are probably underrepresented in opinion polls of all sorts. In most voting intention polls this doesn’t matter, as people on the very fringes are also extremely unlikely to vote… but if the Scottish referendum does manage to engage some who were previously totally detached and, crucially, those people vote in a substantially different way to other people of similarly marginal demographics, then it could be a source of error.

The second potential pitfall is differential response. Much of the media discussion around this has called it “shy Noes” – people who want to vote no but are reluctant to admit it to pollsters. That’s possible, but it should be much less of a problem with online polls when people are giving their opinion to an impersonal computer screen. I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother? Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.

So, how confident am I about the polling in the Scottish referendum? Well, I suppose I’m fairly confident – if there was anything I thought we were doing critically wrong we’d have corrected it. If I had to put money on the result, I’d certainly back the polls, but the potentials for error are there. We’ll know on Friday if they’ve been avoided.


199 Responses to “Will the polls get the Scottish referendum right?”

1 2 3 4
  1. RICH

    Why do we expect them to know more than anyone else?

  2. TARK

    Miliband was late and so the crowd of press brought attention; BBC showed that it was mostly just too many people and the atmosphere was not really intimidation, Miliband seemed embarrassed at the suggestion.

  3. Betfair.

    Possible reasons:

    1 – They are part of the conspiracy by the PM
    2 – They have seen tonights polls (or private polls) and know something we don’t
    3 – The money paid out warrents the publicity gained

    Take your pick!

  4. @Paul

    There are always a small number of rejected or spoiled ballots. Estimating these will total 1% of all votes cast on Thursday may be a bit on the cavalier side, given that on an 85% turnout, 1% is likely to be around 36,425! But you never know…

  5. How to vote?

    Consider the heart. Nationalism is not boring logic; that’s the No campaign.

    Nationalism is an emotion based on language, culture and history – half the countries of the world wouldn’t exist if it was just about rational economy. (Mind you I think Scotland would work fine without England.)

    And it would be a liberal, left of centre country? Do you really want right wing Tories to still have power over Scotland?

  6. Well I think the government has a very good idea of who is going to win, based on the street by street returns from the BT campaign. Definitely signs of panic – the vow and talk of devolving power over the whole country looks like something pulled out of the hat as an urgent response to a problem without having been properly thought through.

    So I think it is either very close, or pointing to a yes win, for now I will go with very close.

  7. Re tuition fees:

    I heard Nicola Sturgeon saying that they would apply to the EU for exemption of the ruling that states all EU citizens should be treated the same.

    The official response is given here:

    http://www.yesscotland.net/answers/what-will-happen-tuition-fees-independent-scotland

  8. @Richard

    Awful logic

    We could say the constant YES flashmobs and so on are signs of THEIR panic that they are losing.

  9. Without a doubt no future UK government would ever grant another referendum over its break up after this divisive and extremely damaging campaign.

    I think the country as a whole is now so deeply wounded and scarred that it won’t contemplate an EU referendum either.

    UK will need quite some time to reflect on itself and in doing so it requires stability.

    I think the way the campaign has gone actually encourages those who want change even more.

    The campaign has has been lively, and anyone wanting real change to come would be unwise to let the Government off the hook.

    If you want real change, keep picking at the same wound, and don’t let it heal.

  10. RICHARD.
    Good Evening to you.
    I agree with your analysis here.
    The BBC Journalist at 17.30 ish was suggesting that the Government is seriously worried.
    The TIMES has a powerful attack on the PM’s approach.

    Gordon Brown is on fire, like in days of old, before he had to do New Labour’s dirty work, under John Smith 1992-94, when he faced down the Left’s spending pledges. It is funny how he looks energised and younger.

  11. Bramley: “Re: Tuition fees, I heard Nicola Sturgeon saying that they would apply to the EU for exemption of the ruling that states all EU citizens should be treated the same.”

    Yeah, good luck with that, ‘applicant country’.

  12. As far as i can see the only rational position is that one cannot tell which way the vote is going to go. The polls say it is very close, AW says we can’t tell if the polls are wrong and the polls only have to be very slightly wrong for Yes to win.

    That said if I had to guess I would guess that ‘No’ wins My reasons are a) that’s what most of the polls are currently saying b) I think that Crosstatt is right that there is a ‘structural majority’ fir no (well a natural majority as I am not sure what a structural one is).c) I think that the narrowing of the polls will affect yes and no in different ways.

    To explain C, I guess that both sides will be more motivated to vote by the narrowing polls, one because they sense they might win and the other for fear of losing. The Yes side will however contain a number of people who are not naturally enthusiastic about independence but who want to register their disgust at Mrs Thatcher, Tony Blair, the English and so on. When the likelihood of a yes win is low the rational way for these people to vote is ‘yes’. It makes their voice heard, and may extract concessions. As the likelihood of a Yes vote rises, a vote for yes ceases to be a no brainer and becomes something that may have consequences. So I would expect some of these people to think twice, although having done so they may obviously go ahead nevertheless

    On this analysis a stalemate in the sense of no nationalist surge should favour the No side and it would take the euphoria of a shift in the polls and cheering crowds to carry the yes side over the line.

    So if will be interesting to see if this shift occurs.

  13. Anthony
    I have a post in mod – no idea why but could you release it please ?

  14. @Fraser

    The Graun, BBC and ITN all went with ‘intimidation’ and Miliband said there were ugly aspects to the Yes campaign. So I’ll go with that.

  15. Robin, there really is an ocean of difference between what the ‘SNP vision’ offers and what they will most probably be able to deliver. The number of undeciders/non-nats who have jumped to Yes astonishes and depresses me.

    If/when there is an EU referendum and UKIP try this, I would bet my home that the result would be a massive pro-EU vote from the rest of the UK.

  16. Robin, Steve2

    Well quite !

    It’s frustrating that journalists never press home the point but perhaps if they did, there would be banners & protests demanding their sackings too.

    The land of milk & honey where nothing bad will ever need to happen because em well, ummm, errr awaits…..

    I hope No wins but the closer it gets, the more anxious I am becoming. My family still live there & are divided between No, Yes & ‘what’s the point, it won’t change anything’ while I’m in England trying desperately to persuade them to vote No.

  17. TARK

    Not what I saw said on BBC News and Miliband directly said it was not intimidation today but he felt it had happened elsewhere.

  18. Richard

    “based on the street by street returns from the BT campaign”.

    That assumes that they have had the ground troops to actually conduct a canvass. They may well have in other parts of Scotland for all I know, but the experiences of my family in Ayrshire, Glasgow and the Black Isle suggests that they haven’t.

    Social media is full of pictures from both sides of activists holding placards and claiming huge support. I suspect that may be a wee bit of propaganda. :-)

  19. “Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.”

    There’s not much they can do to make sure that they’re not getting not enough No voters?

    That’s a LOT of negatives!

  20. Skippy
    “We could say the constant YES flashmobs and so on are signs of THEIR panic that they are losing.”

    Obama had a healthy lead in later stages of 2012 yet there were flash mobs at Romney events/conferences in Boston.

    I think it is close but the momentum which started from that fractious second TV debate is not with Yes any more. They just couldn’t deliver or convince on their policies in the final build up to the vote. They had zero answers in the face of big heavy weight interventions by banks, businesses, oil companies, NHS staff, EU leaders and so on and ultimately if you are a Yes campaigner, you could say Scots were intimidated into voting No or if you are unionist, you could say head prevailed over heart.

  21. @ Martin

    “@Roykite – we get another referendum if the vow isn’t upheld. They talked about this last night on ‘Scotland tonight’ (David Torrance and another lady from the herald). They both felt that the SG would legitimately be able to call another referendum either if the vow wasn’t upheld or in leaving the EU.”

    This assumes the Scottish vote to stay but the rUK votes out. Judging by how the Scottish voted in the AV referendum, much as Salmond likes to socially engineer, I don’t think there is a much difference as he would like to believe. And I ‘think’ the Scottish Social Attitudes survey also supports this theory. If someone has the figures handy that would be great.

  22. Jack

    How to vote?

    Consider the heart. Nationalism is not boring logic; that’s the No campaign.

    Nationalism is an emotion based on language, culture and history

    Actually that’s not the case and we have polling to prove it. The ICM-Guardian poll I refereed to earlier

    http://www.icmresearch.com/data/media/pdf/2014_sept_guardian_scotland_indep_poll.pdf#page=24

    asked Yes and No voters You say you are going to vote Yes/No. Which two or three of the following best describes why you plan to vote that way? and gave them the same list to choose from. 41% of Yes voters gave “Your feelings about Scotland” – their second most popular after “Your feelings about Westminster, and the type of politicians there” (51%).

    But 53% of No voters gave “Your feelings about the United Kingdom as a whole” – and indeed 16% also said “Your feelings about Scotland”. So it is clearly more No voters who are making a nationalist decision that Yes voters.

  23. Is a vow stronger or weaker than a cast-iron guarantee?

  24. PETE B.
    A vow is stronger, if it is based on religious faith. LOL.

  25. @Pete

    One comment I saw on Twitter:

    “1st ‘vow’ already broken. How can they start timetable to new powers on 19th Sept when Parliament is in recess until 13th Oct.”

    Fair point.

  26. It should have been ‘I referred’ in that last comment, not ‘I refereed’. Though what it feels like on UKPR sometimes…

  27. Interested

    Judging by how the Scottish voted in the AV referendum, much as Salmond likes to socially engineer, I don’t think there is a much difference [in attitudes to the EU] as he would like to believe.

    That may be true for some matters, but not the EU.ICM asked in August:

    http://www.icmresearch.com/data/media/pdf/2014_aug_scotlandpoll9.pdf#page=66

    In the event that Scotland does become independent, should it or should it not apply to be a member of the European Union? and it was agreed by 50% to 31%. Yes voters were keenest (59-27), but so were No ones (49-34). Only the DKs were close (35-32). In contrast all-GB polls tend to be evenly split at the moment.

    It’s also worth pointing out that Scotland being pro-EU is something shown fairly consistently, both in specific polls like this, but also in regional cross-breaks (London is the other pro-EU Region of course).

  28. I haven’t seen any discussion of the likely effectiveness of the two sides in getting the vote out. That would depend not only on feet on the ground, organisation and enthusiasm, but on quality of canvass returns.

    That could make a 2-3% difference, which might be enough to swing it. Does anyone have a view on this?

  29. I came off the fence around 6 months ago and backed independence.

    I’m going to come off the fence for a second time and predict the majority of the Scottish electorate will vote YES.

    My final prediction is YES 53% no 47%.

    I’m not at all convinced (Sorry AW) the pollsters have been picking up a true reflection of what is happening on the ground.
    The problem Better Together have is that they are shuffling in the establishment right into peoples faces.

    A lot of the people who have never voted or at least not voted for years were fed up with Westminster. I really do think this will be a defining matter on securing a YES victory.

    So there you have it….I’m going for a YES victory.

  30. Roger Mexico,

    Although that question isn’t actually the same as “Should the UK be a member of the EU?” You can imagine someone thinking that (a) a country like Scotland would be better-off in the EU and (b) think that the UK could do without the EU.

  31. ROBIN
    It doesn’t matter what DC (cast iron guarantee) and NG (pledge) say – they won’t be in government after May
    ____________

    Yes it does matter because when Scotland votes YES I think you can say ta ta to EM

  32. BBC R4 had a lengthy analysis of the implications of the vow in the rest of the UK. Tory backbenchers not happy, and regional movements from Yorkshire and Cornwall interviewed at length, as well as Plaid C.

    My prediction last night that the deal on offer would stir things up elsewhere has already come true – even faster than I thought.

  33. I also detected and a rather less hostile approach from ITN towards YES Scotland tonight with their closing statement regarding Scotland’s contribution towards the treasury and that the Scots are not being subsidised. I they too are also wakening up to the fact that Scotland will vote YES,

  34. Statgeek, Parliament is not the start of the legislative process.

  35. If we get a No vote, I also think G Brown has played a partisan blinder.

    I’m fascinated to know if he was pre announcing all the latest offers with the agreement of Downing Street, as I suspect he was out on a limb, knowing Cameron couldn’t question it with 2 days to go.

    Whether the greater powers plan is good for the whole UK we’ll leave aside, but I think Brown was thinking of May 2015. He wants the union saved, but not by Cameron. Equally, he wants to neutralise any SNP attack on Labour in Scotland.

    If he really has brought them enhanced powers and the ability to definitively protect the NHS (which they have already, but we’ll let that pass) in Scotland, Labour can argue that in 2015 it’s either Labour or Tories, forget the SNP.

    I suspect this may be why Brown is forcing the issue and is very keen on setting out a quick timetable. If there is a Yes vote, then it’s obviously all for nought, but with Cameron staring down the barrel of a break up under his watch, I rather think Brown has outmanouvered him and acted to try and preserve Labour’s northern flank, in what looks like being a tight general election.

  36. I was listening to a supporter of the No campaign being interviewed on the BBC Radio 5 Live Drive programme, yesterday. The No Supporter said that Scotland sells more to England than it does to the rest of the world combined.

    I thought to myself,is that good? I later remembered something from my economics course. This was the comparison of firms which chose to grow by expanding their own business (organic growth) and those which expanded by merging with other firms. Several studies showed that growth by the “organic” route i.e. expanding your own business, was higher than the growth obtained by merging with other companies.

    Of course these studies refer to the the growth of firms, not to the growth of countries. Therefore one cannot conclude that Scotland’s growth, and England’s growth are slower than they would be if they had not merged.

    However, one question, has struck me throughout the referendum campaign. If Scotland is such a feeble country in economic terms as some of the No campaigners claim, then why does England want to keep Scotland bound in to a union? Why does England try to buy the support of Scotland by offering incentives?

    In some old melodramas – the Sherlock Holmes story, The Speckled Band is an example – the villain tries to stop their daughters marrying because if they did so, the father, would lose a large part of the family inheritance.

    Of course the Scottish referendum campaign is not like that.

  37. @AC

    Scotland going independent will (a) badly damage DC (the radio this morning was once again saying that he will face calls for his resignation); (b) lose Labour far fewer seats than would be necessary to stop them winning the election;

  38. @Steve

    Take us through the process then.

  39. Allan Christie,
    “A lot of the people who have never voted or at least not voted for years were fed up with Westminster. I really do think this will be a defining matter on securing a YES victory.”

    Spot on. This group are much more likely to vote YES, and yet are hard for pollsters and campaigners to reach. A lot depends on whether they will actually be able to drag themselves to the polling station.

    It’s a similar situation to UKIP in other elections. They too are reaching disaffected voters, and it’s one reason why the Tories can’t count on UKIP voters returning to the fold, because most of them weren’t in the fold to start with.

  40. robin

    don’t waste your time.

    Re sturgeon/applying to EU so that they can continue to charge only English students.

    Playing the game of yes will win, would that application be before they apply for EU membership or afterwards?

  41. ALEC

    Ever since Scotland got devolution things have been stirring up..That’s why I said away back when I first started commenting on this website that the best way forward for the UK was either full independence for Scotland or no devolution..ie direct rule from Westminster. Anything else in the middle will create resentment..I was right.

    Anyway…Ive just been looking into the SNP archives.

    Free in 93 but it was 3 MP’s in the 92 election.

    Heaven in 97…I don’t think it was seven.

    Hard to believe a party which won just 3 MP’S little over 20 years ago now has the world’s attention and is on the brink of achieving it’s ultimate goal.

    Alex Salmond. Margaret Ewing and Andrew Welsh were the 3 nationalist MP’s elected in 1992. :-)

  42. statgeek (fpt)

    Very strange sampling on this poll. What’s going on?

    Today’s sample as % of previous 10 poll average:

    UK: 86%
    Lon: 96%
    RoS: 81%
    Mid: 94%
    Nor: 91%
    Sco: 66%

    I mentioned this previously. The response from YouGov’s Scottish panel members had been so large (and therefore needed to be weighted down so much) in the last week that assumed extra were being requested for separate analysis.

    It doesn’t seem to have been, and instead it looks as if instead of only say 40% responding to survey requests when sent out perhaps 90% are swept up in the excitement and ‘playing a part’. Possibly in the latest sample they decided to send out fewer requests to members in Scotland to counterbalance the current high response rate. Or if there was a separate private analysis, maybe it is no longer needed.

  43. ROBIN
    @AC
    Scotland going independent will (a) badly damage DC (the radio this morning was once again saying that he will face calls for his resignation); (b) lose Labour far fewer seats than would be necessary to stop them winning the election
    __________

    I don’t doubt it will damage DC, I’m on about the Tory party itself and they may benefit from no Scottish MP’s in Westminster. Remember they are 59 MP’s in Scotland, only one is a Tory so that will be a +58 for the .

    Labour flung everything into this campaign and even temporally put GB back in as PM so they will take knock when the Scots vote YES.

    Who know…Maybe UKIP and the Greens might form a new government. ;

  44. PETE B

    “Spot on. This group are much more likely to vote YES, and yet are hard for pollsters and campaigners to reach. A lot depends on whether they will actually be able to drag themselves to the polling station.
    It’s a similar situation to UKIP in other elections. They too are reaching disaffected voters, and it’s one reason why the Tories can’t count on UKIP voters returning to the fold, because most of them weren’t in the fold to start with”
    ___________

    Absolutely and even though I think the UKIP VI will fall back towards the 2015 rUK election they have however reached out to voters who have been disconnected with the main parties and Westminster itself. Look at the stunning UKIP results at the locals and European elections .

    If they can drag themselves out of bed to vote UKIP then I’m pretty sure the same will happen in Scotland for the YES vote.

  45. @Roger

    My little conspiracy alarm was wondering what London and Scotland had in common to get low samples. Lower than average UKIP support?

  46. Polly wolly stop telling people that they are wasting there time with regards to my comments.

    I might just boot you up the arse.

  47. @Number Cruncher

    Those Local Authority numbers look really good. Thanks for all your work.

    I am willing to bet that the 3 Local Authorities you have as the lowest Yes vote: Borders, D&G and East Renfrewshire will be the 3 lowest Yes votes in reality as well.

    The big unknown question I think is places like Glasgow, Renfrewshire and Inverclyde will they behave as your model suggests or have a significantly higher Yes vote due to high turnout amongst marginalised pro Yes voters.

    For Yes to win the West Coast Labour strongholds will need to poll more strongly for Yes than the model suggests. If any of them are coming in below the percentages you give here then that would point to a no vote for me.

  48. @R&D
    “Re sturgeon/applying to EU so that they can continue to charge only English students.
    Playing the game of yes will win, would that application be before they apply for EU membership or afterwards?”

    It’s a total non-starter in any event.

    There is no need for NS to apply for a derogation from the EU prior to effective independence, as in the interim Scotland would remain part of the UK, and the EU has no jurisdiction as the matter is a wholly internal situation. There is no Community/Union dimension.

    If NS applies after effective independence, for a ruling on whether Scotland on becoming an EU member state could continue with the policy, the EU would tell her to get lost. I don’t think NS or the SNP or Yes generally has thought this through. Discriminating between nationals of other member states is as fundamental a principle of EU law as you will find. There can be no objective justification for such discrimination and a derogation would simply not be granted.

    I feel a bit sorry for NS here as the problem with the policy isn’t that Scotland does not charge its students fees but that rUK does. I also have sympathy for the SNP’s reasons for wanting to maintain the status quo on this issue – a desire to ensure greater access for Scottish students to attend university. But this is just a complete no-no.

  49. Salmond has just assured the shipbuilders that the UK will still build the next class of frigates on the Clyde after a Yes vote.

    Astonishing.

  50. Alan
    There might also be another parallel between the SNP and UKIP. As you point out, SNP had 3 MPs 20 years ago and now rule the roost in Scotland. Those who dismiss UKIP as a flash in the pan or protest vote might yet have a shock in 20 years time.

    Anyway, I hope the refendum vote is Yes, but it will be very close. I wonder how big an effect postal vote fraud will have, or is that more an English problem?

1 2 3 4