The Scottish polls at the end of last week and the weekend were broadly clustered around a small No lead. There are a couple of days campaigning left, but perhaps a more likely route to a YES victory is if the polls are underestimating the level of YES support for some reason. Over the last couple of days I’ve seen several blogs or articles pondering whether the polls could be wrong, could they be underestimating YES or NO?

It would be hubris to suggest the polls couldn’t be wrong. Obviously they can. At most elections there are polls that perform better or worse than their peers, some of that is better methodology. When the polls are close most is probably just normal sample variation. That’s a matter for another time though, here I’m pondering more about the possibly that all the polls are wrong, the potential for a systemic bias with everyone a bit too yes or a bit too no. This is possible too – think of the way all polls overestimated Lib Dem support in 2010, or most famously how all the polls overestimated Labour support in 1992. How likely is that?

The Scottish referendum is a bigger challenge for pollsters than an election would be because it’s a one-off. In designing methodology for voting intention the experience of what worked or didn’t work at previous elections weighs heavy, and most companies’ weighting schemes rely heavily upon the previous election – if not directly through weighting by recalled vote, in using the data from the previous election in designing and testing other weighting targets. For a referendum you can’t take that direct approach, pollsters needed to rely more on modelling what they think is an accurate picture of the Scottish electorate and hoping it reflects the Scottish people well enough that it will also reflect their referendum voting intentions – it’s complicated because Scotland has a complicated electorate. Scottish voters have two Holyrood votes and a Westminster vote, and they use them all in different ways with different political loyalties. Within the space of a year Scotland managed to be a Labour stronghold at Westminster and to produce a SNP landside at Holyrood – using either election alone for weighting gives a rather different picture of what the Scottish electorate are like, even though you are trying to model the same population. Different companies have arrived at different methods of political weighting to deal with the issue – Survation, ICM and TNS weight by Holyrood recalled voted alone, YouGov weight by Holyrood recalled vote with a nod towards 2011 Holyrood voters who backed Labour in 2010, Opinium weight by Holyrood and Westminster recalled vote, Panelbase weight by Holyrood and European recalled vote, Ipsos MORI don’t use political weighting at all. Despite the variance they have all converged to produce the same sort of result, and that gives me some confidence – if there was a particular skew from being online or from using Holyrood recalled vote we would expect to see different results.

Most speculation about whether the polls might be wrong has – rightly in my view – concentrated on two particular issues. Very high turnout and differential response rate.

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election. This poses a problem. Any professionally run opinion poll will make every effort to get as representative a sample as possible, but in practice there are limitations. People on the very fringes of society, people struggling in absolute poverty, those utterly detached from mainstream politics and civic society – people on the extreme edge are probably underrepresented in opinion polls of all sorts. In most voting intention polls this doesn’t matter, as people on the very fringes are also extremely unlikely to vote… but if the Scottish referendum does manage to engage some who were previously totally detached and, crucially, those people vote in a substantially different way to other people of similarly marginal demographics, then it could be a source of error.

The second potential pitfall is differential response. Much of the media discussion around this has called it “shy Noes” – people who want to vote no but are reluctant to admit it to pollsters. That’s possible, but it should be much less of a problem with online polls when people are giving their opinion to an impersonal computer screen. I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother? Issues of differential response can be mitigated through careful sampling and political weighting but again, it can only go so far. Pollsters can make sure they aren’t getting too many people who voted SNP in 2011, but there’s not much they can do to be 100% certain they are aren’t, for example, getting too many Yes voting Labour voters and not enough No voting ones.

So, how confident am I about the polling in the Scottish referendum? Well, I suppose I’m fairly confident – if there was anything I thought we were doing critically wrong we’d have corrected it. If I had to put money on the result, I’d certainly back the polls, but the potentials for error are there. We’ll know on Friday if they’ve been avoided.


As well as the Ashcroft and Populus polls earlier today we also have the monthly ICM poll for the Guardian tonight, reported here. Topline figures with changes from a month ago are CON 33%(+2), LAB 35%(-3), LDEM 10%(-2), UKIP 9%(-1), GRN 7%(+3). It shows a sharp narrowing of the Labour lead, but it’s almost certainly a reversion to the mean: the previous ICM poll had the Labour lead jumping up to 7 points when it had been previously showing Labour and the Conservatives pretty much neck and neck.

The poll also asked about English and Welsh attitudes to the Scottish referendum and to further devolution. As we’ve seen elsewhere, there is little support for a currency union with an independent Scotland amongst the rest of the UK – 27% of people say the remainder of the UK should negotiate a currency union, 63% they should not. Asked about more devolution in England, via regional assemblies or an English Parliament 45% think it would help their area of England, 42% that it would not.


A couple of tight polls today. The weekly telephone poll by Lord Ascroft has topline figures for CON 33%, LAB 33%, LDEM 9%, UKIP 14%, GRN 6% (tabs here). Meanwhile the twice-weekly online poll by Populus this morning had figures of CON 34%, LAB 35%, LDEM 9%, UKIP 13%, GRN 3% (tabs are here)

Along with the Ipsos MORI poll last week showing the Conservatives just ahead one might be forgiven for thinking that adds up to a narrowing of the polls. I’d be a bit cautious – the YouGov daily tracker had a three point lead at the weekend and a couple of sixes last week, the Opinium poll on Sunday had an eight point Labour lead. I think we’re probably just seeing a couple of low polls randomly turn up at the same time… though of course, time will tell.


With what I assume are all Sunday’s Scottish polls in, where do we stand? Looking across the board at all six companies polling, two of them using two different modes, we actually have a broadly consistent picture. Excluding don’t knows, the Yes shares in the 8 different companies/methods are:

ICM (online) 54%
Panelbase (online) 49%
ICM (phone) 49%
TNS (face to face) 49%
YouGov (online) 48%
Opinium (online) 47%
Survation (online) 47%
Survation (phone) 46%

Seven of the polls are clearly clustered around a small lead for the NO campaign, with the one exception that rather odd looking ICM online poll with a smaller sample size than their usual online efforts. A lead of just a couple of points in a single poll is within the margin of error, but in this case all but one poll is showing NO ahead, so I think we can reasonably say that the polls are giving NO a genuine but small lead.

If the polls are broadly correct, and if nothing changes in the last five days, then NO look like they’ll have a narrow win… but of course those are two very significant ifs. It’s certainly possible for a race this tight to change within a few days and there have certainly been occasions in the past when the polls have had a systemic error of a couple of points in one direction or the other.


It’s the last Sunday before the referendum so we can expect several polls tonight (in fact, the chances are we’ll get lots today, and then comparatively few until Wednesday when there will be a glut of eve-of-referendum polls). I am expecting at least three today – Survation, Panelbase and Opinium – and I’ll update as they appear.

First out is Survation, conducted for the Better Together campaign. Their topline figures are YES 41%, NO 47%, Don’t know or refused 12%. Excluding the don’t knows and won’t says that works out at YES 46%, NO 54%. Full tables are here. I haven’t included any changes since last time as unlike all Survation’s previous Scottish referendum polls this one was conducted by telephone rather than online. It means we couldn’t confidently conclude anything from any change, though for what it’s worth it wouldn’t be showing any significant change anyway, Survation’s last online poll had YES 47%, NO 53%.

Later on we have an Opinium Scottish poll for the Observer, due at 8 o’clock, and a Panelbase for the Sunday Times.

UPDATE: The second Scottish poll of the evening is one we weren’t expecting – an ICM poll for the Sunday Telegraph here. The fieldwork was done at pretty much the same time as the telephone ICM poll for the Guardian yesterday, but this one was conducted online and has significantly different figures. Topline figures with changes from ICM’s last online poll are YES 49%(+11), NO 42%(-5), Don’t know 9%. Excluding don’t knows this works out at YES 54%(+9), NO 46%(-9). This echoes the large shift towards YES over the last month that YouGov and TNS have shown, but the significant lead for the YES campaign is in contrast to other polls, which are showing a small lead for the NO campaign.

Note that the sample size for the poll was only 705, smaller than usual but not obscenely so (a sample size of 705 increases the margin of error to 3.7%, so less precise than 1000, but not by a vast amount). Update with the Opinium poll coming up very soon….

UPDATE 2: The Opinium poll for the Observer is also out and has figures that are very much in line with the main pack – YES is on 45%, NO is on 49%, 6% say don’t know. Excluding don’t knows that works out at YES 47%, NO 53%. This is Opinium’s first outing in the Scottish referendum campaign, so no trend data. Full results are here, and Opinium have a note on methodology here – in Opinium’s usual GB polls they do not use any political weighting, but in their Scottish polling they are weighting by both 2011 recalled Holyrood vote AND recalled 2010 Westminster vote.

Still to come tonight we have at least a Panelbase poll for the Sunday Times.

UPDATE 3: What is presumably the final Scottish poll of the night is Panelbase for the Sunday Times, and has a wafer thin NO lead. Topline figures appear to be YES 49%(+1), NO 51%(-1) (they are being widely quoted at 49.4% to 50.6%, but quoting decimal places when you’ve a margin of error of plus or minus 3 whole percentage points always seems downright silly to me!). I’ll update again shortly with a roundup of where we stand.