The Press Association are reporting that “The majority of people from across the political spectrum believe Scotland should be responsible for raising most of the money it spends, according to research from an independent think-tank.”
Because it is on the Press Association feed, this is then repeated verbatim by various other newspaper websites here, here, here, etc, etc, all labouring under the misapprehension that because the Press Association reports a poll it is meaningful. They are wrong.
The “poll” was conducted by Reform Scotland, a think tank that published proposals for devolution plus earlier this year. The “sample” was drawn from people on Reform Scotland’s mailing list or following them on twitter. Needless to say, this is not a method likely to provide a representative sample of the Scottish public as a whole.
I hate to write as if addressing morons, but sadly it sometimes appears as if it is necessary. People who have signed up to follow a think tank that has proposed a devolution plus plan are, firstly, far more likely to be interested in politics (the vast majority of normal people are not on the distribution lists for think tanks!) and secondly, likely to be pre-disposed towards further devolution of power towards Scotland (for what it’s worth, the poll is also three-quarters male, only 10% over 65+ and has more Tory identifiers than Labour ones).
To give them some credit, Reform Scotland themselves haven’t claimed it is a representative poll, saying “We do not claim that this poll is totally scientific as it was self selecting. However, the responses, particularly those broken down by party affiliation, are very interesting, in particular”. Alas, the reality is that these caveats never get picked up by journalists, and such surveys inevitably end up being misreported as representative meaningful polls. For the record, the party breakdowns are not of any meaning either, since in the same way the poll overall will be grossly biased towards people with an interest in Scottish politics and a predisposition towards greater devolution, so will each of the party crossbreaks (i.e. the Labour voters in the sample will be more political and more in favour of further devolution than the average Labour voter, ditto other parties. They are also grossly demographically skewed towards younger men, and apart from the SNP have sample sizes under 100).
Over on the British Polling Council’s website there is an article written by Peter Kellner several years ago titled “A Journalist’s Guide To Opinion Polls”. Amongst other things, it gives guidance to journalists on when to take a poll seriously, and when to bin it. It is still flawless advice today:
“If the poll purports to be of the public as a whole (or a significant group of the public), has the polling company employed one of the methods outlined in points 2,3 and 4 above [quasi-random or quota sampling]? If the poll was self-selecting — such as readers of a newspaper or magazine, or television viewers writing, telephoning, emailing or texting in — then it should NEVER be presented as a representative survey.