ComRes have done their monthly online poll for the Independent on Sunday. Topline figures are CON 31%(+2), LAB 34%(-1), LDEM 7%(nc), UKIP 19%(nc), GRN 4%(nc). For clarification, given some of the misunderstandings on Twitter earlier today, this is using ComRes’s normal methodology and prompting, they haven’t changed anything (I have no idea if they intend to do so or not… though I expect they’ll be getting a lot of people asking them tonight!). The sample size however was smaller than usual, as with the other half of the sample ComRes carried out an experiment asking the voting intention question including UKIP in the main voting intention prompt. The result using that different method was CON 29%, LAB 31%, LDEM 7%, UKIP 24%, GRN 5%.
Now, I should underline the importance of noting that this is just one poll. It is comparing two samples of 1000 or so people, with the usual margins of error that implies – so not all the difference will necessarily be prompting, some could just be normal sample variation. Please don’t go away with the idea that prompting for UKIP will always has the effect of bumping up UKIP by 5% – it’s just one data point. I think it probably does make a difference (we’ve tested in the past), but five points does seem rather high. Also remember that prompting may affect different methods differently, so the way it affects a ComRes online poll using their methods would not necessarily reflect the way it would affect any other poll (I am personally intrigued by the possibility that prompting may have a different impact in telephone polls, where people may feel obliged to pick one of the options offered by a human interviewer, than in an online poll where it’s just clicking through to another list of options – but obviously I don’t have phone polls to test it on!)
Knowing that prompting does make a difference – something that pollsters knew anyway – doesn’t actually get us any closer to an answer to the real question though, whether prompting for UKIP produces more or less accurate results in GB election polls. It the ComRes figure of 19% more or less accurate than the figure of 24%? Whether polls prompt or not for UKIP is often a issue that produces a lot of comment. Part of that is from people whose concern is, shall I say, more to do with maximising the reported level of support for UKIP than it is to maximise the accuracy of polling. Part of it is that, prima facie, it does seem somewhat strange that a party (normally) running in third place isn’t prompted for when the party that’s (normally) running in fourth place is. Another part is people looking for an explanation for the big difference in reported levels of UKIP support between different pollsters; typically the companies showing the highest levels of support, Survation and Opinium, show UKIP at about twice the support of ICM or MORI, who typically show the lowest. In the latter case I think the attention is misplaced – the reason for the biggest differences in levels of UKIP support in the polls appears to lie elsewhere – companies like Opinium manage to show some of the higher figures without any prompting! Rather they appear to be a contrast between telephone polling and online polling, for some reason online polls show consistently higher levels of UKIP support than telephone polls. That may be something to do with the mode (perhaps people are more ready to admit they are voting UKIP to an anonymous computer screen than to a human interviewer) or it could be something to do with sampling (for some reason phone samples have fewer of the sort of people who vote UKIP than online samples do).
As a pollster it is more important that methods produce the most accurate results than it is whether they appear “fair” (and certainly it’s more important to be accurate than to produce the higher possible score for UKIP!). The fact is that there isn’t a hard and fast rule about when you do and don’t prompt, we don’t have the evidence to say the cut off point is x% support, or y place, or z number of MPs. It’s a matter of judgement. We know from experience over the last couple of decades that prompting for smaller parties tends to overestimate their support (probably because it gives them a prominence and perception of equality with the major parties that may not be there among the general public), we also know that in the 1980s NOT prompting for the Lib Dems used to underestimate their support, so getting it wrong either way can produce error. Sometimes you can get it wrong by prompting, sometimes you can get it wrong by not prompting. There is no real way of knowing when a party switches from a position where prompting risks overestimating them to one where not prompting risks underestimating them – but clearly we are equally keen to avoid both errors. If UKIP establish themselves to the point that they have lots of MPs, consistent support over time, have known people and policies, are treated as a major established party that is given equal treatment by the BBC and OfCom and so on the time will come when the risk of not prompting outweighs the risk of prompting (it has already come, for example, in European elections)… but when you reach that point? It’s a judgement call.
It’s in a bigger context too. The last general election was held in the middle of “Cleggmania” and a surge of enthusiasm for the Liberal Democrats. The polls overestimated their support. The European elections earlier this year saw a great big surge of enthusiasm for UKIP… and of polls in the last week all but one company overestimated their support. In the Scottish referendum I don’t think anyone could deny that the YES campaign were the more enthused, and the polls seem to have all slightly overestimated their support. I may very well be reading something into these that isn’t there, but you get my drift – polls may be overestimating support for parties and movements that have particularly enthusiastic and zealous supporters. There’s also that unexplained difference in UKIP support between telephone companies and online companies, and what might be behind that. Getting UKIP right at the next election is the big challenge facing pollsters, but its about more than just prompting.