What effect did MORI’s wording have?
“In the next election for Mayor of London, the present Mayor, Ken Livingstone is standing for re-election as the Labour Party candidate. Boris Johnson is standing for the Conservatives, Brian Paddick is standing for the Liberal Democrats, Siân Berry is standing for the Green Party, and there will be other candidates too. In the election for Mayor, voters will have two votes, one for their first choice as Mayor and one for their second choice. If the election were held tomorrow, which candidate would be your first choice?”
A couple of people have raised questions in my comments and over on Mike Smithson’s site about the wording MORI used in today’s mayoral poll – reproduced above. People raised two points, firstly, could the extra mention of Ken Livingstone being the incumbent have helped him and, secondly, what might the effect be of prompting people with Sian Berry’s name?
In theory reminding people of Livingstone’s incumbency could help him (or indeed hinder him!), but I suspect that in practice the effect wouldn’t be large – most people likely to vote probably know Ken is the mayor. Obviously we can’t actually tell without trying parallel questions with alternate wordings, but I find it unlikely. Certainly it can’t explain any of the difference between this and the YouGov polls, since YouGov also preface their question with text saying that the post is currently held by Ken Livingstone.
More interesting is the effect of including Sian Berry amongst the names in the prompt. In contrast to MORI’s wording, in YouGov’s question respondents were shown only the names of Livingstone, Johnson and Paddick – they were only given the names of the other candidates after opting for “another candidate”. This MORI poll found Sian Berry at 5%, the highest that any of the minor party candidates has managed in any of the polls so far. Was that because of the prompting?
In this case we do have a very direct example of what effect including the Green party in the prompt has. In last year’s elections to the Scottish Parliament YouGov prompted only with the Conservatives, Labour, Lib Dem and SNP. A few weeks before the election the Green’s were showing only 4% in YouGov’s polls. In fact, overall the figure for the minor parties looked incredibly low compared to the result at the previous Scottish election. As a result Peter Kellner decided to change YouGov’s question wording halfway through the campaign to include the Greens and other minor parties in the prompt. The result was that in the next YouGov poll a week later, the level of Green support jumped up to 9%. In other words, including the Greens in the prompt more than doubled their support.
If we know including minor parties increases their support in polls, that leaves the question of whether it produces more accurate answers? In the case of YouGov’s Scottish polls it probably didn’t – their final poll had the Greens on 9% when the Green party ended up getting only 4%, so they’d probably have been better off leaving them out of the prompt. On the other hand, polls that don’t prompt at all tend to underestimate the level of Lib Dem support. Unfair though it seems, the right approach seems to be to prompt by the name of the big three parties, but not all the minor parties.
In the case of this particular poll, the decision to include Sian Berry in the prompt, but not the other minor parties, seems very strange given that the Green party finished behind UKIP, Respect and the BNP at the last election.
I do some work with Behavioural Economics and I would say from this PoV that the Mori question is framed in such a way as to influence the result (I’m not saying it is deliberate). “Present Mayor, Ken Livingstone … first choice … Who would be your first choice?” will tend to exaggerate support for Ken.
I can’t see the YouGov question on the website – the reports all begin “Boris Johnson Conservative”. I would predict that a question which began with BJ and then asked about first choice would exaggerate support for him.
It may be “irrational” but it’s human nature.
Also do pollsters say who their client is?
Ah, but the Scottish ballot paper was explicitly designed by Douglas Alexander to reduce the incidence of people splitting their vote (in practice almost always that means voting for a larger party for the constituency and a smaller one for the regional list, which is the figure you’re discussing).
You’ll remember the chaos on election night last May. Although the Gould report bottled it, the US observers were absolutely clear: the overwhelming bulk of the spoilt papers were over-votes on the regional side, and the overwhelming bulk of those were X – large party, X Greens.
You’re right, though, that wouldn’t have taken the party to 9%, I’d say more like 6% (and enough to have seen perhaps two more Green MSPs).
If someone can explain to me why the best option isn’t just to present the same list of candidates that will be on the ballot, I’ll be delighted to hear it, especially for YouGov given their methodology.
“the overwhelming bulk of the spoilt papers were over-votes on the regional side, and the overwhelming bulk of those were X – large party, X Greens.”
Nearer the time it was reported that the over-whelming bulk were votes for SNP-Solidarity. Where can I find the figures you have about this?
James – both times YoUGov have tried it, the Scottish elections and the European elections in 2004, it ended up grossly overestimating the suppport for the smaller parties who were prompted for. It might be fairest, but it doesn’t actually produce accurate results.
I suspect the reason is that the vast majority of people don’t make their mind up who to vote for with their pen hovering over the ballot paper with a list of all the candidates before them; they make their mind up in advance before going to the polling station when the larger parties have far more prominance.
My limied understanding is that various things can influence the respondent’s answers incuding the order of the questions, the context of the questions, the phrasing of the question and extraneous factors (time of day, location, what was on the news, etc. etc.). Therefore there isn’t a right way to word a question.
I’m not a market researcher but my direct marketing experience says that the most influential factor in response will be the list – who we talk to, where the list is sourced and the past behaviour of those on the list (e.g. when planning a direct mail campaign you would prefer lists with a record of response such as a mail order buyers list over compiled lists or lists created for other purposes). The offer is the next most significant factor – what we’re asking the recipient to do. And only then does copy become a factor.
My understanding of polling is that you select the model closest to the real result and work from there rather than try to construct some kind of theoretical model of perfection. The question above doesn’t look too good and may be a factor in the poll findings but I suspect less so than many of the other considerations I’ve noted above.
I’d suggest that the fact the question highlights that voters have both a first and second preference vote could be more important… did YouGov and ICM actually stress this in the same way? Obviously it would have come up, but what proportion of voters pass on offering a second preference altogether? And could that have some baring on the polling and the eventual result?
As for Mori flagging up the Greens, I’d tend to agree that any of the minor parties being mentioned in such a way is likely to boost their share in any given poll… in most elections the three main parties will dominate the debate and as such the minor parties find it hard to get noticed as a result their support is depressed, but mentioning them in the way Mori has provides them with a profile they simply won enjoy during the campaign.
Could then mention of Livingstone’s incumbency and the Greens reflect the fact that it’s a poll conducted for Unison?
Wonder if anyone in the Green Party thinks that association with Livingstone may not be to its advantage?
The problem with all these polls is that they force people to express an opinion on the Mayoral Election itself.
Whereas in practice many will vote on national issues (less so than in local elections, but many nonetheless).
On the actual ballot paper – are the candidates listed in alphabetical order? If so, is that by party or by surname?
Has anyone done a study as to whether people with surnames starting with A or B tend to outpoll people whose surname starts lower down the alphabet?
In most UK elections (I don’t live in London so I haven’t seen the London ballot paper) candidates are listed alphabetically by surname.
There have been lots of studies across the world over many years that have supported the idea that there is an advantage to being towards of the top of the ballot.
However, it is not uniform – there is a greater effect when parties are not listed on the ballot paper (they are in the UK), there is a greater effect in small, less publicised races. In the USA people will vote for many positions on a single ballot paper – they’ll have, for example, one vote for President, then governor, then deputy leiutenant, then members of the school board and so on down. Studies found a much greater effect there on the races lower down the paper (e.g. people might have a firm idea who they want as president, and then just tick the top candidate for dog catcher).
The most recent UK evidence I can find is from Rallings, Thrasher and Borisyk in 2006. They found there was a significant difference in multi-member wards with candidates from the same party – if three candidates from a party were contesting a council ward in the 2006 London elections, 58% of the time the one first in the alphabet came highest of the three. 58% the one last in the alphabet came third.
What they didn’t find was any effect on finishing positions in single member seats, suggesting that in the UK there is an effect within party slates, but not overall.
Thanks Anthony. Interesting.
Anthony:
I’m still interested in the bias issue in the questions. In normal polling you rotate the order specifically to eliminate bias. The Mori wording, featuring Ken first and then going on about First twice is, in my view, biased to Ken. Can you post the you.gov wording please?
And do both Mori and YouGov then list the candidates in (randomised) order, or does the nice lady from Mori simply then wait for the person to respond to her question?
Well there are some indications that “Alex Salmond for First Minister” worked for us; because he is.
If we put up two candidates in my ward in 2011 I’ll try to get my colleague to have a name like Zorro……
Peter.
>>Nearer the time it was reported that the over-whelming bulk were votes for SNP-Solidarity. Where can I find the figures you have about this?
I’ve got the audio from the Today programme on the Saturday morning after the election during which the American observer tells Humphries that. If you want a copy of that, let me know. I may blog it, now this thread has reminded me about it.
I don’t have figures, though: another failing of Gould.