On Friday the BPC/MRS inquiry into the polls at the 2015 started rolling. The inquiry team had their first formal meeting in the morning and in the afternoon there was a public meeting, addressed by representatives of most of the main polling companies. It wasn’t a meeting intended to produce answers yet – it was all still very much work in progress, and the inquiry itself isn’t due to report until next March (Patrick Sturgis explained what some see as a very long time scale with reference to the need to wait for some useful data sources like the BES face to face data and the data that has been validated against marked electoral registers, neither of which will be available until later in the year). There will however be another public meeting sometimes before Christmas when the inquiry team will present some of their initial findings. Friday’s meeting was for the pollsters to present their initial thoughts.

Seven pollsters spoke at the meeting: ICM, Opinium, ComRes, Survation, Ipsos MORI, YouGov and Populus. There was considerable variation between how much they said – some companies offered some early changes they were making, some only went through possibilities they were looking at rather than offering any conclusions. As you’d expect there was a fair amount of crossover. Further down I’ve summarised what each individual company said, but there were several things that came up time and again:

  • Most companies thought there was little evidence of late swing being a cause. Most of the companies had done re-contact surveys, reinterviewing people surveyed before the election and comparing their answers before and afterwards to see if they actually did change their minds after the final polls, and most found little change that cancelled itself out, or produced negligible movement to the Tories. Only one of the companies who spoke thought it was a major factor.
  • Most of the pollsters seemed to be looking at turnout as being a major factor in the error, but this covered more than one root cause. One was people saying they will vote but not doing so, and this not being adequately dealt with by the existing 0-10 models of weighting and filtering by likelihood to vote. If that is the problem the solution may lie in more complicated turnout modelling, or using alternative questions to try and identify those who really will vote.
  • However several pollsters also talked about turnout problems coming not from respondents inaccurately reporting if they vote, but from pollsters simply interviewing the sort of people who are more likely to vote, and this impacting some groups more than others. If that’s the cause, then it is more of problem of improving samples, or doing something to address getting too many engaged people in samples.
  • One size doesn’t necessarily fit all, the problems affecting phone pollsters may end up being different to online pollsters, and that the solutions that work for one company may not work for another.
  • Everyone was very wary of the danger of just artificially fitting the data to the last election result, rather than properly identifying and solving the cause(s) of the error.
  • No one claimed they had solved the issue, everyone spoke very much about it being a work in progress. In many cases I think the factors they presented were not necessarily the ones they will finally end up identifying… but those where they had some evidence to show so far. Even those like ComRes who have already made some initial conclusions and changes in one area were very clear that their investigations were continuing, they were still open minded about possible reasons and conclusions and there were likely more changes to come.

Martin Boon of ICM suggested that ICM’s final poll showing a one point Labour lead was probably a bit of an outlier and in that limited sense was hence a bit of bad luck – ICM’s other polls during the campaign had shown small Conservative leads. He suggested this could possibly have been connected to doing the fieldwork for the final poll during the week, ICM’s fieldwork normally straddles the weekend and the political make up of C1/C2s in his final sample was significantly different from their usual polls (they broke for Labour, when ICM’s other campaign polls had them breaking for the Tories) (Martin has already published some of the same details here.) However, bad luck aside he was clear about there being a much deeper problem in that the fundamental error that had affected polls for decades – a tendency to overestimate Labour – has re-emerged.

ICM did a telephone recall poll of 3000 people who they had interviewed during the campaign. They found no significant evidence of a late swing, with 90% of people reporting they voted how they said they would. The recall survey also found that don’t knows split in favour of the Conservatives and that Conservative voters were more likely to actually vote… ICM’s existing reallocation of don’t knows and 0-10 weighting by likelihood to vote dealt well with this, but ICM’s weighting down of people who didn’t vote in 2010 was not, in the event, a good predictor (it didn’t help at all, though it didn’t hurt either).

Martin’s conclusion was that “shy Tories” and “lazy Labour” were NOT enough to explain the error, and there was probably some deeper problem with sampling that probably faced the whole industry. Typically ICM has to ring 20,000 phone numbers in order to get 1,000 responses – a response rate of 5% (though that will presumably include numbers that don’t exist, etc) and he worried again about whether our tools could get a representative sample.

Adam Drummond of Opinium also provided data from their recontact survey on the day of the election. They too found no evidence of any significant late swing, with 91% of people voting how they said they would. Opinium identified a couple of specific issues with their methodology that went wrong. One was their age weighting was too crude – they used to weight age using three big groups, with the oldest being 55+. They found that within that group there were too many people who were in their 50s and 60s and not enough in their 70s and beyond, and that the much older group were more Tory. Opinium will be correcting that by using more detailed age weights, with over 75s weighted separately. They also identified failings in their political weightings that weighted the Greens too high, and will be correcting that now they have the 2015 results to calibrate it by.

These were side issues though, Opinium thought the main issue was one of turnout, or more specifically, interviewing people who are too likely to vote. If they weighted the different age and social class groups to the turnout proportions suggested in MORI’s post-election election it would have produced figures of CON 37%, LAB 32%…. but of course, you can’t weight to post-election turnout data before an election, and comparing MORI’s data at past elections the level of turnout in different groups changes from election to election.

Looking forwards Opinium are going to correct their age and political weightings as described, and are considering whether or not to weight different age/social groups differently for turnout, or perhaps trying priming questions before the main voting intention. They are also considering how they reach more unengaged people – they already have a group in their political weighting for people who don’t identify with any of the main parties… but that isn’t necessarily the same thing.

Tom Mludzinski and Andy White of ComRes offered an initial conclusions were that there was a problem with turnout. Between the 2010 and 2015 elections actual turnout rose by 1%, but the proportion of people who said they were 10/10 certain to vote rose by 8%.

Rather than looking at self-reported levels of turnout in post-election surveys ComRes did regressions on actual levels of turnouts in constituencies by their demographic profiles, finding the usual patterns of higher turnout in seats with more middle class people and older people, lower turnout in seats with more C2DE voters and younger voters. As an initial measure they have introduced a new turnout model that weights people’s turnout based largely upon their demographics.

ComRes have already discussed this in more detail than I have space for on their own website, including many of the details and graphs they used in Friday’s presentation.

Damian Lyons Lowe of Survation discussed their late telephone poll on May 6th that had produced results close to the election, either through timing or through the different approach to telephone sampling they used. Survation suggested a large chunk of the error was probably down to late swing – their recontact survey had found around 85% of people saying they voted the way they had said they would, but those who did change their minds produced a movement to the Tories that would account for some of the error (it would have moved the figures to a 3 point Conservative lead).

Damian estimated late swing made up 40% of the difference between the final polls and the result, with another 25% made up from errors in weighting. The leftover error he speculated could be caused by “tactical Tories” – people who didn’t actually support the Conservatives, but voted for them out of fear about a hung Parliament and SNP influence and wouldn’t admit this to pollsters either before or after the election, pointing to the proportion of people who refused to say how they voted in their re-contact survey.

Tantalisingly, Damian also revealed that they were going to be able to release some of the private constituency polling they did during the campaign for academic analysis.

Gideon Skinner of Ipsos MORI‘s thinking was still largely along the lines of Ben Page’s presentation in May that was (perhaps a little crudely!) summarised as lazy Labour. MORI’s thinking is that their problem was not understating Tory support, but overstating Labour support. Like ComRes, they noted how the past relationship between stated likelihood to vote and actual turnout had got worse since the last election. At previous elections they noted how actual turnout had been about 10 points lower than the proportion of people who said they would definitely vote; at this election the gap had been 16 points.

Looking at the difference between people’s stated likelihood to vote in 2010 and their answers this time round the big change was amongst Labour voters. Other parties’ voters had stayed much the same, but the proportion of Labour voters saying they were certain to vote had risen from 74% to 86%. Gideon said how this had been noticed at the time (and that MORI had written about it as an interesting finding!), but it had seemed perfectly plausible that now the Labour party were in opposition their supporters would become more enthusiastic about voting to kick out a Conservative government than they had been at the end of a third-term Labour government. Perhaps in hindsight it was a sign of a deeper problem.

MORI are currently experimenting with including how regularly people have voted in the past as an additional variable in their turnout model, as we discussed in their midweek poll.

Joe Twyman of YouGov didn’t present any conclusions yet, just went through the data they are using and the things they were looking at. YouGov did the fieldwork for two academic election surveys (the British Election Study and the SCMS) as well as their daily polling, and all three used different question ordering (daily polling asked voting intention first, SCMS after a couple of questions, the BES after a bank of questions on important issues, which party is more trusted and party leaders) so will allow testing of the effect of “priming questions”. YouGov are looking at the potential of errors like “shy Tories”, geographical spread of respondents (are there the correct proportion of respondents in Labour and Conservative seats, safe and marginal seats), are respondents to surveys too engaged, is there panel effect and dealing with turnout (including used the validated data from the British Election Study respondents).

Andrew Cooper and Rick Nye of Populus also found no evidence of significant late swing. Populus did their final poll as two distinct halves and found no difference between the fieldwork done on the Tuesday and the fieldwork done on the Wednesday. Their recontact survey a fortnight after the election still found support at Con 33%, Lab 33%

On the issue of turnout Populus had experimented with more complicated turnout models during the campaign itself – using some of the methods that other companies are now suggesting. Populus had weighted different demographic groups differently by turnout using the Ipsos MORI 2010 data as a guide, and they also had tried using how often over 25s said they had voted in the past as a variable in modelling turnout. None of it had stopped them getting it wrong, though they are going to try and build upon it further.

Instead Populus have been looking for shortcomings in the sampling itself, looking at other measures that have not generally been used in sampling or weighting but may be politically relevant. Their interim approach so far is to include more complex turnout modelling and to add in disability, public/private sector employment and level of education into the measures they weight by to try and get more representative samples. Using those factors would have given them figures of CON 35%, LAB 31% at the last election… better, but still not quite there.


170 Responses to “The Polling Inquiry public meeting”

1 2 3 4
  1. @Mr Jones

    I also think the EU are desperate to keep the political dream of a United Europe alive.

    They haven’t been working all these decades, just to let Greece spoil the party.

  2. @Martyn – a typically bombastic reply, but much is poorly understood or missing the all important context.

    “The responsibility for taking loans freely entered into between informed legal adults lies solely with the borrower.”

    We will disagree on this. Caveat emptor, would be my general approach. If a lender/loan shark lends to a debtor an amount he knows cannot be repaid, then I have few moral qualms about seeing the lender lose something. The only question for me in such cases is what the systemic risks are and whether others would be hurt. Borrowers do have responsibilities too, obviously, but your world of black and white is childishly simplistic, in my view.

    “Absolute amounts are relevant. They can’t afford the army/navy they have. So they should make it smaller, fast.”

    They’ve reduced total military spending from 2.8% of GDP to 1.68% in five years, even as their GDP has shrunk by 25%. I’d say that’s a pretty fast cut.

    Oh – and don’t forget that the German government government was still lending the Greeks money to buy brand new frigates (and applying great pressure on them to do so) after the crisis hit. Oh that’s right – they were made in Germany. Very fiscally responsible.

    ” We should be working on ensuring that the 320 million hardworking taxpayers of the Eurozone should get back the 360 billion euros that the Greek state borrowed.”

    As @Laszlo points out, The Eurozone volunteered to take on Greece’s commercial debt. The banks lent to Greece because they were assured that come what may, the EZ would stand by all it’s debts.

    The real question for you should be why the EZ taxpayer is shouldering the burden of bad debts in the private sector? I thought the markets were meant to function themselves, but that only seems to work when people are making a profit?

    The banks were sloppy with their credit checks, and really don’t deserve much sympathy. The taxpayers of the EZ should be asking their own leaders why such an inept currency system was ever devised.

    “You are dictating terms on how other people retrieve their money from debtors. You may consider them immoral but they are within their rights to demand repayment on previously agreed terms.”

    I agree – they have a perfect right to demand what they want. Creditors can sometimes be completely stupid in their demands, looking for their pound of flesh rather than seeking a path that will give them a much better chance of ultimately being paid. [Remember that Syrzia has repeatedly stated that they wish to repay their current debts, and have not been asking for more write offs].

    The Euro – yes I agree. Greece really needs to get out but can’t. It’s dreadful mess that the orginators of the Euro must take great responsibility for.

    Apart from that, we won’t agree, and I suspect we are taking up Anthony’s patience debating something of little relevance to polls.

  3. @CMJ – 9.12 yesterday evening

    You suggest that an English Labour Party would be putting the horse before the cart. Surely that’s the right order – for putting the cart before the horse would be far less productive!

    As to the situation as it currently stands, it seems to me that the Labour party as a whole is still trying to work through the implications of devolution when the rest of the UK’s political set up has been up and running with the new situation for, well, roughly sixteen years!

  4. @Alec and Martyn

    Whilst I agree entirely with Alec on the Greek situation, I wonder whether this on-going saga, and the mess being made of the situation in Calais, will combine to influence people’s views on the EU. Polling may well be involved here!

    Re: Calais
    Personally I think that people should be given the freedom to go wherever they like, as long as they obey the general laws of the society to which they come. Surely we need enterprising, intelligent, committed people in our economy. Why we’re determined to keep them out of the country is beyond me!

  5. LASZLO

    @”none of the measures would deliver sustainable financial flows .

    In the second half of last year Greece was the fastest growing economy in the EZ. It is now back in recession.
    A primary budget surplus ( before debt interest) was predicted at 2.6&% GDP this year-without additional fiscal measures. A deficit is now in prospect.

    So now Athens needs to find 8 bn euros of budget measures to hit a primary surplus target of 1% this year & 2.5% next year. This compares with 2.5 bn euros the creditors were demanding before Syritza came to power.

    Tsipras has wasted 6 months -and now proposes a retrospective 12% tax on 2014 corporate profits & 3.5% tax on salaries to be paid by employers.

    The Bank of Greece estimates that Tsipras’s proposals will reduce growth by 2 percentage points.
    The counter proposals by lenders are forecast to reduce growth by 1 percentage point.

  6. Labour leadership candidates strategies analysed.

    If liz comes third yvette wins,but if liz comes second andy wins ?Change in strategy for Burnham ?

  7. @John B

    I meant they were putting the cart before the horse!

    I agree they are way behind the curve. They have much catching up to do.

  8. @Colin – the implication of your post sounds like you lay the blame for the return to recession in Greece at Syrzia’s door.

    If so, that’s very unfair. They only came to power on January 25th, so the latest Q1 GDP figures can hardly be blamed on their actions.

    Many would argue [stuff unrelated to public opinion or polling and likely to provoke another tiresome left-right economic argument that has no place here – AW]

  9. Perceptive commentary by Gerry Hassan on LiS

    http://www.compassonline.org.uk/downfall-the-rise-and-fall-of-the-house-of-scottish-labour/

    In a climate of social democracy facing hard times there can be no guarantees of success, but Scotland and even the SNP need a vibrant, confident opposition politics. Politics have changed so much and so quickly in Scotland, that it is by no means certain that Labour can even be sure that it will remain the leading opposition party. Such are the disorientating scale of changes which Labour have to understand and adapt to, if they are to have any kind of real and relevant future.

  10. @Alec

    I read your response before AW started snipping us. In deference to the kind man who did a nice writeup on a meeting I couldn’t attend, I will put my response in a little box, put it over the horizon, and we shall admire the glow from afar.

  11. On topic from the DT:

    h ttp://www.telegraph.co.uk/news/general-election-2015/politics-blog/11695816/Why-did-the-polls-get-it-wrong-at-the-general-election-Because-they-lied.html

    I don’t think anyone will be surprised by the author.

  12. 07052015
    I don’t think it matters what I read, I know the GE results. Your loyalty to the left always leads you to delusion. Quite why you think I know nothing of London I could not say, I was born in Victoria Park Road Hackney. Despite what Prospect might opine, we still have Mr Johnson as Mayor and are most likely going to have Mr Goldsmith replacing him. Bow bells are not going to ring for Labour any time soon me ole cock sparra.

  13. Sky have a poll on the EU. Whether it is possible to have a group of Sky customers where “Data are weighted to the profile of the population.” is actually representative of the electorate is another matter.

    http://news.sky.com/story/1507772/economy-key-to-eu-vote-sky-survey-suggests

  14. Britons will not vote to leave the European Union if they fear it will damage the country’s economy, a Sky survey indicates.

    That’s the strap line, and a statement of the obvious in my view.

    No matter how much the ‘No’ campaign talks about immigration the like, it not enough unless people feel certain jobs won’t be at risk.

    I personally think this is a very high obstacle to overcome.

  15. @RolandG

    “…..and are most likely going to have Mr Goldsmith replacing him.”

    Sorry to spoil your evening but the next mayor is still most likely not to be a Conservative. Goldsmith is the individual favourite at 15/8 but that’s only because he’s a shoe in for the Tory nomination, whereas the odds are pretty evenly shared between the two leading Labour candidates. So in terms of which party is most likely to win, your best odds at the moment are 8/15 Labour v 2/1 Conservative.

  16. @Hannah

    That DT columnist has committed the ultimate sin and traduced pollsters and the polling industry in general. So I look forward to AW being so appalled that he reinstates our ability to discuss him and his opinions, subject to this being always done in highly pejorative terms. Although the latter would happen anyway.

  17. “I read your response before AW started snipping us…”

    ———

    I missed it, and I was just getting into it as well. (Even though it did lack the prerequisite Scots dimension, but prolly would have been worked in at some point).

  18. I have just sent the following submission to the BPC/MRS inquiry.

    1. As an amateur psephologist, I have for some years contributed to the UKPollingReport website. Therefore I would like to submit opinions for consideration by your inquiry.

    2. I attended the meeting on 19th. June at which your inquiry was launched.

    3. I do not have data to submit for your consideration. My comments are conceptual in nature.

    4. My academic background is in psychology, particularly occupational psychology. In this field, specialised techniques are used to ensure that measures such as psychometric selection tests are constructed, so far as possible, impartially.

    5. The opinion polls carried about before the 2015 election did not appear to be constructed using techniques as rigorous as those for psychometric measures. Many of the variables included appeared to have been chosen for historical reasons or to reflect particular clients’ specific interests.

    6. Variables that appeared to be less accurate in 2015 than in the past included socio-economic class and geographical region. At the meeting on 19th. June, it was also suggested that public and private sector workers may have voted differently.

    7. In relation to social class, occupational sector and level of educational achievement may be more relevant than socio-economic class in the traditional sense. These measures may actually be intermediate variables moderating the effect of socio-economic class. Empirical research is required to establish which of these variables now best predict voting behaviour.

    8. Opinion polls have for long differentiated the effects of geographical location on voting behaviour using the standard geographical regions. Clearly there were substantial regional effects on voting behaviour, notably in Scotland. However, it appears that there may now also be large variations between locations within regions, e.g. between city centre, suburban and rural areas.

    9. Empirical work is required to disentangle regional differences in swings from the increasing polarisation of voting behaviour between safe Labour, marginal and safe Conservative seats.

    10. One reaction to increasing differences between constituencies has been the development of “Ashcroft” polls, using larger samples in less frequent surveys of selected seats. This is a welcome development; but work is needed to establish optimal sampling sizes for such polls and the frequency with which they are conducted. The selection of constituencies for such polls needs to be randomized or determined using scientific quotas, rather than reflecting individuals’ interests in particular seats.

    11. Whilst there are increasing geographical differences between constituencies, changes in voting intentions over short time periods appear to have reduced markedly compared to the past. It is doubtful whether very frequent opinion polling, using standard questions to produce time series data, is cost-justifiable on the scale of recent practice.

    12. Whereas in the past voting intentions appeared to change gradually, during the past parliament there were large and sudden changes. One example would be the fall in Liberal Democrat support in 2010 after they reversed their policy on university tuition fees.

    13. Opinion polls have traditionally been constructed using linear assumptions based on least-squares methodology. Such measures are reasonably straight-forward to construct. They are also readily explicable to users with a reasonable level of numerical ability. However, investigation is required to check that the assumptions used for opinion poll construction are justifiable.

    14. Specifically, some of the changes in political opinion during the 2010 – 2015 parliament would appear to have been catastrophic in the mathematical sense, resulting in non-linear change. This presents major challenges for opinion poll construction.

    15. There are similarities in the breakdown of linearity in changes of political opinion with the discontinuities in financial and economic behaviour associated with the banking crisis of 2008.

    16. It appears likely that there will be non-linear changes in intended voting behaviour between now and the 2020 election. Not least, the current state of the political parties is such there is a considerable possibility of major re-organisations. The major challenge to the incumbent Government at the next General Election may come from a political party which does not yet exist. This will cause major challenges for pollsters who largely rely on time series data.

    17. One answer to increasing unpredictability in voting intentions may be to place greater reliance on qualitative methodology rather than current quantitative techniques. However, such qualitative reporting should be based on the development and use of systematic methodology to establish the validity and reliability of such work.

    18. I am inclined to agree that there is a “shy Tory” effect which distorts predictions because of differential turnout and because voters, for whatever reason, do not report their voting intentions accurately. This effect does not appear to result from late changes in voting intention. However, it would appear that this “shy Tory” effect may vary between different elections and may have been particularly large in 2015. I am not sure what, if anything, can be done about this.

    19. Whilst the inquiry is concerned that the 2015 opinion polls got the result wrong, it should be acknowledged that there are inherent limits to the accuracy of questionnaire studies. I suspect that political opinion polls compare favourably with other questionnaires in terms of their reliability and validity and in terms of the proportion of the variance accounted for. The inquiry could investigate this by comparing opinion polls with market research polls more generally.

    20. No opinion poll can be proof against corrupt practices. There is too much concern at present about corruption in British elections, for instance in relation to excessive postal and, even worse, proxy voting and the unverifiable casting of votes from overseas. Politicians appear to be far too relaxed about current voting practices. I hope the inquiry will not shy away from considering such matters where appropriate, and if necessary from commenting in strong terms.

    20. Finally, censorship by legal or other means of political opinion polling, including recent proposals in parliament to restrict polling just before elections, is totally unacceptable on practical grounds as well as matters of principle.

1 2 3 4