UKPR Polling Average

Very soon I am going to launch a weighted average of the polls, the UKPR polling average. I’ve thought long and hard about this because generally speaking I don’t like polling averages. There is no statistical justification for a polling average – the different companies do slightly different things, they weight differently, ask different questions and include people who are more or less likely to vote and more or less certain for whom. Averaging them together isn’t the equivalent of one big poll with a smaller margin of error, it’s just mishmash of different methodologies. Neither does an average get you the better results – the true picture isn’t normally the average of the polls, in fact, when compared to elections the poll that’s worst for Labour tends to be the best.

So, with all that in mind why am I doing it? Two reasons: the first is that there is demand for it, and if I don’t provide it other people will, and will do it less well. My firm belief is that the best way to follow the polls is to look at individual pollsters, see what their trends are and understand the differences in approach that result in the differences between them. I hope that regular readers here will always judge polls in this manner. The reality is that, especially as we approach the next election, is that many people (and newspapers!) can’t be bothered to do that, and just want one nice figure that shows them what “the polls” show. Since someone will provide it, I thought I’d better do it properly.

A second reason is that it allows me to put up a running projection of what the current polls would translate into on a uniform swing. At the end of the day, the one question a lot of people want to know the answer too when they look for polls is “if there was an election now, what would the result be?” To do that, we need a figure representing what “the polls” show.

In the past there have been a couple of different approaches to polling averages. The first is a straight rolling average of the last five polls – what the graph on my voting intention page shows. That is vulnerable, however, to being skewed by having a lot of polls from a particular pollster in it. Secondly you can take the average of the latest poll from each company. That has the downside of what you do with companies who poll irregularly, or what happens if they miss a poll. You also miss out on good quality data from pollsters who produce lots of polls, while include aging stuff from pollsters who produce data less regularly (as Nate Silver, who produces the US averages on www.fivethirtyeight.com said when he justified including data from more than one poll from the same pollster in his averages “getting SurveyUSA’s sloppy seconds may be as good as getting virgin results from a lot of pollsters”). Another quite common approach is to weight data according to sample size, something I particularly dislike since in the past the polls with the largest sample sizes have certainly not been the most accurate (ICM, for example, who have one of the most enviable records, normally have the smallest sample size).

What the UKPR Polling average will do is weight the polls that go in according to how recent they are and the track record of the polling company. They will also factor in some methodological and transparency issues and whether there are other polls of the same company in the average (as a compromise to stop mulitiple polls from the same company having too much effect). I’m never going to like polling averages, but given they are going to exist, I can at least provide the very best one I can.


42 Responses to “UKPR Polling Average”

  1. Anthony: many thanks. This may go some way to answering the question i asked the other day abour WMA that is often supplied by NBeale. It’d be interesting to see if your methodology is differnet from his. It would also be an interesting counter to many of the more emotional contributors to this blog who jump up and down every time a poll (should I say POLL) does/doesn’t fit their political bias

  2. NigelJ – actually I sort of hope it doesn’t on the latter point! My recommendation would still be that people judge the polls as a group, looking at the trends within each pollsters’ data. I certainly wouldn’t support people countering that by pointing at the “average”.

    It’s only ever going to be a shorthand for the polls are saying, it will never be as good as looking at the bigger picture :)

  3. Anthony,

    just a few suggestions.

    It might be good if on the UKPR PA page the latest polls was illustrate with the variation from the average. that would give a quick indication of whether we were seeing a real change.

    You could also archive these so that we could if we wished see how each poll over time varied from the average at that time.

    I would of course like to see a regional map but as different polsters have different regions then I think although indicative it would probably be misleading.

    As to presentation I think the line graph works well but how about a pie chart which would show not just the current position but the nmaximum and minimums for the main partys.

    That might let people quickly see not just where the average says we are but also what things like the Labour minimum are from just one picture.

    if you were (as you seem to suggest) tieing it in to seat projections I’d also like to see a stacked bar graph of seats. so you could see either Regional; the seats for each party in each region or Party; each parties seats divided by region.

    again it’s about giving the general public and the media the quickest easy to understand picture of what the average says the impact would be.

    Peter.

  4. I won’t be doing fancy things with the average – as you’ll have guessed from the article, I regard it only as a nice shorthand for, as you say, “giving the general public and the media an easy to understand picture”.

    I like the stacked bar chart idea – I don’t have the skills to do a dynamic map that colours seats according to a projection, but a stacked bar chart would be quite nice. It won’t be there to start with – the new pages are all mostly done and even after that, the first thing in the “to do in the future” queue for the projection is treating Scotland seperately (thought you’d like that), but i’ll squirrell the idea away for the future.

  5. Nice work Anthony. I look forward to seeing it.

    Hopefully it’ll encourage some people on here to discuss polling results rather than debate their own political opinions.

    Incidentally Anthony, maybe you can give a simplistic explanation as to why the Tories need to finish 5% ahead of Labour ‘to win’. I recently struggled to explain it when somebody asked me.

  6. Another thing I’m going to put back on the site is a link to this article explaining it

    http://ukpollingreport.co.uk/guide/electoral-bias

  7. Interesting idea, and I’m sure I’ll soon be an avid reader of its updates.

    I’m curious though about the uniform swing projection; why do you think this will be useful? In particular, I think the Liberal Democrat figures are more likely to mislead rather than to clarify?

  8. Have found some web based graph widgets here http://www.ibm.com/developerworks/xml/library/x-ajaxxml1/index.html

  9. Nice idea!

    Already there are several helpful(?) suggestions, so please forgive another addition to the wish list.

    I’d prefer a line graph with the individual monthly? points indicated, and with error bars shown at each point that’s plotted.

    For the reasons you explain, it will never be possible to construct a really “scientific” average – but the error bars, part of any genuine piece of statistical interpretation, would perhaps be a built-in warning not to get over-excited by small apparent variations that are often well within the “experimental uncertainty”.

  10. The short version is that the uniform swing projection will be there becuase it is the “accepted version” that everyone is used to. It won’t involve any human judgement or error – just a flat formula.

    However, I’m also putting up a second projection, which I’ll add lots of bells and whistles to in order to try and push the envelope and get what I actually think is the best projection. At the moment that’s going to have an incumbency factor in seats with newly elected MPs or MPs standing down and some regional variation based on the big marginal poll PoliticsHome produced earlier this year (obviously it’s a big assumption to make that the pattern has remained the same, hence doing this in tandem with a straight uniform swing). The next thing beyond that will be seperate treatment of Scotland, but it will always be a work in progress.

  11. JohnH – I’m not planning on graphing the average at all at the moment (and I doubt the software could handle error bars anyway). I think if I think add them it would add spurious statistical legitimacy to something that really is a media shorthand.

  12. …And it’s up there in the corner on the front page :) (anyone wondering, the seat guide part of the site will change to match later on, I just need to sort out the registration stuff over there)

  13. Thanks. I think there’s a risk in feeding an assumption that just because a number exists, it is meaningful – as with uniform swing calculations – but it’s good to know you’ll also be doing a different calculation.

  14. Now it’s up there you can see that the biggest difference is makes is that projects the Lib Dems would have 6 more seats in a election tomorrow than on a straight uniform swing.

  15. Wow that was quick!!!!

  16. The projection on the top right, is that based on the last poll or the new average? Seems rather low for the Conservatives.

  17. Gary, it’s based on the new average of the polls and a Tory lead of 9 points or so really does only scrape an overall majority (and with a low Lib Dem score. A higher Lib Dem score would give a hung parliament)

  18. May I comment how pleasing to the eye the new layout is. I am very impressed.Well done.

  19. I knew the Conservatives needed a big lead, given the way party supporters are distributed around our island, but I always assumed that if the Conservatives could get over 40% then that uneven spread of party support would be cancelled out. Interesting, interesting. Site looks great by the way. The additional information is much appreciated. I can understand why you were not keen to add a running average but for followers of polls and therefore the maths associated with them I think it will be a welcome addition to discuss (argue about).

  20. Maybe you’ll achieve the ultimate, and have so many different statistics that everyone will at last be able to find something that fits their prejudices… erm… considered opinions. (Well, once you’ve got separate results for Scotland sorted)! :)

  21. Well done son – the UK needs its own RCP Average, and the UKPR Average will do just nicely.

    You’re right to be skeptical of polling averages of course – methodologically speaking that is. But then, the RCP averages got it right, state by state and nationwide – so perhaps by taking the average of well conceived polls we really do distill some common sense…

    N

  22. Will you also be weighting by demographics as Nate does on 538?

  23. Thank you for the explanation Anthony. I’m surprised to see a 9 point lead would only give a 10 seat win even after the boundary changes.

    Your point about tactical voting is interesting. I guess an end to anti_tory tactical voting and a rise in the SNP vote could boost the majority even on a 9% ‘win’.

  24. A quick calculation shows that the average conservative percentage points is just under 43 (based on th elast 20 polls) – pretty close to the most recent figures, and possibly within margin of error. Could this suggest that this is pretty solid, and that the real movement is between lab, libdem and “others”. Perhaps the SNP/lab movements showing an effect, or lab voters returning from the LibDem at the prospect of a Tory govt? Meanwhile the defectors from Lib Dem to Tory have stayed there.

  25. Al – I wish.

    My understanding is that Nate didn’t weight them by demographics: rather, alongside the polling figure for each state he also produced a projection based on the demographics of each state. State A has x% African Americans, y% blue collar workers, therefore it should be voting like this. His figures for each state then took into account both the demographic projection and the polling figures.

    To do that he needed seperate polls for lots of states so he could do regression analysis based on comparing support in each state with the demographics of each state.

    We don’t have loads of regional polls in the UK, so the raw material to do the sort of thing I think Nate did is sadly absent.

  26. ….Sorry forgot to say that the average Con percentage points over the last 20 polls is also very close to Anthony’s current WMA – which was the point I was trying to make, in order for it to be relevent to this thread!

  27. Was never very good at maths, and weighted averages give me a headach, so just to check I understand it just looking at the Conservative figures the calculation is (41 * 0.9) + (43 * 0.69) + (45 * 0.14) + (42 * 0.46) + (39 * 0.27) divided by (0.9 + 0.69 + 0.14 + 0.46 + 0.27) which is then rounded to nearest whole value.

  28. If it’s working as I meant it to, yes :)

  29. It looks like it is working to me. And I must admit I quite like it, looking at the latest poll is very good but I also like to llok at past polls to spot any trends. I tend to look at polls by the same polling company and then at all polls, not in great detail but just to get an quick impression as to where the polls seem to be heading. Would it be possible to have a list of past “UKPR Polling Averages” or is that taking things a bit too far?

  30. Very nice looking site, and an impressive bit of work. I’d like to ask your professional opinion about how Electoral Calculus do their projections. They’re fairly open about the methodology. In the past, I’ve tended to use them as a guide to the overall position, though they’re weak on the NATs and ‘Others’.

  31. I tend to view the average as another poll with no more accuracy or importance than any other: it gains some accuracy from what can be reasonably assumed to be a larger sample size but loses it due to the potentially conflicting metholodogies involved (possibly making it less accurate overall, who knows)

    but I do like the estimated GE election result and the new look.

  32. Anthony – Why include BPIX which is not a member of the British Polling Council and never discloses anything about how it produces it figures? You then discriminate against MORI even though it is transparent but you don’t happen to like its methodology.

    These two elements completely discredit what you are trying to do.

    I think this whole concept is flawed and you of all people should have stood out against it.

  33. The subjective element undermines my confidence in the weighted averages.

    If there was a sound statistical basis for assessing which is the “best” method, it might be acceptable, but I assume there is not. If the adjustments for “reliability” were dispensed with, how different would the result be? Put another way, is it worth introducing the subjective element, which will attract criticism, for the difference it makes?

    I posted a similar message earlier, but it seems to have gone astray.

    Ernie

  34. Not at all sure about this.

    I have been pretty convinced by the arguments against which you outline in your first para Anthony.

    But you have been honest about your reasons for doing it & your last two sentences seem fair.

    Provided we all realise that it is the product of entirely subjective & selective weightings. ( maybe all weightings are subjective?)

    The risk you run is the one you identify-those who crave a “sound byte” simple set of numbers will use it as a “truth” which you know it is not.

  35. Ernie – to be honest, the age of the poll and the penalty against BPIX for us not being able to judge their methodology properly are the heaviest factors. The impact of pollsters’ past records is comparatively small simply because at recent elections they ALL have good records.

  36. The problem with projecting average party ratings into how big an overall majority (or lack thereof) is going to be, is that the swings are likely to be larger in marginal seats – therefore a division of 42:33:16 may not lead to a Tory majority of just 10 – it would almost certainly be rather more and such a projection, however well-intentioned, could be misleading.
    It’s interesting though!

  37. Andy – that’s the point of producing the second version of the projection, which has varied swings in different regions in the pattern we saw in the big PoliticsHome poll of marginal seats.

    At the moment I don’t have any evidence for different swings in safe seats since by definition that poll only included marginals, but it’s a work in progress.

  38. I think the point about BPIX is important as they produce results which are consistently biased in one direction, so whatever weighting is used to minimise distortions it won’t be eliminated.

    As non-members of the BPC they do not pass the test of transparency, so their distortions cannot be placed in context and to include them at all has the consequence of prejudicing wider understanding.

    Perhaps a rolling 12-month and 3-month average of individual pollsters’ polls could be flagged up against their current poll to give an indication of current trend from each different polling sources, which would balance out the question of problematic timings and weightings for an average – the constant refrain is after all that it is not just the gaps that count, but also the overall totals.

  39. On the question of differential swings in marginal seats, doesn’t this depend on one’s definition of marginal?

    Perhaps it would be informative to have a glossary of accepted definitions by this site, so as to avoid confusion. If marginals are statistically defined then the difference can be calculated.

  40. Anthony,

    Out of interest which version ( or both) stacked graph do you think would be the more usefull,

    Party Stacks; all the same colour by with regional dividing lines so we could see the scottish Labour seats in the Labour stack, or

    Regional Stacks; A stack for every region with the scottish stck showing the deifferent seats by party colour.

    The regional one would be most useful for me and certainly the clearer where as the SNP stack on a party one would fairly obviously only have one region in it.

    Would you include NI as a stack even if the polls didn’t cover it?

    Peter.

  41. “the first thing in the “to do in the future” queue for the projection is treating Scotland seperately ”

    Yes please do.

    We aren’t in much doubt who the next PM will be, but the situation in Scotland is very fluid.

    Labour are on the way down but the LibDems are doing even worse for no obvious reason and the Conservatives are dying off from old age. The Socialists are spending more time with their lawyers and the Greens’ progress has been set back by “Alex Salmond for First Minister” which won’t happen again.

    Are the SNP on the way up or is it just that they have to get the disaffected Labour votes because nobody else does? Is it enough to make any real difference in FPTP? Have they peaked already or was Glasgow East an outlier? Even if they became the most popular government for 50 years would it make any difference to the level of support for independence?

    All these questions are where the polls can give us some indication of the way things are going, and that’s what they are for isn’t it?

  42. Anthony,

    Any chance of a seperate Scottish polling average, or are the polls too infrequent for it to be valid?

    Peter.