What Went Wrong

Today YouGov have put out their diagnosis of what what wrong at the election – the paper is summarised here and the full report, co-authored by Doug Rivers and myself, can be downloaded here. As is almost inevitable with investigations like this there were lots of small issues that couldn’t be entirely ruled out, but our conclusions focus upon two issues: the age balance of the voters in the sample and the level of political interest of people in the sample. The two issues are related – the level of political interest in the people interviewed contributed to the likely voters in the sample being too young. There were also too few over seventies in the sample because YouGov’s top age band was 60+ (meaning there were too many people aged 60-70 and too few aged over 70).

I’m not going to go through the whole report here, but concentrate upon what I think is the main issue – the problems with how politically interested people who respond to polls are and how that impacts on the age of people in samples. In my view it’s the core issue that caused the problems in May, it’s also the issue that is more likely to have impacted on the whole industry (different pollsters already have different age brackets) and the issue that more challenging to solve (adjusting the top age bracket is easily done). It’s also rather more complicated to explain!

People who take part in opinion polls are more interested in politics than the average person. As far as we can tell that applies to online and telephone polls and as response rates have plummeted (the response rate for phone polls is somewhere around 5%) that’s become ever more of an issue. It has not necessarily been regarded as a huge issue though – in polls about the attention people pay to political events we have caveated it, but it has not previously prevented polls being accurate in measuring voting intention.

The reason it had an impact in May is that the effect, the skew towards the politically interested, had a disproportionate effect on different social groups. Young people in particular are tricky to get to take part in polls, and the young people who have taken part in polls have been the politically interested. This, in turn, has skewed the demographic make up of likely voters in polling samples.

If the politically disengaged people within a social group (like an age band, or social class) are missing from a polling sample then the more politically engaged people within that same social group are weighted up to replace them. This disrupts the balance within that group – you have the right number of under twenty-fives, but you have too many politically engaged ones, and not enough with no interest. Where once polls showed a clear turnout gap between young and old, this gap has shrunk… it’s less clear whether it has shrunk in reality.

To give an concrete example from YouGov’s report, people who are under the age of twenty-five make up about 12% of the population, but they are less likely than older people to vote. Looking at the face-to-face BES survey, 12% of the sample would have been made up of under twenty-five, but only 9.1% of those people who actually cast a vote were under twenty-five. Compare this to the YouGov sample – once again, 12% of the sample would have been under twenty-five, but they were more interested in politics, so 10.7% of YouGov respondents who actually cast a vote were under twenty-five.

Political interest had other impacts too – people who paid a lot of interest to politics behaved differently to those who paid little attention. For example, during the last Parliament one of the givens was that former Liberal Democrat voters were splitting heavily in favour of Labour. Breaking down 2010 Liberal Democrat voters by how much attention they pay to politics though shows a fascinating split: 2010 Lib Dem voters who paid a lot of attention to politics were more likely to switch to Labour; people who voted Lib Dem in 2010 but who paid little attention to politics were more likely to split to the Conservatives. If polling samples had people who were too politically engaged, then we’d have too many LD=>Lab people and too few LD=>Con people.

So, how do we put this right? We’ll go into the details of YouGov’s specific changes in due course (they will largely be the inclusion of political interest as a target and updating age, but as ever, we’ll test them top to bottom before actually rolling them out on published surveys). However, I wanted here to talk about the two broad approaches I can see going forward for the wider industry.

Imagine two possible ways of doing a voting intention poll:

  • Approach 1 – You get a representative sample of the whole adult population, weight it to the demographics of the whole adult population, then filter out those people who will actually vote, and ask them who they’ll vote for.
  • Approach 2 – You get a representative sample of the sort of people who are likely to vote, weight it to the demographics of people who are likely to vote, and ask them who they’ll vote for.

Either of these methods would, in theory, work perfectly. The problem is that pollsters haven’t really doing either of them. Lots of people who don’t vote don’t take part in polls either, so actually pollsters end up with a samples of the sort of people who are likely to vote, but then weight them to the demographics of all adults. This means the final samples of voters over-represent groups with low turnouts.

Both methods present real problems. May 2015 illustrated the problems pollsters face in getting the sort of people who don’t vote in their samples. However, approach two faces an equally challenging problem – we don’t know the demographics of the people who are likely to vote. The British exit poll doesn’t ask demographics, so we don’t have that to go on, and even if we base our targets on who voted last time, what if the type of people who vote changes? While British pollsters have always taken the first approach, many US pollsters have taken a route closer to approach two and have on occasion come unstuck on that point – assuming an electorate that is too white, or too old (or vice-versa).

The period following the polling debacle of 1992 was a period of innovation. Lots of polling companies took lots of different approaches and, ultimately, learnt from one another. I hope there will be a similar period now – to follow John McDonnell’s recent fashion of quoting Chairman Mao, we should let a hundred flowers bloom.

From a point of view of an online pollster using a panel, the ideal way forward for us seems to be to tackle samples not having enough “non-political” people. We have a lot of control over who we recruit to samples so can tackle it at source: we record how interested in politics our panellists say they are, and add it to sampling quotas and weights. We’ll also put more attention towards recruiting people with little interest in politics. We should probably look at turnout models too, we mustn’t get lots of people who are unlikely to vote in our samples and then assume they will vote!

For telephone polling there will be different challenges (assuming, of course, that they diagnose similar causes – they may find the causes of their error was something completely different). Telephone polls struggle enough as it is to fill quotas without also trying to target people who are uninterested in politics. Perhaps the solution there may end up being along the second route – recasting quotas and weights to aim at a representative sample of likely voters. While they haven’t explicitly gone down that route, ComRes’s new turnout model seems to me to be in that spirit – using past election results to create a socio-economic model of the sort of people who actually vote, and then weighting their voting intention figures along those lines.

Personally I’m confident we’ve got the cause of the error pinned down, now we have to tackle getting it right.


274 Responses to “What Went Wrong”

1 3 4 5 6
  1. Mark W

    “To compare the two and conflate the difference into some frothing anti BBC story is reckless with facts and daft. ”

    Not really, one petition has had a lot of news hype, the other very little, which to my mind makes it more significant.

    Having said that I think both petitions are wrongly motivated and should be ignored.

  2. Mark W
    “…frothing anti BBC story…”

    I didn’t mention the BBC, and ToH only said that it was odd that they missed the story. Hardly ‘frothing’. I can do frothing if you want, but Anthony might ban me.

  3. @AU.

    Hey, if the Chinese will put up the capital and give us a 10% mates rates discount on the construction costs, they can call it anything they want….

  4. To go back to the issue of the thread (having been absent for a few days) can I address the issue of how much impact Individual Electoral Registration had on the collective polling failure at GE 2015.

    Superficially, the finding from AW’s study is that electoral registration had no net impact – that is, people who said they would vote Conservative but turned out not to have been registered were just as prevalent as those who said they would vote Labour and turned out not to be registered.

    However, I would dispute whether any such conclusion on the impact of the changes to electoral registration can be drawn from studies of YouGov panelists (not that AW was actually seeking to draw such a conclusion, should be said). The fact is that the YouGov panelists selected for the follow up study were all, by definition, people who at some point had registered online to join the YouGov panel. They have, therefore, demonstrated their proficiency at being able to register individually online for something. So if we are searching for the sort of people who displayed a lack of proficiency at being able to register invidually online for anything, including registering individually to vote, YouGov panelists really are the very sort of people that we should NOT be looking at to test the question. Note also that online registration was the overwhelming means by which people did register for GE2015, where they were not already automatically matched.

    Nor has the study explained all of the YouGov polling failure. What accounts for about one-third of the unexpected Conservative lead remains unexplained. So the jury is still out. Hopefully BES follow up sampling tested against the marked register may yet shed some light on the impact of IER, in so far as it will (I hope) not be confined to samples requiring a-priori online registration.

  5. Re IER

    Might it not be the case that it has had the effect of reducing the quantity of fraudulent registrations?

    I don’t have any evidence for this, but it seems just as plausible as that there are vast numbers of potential voters who have not registered.

    Perhaps a bit of both.

    I suppose it’s nigh on impossible to prove either way, as we don’t even know how many people there are in the country.

  6. Survation – “New Polling Indicates Lack of Public Support for Proposed Privatisation of Land Registry, Green Investment Bank, National Air Traffic Services and legacy Student Loan Book”

    http://us1.campaign-archive1.com/?u=e17762efe2cccb1f0ed943c1f&id=6854837ccf

  7. At the moment the Trump petition is at 540,000 and still rising rapidly. By any stretch of reasoning this is a very high number which was achieved very quickly and I think worthy of news comment, especially at it may concern the next POTUS.

  8. Another petition was to ‘Accept more asylum seekers and increase support for refugee migrants in the UK.’. That achieved 447,804 signatures and also had little attention from the BBC.
    Some stories are more newsworthy than others, bias has little to do with it, rather what is likely to grab the public’s interest. Trump makes the headlines, which is what he wants

  9. OLD

    Don’t worry they will still go ahead.

  10. OLDNAT

    Apologies for incorrect name.

  11. Discussion on this thread seems to have more or less fizzled out, so maybe I can raise a topic that’s caught my attention.

    I was wondering about the implications of a Brexit for RoI/NI. The possible reintroduction of a hard border has been mentioned, but I wonder if a different political settlement might be prompted.

    Looking at the demographics of NI on wiki, I see that up to age 40, catholics now outnumber protestants/other Christian. That’s presumably down to birthrate plus a little help from Polish immigration (and maybe differential
    emigration rates). On current trends, catholics will overtake protestants some time in the next 20 years.

    That much is well known. But a Brexit would face NI with an existential decision, especially if a majority voted to remain in the EU (as polls suggest).

    So it would be very interesting to seeing polling asking two questions:

    1. If there was a referendum tomorrow on whether to remain in the UK or join the RoI, would you vote to remain in the Uk. Yes/No.

    2. If the result of the EU referendum was that the UK voted to leave, but NI voted to remain, how would you vote in a further referendum on whether to remain in the UK and leave the UK, or join the RoI and remain in the EU.

    I haven’t been able to find any polling along these lines. Has anyone seen any?

    It does seem to me that this could become a major issue.

  12. @ Neil J

    Interesting. Don’t see a point to it though if it’s successful, I wonder what this will do to his holdings in Scotland (probably make the Scots happy).

    This is how you deal with folks like Trump.

    http://sclarita.com/2015/12/10/la-county-supervisor-sheila-kuehl-calls-irate-speaker-an-asshole-and-its-awesome/

    That’s my County Supervisor! (She beat a Kennedy)

  13. SOCALLIBERAL, I agree, I doubt it will even be called for debate in Parliament. Personally think we are better confronting such people than banning them, Trump is revelling in the publicity.
    Love the link, brilliant

  14. Here’s an interesting focus group in the US, regarding Donald Trump:

    http://www.msnbc.com/rachel-maddow-show/trump-focus-group-ive-never-seen-anything

    It’s fascinating in a watching-a-car-crash sort of way.

  15. Irony is, that even if ToH is right about the equality thing – that it’s only pulling apart at the extremes and others are growing closer – this doesn’t necessarily give us any cheer.

    For example, since we seem to be going down the road of progressively eradicating the middle class, along with the over competition among the elite as seen in China, then this would fit Howard’s account. The elite growing further apart from the rest, and the rest closer together, since they’re all being brought down from the middle.

    But that’s not exactly ideal, a few in the elite way ahead and the rest all much closer together in their serfdom…

  16. @Somerjohn

    NI won’t leave for the same reason Scotland won’t – money.

    @Catmanjeff – “It’s fascinating in a watching-a-car-crash sort of way.”

    He’s the Corbyn of the United States. The more people criticise him the more his supporters think this is proof that he’s on the right track. And no amount of wild stuff (holocaust-denying friends, Hoxha, Mao on Corbyn’s part, anti- muslim anti-mexican stuff on Trump’s and both in thrall to the murderous IRA) will make them budge because both sets of supporters share admiration for their guy daring to say aloud what they think. They each point to one or two things (anti-war stuff for Corbyn, progressive taxation on Trump’s) as though that exonerates the rest.

    And each set of supporters look at the other phenomenon with consternation, not realising that the population looks at them in that way!

  17. CARFREW

    You do seem down. I think you need to change your mindset. Think of a way to become one of the elite and then go for it with all the brains and energy, you obviously have.

    I mean this advice in the best way. It’s the same advice I gave my children and now my grandchildren.

  18. CANDY
    Hello to you, from a windy Bournemouth.
    I agree with your post at 12.07PM.

    It is like hard line theology; people who dissent are regarded as heretics to be disdained, and the believers really think that they can persuade the unconverted to follow their creed, not understanding, in Labour’ case, that ‘working people’ are not ‘left wing’.

  19. @ToH

    You’re a bit late to the party with that advice Howard!! The problem is that with no middle ground, more peeps are forced to try and compete to be in the elite. Lacking sufficient elite roles, the resulting over competition amongst the elite makes even being in the elite an overly precarious, stressful, potentially illness-inducing place to be…

    We’re seeing it already, the way peeps are forced to take internships, to pay for a chance to be in the elite. It’s further along in China, where the ante keeps being upped. Initially it was just a few hours extra tuition, now parents have to trade their homes to get their kids a degree abroad, but when they return, it still gives no guarantee of a decent job…

  20. CARFREW

    That’s only true if you continue to take a negative view of life. My advice worked for me, worked for my son, and looks like working for those of my grandchildren who follow it..

  21. TOH

    After crawling around in the cellar, insulating my daughter’s scullery floor yesterday, and putting up a huge Xmas tree today – your first version of my moniker seems distinctly more accurate. :-(

  22. CARFREW

    @” a few in the elite way ahead and the rest all much closer together in their serfdom…”

    Not sure that your analysis is shared by professionals in the field:-

    http://www.bbc.co.uk/news/uk-22007058

  23. @Colin

    Lol, that analysis doesn’t challenge my point. It does summat different, provides a more nuanced breakdown of the class structure, and how since deindustrialiation traditional working class jobs have gone etc., more people in services etc., rather than being about inequality.

  24. @ToH

    Lol, over-competition amongst the elite doesn’t suddenly disappear with a positive views on life!! Chinese parents won’t suddenly stop having to risk their homes on adopting a sunnier disposition.

    But I’m ok with my lot. It isn’t all about how I personally feel about things, or whether my family are ok… That’s a rather narrow, self-centred outlook…

  25. @ToH

    And I’m sure your approach worked for you. Many approaches worked for boomers, they had so much lined up for them. And their offspring could bemefit somewhat too. But how well it works in the future is summat else…

  26. Candy
    Speaking as a Corbyn supporter comparing us and Trump’s backers is a bit of a stretch. Now obviously I know you’re not comparing us on the policy front (I think we can all agree Corbyn and Trump are polar opposites if such a thing exists) but there are some pretty massive differences in most other areas one being that you’ll find Trump draws support from the dame demographics that UKIP does over here while the line about Corbyn only appealing to metropolitan liberals and students is probably not totally unjustified.

    But also you’ll probably find that Corbyn supporters (however much you disagree with them) are reasonably well informed about the problems facing us (climate change, massive inequality etc) and also aware of what the media like to CLAIM are the problems (immigrants, benefit scroungers etc) Trump supporters though from what I’ve seen (and heard from relatives in Wyoming who happen to be staunch Republicans but hate Trump) tend to be more of the radical tea party variety. Deeply religious, fiercely patriotic, very conservative and generally not very well educated.

    The only real similarity is their both backers of a unorthodox “anti establishment” figure. If you want a US comparison it has to be with supporters of Bernie Sanders.

  27. From some of the ill-informed nonsense I’ve seen shared on social media about the coalition air campaign in Syria, I wouldn’t necessarily say that all Corbyn supporters are well-informed. They appear to believe that the USA barrel bombs civilians, that UK pilots deliberately set out to kill children, etc, etc.

    But yes, in the “I live on a completely different planet to Earth” stakes there’s nothing quite like the American far-right.

  28. I’m pretty sure this much smugness is going against the spirit of the recent climate change talks…

  29. * smogness.

  30. I think those representatives of the Syrian Opposition who demonstrated outside the Stop The War bash are the “informed” ones.-and their message to Corbyn & his friends was clear.

    I thought the interview with Peter Tatchell was interesting too-he was demonstrating with them. He said STW were right on Iraq-but on Syria they have “lost the plot”.

    Of course that depends on the nature of their plot. If “Don’t Bomb Syria” applies to Assad as well as everyone else-that is at least a balanced position in support of a ceasefire.

    But somehow they manage to avoid mention of Assad & Russia.

  31. CARFREW

    “But I’m ok with my lot. It isn’t all about how I personally feel about things, or whether my family are ok… That’s a rather narrow, self-centred outlook…”

    I agree with you, which is why I always try to spread the word. Getting on in life is very dependent on attitude. With the right attitude it is often possible to move mountains.

  32. Colin,

    Next you’re going to expect CND to protest outside the Russian embassy. (The radical splitters from CND actually did this in the 1960s.)

  33. @Rivers10
    “The only real similarity is their both backers of a unorthodox “anti establishment” figure. If you want a US comparison it has to be with supporters of Bernie Sanders.”

    Precisely.

    One other significant problem Trump has (as pointed out by Karl Rove) is that while among Republican voters he has a high floor (around 22%), he also has a low ceiling (around 35%). So he isn’t even that popular overall among Republican voters. Indeed, 27% said they would vote Trump come what may, whereas 26% said they would vote against Trump come what may.

  34. There’s a VI poll at 7.30pm by the way for the Sunday Mirror and IoS. The last one had the Tories +15.

  35. CON: 40% (-2)
    LAB: 29% (+2)
    UKIP: 16% (+1)
    LDEM: 7% (-)
    GRN: 3% (-)

    #RedTide

  36. Comres/IoS/Sunday Mirror

    Con 40 (-2)
    Lab 29 (+2)
    LD 7 (-)
    Ukip 16 (+1)

    Took these numbers from Mike Smithson. He hasn’t disclosed those of any other party.

  37. Neil
    Careful of what you read online, many online commenters (particularly on social media or certain forums) are either trolls or over zealous keyboard warriors (that applies to all political persuasions) What I mean was the “average” supporter who in the case of Corbyn is probably like me in being opposed to airstrikes in Syria for various reasons but none of them being cos they feel it would just be a smokescreen to randomly bomb civilians (although collateral damage is real and one of the reasons many were opposed)

    Trump supporters however just have their heads in the clouds over various issues. The banning Muslims thing is a great example, such a policy would literally solve no problem anywhere and it could be very easily proven, yet these people still lap it up.

  38. RAF
    That doesn’t surprise me, as I said my American relatives are staunch Republicans but they hate Trump, they’ll almost certainly still vote for him though if nominated since the two leading Democrats they hate even more. They think Hillary is an unprincipled quack and Bernie is a commie. Suffice to say they’re not looking forward to 2016.

  39. @TOH
    @Carfrew

    I don’t know whether this has any relevance to your discussion, but I remember reading somewhere ( please don’t ask me for the source ) that the median American male earns the same today in real terms as he did in 1976.

    Note ‘median’ not ‘average’.

    The rich in America have got very rich, and women are doing better, but the middle American male is no better off then he was forty years ago.

    I doubt that is true of this country, but I have no idea. I just thought it was a stunning statistic, which is why I remembered it.

  40. @ToH

    Well, like I said. You could have many different kinds of attitude and still do ok, as a boomer, with cheap rent, cheaper housing, cheap utilities, free degrees, full employment, job security, lavish pensions, big house price gains, more welfare support etc.

  41. @Millie

    Yeah, seems to be much the same here too. Wages flatlined in real terms, but cost of essentials has rocketed, less job security etc.

    I’d look for some data, it I’m busy watching a friend at a gig, with a glass of Merlot…

  42. Millie

    Can’t remember the exact figure, but your recall may also be of mine – a stunning analysis by a frmale American prof.

    IIRC it was based on four factors.

    1. the end of post-war era thinking, during which companies provided the benefits (health insurance, pensions etc) that the State often provided elsewhere

    2. a real terms reduction in employee wages, which resulted in both adults in a family having to work – there was no “reserve” earning capacity in the family.

    4. the geographic re-distribution of people into comutersville, where, with virtually no public transport, 2 cars were the minimum need, hence increased expenditure involved in work.

    The net result being that the average family, instead of being able to save from their regular income, were constantly juggling to match income/expenditure. A comparatively small crisis, could result in financial disaster.

    3.

  43. @Carfrew

    The study consisted of 610 managers who evaluated video recordings of an interview between actors playing a managers and a prospective hire. All the scripts were the same but the actors ordered either a Coke or a Merlot. Regardless of the choice of beverage for the manager, when the job seeker ordered Merlot he was perceived as less worthy of being hired and less “intelligent, scholarly and intellectual.” – See more at: http://blog.legalsolutions.thomsonreuters.com/top-legal-news/summer-associates-beware-of-the-merlot

    I merely report the facts – I don’t comment upon them…

  44. Carfrew

    The golden age you think that we baby boomers lived through, didn’t really exist you know. Other than the problem with the income to house price ratio being massively worse now, than then, virtually everything is better now. And the housing crisis is caused by there being a lot more people in the country now.
    Count your blessings as my mum would say and get on with life. You only have one.

  45. Robert Newark

    “get on with life. You only have one.”

    Jeez! How dismissive of rencarnationists can you get? :-)

  46. @Robert

    Lol, spoken like a true boomer!!

    It did happen, because I benefitted quite a bit myself. Eg no tuition fees for me at Oxford. Cheap housing, cheap bills etc. etc.

    Missed out on stuff like full employment, and I didn’t bother with the privatisation shares…

  47. And why the assumption I’m not getting on with life? To me, finding ways to improve things IS getting on with life. As opposed to just sticking one’s head in the sand and assuming anything positive is all down to one’s own wondrousness…

  48. @Rivers10

    I’m sure Trump’s supporters would be as horrified to be compared to Corbynistas as you are to be compared to Trumpers (or whatever they call themselves).

    But there are very strong similarities.

    For example I notice you are neatly gliding over Corbyn’s penchant for Mao, Hoxha, the IRA and assorted weirdos. The exact same way that Trump’s supporters gloss over his peccadillos.

    In terms of economics, Trump is the most moderate of the Republicans – shame he is such a bigot. And Corbyn might be “against war” – shame he spoiled it by ardently courting every violent terrorist group he meets, isn’t it?

    What Corbynistas and the Trumpers both overlook is that the negatives outweigh the positives. It’s the hypocrisies that grate.

    There is a reason that people believe Trumps stance on muslims has nothing to do with terrorism – he funded the IRA who are terrorists. There is a reason that people accepted a peace argument from Charlie Kennedy but can’t accept one from Corbyn. Corbyn actually invited the IRA to visit Parliament two weeks after the Brighton bombing. This is not the behaviour of a man of peace – good people don’t rub salt in the wounds of the recently bereaved or behave in such an insensitive way. Can you imagine the French opposition inviting ISIS to the French assembly after the Paris attacks? Yet both attacks were of similar magnitude because they both tried to take out an elected leader. (It’s a complete cop-out to say the IRA killings were ok because they had a political agenda, so do ISIS, they too are trying to form a state. It’s also a cop-out to say the IRA killings are acceptable because they are not brown).

    Both groups fervently believe that no-one will notice their heroes negatives and that they don’t matter. Both groups refuse to acknowledge that these negatives are building a natural majority against them.

  49. “Jeez! How dismissive of rencarnationists can you get? :-)”

    ———-

    In a parallel universe right now, another version is playing out. Robert and ToH are worried about the ladder pulling, you’re a unionist, and AW doesn’t mod food-related posts…

    And everyone understands margins of error, and regression to the mean, and CUSUM, and the headlines never misrepresent polls, and every peeps are all into cricket and Thorium and are very concerned about the storage situation…

  50. @ Millie

    Thanks muchly for the link. Shall read it tomoz once Merlot has worn off…

1 3 4 5 6