What Went Wrong

Today YouGov have put out their diagnosis of what what wrong at the election – the paper is summarised here and the full report, co-authored by Doug Rivers and myself, can be downloaded here. As is almost inevitable with investigations like this there were lots of small issues that couldn’t be entirely ruled out, but our conclusions focus upon two issues: the age balance of the voters in the sample and the level of political interest of people in the sample. The two issues are related – the level of political interest in the people interviewed contributed to the likely voters in the sample being too young. There were also too few over seventies in the sample because YouGov’s top age band was 60+ (meaning there were too many people aged 60-70 and too few aged over 70).

I’m not going to go through the whole report here, but concentrate upon what I think is the main issue – the problems with how politically interested people who respond to polls are and how that impacts on the age of people in samples. In my view it’s the core issue that caused the problems in May, it’s also the issue that is more likely to have impacted on the whole industry (different pollsters already have different age brackets) and the issue that more challenging to solve (adjusting the top age bracket is easily done). It’s also rather more complicated to explain!

People who take part in opinion polls are more interested in politics than the average person. As far as we can tell that applies to online and telephone polls and as response rates have plummeted (the response rate for phone polls is somewhere around 5%) that’s become ever more of an issue. It has not necessarily been regarded as a huge issue though – in polls about the attention people pay to political events we have caveated it, but it has not previously prevented polls being accurate in measuring voting intention.

The reason it had an impact in May is that the effect, the skew towards the politically interested, had a disproportionate effect on different social groups. Young people in particular are tricky to get to take part in polls, and the young people who have taken part in polls have been the politically interested. This, in turn, has skewed the demographic make up of likely voters in polling samples.

If the politically disengaged people within a social group (like an age band, or social class) are missing from a polling sample then the more politically engaged people within that same social group are weighted up to replace them. This disrupts the balance within that group – you have the right number of under twenty-fives, but you have too many politically engaged ones, and not enough with no interest. Where once polls showed a clear turnout gap between young and old, this gap has shrunk… it’s less clear whether it has shrunk in reality.

To give an concrete example from YouGov’s report, people who are under the age of twenty-five make up about 12% of the population, but they are less likely than older people to vote. Looking at the face-to-face BES survey, 12% of the sample would have been made up of under twenty-five, but only 9.1% of those people who actually cast a vote were under twenty-five. Compare this to the YouGov sample – once again, 12% of the sample would have been under twenty-five, but they were more interested in politics, so 10.7% of YouGov respondents who actually cast a vote were under twenty-five.

Political interest had other impacts too – people who paid a lot of interest to politics behaved differently to those who paid little attention. For example, during the last Parliament one of the givens was that former Liberal Democrat voters were splitting heavily in favour of Labour. Breaking down 2010 Liberal Democrat voters by how much attention they pay to politics though shows a fascinating split: 2010 Lib Dem voters who paid a lot of attention to politics were more likely to switch to Labour; people who voted Lib Dem in 2010 but who paid little attention to politics were more likely to split to the Conservatives. If polling samples had people who were too politically engaged, then we’d have too many LD=>Lab people and too few LD=>Con people.

So, how do we put this right? We’ll go into the details of YouGov’s specific changes in due course (they will largely be the inclusion of political interest as a target and updating age, but as ever, we’ll test them top to bottom before actually rolling them out on published surveys). However, I wanted here to talk about the two broad approaches I can see going forward for the wider industry.

Imagine two possible ways of doing a voting intention poll:

  • Approach 1 – You get a representative sample of the whole adult population, weight it to the demographics of the whole adult population, then filter out those people who will actually vote, and ask them who they’ll vote for.
  • Approach 2 – You get a representative sample of the sort of people who are likely to vote, weight it to the demographics of people who are likely to vote, and ask them who they’ll vote for.

Either of these methods would, in theory, work perfectly. The problem is that pollsters haven’t really doing either of them. Lots of people who don’t vote don’t take part in polls either, so actually pollsters end up with a samples of the sort of people who are likely to vote, but then weight them to the demographics of all adults. This means the final samples of voters over-represent groups with low turnouts.

Both methods present real problems. May 2015 illustrated the problems pollsters face in getting the sort of people who don’t vote in their samples. However, approach two faces an equally challenging problem – we don’t know the demographics of the people who are likely to vote. The British exit poll doesn’t ask demographics, so we don’t have that to go on, and even if we base our targets on who voted last time, what if the type of people who vote changes? While British pollsters have always taken the first approach, many US pollsters have taken a route closer to approach two and have on occasion come unstuck on that point – assuming an electorate that is too white, or too old (or vice-versa).

The period following the polling debacle of 1992 was a period of innovation. Lots of polling companies took lots of different approaches and, ultimately, learnt from one another. I hope there will be a similar period now – to follow John McDonnell’s recent fashion of quoting Chairman Mao, we should let a hundred flowers bloom.

From a point of view of an online pollster using a panel, the ideal way forward for us seems to be to tackle samples not having enough “non-political” people. We have a lot of control over who we recruit to samples so can tackle it at source: we record how interested in politics our panellists say they are, and add it to sampling quotas and weights. We’ll also put more attention towards recruiting people with little interest in politics. We should probably look at turnout models too, we mustn’t get lots of people who are unlikely to vote in our samples and then assume they will vote!

For telephone polling there will be different challenges (assuming, of course, that they diagnose similar causes – they may find the causes of their error was something completely different). Telephone polls struggle enough as it is to fill quotas without also trying to target people who are uninterested in politics. Perhaps the solution there may end up being along the second route – recasting quotas and weights to aim at a representative sample of likely voters. While they haven’t explicitly gone down that route, ComRes’s new turnout model seems to me to be in that spirit – using past election results to create a socio-economic model of the sort of people who actually vote, and then weighting their voting intention figures along those lines.

Personally I’m confident we’ve got the cause of the error pinned down, now we have to tackle getting it right.

274 Responses to “What Went Wrong”

1 2 3 6
  1. Nope.

    Late Tory Swing or ‘shy tory’ factor

  2. Interesting stuff – though you won’t be surprised that I ask a question about the Scottish crossbreak! :-)

    Prior to the May election, you were experimenting with weighting the Scottish sample by some Scottish demographics, rather than just part of the GB demographic.

    Is that continuing?

  3. I think you’re right and you’re wrong. The analysis of past mistakes is probably quite near the mark but that does not mean that the correct result is attainable.

    Those who are not interested in politics fall into a number of categories but can be simplified into those who will never be interested and those who will be interested if they are directly affected. This latter group are subject specific and thus difficult to pin down. In fact it is worse than that because once you move beyond their interest in a specific topic they revert to what may be a completely counter intuitive position on everything else. Life was so much simpler before TV.

  4. Oldnat – all in good time :)

  5. Anthony

    Is one allowed to say “Bastard!” to one’s host? :-)

  6. “There were also too few over seventies in the sample”

    I know that is correct from the perspective of a pollster wanting to get a better method for predicting an election right.

    But this methodological obsession frames the problem the wrong way. Surely your results identify there were too few younger people who contributed to the ballot? Perhaps we should be focussing most of our effort on how it is the election result ends up not reflecting the views of society properly?

    I ask these questions whilst my 15 year old lad is asking me to explain why he will not be allowed to vote in the European referendum, whilst all those that will not have to deal with the long term consequences of the referendum will be able to do so.

  7. Approach 2 seems much more preferable to me.

    With Approach 1 – you get people not that interested in politics and send them a voting intention poll. What then happens? If they complete the poll, do you then include them in the sample at full weight? Surely you shouldn’t – they may well not go to the polling station on the day.

    Following Approach 1 feels like over- engineering the problem – ie you include everyone and then downweight.

    Whereas Approach 2 takes you straight to what you are looking for – if X% of people who voted in GE 2015 are under 25 then when you do a poll you want X% of the poll sample to be under 25.

    OK, you could say turnout could change. But how can you tell? You can’t trust people who say they will vote to actually do so – people say they will do all kinds of things – remember the poll on attendance at church over Christmas.

    Past history must surely be the best guide. If you don’t want to over-rely on one GE then use the last two GEs and weight it 2/3 GE 2015 and 1/3 GE 2010.

  8. “we don’t know the demographics of the people who are likely to vote”

    calculating turnout by grouping council wards by common factors might give a rough ratio if not an exact number

  9. “Breaking down 2010 Liberal Democrat voters by how much attention they pay to politics though shows a fascinating split: 2010 Lib Dem voters who paid a lot of attention to politics were more likely to switch to Labour; people who voted Lib Dem in 2010 but who paid little attention to politics were more likely to split to the Conservatives.”


    This “not paying attention to politics” thing. In the case of LibDems, it benefits Tories. How does it work with other elements?

  10. AW
    Very interesting analysis and exposition of the difficulties of accurate polling. One thought that occurs is that exit polls should record some basic demographic information if at all possible. This would not help immediately, but if it was done in 2020, one would hope that polls for 2025 would benefit.

    David Hodd
    “Perhaps we should be focussing most of our effort on how it is the election result ends up not reflecting the views of society properly?”

    The main way that it didn’t was that UKIP were vastly under-represented in MPs compared to their popular vote. One solution would be to reserve some seats to be allocated according to popular vote, and that would avoid pure PR.

    “I ask these questions whilst my 15 year old lad is asking me to explain why he will not be allowed to vote in the European referendum, whilst all those that will not have to deal with the long term consequences of the referendum will be able to do so.”

    Surely any parent or grandparent will take into account the likely future for their descendants, and will have more experience and wisdom than a 15-year-old. As a young man at the last referendum, the only reason I could think of to stay in was because of the promised cheap beer and fags which never happened. Nowadays I would take a broader view. :-)

  11. “Surely any parent or grandparent will take into account the likely future for their descendants, and will have more experience and wisdom than a 15-year-old.”


    Lol, have you not met any boomers?

  12. Carfrew
    If a boomer means someone born in the first few years after the war, then I’m one myself. The only ones I’ve met who aren’t too concerned about the future are those with no kids. But I can see how growing up with the fear of imminent nuclear destruction at any moment might make some of us think that the risk of leaving the EU is pretty trivial.

  13. @Pete B

    Lol, it’s ok, I can tell you’re a boomer Pete!! If you want to see the boomers as a wondrously self-sacrificing sort, as opposed to the generation that benefitted from the sacrifices of their ancestors while going on to pull up the ladder behind them on everything from tuition fees to pensions and much else besides then I’m not the kind of guy to go overboard challenging that. Even if it also gives the lie to that idea about the wisdom of the elders.

    I was 15 myself when I realised that the wisdom of the elders was not necessarily something one can rely on. Specifically some of my Oxbridge-educated teachers. (Mind you, some of them were on my side and saw my point…)

    But don’t take my word forrit. Check out what Max Planck had to say about how science progresses. (Ironically he was older when he said it…)

  14. (Well I assume he was older, I haven’t checked…)

  15. Carfrew
    I can’t be a**ed arguing over details of government policy and exactly which generation caused the bits you don’t like, but I’m certainly prepared to concede that the post-war welfare state (created by those born in Victorian times by and large) was over-generous in both state benefits and things such as pensions in nationalised industries such as teaching, even though my own mother still benefits from it.

    I also agree that Oxbridge-educated teachers will not necessarily know much about real life, especially if they went straight into teaching.

    I’m not interested in what Max Planck has to say about progress in science because it’s not relevant.

  16. Carfrew

    As you may be aware, I have said often that my generation did unreasonably well out of the “system”.

    At the same time, we are greatly outnumbered by the younger generations, so why have you lot been so incompetent as to –

    1. swallow the propaganda that “we” are all deprived, poverty stricken victims (that was our parents’ generation)


    2. Let us screw you rotten?

  17. @Pete B

    I know you can’t be areed debating it too. Me neither!! What’s the point? There’s so much material on my side of the argument.

    I must clarify however: It’s not simply a case of being over-generous in terms of the welfare state as you put it. It’s lots of things like full employment, for eggers, which kept wages higher too, and not boosting house prices into the stratosphere, or letting utility companies ramp up prices on necessities (while using the revenue to hoover up more of the market etc. etc. etc.)

    And it’s not historical, as you suggest by using the past tense, but current, as we see with pensions and winter fuel payments etc.

    This is without getting into the fact it may not be “generosity” at all, but with multiplier effects these things may leave the state better off. Which is why despite an economy hammered after the war and a big debt to pay off we had two decades of growth with such policies before the oil crisis hammered us (after the neolibs sold off the oil buffers).

    The Oxbridge teacher thing wasn’t an issue about real life, but about the teaching itself. (Though thinking about it, not so sure it was the Oxbridge ones who were necessarily the greater issue…)

    What Max Planck had to say is relevant, because it doesn’t just apply to science, or I wouldn’t have mentioned it.

  18. @oldnat

    Well I’m kinda on the borderline in terms of being a boomer. By some measures not, by other measures I might be.

    A good deal of the change of course happened in the Eighties, when the younger couldn’t vote or hadn’t even been born yet. Effects still working through now decades later eg deregulated banking and the Crunch. Many of these changes in stacking the deck thus leave youngsters too preoccupied with just getting by to really take the fight to now well-ensconced and RETIRED boomers who with modern medicine and nice pensions have plenty of time to feather their nest further.

    Even among the more elite now, they have to worry about paying for an internship and stuff. The boomers secured more power, is the point. But beyond that, the left have a tendency to split their vote. And… Other things… it’s easier to pull provision apart than it is to set it up… Interesting question really…

  19. @oldnat

    There is also what happened to the media in the eighties. If you have to dig and delve elsewhere for proper info. while even as a graduate struggling to get by working sixty hours a week on miminum wage, well… It’s asking quite a lot.

    But a lot of youngsters are nonetheless politically engaged… But those in middle age need to line up alongside. If the media successfully hypes immigration as the issue for example instead of other government policy, then UKIP get votes instead and the boomer party continues…

    …is another reason why…

  20. Carfrew

    While I jest about it, I think the social consequences of what has developed [1] are very serious.

    A relatively small group of people benefitting from inherited wealth (and the much more important wealth in “social capital” that grandparents can pay for) can always be outvoted by the majority to create a fairer system. [2]

    When the majority (at least of those that vote) have inherited wealth (or the prospect of it) in the form of housing bought by the previous generation, then the system changes radically.

    Little of the increased value of housing has been “earned” by the owner in the form of home improvements. Most of it comes from increased land values – often enhanced by public spending like HS1.

    If the basis of wealth (land) isn’t taxed, then it falls on income and spending – the areas in which less wealthy sections of society are most penalised.

    Can this be fixed? Yes. Will it be? No.

    So like many of my contemporaries, I’ll carry on investing in social capital for my grandkids, so that they will be the beneficiaries in an inegalitarian society.

    [1] Of course, my lot planned nothing – we just happened to be in the right place at the right time, and benefitted from the inability of the “system” to adjust fast enough to changed circumstances – and then the cynical use by politicians of voting demographics. We are just as incompetent as every other generation.

    [2] I’ve been re-reading Agatha Christie recently, and her references to how the “Upper Middle Class” (if I’ve got that English term right) were devastated by inheritance tax in the post-war years are interesting.

  21. @ Pete B

    Voting age- The logic of who will be most effected by a vote on the EU would suggest that a one year old should vote as he or she will be effected for even longer.
    Perhaps voting should relate to paying income tax, which would prevent the unemployed and prisoners from voting and anyone who is still in education unless like my grandson you are working part time to fund University costs.
    When I first purchased a house having accumulated a deposit serving in the Army for three years and saving a deposit only 10% of the population were able to afford to buy a house. My parents bought their first house a few years before me and I needed them to guarantee my mortgage.
    I helped my children in the same way to buy their homes.
    Having been self employed and spent 40 years working often 7 days a week and having only a couple of holidays, until retiring, any capital we have accumulated will be shared equally between our six grandchildren.

  22. Thanks for the explanation of the “What went wrong” report. I look forward to reading the next one on December 2020.

  23. Glad you found it Anthony-looking forward to more accurate polls in future.

    The irony of failing to consult the old is not lost on a site which featured TOH’s minority opinion :-)

  24. The way we are going with life expectancy, TOH is barely middle aged.

  25. Colin

    The most interesting thing to me is that so many who said I was deluded at best or just plain daft no longer post. I particularly remember one person who was really scathing about my forecast the day before the election and has never posted since.


    Your as old as you feel and I must say generally I feel better now (at 75) than I did at 60 despite my ongoing health problems.


    Loved your piece about the baby boomers, much of it correct,except your comments about care for the next generations. It seems to me from watching other families that the modern generation is much less caring about the future generations than my generation. However I agree this is not the place to debate that.


    Thanks for another interesting piece. I think it probable that which ever approach you will choose the forecasts will improve. However personally I will continue to base my own forecasts on the detailed polling information on questions about the economy, leadership, national security .

  26. TOH

    Lets hope the wisdom of old age now pervades the whole system!!

    Must say that I envy those who are mired in grief at the wealth they have taken from their descendants.

    In our household its the other way around-though an entirely voluntary arrangement I hasten to add.:-)

  27. COLIN

    Indeed, same here.

  28. I would like to thank Anthony for all the work and for sharing the analysis.

    It seems highly plausible with a very good chance of being correct. I hope it will lead to practical changes.

    I am looking forward to seeing the acid test in 2020.

    How many of the current political leaders will be in post by then I wonder?

    Thank you Anthony.

  29. Pollsters, who spend their days sifting through data and putting it to work had not clocked that people who respond to polls are more engaged? It’s not exactly a revelation.

  30. http://www.realclearpolitics.com/epolls/latest_polls/president/

    Just in case we thought it was only the UK which had polling issues, what is going on in the USA?

    The Iowa Republican polling is to say the least strange! Trump either 13 points ahead or 5 behind!

  31. This seems a logical analysis, with fascinating findings.

    I think the second approach will be better, but pollsters may need to change methods more frequently to cope with changing social habits. For instance, answering phone calls may be considerably affected if cold-calling can be stopped or much reduced.

  32. American polling is a mire, as I recall. There are good polls in there but you have to hang around for a while to be able to pick them out. Sample sizes seem to be smaller in general too, which doesn’t do wonders for the margin of error.

  33. The race to select the Republican Presidential Candidate is fascinating but terrifying.

    If Donald Trump wins the Presidential Election, I’ll volunteer quite happily to catch the next one way space flight off this rock.

  34. Another thought on Donald Trump.

    Given the type of action he is calling for (including other recent announcements), then surely the Home Secretary should prevent himn from entering the UK, as he represents a threat toour national security?

    Other such out-spoken people have been barred in the past.


    I mentioned this the other day. Read “The Plot Against America” by Philip Roth.

    Chilling parallel with a fictional Lindbergh presidency in 1940.

  36. “Breaking down 2010 Liberal Democrat voters by how much attention they pay to politics though shows a fascinating split: 2010 Lib Dem voters who paid a lot of attention to politics were more likely to switch to Labour; people who voted Lib Dem in 2010 but who paid little attention to politics were more likely to split to the Conservatives.”


    I think this is largely correct, but there is also one missing piece of the jigsaw.

    I speak as someone who is very interestd in politics – and have always voteed, not just in general elections but all elections since I turned 18 (now in my 30s).

    I first discovered Populus, YouGov when I was really hard up – I mean back-against-the-wall hard up.

    I signed up for loads of polling sites just to try and make ends meet.

    There are loads of long, boring consumer panel polls where you get asked loads of ridiculous questions (‘if this toothpaste was a person….”)…that take ages to fill out and the reward is a pittance – if you don’t get screened out after 10 minutes.|

    I don’t do these anymore.

    I still do Populus/Yougov (although haven’t had anything from YouGov for months). I LIKE telling people what I think – and also vote in, for example, Daily Mirror polls (for which I’m not paid).

    In my experience, Populus in particular pay the best of any survey site – political or otherwise – on the web.

    I’m sure there are many like me that found te polling sites searching “how to make cash online”.

    The point I’m trying to make is that while it’s correct that the pollsters over-represent the politically active/politically engaged, I suspect that they even more so over-represent those tat are politically active/politically engaged and hard up.

  37. @MARK

    I think you may have cracked it.

  38. The question is:

    Why was the polls in Scotland so accurate but the UK polls so inaccurate?

    What was different in the Scottish sampling compared to the UK sample.

    The theory of ‘too many politically engaged’ and ‘too few over 70s’ would surely apply to Scotland as well?

    Unless this difference can be explained then I’m not sure the theory holds water.

  39. “The theory of ‘too many politically engaged’ …..would surely apply to Scotland as well?”

    Just a theory, but, from what I have heard from my Scottish friends – and political pundits – is that the indy referendum got a lot of people / a lot more people politically engaged.

    If that is true, then the pool of politically engaged people would be a lot larger than the rest of the UK, thus minimising the amount that politically engaged people were over-represented.

  40. @Mark

    And what this explanation is implying is that ‘Conservative’ is the default voting position in England. So the ‘not so politically engaged’ vote Conservative, Am I reading this right?.

    In Scotland that would not be the case and the ‘not politically engaged’ vote the same way as the rest of the population??

    Similarity over 70s in Scotland vote the same way as 60-70s

    It is definitely possible but it does highlight how different Scotland is from England in political make up.

  41. Couper @ 2.45 pm

    I don`t think you can generalise on one trend displayed by a cohort of voters 2010 to 2015.

    Those seduced by Nick Clegg`s charms into voting LibDem in 2010, and then by the Tory claims in 2015 that a Labour vote would mean the SNP controlling the UK government, will not permanently vote Tory.

    They might decide it is David Cameron`s fault if we suffer an ISIS massacre, or that he has caused the Cumbrian floods.

    Fickle unthinking ignorant voters are a problem for all of us, politicians, pollsters, decent people.

  42. Couper2802

    “Why was the polls in Scotland so accurate but the UK polls so inaccurate?”

    Fewer elderly Con voters will be one part of it.

    The other part of the story is the wwc in England stopped voting around 2005 but it’s been disguised by very high differential turnout among some of the Lab supporting minority groups.

    Scotland doesn’t have that – or maybe it did for a long time but then they finally decided to switch.

    If so and it was possible to measure the number of years between them stopping voting Lab and switching to SNP that would be an interesting number.

  43. Why don’t polling companies use the marked register to find out more about which people vote.

    Although the turnout is different for local and general elections, you can get separate marked registers for each.

    New people coming onto the register by the time of the next election would be excluded but should not change the overall picture since fewer younger people actually vote, especially not in local elections.

  44. @Carfrew

    You need to be careful about the whole “boomer” thing. In the United States, they are the biggest generation, but in the UK, it’s Generation X that are the dominant generation. Here’s the demographic profile from the 2011 census.


    Gen X is that huge bulge aged between 40 and 50 in 2011.

    Because they didn’t start their families till they reached their 30’s, their children are all school age. So they care about schools and the NHS, but arn’t interested in university fees or the price of houses for young adults – they won’t start worrying about that for another decade. As things stand, they are happy about expensive houses, because they bought at the start of this century and have made some money from them.

    P.S. Gen X moving through the population are causing all sorts of other effects.

    For example, pubs and clubs tend to blame the anti-smoking laws for their demise. The likely real reason is that around about 2001, Gen X hit it’s thirties and started having babies and staying indoors. The generation following them is much smaller and couldn’t fill their shoes in terms of consumption. And the generation after that is even smaller. If the pub industry had looked at the census in 2001, they’d have spotted the looming problem – unfortunately they tend not to be data driven, so just assumed that the next generation would be just as big despite all evidence to the contrary.

    When Gen X finishes raising their children, we’ll probably see a recession in the toy industry, baby clothes etc. And when they hit retirement, downward pressure on house prices as they cash in. They’ve been the big driver in house prices – as this huge generation hit their 30’s in 2001, house prices started to rise as they set up families and also invested in BTL – that whole process will go into reverse because succeeding generations are smaller. The whole housing thing will resolve itself within a couple of decades.

  45. Mr Jones

    Overall turnout figures don’t tell us much about who stopped/started voting, but the Scottish turnout in UK GEs (always over 70% last century) was
    2001 – 58.2% (Lab share of the vote – 43.3%)
    2005 – 60.6% (Lab share of the vote – 39.5%)
    2010 – 63.8% (Lab share of the vote – 42.0%)
    2015 – 71.1% (Lab share of the vote – 24.3%)

    Don’t know if that helps.

  46. if that list was interpreted as Lab voters stopping voting for years (and only a dribble switching) for a long time before they switch en masse

    (and maybe that’s not it at all)

    then maybe c. 14 years is the magic number

  47. So long story short, the Conservatives won thanks to low-information voters.

  48. @Chris is “not politically engaged” the same thing as “low-information”?

    As I watch the increasingly lurid “Don’t Bomb the Little Children” memes swilling around Facebook, I wonder if sometimes people can get very, very engaged based on a very small amount of information that isn’t particularly factual.

    I’m pretty sure someone who reads a decent paper a couple of times a week would have more information than the majority of the population and be able to make an informed choice at elections, despite not having their blood boiling over with anger.

1 2 3 6