What Went Wrong

Today YouGov have put out their diagnosis of what what wrong at the election – the paper is summarised here and the full report, co-authored by Doug Rivers and myself, can be downloaded here. As is almost inevitable with investigations like this there were lots of small issues that couldn’t be entirely ruled out, but our conclusions focus upon two issues: the age balance of the voters in the sample and the level of political interest of people in the sample. The two issues are related – the level of political interest in the people interviewed contributed to the likely voters in the sample being too young. There were also too few over seventies in the sample because YouGov’s top age band was 60+ (meaning there were too many people aged 60-70 and too few aged over 70).

I’m not going to go through the whole report here, but concentrate upon what I think is the main issue – the problems with how politically interested people who respond to polls are and how that impacts on the age of people in samples. In my view it’s the core issue that caused the problems in May, it’s also the issue that is more likely to have impacted on the whole industry (different pollsters already have different age brackets) and the issue that more challenging to solve (adjusting the top age bracket is easily done). It’s also rather more complicated to explain!

People who take part in opinion polls are more interested in politics than the average person. As far as we can tell that applies to online and telephone polls and as response rates have plummeted (the response rate for phone polls is somewhere around 5%) that’s become ever more of an issue. It has not necessarily been regarded as a huge issue though – in polls about the attention people pay to political events we have caveated it, but it has not previously prevented polls being accurate in measuring voting intention.

The reason it had an impact in May is that the effect, the skew towards the politically interested, had a disproportionate effect on different social groups. Young people in particular are tricky to get to take part in polls, and the young people who have taken part in polls have been the politically interested. This, in turn, has skewed the demographic make up of likely voters in polling samples.

If the politically disengaged people within a social group (like an age band, or social class) are missing from a polling sample then the more politically engaged people within that same social group are weighted up to replace them. This disrupts the balance within that group – you have the right number of under twenty-fives, but you have too many politically engaged ones, and not enough with no interest. Where once polls showed a clear turnout gap between young and old, this gap has shrunk… it’s less clear whether it has shrunk in reality.

To give an concrete example from YouGov’s report, people who are under the age of twenty-five make up about 12% of the population, but they are less likely than older people to vote. Looking at the face-to-face BES survey, 12% of the sample would have been made up of under twenty-five, but only 9.1% of those people who actually cast a vote were under twenty-five. Compare this to the YouGov sample – once again, 12% of the sample would have been under twenty-five, but they were more interested in politics, so 10.7% of YouGov respondents who actually cast a vote were under twenty-five.

Political interest had other impacts too – people who paid a lot of interest to politics behaved differently to those who paid little attention. For example, during the last Parliament one of the givens was that former Liberal Democrat voters were splitting heavily in favour of Labour. Breaking down 2010 Liberal Democrat voters by how much attention they pay to politics though shows a fascinating split: 2010 Lib Dem voters who paid a lot of attention to politics were more likely to switch to Labour; people who voted Lib Dem in 2010 but who paid little attention to politics were more likely to split to the Conservatives. If polling samples had people who were too politically engaged, then we’d have too many LD=>Lab people and too few LD=>Con people.

So, how do we put this right? We’ll go into the details of YouGov’s specific changes in due course (they will largely be the inclusion of political interest as a target and updating age, but as ever, we’ll test them top to bottom before actually rolling them out on published surveys). However, I wanted here to talk about the two broad approaches I can see going forward for the wider industry.

Imagine two possible ways of doing a voting intention poll:

  • Approach 1 – You get a representative sample of the whole adult population, weight it to the demographics of the whole adult population, then filter out those people who will actually vote, and ask them who they’ll vote for.
  • Approach 2 – You get a representative sample of the sort of people who are likely to vote, weight it to the demographics of people who are likely to vote, and ask them who they’ll vote for.

Either of these methods would, in theory, work perfectly. The problem is that pollsters haven’t really doing either of them. Lots of people who don’t vote don’t take part in polls either, so actually pollsters end up with a samples of the sort of people who are likely to vote, but then weight them to the demographics of all adults. This means the final samples of voters over-represent groups with low turnouts.

Both methods present real problems. May 2015 illustrated the problems pollsters face in getting the sort of people who don’t vote in their samples. However, approach two faces an equally challenging problem – we don’t know the demographics of the people who are likely to vote. The British exit poll doesn’t ask demographics, so we don’t have that to go on, and even if we base our targets on who voted last time, what if the type of people who vote changes? While British pollsters have always taken the first approach, many US pollsters have taken a route closer to approach two and have on occasion come unstuck on that point – assuming an electorate that is too white, or too old (or vice-versa).

The period following the polling debacle of 1992 was a period of innovation. Lots of polling companies took lots of different approaches and, ultimately, learnt from one another. I hope there will be a similar period now – to follow John McDonnell’s recent fashion of quoting Chairman Mao, we should let a hundred flowers bloom.

From a point of view of an online pollster using a panel, the ideal way forward for us seems to be to tackle samples not having enough “non-political” people. We have a lot of control over who we recruit to samples so can tackle it at source: we record how interested in politics our panellists say they are, and add it to sampling quotas and weights. We’ll also put more attention towards recruiting people with little interest in politics. We should probably look at turnout models too, we mustn’t get lots of people who are unlikely to vote in our samples and then assume they will vote!

For telephone polling there will be different challenges (assuming, of course, that they diagnose similar causes – they may find the causes of their error was something completely different). Telephone polls struggle enough as it is to fill quotas without also trying to target people who are uninterested in politics. Perhaps the solution there may end up being along the second route – recasting quotas and weights to aim at a representative sample of likely voters. While they haven’t explicitly gone down that route, ComRes’s new turnout model seems to me to be in that spirit – using past election results to create a socio-economic model of the sort of people who actually vote, and then weighting their voting intention figures along those lines.

Personally I’m confident we’ve got the cause of the error pinned down, now we have to tackle getting it right.


274 Responses to “What Went Wrong”

1 4 5 6
  1. @Carfrew
    “Well, like I said. You could have many different kinds of attitude and still do ok, as a boomer, with cheap rent, cheaper housing, cheap utilities, free degrees, full employment, job security, lavish pensions, big house price gains, more welfare support etc.”

    You really have got an obsession about the benefits of being a ‘boomer’, and of how hard done by you are. I know some will accuse me of repeating the ‘Four Yorkshiremen’, but here’s a few facts.
    My first house cost £8,700 at a time when I was earning £1600 per year.
    My water bill is cheaper in cash terms than it was 40 years ago.
    Electricity and Gas was more expensive in real terms when they were nationalised.
    Very few people got degrees, me included.
    Job security might have existed in nationalised industries, but not anywhere I worked. There was no such thing as written warnings, you could just be sacked.
    Lavish pensions for a few perhaps. My private and occupational pensions bring in about £400/month after 45 years at work.
    I can’t speak about welfare or rents as I have no experience of either.
    In addition, I had classmates who died of diptheria and TB, and another who was crippled by polio.
    We played on bomb sites, and grew up thinking the world would be obliterated by nuclear war any moment.
    Television, telephones and cars were for the few.
    Class size of 48 in one year.

    So let’s please stop this carping about which generation has had it easier than another. Each generation has had its own advantages and disadvantages.

  2. I’m not necessarily doubting the figures, but the ComRes poll shows unweighted figures of Con 30%, Lab 33%, and initial weighting moves that to Con 32%, Lab 29%. I presume it’s further weighting based on assumed likelihood of voting that gives Conservatives such a healthy lead in the published polls.

  3. According to WIKI , Baby Boomers are people born during the demographic post–World War II baby boom approximately between the years 1946 and 1964, giving an age range between 51 and 69 as of 2015.

    So the ill gotten gains of this group which Carfrew goes on & on about , are just on the point of about being handed on to the Envious Generation which follows it-Bank of Mum & Dad brimming over , Second Homes for free hols, Inflated House Values about to feature in Wills :-)

  4. Pete B
    I wrote something similar to you but then deleted it, so I agree with you. You missed out that uni grants were possible because only 10/15% of kids went to uni. Most left school at 15. Then there was 15% interest rates, 25% inflation with wage rises always lagging. Phone calls were expensive and unless you were lucky were made from a call box at the end of the road. My salary in 1968 was £460 pa which equates to about £8k now. What’s the minimum wage now? Hence the reason why kids nowadays can have nearly new cars and holiday abroad. I had a 50’s A40 held together with empty beer cans, fibre glass and STP stickers and holiday was a week in Bournemouth. Final salary pensions are a big loss but blame successive Chancellors for that.

    I’m not saying that everything is perfect now, of course it isn’t but it never was. Take off your rose tinted specs. Tomorrow is the first day of the rest of your life. Enjoy it.

  5. @Pete B

    I understand why you’d want to close down my posts. You tried that last time the other day. I’m not as censorious, but then I don’t need to be.

    In the end, you have to look at the overall picture, not just cherry-pick. As with the JSA thing a short while back. You were happy with Neil’s idea that it is “possible” to survive on JSA… erm, if the landlord pays the bills.

    But the glaring issue with that is there are not nearly enough landlords willing to do that!!! And you still didn’t have money left over for STs and bulbs etc.

    But to make life easy, in the end, you used to be able to support a family on one income, not so often nowadays. And you shouldn’t discount the benefit of full employment, which makes any job insecurity much easier to deal with. Whereas now peeps dependent on tax credits. The difference is pretty stark.

    And sure, not all boomers benefitted. Eg not all got degrees. But it is peeps who benefitted from free tuition and more besides telling others to change their attitude.

    What were rents like at the time? And hence was it easier to save a deposit too? What proportion of disposable income did gas and leccy take up? The full picture, Pete, is the thing…

  6. (Well ok, maybe not disposable income, but you know what I mean…)

  7. Carfrew
    Until the 80’s and the introduction of 6 month tenancies, no one in their right mind was a landlord except slum landlords like Rachman of the 60’s and Hoogstraaten from the 70’s. There was only council house or slummy student accommodation available.
    I never went to uni. Never wanted to, units were for academics in those days, I was not academic, I was/am, practical. I wanted to enter the world of work and make a career.

    This two people working because life is so expensive, is a misnomer as well. That has happened larger because of the emancipation of women, who don’t want to sit at home all day hoovering and also because people want more ‘stuff’ nowadays. In other words it is free choice.

  8. Carfrew
    I really don’t want to get into a lengthy debate about this, and I can’t remember every detail of every bill I ever had, or what proportion of my income it was.

    What I can say is this, at one point I bought a house when mortgage rates were 10%. My wife didn’t work as we had small children. I had budgetted that we could afford the rate to go up to 12% (they were very volatile in those days – 1978). In the end the rate went up to either 15% or 17%, I can’t quite remember, but it was over 55% of my take-home pay. Our staple fare was turkey-neck stew which my wife bought at the market when it was closing for the night.

    Please just accept that there are are advantages and disadvantages to each generation and as Robert Newark said “I’m not saying that everything is perfect now, of course it isn’t but it never was. Take off your rose tinted specs. Tomorrow is the first day of the rest of your life. Enjoy it.”

  9. @Pete

    If you don’t wanna get into a debate about it, fine. No one is insisting you should!! You don’t have to keep saying it. (But you can if you want to waste your time).

    Anyways, I’ll letcha get some kip and reply in the morning if that’s ok…

    P.s. As for the “obsessed” ad hominem, one cannot escape the significance of boomers in polling terms. It’s not gonna go away…

  10. P.p.s.

    Well, to correct myself again, obviously boomers may go away at some point. But you know what I mean.

    P.p.p.s. Rough that you only earned £1600 in ’78. I mean, I had a holiday job in a factory back then and although not yet an adult got £40 a week, or £2k per Annum pro rata.

  11. Typo. It was £4600.

  12. Candy
    Lol, Trumpers, I can’t see that one catching on however appropriate XD

    Anywho in relation to your main point we (Corbynites) don’t gloss over Corbyn’s supposed skeletons rather we believe the vast majority are either fabricated, exaggerated or taken out of context and the rest are no big deal to begin with.

    You mention the IRA for example, the difference is the IRA (whether you like them or not and for what’s its worth I don’t) are seeking to remove what they see as foreign occupation forces, a claim which is not wholly unsubstantiated given the history of Britain in Ireland. What’s more even if you disregard that, he didn’t invite them in to congratulate them on a well executed terror attack, rather it was an honest attempt at deliberation and negotiation. Its all very well branding someone evil, wicked and the spawn of demons but once you do you throw any kind of peaceful resolution out the window. As the old saying goes you destroy your enemies when you become friends with them.

    As for Mao, Hoxha, Stalin, Hitler, Genghis Khan, Satan and anybody else Corbyn is supposed to be a great admirer off, 99% of those claims are totally unsubstantiated and most of what remains is taken out of context i.e
    Interviewer: Can you name a dictator?
    Corbyn: Enver Hoxha
    Interviewer: What’s your opinion on cupcakes?
    Corbyn: I like them

    Next days mail front page: SHOCK INTERVIEW REVEALS CORBYN “LIKES” HISTORICAL TYRANTS!!!
    Joke but you get my point.

    On the same issue but more seriously its very possible to admire certain aspects of an individual or organisation but dislike them overall. For example I agree with Hitler’s policy to fund research into the health risks of smoking and when possible discourage it amongst the populace. Do I like Hitler though? Of course not.

    Trumpers on the other hand DO gloss over his skeletons and in some instances openly embrace them in all their glory. In fact when it comes to glossing over a politicians skeletons most everyone does it. For example the amount of times on this site alone I’ve witnessed Thatcherite’s gloss over the fact that she supported the Apartheid regime and our current PM visited the country in 89 to campaign to lift sanctions on the country. And before the claims of “they didn’t REALLY support apartheid though” get brought out how about Thatcher’s absolute love for the Pinochet regime in Chile which was responsible for goodness knows how many deaths. Frankly when it comes to skeletons in the closet Corbynites are pretty unique in embracing their leaders history and at least trying to justify it. Just something to think about.

  13. @Robert

    Just wanna say this quickly. I think you have a point about the emancipation thing. But it’s a question of cause and effect. More women want to work, and do so, thus family incomes go up. Then good ole supply and demand kicks in: families have more money, which then allows rents, bills and house prices and other essentials etc. to increase to match the increased income.

    And then you wind up in the situation now where you both have to work, where before it was optional…

  14. Correction: Carfrew (and then I) confused two separate house purchases. No-one’s interested I’m sure, but I just don’t like errors. G’night all.

  15. @Rivers10,

    It seems to me that, like Corbyn and McDonnell, you see murders by Irish Republicans as justified because of Irish history.

    That’s fine. Campaign in 2020 on a “terrorism’s OK if you support the cause” platform. When that fails, maybe you can campaign in 2025 with an armalite.

  16. Neil A

    I read your post in response to that of Rivers10, but I can’t see anything in theirs that would substantiate your response.

    Perhaps you could quote the relevant sections to justify your point?

  17. PeteB

    Your response to Carfrew of 11.18. I could not have put it better myself. I was trying to cheer him up, life is for living and enjoying as much as possible.

  18. Good Morning All from Bournemouth East and AFCB Land.

    PETE B. Turkey-Neck Stew? You were lucky….as that brilliant sketch puts it.

    I think the debate about Jeremy Corbyn in the early hours of this morning on here points to the fact that he and his Shadow Chancellor have said many things which they did not think would be used against them, as they were on the fringes of the old Left, with no prospect of being on any Front Bench in Parliament.
    As with Liam Byrne’s silly joke about the money, the comments by thee two of them, and the platforms they shared will be used against them and the Labour Party to damage the Labour Party deeply.

  19. @ Neil A

    “But yes, in the “I live on a completely different planet to Earth” stakes there’s nothing quite like the American far-right.”

    Tell me about it!

    But you know, I’m starting to realize that the larger problem (and it exists in all political ideologies) is that people often do not question or attempt to filter what they hear, taking whatever they hear as 100% absolute fact.

  20. @ Neil J

    “I agree, I doubt it will even be called for debate in Parliament. Personally think we are better confronting such people than banning them, Trump is revelling in the publicity.”

    He is. There are a couple of theories on why he’s doing this and I’m not sure I agree with or embrace any of them though it’s possible any of of them is right.

    1. He doesn’t believe any of this and is just saying these things to attract the lowest common denominator.

    2. He’s realizing that he might actually become the Presidential nominee and runs the risk of being President, which he actually doesn’t want. So he is intentionally trying to blow the campaign by forcing the GOP to force him out. (Rachel Maddow’s theory).

    3. He’s pushing hard on these things to rally back against the weakness that has been exposed in Iowa.

    “Love the link, brilliant”

    I think it’s how you deal with hate speech.

    She did royally irritate me last year and in 2013 (well actually, it’s one of her friends) but I ultimately voted for her anyway and it’s moments like that I’m glad I did.

  21. I’ve long defended Liam Byrne, on the basis that the joke from the chief secretary was a very long standing tradition which Laws, depending on how you look at it, either wasn’t expecting as a Lib Dem, or decided to exploit for political gain.

    But on May 8th, I was sitting in a fairly grim Wetherspoons in Hodge Hill, opposite some railings still fluttering with “Re-elect Liam Byrne” posters – he got 70% of the vote on a day Labour got hammered nationwide – and couldn’t help thinking “Had to try and be funny, didn’t you Liam?”

  22. New thread

  23. @Mr Nameless if you’re still around.
    At least Laws got his comeuppance

  24. In recent local by-elections when 40,000 people voted, 45% voted Nationalist, 22.5% voted Labour, 20.5% voted Conservative and 9.5% voted Liberal Democrat. Your comments suggest this might be similar to current polls. It would nice to see one some time.

1 4 5 6