I’ve mentioned in various comments over the last few weeks that YouGov were still looking at their post-election methodology and what changes to make. It is a rather more complex task than one might think (especially since all of us in the team took breaks for holidays at various points during May or June to recover from the general election campaign). Anyway, we have finally rolled out the new methodology, and the pre-budget figures yesterday were the first using the updated method. On the whole, the changes are minor and build upon what we were doing anyway. It is mainly updating our weighting figures, and upgrading them where we can – there are no big departures from what went before.

Anyway, below is a copy of paste of my article on the YouGov website explaining it in full:

The polls at the general election were something of a mixed bag. For the first time in the quarter of a century the polls did not overestimate Labour support. YouGov’s own final poll got the Conservative lead over Labour and, therefore, the swing from Labour to Conservative exactly correct. However, at the same time all the pollsters, including YouGov, overestimated the level of Liberal Democrat support.

I am sure there are many debates to come about why the polls overestimated the Liberal Democrats. What we can be relatively sure about it is that it wasn’t due to individual problems with pollsters’ methods – every company got the Lib Dems wrong, whether they polled by phone, online or face to face, and regardless of the political weighting they did or did not use. This was a systemic error. It has been suggested that it is due to a late swing (including on the day abstentions), or young people who hadn’t voted before not actually turning out, or a disproportionate likelihood for polls to be answered by Liberal Democrat supporters or people interested in politics who had seen the debate. Some of these explanations – such as late swing – are not necessarily things that pollsters can do much about.

On election day itself and the days that followed YouGov carried out a massive survey of our panel, contacting just shy of 100,000 panellists. We used this data to analyse exactly what happened and what we could do to make our samples and weightings more accurate, and for the last month we have been testing updated samples and weights on the back of it.

After considering many possible changes, the result is that YouGov will be making modest updates to our methods, rather than any drastic change. Given we actually got the swing from Labour to the Conservatives right, and that the reasons for the over-estimation of the Lib Dem vote may well have been a late swing or a temporary reaction to “Cleggmania”, we needed to be very careful not to over-react. We wouldn’t want to artificially weight down the Lib Dems and end up underestimating their support come the next election.

While there are no big changes though, we are making various tweaks and updates to our methods to bring things up to date. In most cases these are just things that need updating regularly anyway, or where the growth of our panel or the availability of more data on them has allowed us to do things more accurately.

First, we are introducing more advanced social class weighting. Up until now YouGov have only weighted social class by ABC1 and C2DE – essentially a middle class vs working class divide. We have now got more detailed social grade classifications for a large proportion of our panel, so we can switch to weighting AB, C1, C2, DE separately. For those not versed in social classifications, these roughly equate to professionals and managers, clerical and office workers, skilled manual workers, and unskilled manual or reliant upon benefits.

Secondly, we have changed our age weightings slightly. This one isn’t really a result of the election, but because older people are now far more likely to be on the internet. Our top age band used to be over 55s, on the basis that over 55s were one of the hardest groups to recruit to the panel. These days there are more older people on the internet, more older people on our panel, and we have the opportunity to weight them more accurately – so our oldest group is now the over 60s.

Thirdly, our old party ID weightings were based upon the party ID we found at the time of the 2005 election, adjusted to take account of panellists joining YouGov since then. Obviously the election gives us the opportunity to take a new fixed point of reference for party ID, so we have collected party ID from most of our panel afresh, and taken a new snapshot of 2010 party ID to weight to. Over time some people have changed party ID as the Conservative party became more popular, but nevertheless our target weights are not actually that different from the figures we used to weight to. We have updated our newspaper readership targets in the same way, mainly to take account of a growing number of people who no longer read a newspaper.

Moving forward, we are weighting party ID to Conservative 28.5%, Labour 32.5%, Lib Dem 12%, Others 3% and none or don’t know 24%. Note that the proportion of people identifying with the Labour party is still actually higher than the Conservatives, the Conservative lead at the election was due to Labour identifiers voting for other parties or staying at home, and unaligned voters backing the Conservatives.

We are continuing to look at whether to separately weight Labour identifiers who were “loyal” or “disloyal”. At present we will be controlling it in our sampling, rather than weighting as we did during the election campaign. We will keep it under review, in case we need to begin weighting by it again in the future.

Finally, we have made a slight adjustment to our sampling – the only change directly aimed at addressing the Lib Dem over-representation in the election polls. In analysing our elections polls we found that amongst some age groups respondents tended be too educated, with too many having degrees and not enough with few or no qualifications. In future we will specifically sample people in those age groups with low levels of educational achievement to make sure respondents are not too “graduate heavy”.

Taken together, the effect of these changes reduce the recorded level of Liberal Democrat support slightly and increases Conservative and Labour support slightly, but not to any great extent. In test surveys we have asked respondents how they voted in the 2010 election, and the new weighting tends to produce answers within 1% of the actual election result. That isn’t necessarily a guarantee of accuracy, since we know that some of those answers will affected by false recall and we’re not quite sure how that will affect the Lib Dems, but we are confident that the new weighting scheme accurately represents the British public.


121 Responses to “YouGov methodology review”

1 2 3
  1. barney crockett

    “Private rents in NE Scotland are ahead of HB rates at present because of competition from east European migrants.”

    I’d be interested to see the source for your cause and effect description.

    Report comment

  2. Old Nat
    Mass of personal experience
    What do you want to know?
    Before I was a councillor I worked on housing case work, organised interpreting on housing issues, carried out research, helped people get beds and furniture etc
    I now sit on the licensing board dealing with HMOs which is harrowing stuff

    Report comment

  3. After the Budget I’m still struck by the general naivety of this government, and indeed all recent governments, towards cuts. The Coalition are giving every sign of believing their own propaganda – they think they can cut the deficit with a nominal 80:20 percent split of cuts to tax.

    Those in the Westminster bubble (who, to be fair, probably use public services less than the rest of us) have come to regard cutting government spending as a form of virility. With promises to ring-fence large areas of spending, the percentages required in the areas they can cut are enormous: 25 -30%

    Because few of the bubblites have any experience of working in large organisations, public or private, they have the old fashioned top down approach to cuts. You set an arbitrary target and then rely on the “overpaid” managers you are busy denouncing to do the cuts for you. Front line services will be what gets disproportionately cut, not least because so much is already committed to long term spending.

    There are a lot of efficiencies to be made in government spending, but going through the usual cutting rituals is not the way to do it. Most public service workers are full of suggestions for savings in their own areas, but they won’t get listened to in this sort of slash and burn. As Neil A points out, for this level of cuts you have to eliminate whole areas of public service.

    Furthermore all three Parties believe in the models of “saving money” that have been dominant for the last three decades. So contracting out, privatisation, PFI and so on will be given yet another whirl on the merry-go-round. But all the easy gains from these are long gone and so many are in any case illusory.

    A classic example is the DLA system that Jay Blanc explained above. Not only did it cause endless misery and worry – it actually cost more money than the “inefficient” system full of “false claimants” it replaced. No doubt it will now cost much more to put the system back to the way it was. Similar examples can be found in a whole range of fields. And yet Governments go on; reinventing the square wheel.

    Report comment

  4. barney crockett

    “Mass of personal experience” is seldom a substitute for research evidence.

    You are blaming immigrants (from a specific part of the world) for the higher rents in the NE. What specific evidence do you have for that statement?

    Report comment

  5. Actually the naivety of the Coalition also extends to the way they are bringing them in. Not only are they announcing enormous cuts before they know they are even possible, it’s clear that part of the rationale of going in for cuts so heavily now is to “get them out of the way”.

    They hope they’ll be forgotten in five years time, but because they’re so large they’re having to stagger a lot of the welfare cuts. So now the final shock arrives just before the next election. What’s more people tend to remember cuts – the physical reminders are often all around them.

    Ironically it’s tax rises and indeed reductions that the public forget quickly ; they adjust and move on. Who now recalls the 2% standard rate drop that came with the 10% band abolition? For one glorious moment at the end of his speech I though Osborne would reverse it – no such luck, or wisdom.

    Contrast that with that wise left-winger Margaret Thatcher, who went for 50:50 cuts to tax rises and a top income tax rate of 60%.

    If the Coalition were clever, now is the time to start to raise taxes – the VAT increase was inevitable, but nothing like enough. The UK has been under-taxed for many years (especially wealth and the wealthy) as many on this site have shown. Osborne is still too hamstrung with ideology to increase the taxes he should, but when the impossibility of the cuts strikes home, he’ll have no alternative.

    [Last two comments based on a mash-up and update of two I posted before the Budget. I felt like doing some cutting (and pasting) myself. ;) Apologies if they were familiar]

    Report comment

  6. old nat
    Why do you say blame? I don’t blame these good people at all. It is a fact of life.
    You have pointed out that there is a trend for Scotland’s economy to be more concentrated in the east and the deficit figures you quoted earlier lean very greatly on the burgeoning private sector in the north-east centring on oil. This attracts workers both rich and poor. The large majority of the poor arrivals are from eastern Europe, predominantly Poland but increasingly Latvia, Lithuania and smaller numbers from Hungary and Slovakia. They do not in all but the most unusual conditions qualify for housing benefit and mostly rely on private landlords. Rentals are high, above HB levels. These conditions prevail across a wide area here and I am confident similar situatiions exist in England.

    Report comment

  7. I’d sooner see some contrition on the part of YOU GOV for deliberately misrepresenting Liberal policies in their surveys…

    Report comment

  8. barney crockett

    Tut Tut! I didn’t accuse you of blaming the East European immigrants as individuals.

    I simply asked you for any evidence that they were specifically responsible for the high rental costs in the NE.

    I note that you have now retrenched from that position and given a much more accurate statement – that high rental costs in the NE are the product of strong economic activity. Had you thought about it before you posted your original statement, you would have recognised that housing costs in Aberdeen have always been high.

    So we can now agree that your original statement “Private rents in NE Scotland are ahead of HB rates at present because of competition from east European migrants” was wildly inaccurate and an incomplete explanation of the situation.

    Some (not me, of course) might have thought that your original statement was an example of yet another Labour politician blaming East European immigrants for difficulties. I’m glad that you have now corrected that impression, and that you are not one of those who would support language like “British Jobs for British workers”.

    Report comment

  9. Old nat
    I haven’t re-trenched at all. You chose to read something in my comment that was never there

    Report comment

  10. “Even excluding the effect of wider spending and benefit cuts, the squeeze on the worst-off tenth of the population will, according to the Institute for Fiscal Studies, be five times the impact on the richest by 2015.”

    True blue Tories & Orange Liberals pleased.
    Caring Conservatives concerned.
    Left leaning Dems discombobulated.
    Labourites unsurprised.

    The impact on voting will be interesting. 8-)
    I’m not expecting a stampede away from the coalition; rather, I’m expecting a steady drip of lost votes over the coming months.

    Report comment

  11. Wow!
    ALP have replaced Kevin Rudd with Julia Gillard as Australia’s PM… in order to prevent the election of a Liberal/National govt. (and their deficit reduction/spending plans).

    Report comment

  12. YouGove Poll 42/34/17 – very good for Tories, very bad for Lib Dems.

    Report comment

  13. @Roger Mexico – good last post. It tallies with much of what I have said. This budget has not been the big watershed (see Tories on 42% in post budget poll). If the Tories are pleased, they shouldn’t be. They have set themselves up for some terrible pain.

    Incidentally, the split isn’t 4/1 – that’s an Osborne myth. According to the IFS it’s 3/1. When Osborne talks about 4/1 he ignores the previous tightening done by Darling that he has adopted, Even so, I am certain that the cuts at this scale will not be achieved (especially if the Lib Dems are still in the government) and Osborne will either have to return to Darling’s more measured deficit reduction timetable or raise taxes a year or two before an election. The IFS says that the Home Office and other unprotected departments will be looking at 35% cuts. It’s going to be fascinating watching the Tory benches when police forces and prison spending is slashed.

    Another budget highlight Osborned didn’t exactly shout about – the main UK banks will be better off (yes – better off) over the next five years. That’s right – Osborne has effectively rewarded the banks for getting us into this mess and is trying to hide it behind a fig leaf levy. How very Tory.

    Report comment

  14. From the IFS – “”Osborne and Clegg have been keen to describe yesterday’s measures as progressive in the sense that the rich will feel more pain than the poor. That is a debatable claim. The budget looks less progressive – indeed somewhat regressive – when you take out the effect of measures that were inherited from the previous government, when you look further into the future than 2012-13, and when you include some other measures that the Treasury has chosen not to model.”

    I think it’s pretty clear now that Osborne’s claim to not hide things in the small print is nonsense. His big show of accepting 25% cuts is also nonsense – with defence and education limited to 10% it means much higher cuts elsewhere.

    The basic budget rule is that whichever word is used most to summarise the measues, the actual result is diametrically opposite. For Osborne and ‘fair’ think Brown and ‘prudent’. It takes a few days to unpick usually, but today we can see the pattern quite clearly.

    Report comment

  15. Clegg having a hard time on Today – the fact that the progressive elements of the budget are those inherited from Labour.
    Cameron clearly rattled yesterday by Harman’s allegation that he has ‘not been straight’ with the elecorate. Also that Osborne had broken his promise not to hide things in the small print.
    Apart from an ideological desire to shrink the state, there is also a punishing emphasis on pain caused by Labour. How good it is for the country, as opposed to whether it can ‘come good’ over an electoral cycle, will be the test of this drastic budget.

    Report comment

  16. Just to put the Budget in some sort of context :-

    The extra measures announced by GO ( over & above Labour’s measures) ie- £40bn of annual spending cuts and tax increases by 2014/15, amount to 2.2pc of GDP.

    Real public spending will fall by around 4pc to meet the aim of eliminating the structural deficit by 2015/16, a drop of 0.8pc per year in real terms.

    In cash terms, total managed expenditure will continue to rise, from £696.8bn this year to £757.5bn in five years time.

    The 25pc cut pencilled in for un-protected departmental spending is a real terms fall – in cash terms it doesn’t look so bad . The cut implied by Alistair Darling’s last Budget was already 20pc. GO added an extra 5pc over five years.

    As a proportion of output, public spending will shrink from 47.3pc this year to less than 40 pc in five years time.

    Report comment

  17. “400 pounds a week for a house is fair? you have got to be kidding it’s daylight robbery. minimum wage is only 200 pounds a week. HB is the problem, there is no incentive to look for cheaper rents and landlords are guaranteed high rents even if the market can’t bare it”

    I don’t disagree. What I meant is that there is no real justifiable reason for anyone to complain at these capping figures IMO.

    Report comment

  18. Roger Mexico:

    Your posts on the impracticaity of large cuts were right on target.

    I have been a situation where top down imposed cuts made in the way you describe wrecked the budget system and managers confidence in it.

    I have also been part of a small management team which attempted a 5% overall development in one year without recruiting extra management resources while avoiding reckless waste of resources and getting good value for money. We did it, but only by working harder than would have been sustainable in the long term.

    Most organisations can cope with 5% change up or down in a single year. Much more than that and there is inescapably waste or unintended consequences, perhaps leading to crisis management solutions and even higher costs.

    The scale of cuts proposed for some departments where some expenditure is protected and some is fixed, is huge.

    It won’t all fail, and there will be some backtracking. Some objectives will be missed, others compromised or deferred but there will certainly be disasters, and CON/LIB back benchers will have to chose beween loyalty to the coalition and addressing their constituents objections to bizarre unintended consequences of the top-down approach.

    I’m not a nationalist, and I am not trying to make a partisan point, but it could never happen in an independent Scotland which adhered to the Founding Principles, procedures and constitution of the Home Rule parliament.

    Report comment

  19. I agree that organisations, especially public sector ones, are absolutely terrible at implementing budget cuts. Sadly that is a problem of human psychology rather than actual necessity. As a public sector (police) employee I know full well that there are huge, huge areas of waste, and a huge amount of timewasting and laziness. The problem is, have these organisations got management systems robust enough to focus the cuts on the right parts of their business? Experience suggests not. However, as organisations are equally bad at focussing budget increases on the right part of their business, simply rolling back the above-inflation budget increases that have been awarded over the past couple of decades might be a start.

    It’s an old adage that once you add a member of staff, they quickly make themselves “irreplaceable” even if that department once managed to function without them. The work spreads to fill the space available. I am quite sure I work 50% harder than the average person in my organisation, and probably 200% harder than some of the worst offenders.

    Report comment

  20. i know you don’t disagree. you are quite right that is no reason for anyone to complain about these lower caps, in my view they should be lower still and CGT should be higher. property values are far too high and need to come down, no asset bubble can last for ever.
    here in norway i earn a very ordinary wage and use 27% of that on rent, in england i would have to use my whole wage and have two jobs or be content to just sit on the dole. this situation is unsustainable IMO and the state should not be making things worse with HB.
    of course people have to have a place to live and if they have no income then the the state (ie us) must help them but it needs to be done in a way that is market neutral

    Report comment

  21. Neil A

    ” I am quite sure I work 50% harder than the average person in my organisation, and probably 200% harder than some of the worst offenders.”

    It isn’t smart to work hard, but for some folk its hard to work smart.

    I worked with two colleaguesin another department who were both inefficient and lazy. With a well designed spreadsheet I could have absorbed the work of one of them by myself.

    Contrary to the faith based beliefs of the looney right, in my experience there is no difference between the public sector and the private sector in the prevalance of incompetence, stupidity and alcohlism.

    Report comment

1 2 3