I’ve mentioned in various comments over the last few weeks that YouGov were still looking at their post-election methodology and what changes to make. It is a rather more complex task than one might think (especially since all of us in the team took breaks for holidays at various points during May or June to recover from the general election campaign). Anyway, we have finally rolled out the new methodology, and the pre-budget figures yesterday were the first using the updated method. On the whole, the changes are minor and build upon what we were doing anyway. It is mainly updating our weighting figures, and upgrading them where we can – there are no big departures from what went before.

Anyway, below is a copy of paste of my article on the YouGov website explaining it in full:

The polls at the general election were something of a mixed bag. For the first time in the quarter of a century the polls did not overestimate Labour support. YouGov’s own final poll got the Conservative lead over Labour and, therefore, the swing from Labour to Conservative exactly correct. However, at the same time all the pollsters, including YouGov, overestimated the level of Liberal Democrat support.

I am sure there are many debates to come about why the polls overestimated the Liberal Democrats. What we can be relatively sure about it is that it wasn’t due to individual problems with pollsters’ methods – every company got the Lib Dems wrong, whether they polled by phone, online or face to face, and regardless of the political weighting they did or did not use. This was a systemic error. It has been suggested that it is due to a late swing (including on the day abstentions), or young people who hadn’t voted before not actually turning out, or a disproportionate likelihood for polls to be answered by Liberal Democrat supporters or people interested in politics who had seen the debate. Some of these explanations – such as late swing – are not necessarily things that pollsters can do much about.

On election day itself and the days that followed YouGov carried out a massive survey of our panel, contacting just shy of 100,000 panellists. We used this data to analyse exactly what happened and what we could do to make our samples and weightings more accurate, and for the last month we have been testing updated samples and weights on the back of it.

After considering many possible changes, the result is that YouGov will be making modest updates to our methods, rather than any drastic change. Given we actually got the swing from Labour to the Conservatives right, and that the reasons for the over-estimation of the Lib Dem vote may well have been a late swing or a temporary reaction to “Cleggmania”, we needed to be very careful not to over-react. We wouldn’t want to artificially weight down the Lib Dems and end up underestimating their support come the next election.

While there are no big changes though, we are making various tweaks and updates to our methods to bring things up to date. In most cases these are just things that need updating regularly anyway, or where the growth of our panel or the availability of more data on them has allowed us to do things more accurately.

First, we are introducing more advanced social class weighting. Up until now YouGov have only weighted social class by ABC1 and C2DE – essentially a middle class vs working class divide. We have now got more detailed social grade classifications for a large proportion of our panel, so we can switch to weighting AB, C1, C2, DE separately. For those not versed in social classifications, these roughly equate to professionals and managers, clerical and office workers, skilled manual workers, and unskilled manual or reliant upon benefits.

Secondly, we have changed our age weightings slightly. This one isn’t really a result of the election, but because older people are now far more likely to be on the internet. Our top age band used to be over 55s, on the basis that over 55s were one of the hardest groups to recruit to the panel. These days there are more older people on the internet, more older people on our panel, and we have the opportunity to weight them more accurately – so our oldest group is now the over 60s.

Thirdly, our old party ID weightings were based upon the party ID we found at the time of the 2005 election, adjusted to take account of panellists joining YouGov since then. Obviously the election gives us the opportunity to take a new fixed point of reference for party ID, so we have collected party ID from most of our panel afresh, and taken a new snapshot of 2010 party ID to weight to. Over time some people have changed party ID as the Conservative party became more popular, but nevertheless our target weights are not actually that different from the figures we used to weight to. We have updated our newspaper readership targets in the same way, mainly to take account of a growing number of people who no longer read a newspaper.

Moving forward, we are weighting party ID to Conservative 28.5%, Labour 32.5%, Lib Dem 12%, Others 3% and none or don’t know 24%. Note that the proportion of people identifying with the Labour party is still actually higher than the Conservatives, the Conservative lead at the election was due to Labour identifiers voting for other parties or staying at home, and unaligned voters backing the Conservatives.

We are continuing to look at whether to separately weight Labour identifiers who were “loyal” or “disloyal”. At present we will be controlling it in our sampling, rather than weighting as we did during the election campaign. We will keep it under review, in case we need to begin weighting by it again in the future.

Finally, we have made a slight adjustment to our sampling – the only change directly aimed at addressing the Lib Dem over-representation in the election polls. In analysing our elections polls we found that amongst some age groups respondents tended be too educated, with too many having degrees and not enough with few or no qualifications. In future we will specifically sample people in those age groups with low levels of educational achievement to make sure respondents are not too “graduate heavy”.

Taken together, the effect of these changes reduce the recorded level of Liberal Democrat support slightly and increases Conservative and Labour support slightly, but not to any great extent. In test surveys we have asked respondents how they voted in the 2010 election, and the new weighting tends to produce answers within 1% of the actual election result. That isn’t necessarily a guarantee of accuracy, since we know that some of those answers will affected by false recall and we’re not quite sure how that will affect the Lib Dems, but we are confident that the new weighting scheme accurately represents the British public.


121 Responses to “YouGov methodology review”

1 2 3
  1. Maybe there’s some merit to treating a mid-election surge in support differently to one mid-term. It’s just that attitudes to a party or leader are formed slowly during mid-term (a sort of steady drip-drip of grrrr, fed up with this lot, the other lot look not so bad) rather than the rushed and frenzied aspects of an election campaign. Could it be said that an election campaign surge is of less importance than one mid-term? I know, thinking about 1992 stands that thought on its head.

    Knowing our luck we won’t get anything like cleggmania ever again (like we never got 1992 again) and we’ll be adjusting weightings to take account of something that isn’t there next time.

  2. Anthony

    Thanks for that very helpful reply. With the non-registered (or those registered at the wrong address) I’m not sure that even asking them if they are registered gives you the right answer. Especially as the UK still has householder registration, many might have found out they couldn’t vote at the polling station – or when they applied for a credit card.

    These groups (17-24 year-olds and private tenants – with a massive overlap of course) are traditionally the least involved, but remember that most of the Lib Dem surge came from Don’t Knows etc rather than Party switching. What’s more these groups are still I know they’re the groups most under-represented in your panel but they must be there and YouGov presumably try hard to get more – if only for consumer polling (marketing types love them!).

    The under-registration percentages are so high in these groups that it’s probable that many of the the panel members you do have are included (even if they don’t know it). This might explain the residual over-prediction for the Lib Dems.

    Your information on the class / education / newspaper readership interaction was interesting. Class is still of importance – it’s whether the traditional of measuring it is the best possible. Today’s YouGov / Sun poll has the Tory ABC1/C2DE split as 43/38 – significant but hardly class warfare. It may be necessary to have to reexamine the whole topic, particularly as you say with the newspaper readership falls.

    Thanks once again for all the time and care you devote to this site.

  3. Roger, actually I just remembered something else on voter registration. In previous years the British Election Study have crossed checked their data against the marked electoral register to see if interviewees actually voted, and if people were actually on the register. Assuming they do the same this time, then when the data is released that should give a very firm answer on whether there were lots of unregistered people telling pollsters they were voting LD.

    (Oh, and I would be cautious about extrapolating the numbers of young and private tenants who are not registered across the country. The electoral commission’s report is very clear (it has it in big bold text) that the sample of local authorities they looked at was not intended to be representative and you can’t draw conclusions across the country, so young people and private tenants elsewhere may have higher registration rates – not least the sheer awfulness of Glasgow’s figures in the study may be dragging the figures down!))

  4. @Laszlo
    I agree about the HoC post budget debate. It is pointless to expect the Leader of the Opposition, not even the shaow chancellor, to respond immediately to a huge volume of information properly. All we ever get is a set of pre-rehearsed soundbites irrespective of the actual content of the budget.
    It was instructive to see how difficult the BBC economic experts with a bank of computers and researchers found it to respond. GB’s last budget had the removal of the 10p band and the implications were not realised until the next day. I’m sure there are similar unintended (or not) consequences from yesterday.

  5. Anthony

    By the way were the YouGov / Sun VI figures that I referred to above:

    ht tp://today.yougov.co.uk/sites/today.yougov.co.uk/files/YG-Archives-Pol-Sun-results-220610.pdf

    all collected before the Budget? I know that normally you close about 4-5pm, but if there’s another in the next few days it would give us a good indicator of the immediate budget effect.

  6. Aleksandar

    Thanks for that info on Hammersmith and Fulham. I’m not quite sure of the legality of getting the landlords to provide the info (for example can tenants choose to go on the public register or not?), but it’s certainly more efficient. H&F are obviously following Anthony’s advice to increase your registrations as much as possible so you get a better financial allocation from central government.

    Whether it specifically affected the Hammersmith North result I don’t know; Labour did better generally in London than expected. However if other, socially similar, London councils have been following equivalent policies, it might explain some of that trend.

  7. Roger – no, some will have been collected after the budget and a large proportion will have been conducted after most of the budget measures were spread all over the papers – personally I think a comparison from the previous day will be more instructive (I’m also not sure about that Lib Dem figure, it’s looks a touch roguey to me :) )

  8. Anthony

    I agree that the Electoral Commission survey was meant to pick up a cross-section of the local authorities rather than a representative sample, but the age band discrepancies are so large that I suspect there is still some effect. Even the estimate for 25-34 is only 66% accuracy. Still the BES should provide us with some hard info.

    Pity about the Sun survey. I suspect most people aren’t affected by the pre-Budget spin, but are by the actual content. A comparison would have been nice.

    I agree the LD figure looks a bit rogue, especially for the aforementioned 18-24 year-olds compared to previous day. Not that that won’t stop some people gloating over it! ;)

  9. @Roger
    The licensing of HMOs requires the landlord by law to supply a vast amount of information regarding the property and basic info about the tenants – certainly names. The landlord has to inform the tenant that they are filling this form in. The electoral registration dept has used this register to identify and coerce landlords to provide the names of the occupants on an annual basis or be fined £1000 as I understand it. There was a change some years ago regarding the sharing of data between govt depts but I thought that was for preventing illegality rather than just ease.
    I, too, have concerns about this whole data gathering process and its legality. Some people don’t want to be on the electoral roll for personal reasons e.g a victim of domestic violence may wish to ‘disappear’.

  10. @Aleksander

    I’m a bit surprised at the information exchange – the EC website claims that even council tax payment info isn’t passed across. Of course not all private tenancies are HMOs, though those are more likely to have the high turnover.

    I don’t know if it’s a legal requirement to register for the electoral roll or not. The official websites are all ambiguous and there’s a row about it in the talk section of Wikipedia (quite). However if you want some privacy you can opt out of the edited register (which is why I queried if landlords would give tenants this option). The full version only allows limited access.

    For more extreme cases there is something called “anonymous registration” which caters for victims of domestic violence etc.

  11. Roger, well – we have still got YouGov voting intentions with Sunday-Monday fieldwork to compare to the post-budget figures.

    More on that later! :)

  12. @ Roger
    It is a criminal offence not to complete and return the annual canvass forms liable to a fine of £1000 – that’s from a number of different council websites. This also covers false or inaccurate information. The canvas is then checked and used for the electoral roll.
    The legal requirement is not on the individual but on the householder.

  13. Having seen the first reaction to the budget, maybe it will affect the polls after all. Anyone have a view?

  14. @Sue Marsh

    Even the Telegraph are running negative stories about the cuts to Housing Benefit and Disability Living Allowance.

  15. @ Aleksandar

    Thanks for that. I thought it was the case, but all the sites I looked at seemed to be more “return it or you might lose your vote”. Obviously waving the big stick isn’t fashionable.

    They are supposed to be moving to individual registration (the EC has just published guidance), but with local government about to go into budget shock, I suspect it may not be a high priority.

    @ Sue

    Anthony’s promising us unseen riches later (see above). :)

  16. Roger – I thought maybe that was a little hint…
    Would that be fieldwork closing at 4 today? Otherwise probably still a bit soon.

  17. IFS: budget is ‘somewhat regressive’ when you take out measures inherited from Labour, consider the effect beyond 2012-13, and take account of factors that the treasury chose not to model.
    LD deputy leader today lavishing praise on this ‘progressive budget’ at the same time as an anouncement promising further welfare cuts, and estimates that spending in some departments will be reduced by up to a third.

  18. We’ve had Alexander the Great, Attila the Hun and now we have George the Impoverisher.

  19. And Vince Cable holding the litter tray (for the fat cats)

  20. @ JAY BLANC

    “cuts to Housing Benefit and Disability Living Allowance.”

    Is DLA being cut?

    I didn’t think so-just a new regime of medical assessments I think.

  21. @Colin, change from uprating on basis of rpi to cpi is judged to be a cut.

  22. Not convinced so far about explanations for last minute switch (or drpping away) of LD’s on final day.

    I have not got a lot of time to contribute at present due to personal cicumnstances but there has been chat on here previously about leading questions posed by pollsters. I have one for us in the same vein.

    ‘Do you think it is a saving grace that the Compassionate Conservative party was able to form a Government with the right wing Lib Dems and thus curb the Lib Dems’ worst excesses?’

  23. “@Colin, change from uprating on basis of rpi to cpi is judged to be a cut.”

    Ah- I see.

    So the actual rates have not been cut.

  24. I can’t figure out how the well off are to be paying more after the budget.

    We can go buy pretty much all our big ticket items before January (because we have plenty money & a good credit rating).

    So what have I missed? How are we paying our share? 8-)

  25. @Amber,

    The middle classes will be significantly worse off according to the BBC and ITV news. £600 was calculated (on the ITV news, I think) for one supposedly typical middle class family with 2 children and a working mother. Not sure how true this figure is though.

    Regarding the polls, I would personally be surprised if the Tories didn’t suffer quite severely, even in the short-term. Tax rises are never popular, even if they are totally necessary.

  26. By the same calculations a single mother (who also did a low-paid job) will be £20 a year worse off according to yesterday’s calculations on the news. So still worse than before the budget clearly, but no where near as bad as for the middle classes, it seems. But I cannot say if these calculations are accurate. I’m just taking them from the so-called experts on the tv news.

  27. Correction: By the same calculations a single mother (who also *does* a low-paid job) will be £20 a year worse off according to yesterday’s calculations on the news.

  28. Of course, it all depends on consumption too. Those who spend a lot on luxury and other products, will be more severely affected than those who are frugal spenders. I have to say the VAT calculations mentioned for all the families on the news shocked me. They must be currently spending a lot is all I can say! Or perhaps it’s just because I’ve always been very frugal compared to most people.

  29. Colin – Also all benefits are being frozen, another cut to DLA

  30. BILLY BOB

    I heard Simon Hughes on Five Live this morning. Now I always thought that Hughes was a reasonable sort of guy with some left of centre views which would put the Labour Party to shame.

    Well, how wrong can you be – Simon has obviously been seduced by a sniff of power (albeit a bit second-hand in his case) or have we all completely misunderstood what he has been talking about over the past 20 years or so. I wonder how he’s going to deal with all those folks in Bermondsey who are going to be thrown out of their homes when the housing benefit no longer meets the local market costs.

  31. Just seen Harriet H at PMQs… apparently £0 set aside in budget for bringing in an index linked rise in state pension one year earlier than Labour… so just the vat rise then.
    Another broken promise on no family under £40k to lose tax credit… that should have been £30k.

  32. @Sue,

    That’s the one major concern I have about the budget, if I’m being honest. I do think there are quite a lot of un-genuine claimants for disability allowance, but many are genuine. So whilst I agree with the clampdown on un-genuine cases, I do worry that those who are genuine will have their allowances cut, and thus, struggle more financially.

  33. @Colin

    It’s certainly being reported as Cuts to DLA in the press.

    The switch to CPI from RPI will mean a real-terms cut in rates.

    Plus the intent of the new medical tests is to harshly reduce the number of claimants. And there’s not a whole lot of love out there for ATOS’s management of medical testing for the ESA, with recent reports about the exceptionally large successful appeal rate from ATOS medical exams. There are current tests, but are applied by random sampling, not on ever claimant, questions are being asked if these new tests and increase of appeals will cost more than the ‘savings’ as they have with ESA.

    A lot of people are concerned over GO’s comments about the benefit. One of intents of DLA was to move those who could be independent out of care-homes. And the rise in claimants is because it worked, and we have more independent disabled people. GO implied instead that the rise in claimants was from those who shouldn’t be entitled to it.

    Of great concern was ‘getting people off DLA and into work’ since DLA is not an out-of-work benefit, but available to working disabled as well to cover the additional costs they have over able bodied workers. A lot of disabled people now wonder if they might be pushed off DLA if they are working.

    I think that there’s a fair chance that DLA ‘reforms’ will be amended out of the budget. Or reversed before the 2013 estimate for when the new ‘medical testing’ starts.

  34. @ MATT

    ” I do worry that those who are genuine will have their allowances cut, and thus, struggle more financially.”

    Why should people who “genuinely” meet the medical criteria for the various elements of DLA be affected?

    I don’t understand.

  35. Overall, I’d have to say the budget does seem pretty fair to me though. I know I will be adversely affected, as will my family, but I don’t mind as long as everyone else has to tighten their belts as well. I just want this financial crisis to end, even if it means pain in the short-term.

  36. @Colin,

    “Why should people who “genuinely” meet the medical criteria for the various elements of DLA be affected?

    I don’t understand.”

    I meant I agree with the medical criteria, but am worried about the cuts to DLA.

    “And the rise in claimants is because it worked, and we have more independent disabled people. GO implied instead that the rise in claimants was from those who shouldn’t be entitled to it.”

    I think it’s a bit of both.

  37. @Colin

    The main problem is that medical testing for the DWP is outsourced to an Data-Management firm that then runs all the offices and operations of the tests. And the company outsourced to (ATOS-Origin) gets a bonus for meeting a quota of denied requests. But they don’t see any fines if they turn down someone who then wins an appeal.

    This has of course lead to a growing number of successful appeals, which both cause stress and fear for the applicants, and costs the state a huge amount of money from the appeals process.

  38. @DavidB
    One vox pop today saying they hope the landlords will put the houses up for sale if they can no longer collect rent…” we’ve been trying to buy for ages”.
    One more irreverent voice on twitter suggesting that HB cuts will mean a ‘clensing’ of Camden (not to mention Notting Hill)… so hardly the same need for ‘free schools’ then.
    For Big Society read ‘gated communities’ imho.

  39. JAY:-

    DLA claimant numbers have risen by more than 40% since 1997-cost is now £11bn pa.

    Do you think the incidence of relevant disability level have increased-or there has been some slippage in qualifying criteria?

    Do you feel that this ballooning cost should not be subject to review of any sort?

    MATT

    Thre aren’t any “cuts” to DLA. Changing the indexation / freezing payment levels is the equivalent of the public sector pay freeze, and the massive fall in hours worked / pay in the private sector.

  40. @Billy Bob

    I wonder how much the housing market will fall in areas with average rents above the new HB Limit…

  41. Colin:

    Read my reply above. The intent of DLA was partially to help people move from residential care to independent living. The 40% rise shows that it worked, and there’s a corresponding drop in the number of people living in residential care.

    It’s rather dodgy to cite the 40% rise as a rise in disabled, when it’s really people moving from residential care to independent living. Residential care costs the government a whole lot more than DLA does! It’s successfully cut spending in that way!

  42. @ jay blanc

    the market rates will fall to just over the maximin HB. HB is like the CAP it supports prices and distorts the market. time to move to a better system i think, what we have now is socialism for the landlords

  43. BILLY BOB

    So, if you’ve got a family income of £50,000 plus you are going to be down the equivalent of a couple of bottles of wine a week. If you get your housing benefit capped you’ve got a good chance of being out on the streets – graet stuff!

    I think we could be in for a long period of social unrest and it’s all such a pity because the worst elements of the deficit reduction strategy could have been avoided by a temporary 3 pence increase in income tax for three years bringing in around £45 billion.

  44. @Jay Blanc,

    “The intent of DLA was partially to help people move from residential care to independent living. The 40% rise shows that it worked, and there’s a corresponding drop in the number of people living in residential care.”

    I agree that making DLA widely available is a very good thing. However, I also agree that it has been abused by some (unfortunately) as it has become more widely available.

    “It’s rather dodgy to cite the 40% rise as a rise in disabled, when it’s really people moving from residential care to independent living.”

    I agree.

    @Colin,

    “Thre aren’t any “cuts” to DLA. Changing the indexation / freezing payment levels is the equivalent of the public sector pay freeze, and the massive fall in hours worked / pay in the private sector.”

    Those claiming DLA will be a bit worse off, whichever way you look at it, Colin.

  45. @David B,

    “I think we could be in for a long period of social unrest and it’s all such a pity because the worst elements of the deficit reduction strategy could have been avoided by a temporary 3 pence increase in income tax for three years bringing in around £45 billion.”

    A 3 pence increase in income tax would be more unpopular than the VAT rise, according to some recent polls. Whichever way you get the money from people, they are not going to agree, or like it, in most cases.

  46. The housing benefit is only being capped to £280 a week for flats and £400 a week for a house. This equates to £14,560 a year for a flat and £20,800 for a house. Seems fair to me.

  47. I think the taxation changes announced in the budget are only the “and in other news” item here. The departmental spending cuts are where the real pain is, and the dimensions and impact of those cuts won’t become clear for a few months.

    If the Home Office budget is cut by 1/3 in one parliament, that will mean some pretty extreme choices. I don’t necessarily have a problem with that, but you will see some fairly major “baby with the bathwater” changes to the services offered.

    There are quite obvious dangers for both sides. For the Coalition there is the impact on GDP of removing all that borrowed cash from the public sector, and the warfare with the unions that may also result. For the left there is the danger that once the dust settles people may not actually notice the cuts that much at all. The instinct of the right is that most of those public sector “new hires” simply aren’t doing anything useful enough to justify their salaries. In a few years we will know if that’s right!

  48. matt

    400 pounds a week for a house is fair? you have got to be kidding it’s daylight robbery. minimum wage is only 200 pounds a week. HB is the problem, there is no incentive to look for cheaper rents and landlords are guaranteed high rents even if the market can’t bare it

  49. Interestingly, when we see the UK Budget, he new GERS (Government Expenditure and Revenue Scotland) data for 2008-09 has been published today.

    h ttp://www.scotland.gov.uk/Publications/2010/06/22160331/0

    Scotland had a net fiscal deficit of £3.8bn, some 2.6% of GDP.

    The comparable figures for the UK as a whole are £96.1bn and 6.7%.

    If the Scottish media cover tis data accurately, it will be interesting to see how Scots respond.

  50. Richard in Norway
    You may have been away too long. Private rents in NE Scotland are ahead of HB rates at present because of competition from east European migrants. This produces a strong flow of homeless presentations to the local authority with calamatous social results.
    Local authorities have also grown used to using HB to arguably substitute for cuts in social work provision to support people with serious addiction problems etc. More is landing on Councils. As I pointed out before, how keen are Libs going to be stan as Councillors … or voters to vote for them?

1 2 3