I’ve mentioned in various comments over the last few weeks that YouGov were still looking at their post-election methodology and what changes to make. It is a rather more complex task than one might think (especially since all of us in the team took breaks for holidays at various points during May or June to recover from the general election campaign). Anyway, we have finally rolled out the new methodology, and the pre-budget figures yesterday were the first using the updated method. On the whole, the changes are minor and build upon what we were doing anyway. It is mainly updating our weighting figures, and upgrading them where we can – there are no big departures from what went before.
Anyway, below is a copy of paste of my article on the YouGov website explaining it in full:
The polls at the general election were something of a mixed bag. For the first time in the quarter of a century the polls did not overestimate Labour support. YouGov’s own final poll got the Conservative lead over Labour and, therefore, the swing from Labour to Conservative exactly correct. However, at the same time all the pollsters, including YouGov, overestimated the level of Liberal Democrat support.
I am sure there are many debates to come about why the polls overestimated the Liberal Democrats. What we can be relatively sure about it is that it wasn’t due to individual problems with pollsters’ methods – every company got the Lib Dems wrong, whether they polled by phone, online or face to face, and regardless of the political weighting they did or did not use. This was a systemic error. It has been suggested that it is due to a late swing (including on the day abstentions), or young people who hadn’t voted before not actually turning out, or a disproportionate likelihood for polls to be answered by Liberal Democrat supporters or people interested in politics who had seen the debate. Some of these explanations – such as late swing – are not necessarily things that pollsters can do much about.
On election day itself and the days that followed YouGov carried out a massive survey of our panel, contacting just shy of 100,000 panellists. We used this data to analyse exactly what happened and what we could do to make our samples and weightings more accurate, and for the last month we have been testing updated samples and weights on the back of it.
After considering many possible changes, the result is that YouGov will be making modest updates to our methods, rather than any drastic change. Given we actually got the swing from Labour to the Conservatives right, and that the reasons for the over-estimation of the Lib Dem vote may well have been a late swing or a temporary reaction to “Cleggmania”, we needed to be very careful not to over-react. We wouldn’t want to artificially weight down the Lib Dems and end up underestimating their support come the next election.
While there are no big changes though, we are making various tweaks and updates to our methods to bring things up to date. In most cases these are just things that need updating regularly anyway, or where the growth of our panel or the availability of more data on them has allowed us to do things more accurately.
First, we are introducing more advanced social class weighting. Up until now YouGov have only weighted social class by ABC1 and C2DE – essentially a middle class vs working class divide. We have now got more detailed social grade classifications for a large proportion of our panel, so we can switch to weighting AB, C1, C2, DE separately. For those not versed in social classifications, these roughly equate to professionals and managers, clerical and office workers, skilled manual workers, and unskilled manual or reliant upon benefits.
Secondly, we have changed our age weightings slightly. This one isn’t really a result of the election, but because older people are now far more likely to be on the internet. Our top age band used to be over 55s, on the basis that over 55s were one of the hardest groups to recruit to the panel. These days there are more older people on the internet, more older people on our panel, and we have the opportunity to weight them more accurately – so our oldest group is now the over 60s.
Thirdly, our old party ID weightings were based upon the party ID we found at the time of the 2005 election, adjusted to take account of panellists joining YouGov since then. Obviously the election gives us the opportunity to take a new fixed point of reference for party ID, so we have collected party ID from most of our panel afresh, and taken a new snapshot of 2010 party ID to weight to. Over time some people have changed party ID as the Conservative party became more popular, but nevertheless our target weights are not actually that different from the figures we used to weight to. We have updated our newspaper readership targets in the same way, mainly to take account of a growing number of people who no longer read a newspaper.
Moving forward, we are weighting party ID to Conservative 28.5%, Labour 32.5%, Lib Dem 12%, Others 3% and none or don’t know 24%. Note that the proportion of people identifying with the Labour party is still actually higher than the Conservatives, the Conservative lead at the election was due to Labour identifiers voting for other parties or staying at home, and unaligned voters backing the Conservatives.
We are continuing to look at whether to separately weight Labour identifiers who were “loyal” or “disloyal”. At present we will be controlling it in our sampling, rather than weighting as we did during the election campaign. We will keep it under review, in case we need to begin weighting by it again in the future.
Finally, we have made a slight adjustment to our sampling – the only change directly aimed at addressing the Lib Dem over-representation in the election polls. In analysing our elections polls we found that amongst some age groups respondents tended be too educated, with too many having degrees and not enough with few or no qualifications. In future we will specifically sample people in those age groups with low levels of educational achievement to make sure respondents are not too “graduate heavy”.
Taken together, the effect of these changes reduce the recorded level of Liberal Democrat support slightly and increases Conservative and Labour support slightly, but not to any great extent. In test surveys we have asked respondents how they voted in the 2010 election, and the new weighting tends to produce answers within 1% of the actual election result. That isn’t necessarily a guarantee of accuracy, since we know that some of those answers will affected by false recall and we’re not quite sure how that will affect the Lib Dems, but we are confident that the new weighting scheme accurately represents the British public.