YouGov have been dominating the polling in the pre-election period since the daily polling began, especially since the other companies still seem to be on their “peacetime footing”. At some point many of them will also step up the frequency of their polling as we get closer to the election, but for now YouGov is getting the attention, and two parts in particular of their methodology have been provoking a lot of questions.

Weighting Labour identifiers

Shortly after starting the daily polling YouGov made a slight change to their political weighting. Previously their political weighting was just on party ID. It’s important to note that party ID is NOT the same as past vote: some people identify themselves with one party, but abstain or vote for someone else – perhaps tactically, perhaps as a protest (e.g. “I’m Labour through and through but I’m voting for the Lib Dems because of Iraq”). Nor is it the same as voting intention….

In 2005 only 72% of people who said they identified with the Labour party said they actually voted for the Labour party. 13% voted Lib Dem, 9% stayed at home, 6% voted for other parties (this was mostly a Labour phenomenon, the overwhelming majority of people who identified with the Lib Dems or Conservatives also voted for them).

To take account of this YouGov have always sampled loyal Labour ID (people who identify with Labour and voted for Labour in 2005) and disloyal Labour ID (people who identify with Labour, but did not vote for them in 2005) separately. In the past, they never felt the need to also weight by it – the other weightings were enough to keep it in broadly the correct proportion.

The switch came about on the back of the testing of daily polling before its launch. One day produced an outlying result and further investigation showed that the cause seemed to be too many “Loyal Labour ID” respondents. To prevent that happening again, we decided to start weighting the loyal and disloyal Labour ID respondents separately as an extra control on the representativeness of the sample.

Naturally before we actually did it on any published polls it was heavily parallel tested. We reweighted three weeks worth of polling using both the new and old weights, and in almost every case it made no difference at all to the topline figures (in most cases the changes were in the region of 0.1%). So it doesn’t make the results more Labour or more Conservative, all it does it make things a tiny bit more stable.

Degree of weighting

The other issue that has aroused comment is the amount of weighting YouGov do to their samples. As I’ve said several times before, this doesn’t actually matter that much when it comes to final results – what is important is what the demographics of the sample are AFTER weighting, is that representative? Having to do more weighting does reduce the effective sample size of a survey, so risks making it more volatile, but that’s about it – and so far YouGov’s daily polling has not been unduly volatile.

However, for those who are interested the YouGov figures in the last few weeks have needed more weighting than in previous months. The main reason is the shift in the way YouGov sample. People used to be invited to specific surveys – people got an email inviting them to do a survey about, say, fish, they followed the link and did the survey about fish. If the fish survey was already closed, they got sent away. This allowed YouGov to invite samples that were pretty accurate demographically to begin with and reduce the need for weighting afterwards, though some weighting was still needed because you got different response rates from different groups. This was a plus, but it also had disadvantages – you could end up with too many respondents, or panellists could end up being sent away after taking the trouble to respond to an email. You also couldn’t do very accurate one day surveys since “fast responders” were different to “slow respondents”.

So last autumn YouGov switched to a new system. Now people are sent a non-specific invite and, when they arrive at the YouGov system, they are allocated to whichever survey needs someone in their demographic group (for example, were I to arrive at the site the system would look at all the open surveys and see which one’s quotas were most in need of a 25-40, middle-class, Times reading man from the South East, and send me over there). What this means is that surveys don’t get excess replies, respondents never get sent away empty handed, and that even very fast surveys get an even mix of fast and slow respondents, allowing accurate one day surveys.

However, because you can’t tell who is going to respond to which survey, it also makes it more difficult to calculate the proportions of people to invite to get a sample that needs the least possible amount of weighting – hence the slightly higher levels of weighting (though it’s worth saying they remain relatively low compared to some of the weights needed for quasi-random samples).

They’ll come down over time as the panel team get used to the patterns of response amongst different groups (looking at recent results, for example, we could probably do with inviting fewer men over 55 – too many are responding!) but in the meantime the effect upon polls should be just to make them a tiny bit more volatile.

68 Responses to “YouGov methodology tweaks”

1 2
  1. The problem is, Kyle, which street, when and how many people? If you could ask 10 people each on a representative 30 streets from 300 representative contituencies you might get something resembling a decent sample, but depending on when you polled you would probably miss out those who were working, or those who can’t be bothered to get out of bed in the morning, or those who are disabled, or those who drive everywhere and never walk etc.

  2. Thank you for your introduction, Anthony, and all your ever-courteous replies, and thank you everyone else for making this such a good place to visit (at least for electorally obsessed visitors!).

    Anthony – Would one way of addressing the concerns people are raising about what weighting does to YouGov accuracy be to compare the performance of the pollsters in the Local Govt and Euro elections that have happened since the 2005GE?

    Concerning polls versus canvassing, as an inveterate canvasser I have to say that I trust the polls far more, mainly because the critical variables are largely controlled.

    For a start, as a canvasser, I am not neutral and clearly do not appear to be so, wearing my party colours, carrying a clipboard etc. This produces a severe case of the ‘observer effect’ on the person being canvassed, not only on what they say/don’t say about our our party and candidate, but also what’s said/not said about other parties and their leadership figures.

    My experience is that – politely approached on their own doorsteps – few people want to appear rude, even if they disagree with you. Often they will choose a line that won’t lead them into conflict, so Tory canvassers will find much greater numbers of expressed anti-GB sentiment than necessarily reflects its truly prevailing level, let alone the level of the likely Labour vote.

    As with all parties, I guess, my basic aim as a canvasser, under party instructions, is to try to ascertain/confirm our solid, soft and possible support, and to help us to follow this up all the way to the closing of the polls. Apart from recording voting intention, if expressed, and noting any dangers from being bitten by their dogs, or (very occasionally) themselves, we do not have time for any further interest in the views of other parties’ firm supporters.

    Our ‘brains’ in constituency HQ work out what the returns (and other indicators) mean in terms of how well we are doing, where we need to step up our efforts in canvassing, targeted leafletting etc. All very quiet and rational. It seems to have been working well for us for some time past.

  3. You’re probaly right ,Neil. There are so many factors to consider, But I do think there should be a minimum number of people asked. No poll should have a sample less then 1000.

  4. @William and others

    you clearly haven’t been watching closely enough. 8 points nationally probally equates to ten points in the marginals. Which means a Tory landslide.

  5. To Ken at no1 –
    So when Simon Hughes along with labour voted not to have a referendum on the EU Constitution you were well chuffed then? Was that so? Hughes voted against a major plank of conservative policy and constitutional principle. But you will by your account still vote for him – a man who broke an important solemn constitutional undertaking.

    I only raise this in the context of deducing how people think when it comes to voting. Your confession is one of the most interesting I have come across.

  6. Interested in this “Marginals Polling”. I seem to recal that all sorts of polls last time suggested that marginals were behaving differently to the rest of us. How much evidence is there that it turned out this way?

  7. Demographics?

    – in the last few years we have seen 3 million immigrants and 1 million (at least) otherwise effective British workers thrown onto benefits. NEETS have risen . People who could work but are not and either are on some benefit or other or are just lost to the system have risen.

    That demographics. What do YouGov and Kellner have to say about that? Kellner should have a view – his wife used to be in govt. He should have an accurate line on where to take his sample.

  8. “they may be more likely to alter their views in response to events and news coverage” … etc

    GRI – I believe I made a similar point a while ago. Woove point and re iraq Afghanistan.

    What is the point of trying to second guess WNV’s.

    On the above quoted point – it makes me think about those who did not vote labour over Iraq and Afghanistan in 2005.
    Has anything changed? Has the war and its aftermath miraculously vanished? Have the dead come to life? Have the govt personnel changed? Well Blair is no more but Brown has said it was all a good idea he supported it …. so why should any of these people decide to vote labour this time? Does this affect this strange principle of loyal disloyal labour?

    Of course they could vote Labour to keep conservatives out and hang their principles. A prospect which should give UKIPers some food for thought.

  9. @Bryan Coombe

    Watching very closely but I see no Williams here.

    Do you mean Wm.Hague – the William who really wasn’t watching closely enough?

  10. Sorry the, ‘Woove point and re iraq Afghanistan’ garbage is just mad keystroke play !

    Even captcha seems obsessed with YouGov —- YGGV

  11. Nice one, Trevorsden

    But 58R2?

  12. So does this explain why I get halfway through a survey, then I get an error message, telling me to come back later, then directs me to the home page. When I try the email link again it tells me the survey is closed.
    Frankly this is very annoying – it has happened 6 times and I have only managed to complete 2 or 3 surveys successfully. I will be getting a telegram from the Queen before I get my £50. Quite frankly this smacks of smart practice and is a waste of my time. I do incidentally apologise for being a white, christian, over 55 year old who votes Tory!

  13. @TREVORSDEN………..To Ken at no1 –
    Nobody’s perfect – but confession !? I like my politics local first and foremost. The big issues, the juggernaut is out of control. Politicians work on two levels, my bloke is good locally, if I thought there was a chance of a Tory candidate winning I’d vote for them, If I voted Tory it would be a vote for Labour here, my LD vote is to keep Labour out. :-)

  14. @TREVORSDEN……………..My previous – I’d vote Tory if there was a chance of winning the seat , what a sacrifice in the interest of the greater good eh ! :-)

  15. And, Ken, as you both maintain a Lib Dem MP, you are enhancing the prospect of one day voting under STV, and thus being able to return ‘home’, to vote Tory and Labour respectively.

    Good sense all round!

  16. I have been on the YOUGOV panel almost since it started but have never been asked to update my profile, although my circumstances have changed substantially over the the last 10 years. An out of date panel profile must surely be a source of error, now that so much weighting is necessary?

  17. “As I’ve said several times before, this doesn’t actually matter that much when it comes to final results – what is important is what the demographics of the sample are AFTER weighting, is that representative?”

    This is a little misleading, since it assumes that there is a minimum sample size in each cell. If you’re very short of, say, DE women aged 65+ then you’ll be applying terrifying factors. I’ve always been more comfortable when weighting is applied to *downweight* over samples in specific sub-groups rather than *upweighting* those underpresented in the survey.

  18. The only poll that matters is the election. Personally, I would ban opinion polls. They have had an adverse effect on the economy, as the market sniffs a hung parliament or a Labour victory. They may also have an anti-democratic effect, the trends themselves becoming a thing that influences people’s voting intention. Frankly, I am sick of hearing about them and would rather hear more from all the parties on what they really will do if they win the election. It’s not just the Tories who aren’t being clear about that.

1 2