Last summer I did a post looking at the “house effects” of each polling company, that is – the party partisan effects of each company’s methodology. We know, for example, that ICM reallocate their don’t knows which helps the Lib Dems, that Survation prompt for UKIP, that online companies tend to have slightly different results from telephone ones, that companies have different approaches to turnout and weighting and squeeze questions that all have impacts on their results. Some of these, like the reallocation of don’t knows, are extremely easy to quantify from looking at the tables published by the pollsters. Others are impossible to measure except by looking at the long term differences between different companies results. This is attempt to roll it all up and give a brief picture of which companies tend to give results that are better for Labour, which tend to give higher results for UKIP and so on.

This is important for understanding trends. One of the most commons errors I see on twitter is people forgetting the house effects of different pollsters, and therefore suddenly thinking that the Labour lead has fallen when an ICM poll appears (when actually ICM always tend to show lower Labour leads) or grown when a TNS BMRB is published (when actually they always tend to show a bigger Labour lead).

House effects are not set in stone, they are a result of how the political situation interacts with each pollster’s methodology, so they change with the political weather. For example, ICM’s reallocation of don’t knows tends to help whichever party has lost support since the last election – back before 1997 it helped the Tories, from around 2002 it started helping Labour, these days it is most helpful to the Liberal Democrats. Filtering by likelihood to vote tends to help the Tories, whose supporters come from socio-economic groups that are more likely to vote… but it can help any party if their supporters are particularly enthusiastic.

Anywhere, here is an updated chart based on polls over the last six months*. Companies to the left tend to show bigger Labour leads, companies to the right tend to show smaller Labour leads. Rather than looking at differences in Liberal Democrat support as I did last year, I’ve looked at differences in UKIP support, which tend to be much starker between pollsters:

A few things worth noting:

It’s noticeable how the “new online companies” are clustered in the top left of the graph – there are differences between them, but they all tend to show the highest UKIP scores and the highest Labour leads. Compare them with the three traditional telephone polls from ICM, ComRes and Ipsos MORI in the bottom right, showing the lowest UKIP scores and the lowest Labour leads.

Secondly look at the range, once you put six month’s data in all there is not a vast difference in the size of the Labour leads different companies show. If we leave aside Angus Reid, who only conducted a couple of polls at the start of the year, and look at just the regular companies there is only a 3 point gap between Opinium and TNS BMRB who show the biggest Labour leads and ICM and ComRes online polls that show the smallest Labour leads. In terms of the Labour lead, the polls are not really “all over the place” as people are want to say.

Contrast that with UKIP, where there really is a gulf between what different pollsters show – the average difference between ICM (who tend to show the lowest levels of UKIP support) and Survation (who tend to show the highest) is about 7 points.

(* As with last time the method I use is to take a rolling average of the YouGov daily poll as the comparison point, and then see what the average difference is between each companies’ polls and the YouGov average at the same time (the reason for this rather complicated approach is that it irons out any differences that stem from the timing of polls. If you just took a crude average of each company’s polls it would be skewed, by example, for a company that did several polls in the last month when UKIP was on a high but few polls early in the year when they weren’t). I then relate this the average across all the companies so that YouGov don’t automatically end up in the middle of the graph. It’s important to note that all the differences are relative to each other… so being in the middle of the chart doesn’t necessarily mean you are right.)


192 Responses to “House effects – UPDATED”

1 2 3 4
  1. There ought to be statisticians among us who could definitively tell us what randomness/MOE factors come in to working out a likely Labour lead on YouGov- then we don’t have the arguments about what the lead is! Just by throwing dice you probably wouldn’t have to wait long before two ones or two sixes came up in a row if you are throwing dice 5 times a week as YouGov are doing.

    Personally I would have said the average of about 2 weeks of polls would be about right unless there are any massive changes like budgets or double dips or whatever to factor in. I doubt even 1/2% of the electorate would change their voting intention in any given week without a big story, therefore it has to be such a gradual change as to be imperceptible until at least a month has gone by.

    So I don’t think a Yougov poll from 2 weeks ago is any less relevant to a Yougov poll from yesterday in calculating the current Lab lead.

  2. Norbold

    Pah – you can prove anything with facts.

  3. @Sine Nomine
    Another way of looking at that poll is to divide people into those wanting a Tory govt., and Labour govt. or neither.

    Doing that, we see that ‘the people’ want a coalition above either Tories or Labour in unfettered power.

    You have to face the fact that the Tory party is a minority party (as are Labour, the Lib Dems, UKIP and all) – none of our current parties can make any credible contention that they speak for ‘the people’, which is why it is so ridiculous when some (mostly right wing Tories) claim to…

  4. @Sine Nomine

    “So by a small majority the people (perhaps reluctantly) want to see a Majority Conservative Government in 2015”

    A more realistic statement might be:

    ‘Based on this poll, if forced to choose, more people want a Conservative majority than a Labour majority, but both groups are in the minority.’

    :)

  5. @Carfrew

    “Lol, after all the straw man guff, now you’re having a pop at my “tone”. Still, given you found nothing to challenge in the substance of my post, we can gladly move on…”

    You flatter yourself. There is plenty of substance to challenge. But the fact is that I have better things to do.

  6. Looks like Carney is already showing that he knows his history

    http://m.bbc.co.uk/news/business-23177463

    Implicitly saying he’s going to steer away from simple inflation-targeting?

    Nominal GDP targeting by the back door?

  7. @Statgeek – can’t really argue with that – spot on i would say.

    @BIGFATRON – so if thats the case why did so many people whinge and moan that ‘nobody’ voted for a coalition government!
    When in fact according to you they did indeed vote for a coalition Government – am I correct in that observation?

  8. LEFTY

    Sterling down on the statement.

    As one of the Comments says:- “Increased exports in the short run, lower mortgages in the long run . . . not a bad thing.”

    …………not if you forget to include “screwed savers”

  9. Colin

    We’re all in it together.

  10. I do appreciate AW’s attempts to exclude partisan comments as I feel it produces a better quality of debate .

    However, if the basis of the Conservative Parties criticism of Labour, which is of course directly intended to influence Voter Intention, is to rest on the allegation the Labour is A puppet of Trade Unions, as would equally be the case if Labour’s case against the Conservatives was that they are in the pockets of their wealthy paymasters, then it will be difficult not to at least refer to partisan issues all be it in a non partisan way!

    While the latter doesn’t appear to be generally the case (as yet) the former is becoming more and more central to the Tory Case. Possibly because of the influence of Lynton Crosby.

    Even Colin indicated that this was being overplayed.

  11. LEFTY

    Some are more in it than others-as someone once nearly said.

  12. There’s a lot of uninformed assumptions about the Falkirk affair. EM should eschew Mandelsonian siren voices to play to the Daily Mail gallery. There are few votes there for Labour. Many more will be lost if EM denies the current and thoroughly reasonable efforts being made by several major unions (not just those with traditional left wing leaders) to re-forge the links between TUs and the LP at individual member level in the constituencies. McClusky’s statement about getting more working class Labour MPs was couched provacatively. It probably had to be to get people to sit up and take notice. Just precisely what is wrong with the basic concept that in a democracy wealth earners AKA lower paid workers, who have seen their real wage levels mercilessly slashed for many years, should try to ensure that there are slightly fewer ex policy advisers, ex bankers, and ex party hacks representing them and one or two more who have done real jobs such as Alan Johnson and Denis Skinner from opposite sides of the LP ? Of course the silver spooned ones will try to make an issue of this but like EM I find the hypocrisy of a tiny male elite who still dominate our democracy 181 years after the 1st Reform Act hard to stomach. Stand up to it Ed !

  13. Regarding the validity of older data, I was considering adding in some form of weighting to the MAD data.

    Perhaps a weighting of 3 for the most recent two weeks, 2 for weeks 3 & 4, and 1 for weeks 5 & 6.

    e.g. the Con MAD for the weeks up to Sunday:

    1 & 2: 31.4%
    3 & 4: 30.0%
    5 & 6: 29.5%

    (3 x 31.4) + (2 x 30.0) + (29.5) = 183.7 (divided by 6 = 30.6%)

    The current MAD for the same six weeks by the normal method comes out at 30.1%. Bear in mind that each two week (10 polls) period will have it’s own MAD value, as opposed to calculating the MAD of the thirty polls over six weeks.

    Good idea? Bad idea?

  14. Colin

    They are that. My sympathies lay first with the million kids who have left school since the crash and are on the dole. They come first and the rest of us can form an orderly queue behind them.

  15. WELSH BORDERER

    I may well have misunderstood the whole thing-but I thought “Special Measures” had been invoked by the Central Labour Party in respect of the local Party in Falkirk because individual membership applications were submitted by Unite ( and not the individual) , with subscription fees paid by Unite ( and not the individual).

    If this is within Labour Party Rules, one wonders why Special Measures were imposed.

    If it was not within the rules , one wonders how it all came to be processed.

    And so far as involving lower paid workers in the democratic process is concerned-does the significant funding & campaigning support supplied by Unite to their chosen PPCs , not disadvantage Prospective Labour Candidates who are excluded from such patronage & support ?

  16. LEFTY

    agreed.

  17. STATGEEK

    Have you read AW’s weightings-the ones he uses to produce “UKPR Polling Average” ?

  18. So spending on illegal immigrants using the health service costs each taxpayer a little less than 1 penny a week, I can see that’s enough to start a revolution

  19. @Richard in Norway – its not JUST Illegal immigrants this is aimed at – and anyway its the whole Principal that matters.
    Hunts statement ended with the classic phrase which should sum it all up.
    ‘We have a National Health Service NOT an International Health Service’ – anyone care to disagree with that?

  20. ‘We have a National Health Service NOT an International Health Service’ – anyone care to disagree with that?

    Of course I disagree with Crosby’s soundbite. We need to coperate with the rest of the world to control disease, and we have a duty of care to people in the is country who fall ill or get hurt.

    It really isn’t good enough to treat “foreigners” who fall ill differently, ESPECIALLY if the efforts to prevent it (principled or otherwise) cost more than the saving and have other knock-on effects.

  21. What a load of tosh NICKP – you know full well that this has nothing at all to do with what you state!

    Its the main overriding principal that somehow we have to stop ‘just anyone’ coming here to make use of our fantastic NHS (for all its faults its still the best!)
    Its being abused to some tune and we probably only know a half of whats going on.
    Something had to be done just like the Welfare State situation – most people know these situations couldn’t carry on regardless.

  22. @Sine Nomine
    “Its being abused to some tune ”
    —————————
    In your own words – tosh.

    If ministers really want to make savings then they should concentrate their efforts on the areas where the greatest savings can be made.

    Possible target groups are : –

    1. Older folk who generally need more medical care than younger people. But would they dare risk the effects on VI among this group? I think not.

    2. Smokers – who cost the NHS a great deal due to self-infliceted damage.

    3. Heavy drinkers – ditto.

    4. Grossly overweight folk ( in cases where their condition is due to avoidable over-eating).

    5. Drug users.

    In all cases though what would be the effect on VI of asking the above to pay extra for their care ?

    In any case we are talking about a paltry £12 million nett p.a. ‘loss’. That is before taking into account the extra cost of administering the crackdown’ .

  23. @Colin

    Just found it.

    “Time: A poll will receive a weighting of 1 on the day the fieldwork is completed – after that the weighting decays by 0.05 a day, so that on the 21th day the poll is given a weighting of zero and dropped from the calculation.”

    Similar idea, but I doubt if I could work that into the MAD system, as we’re talking groups of polls. Food for thought though.

  24. @ Colin,

    Does the significant funding & campaigning support supplied by Unite to their chosen PPCs, not disadvantage Prospective Labour Candidates who are excluded from such patronage & support?

    Of course; that’s the point of providing support. It applies as much to any individual volunteering for their preferred candidate as it does for a union. If another candidate wants to enjoy the competitive advantage of my labour/Unite’s printer, they need to win my vote/Unite’s endorsement.

    Unite is so big and hegemonic that I think there is a valid concern about it crowding out other voices (understandably it’s the other affiliated unions who are most frustrated about this, along with Blunkett, Mandelson et al. who appear to be suffering flashbacks from 1970s PTSD), but your argument seems to be that supporting a candidate is in principle a corruption of the selection process. Isn’t figuring out which candidate commands the most support within the party the whole purpose of the selection process? Otherwise they could just pick a name out of a hat.

  25. @StatGeek

    You need a way to differentiate between old data that is needed to smooth out MOE, and old data that should be ignored because there has been an underlying event that has shifted VI.

    Is there a way to tell the 2 apart?

    We could start by looking at what events have shifted polls recently:

    http://upload.wikimedia.org/wikipedia/en/9/96/UK_opinion_polling_2010-2015.png

    The UKIP steps seem to correspond to by elections

    http://www.google.co.uk/trends/explore#q=ukip&date=6%2F2010%2032m&cmpt=q

    (Check news headlines box to help see what event triggered surge in web interest, then try and match that to the polling graphs)

    George Osborne seems to drive declines in Tory fortunes:

    http://www.google.co.uk/trends/explore#q=george%20osborne&date=6%2F2010%2032m&cmpt=q

    So as a start, it looks like you should use less history after a by election, and after a budget speech?

  26. Statgeek I like your current method as it smooths out any short term blip if you weight then a boost due to G8 or conferences get extra weights when 2 weeks later the impact is blown out or at least faded.

    Also very sensibly imo you drop off those with too much deviation if you weighted would not this be more complex as you have to do a mean for each week then eliminate outliers and then assume 4 (or 5 incl Sundays) at the average for each week and then weight as per formula?

  27. Tom Watson has stood down from the Shadow Cabinet.

    http://www.bbc.co.uk/news/uk-politics-23186851

  28. Yep, Labour’s Tom Watson resigns.

    Are Labour getting in a tangle here?

  29. Tom Who – as far as Jo/e Public concerned

    More people know the Golfer?

  30. I should add: the real problem with Labour’s selection process is how impenetrable it is to anyone who isn’t an insider. Openings are often poorly advertised and the selection windows are too short, so people who aren’t monitoring the party closely and ready to drop everything on a moment’s notice to campaign are excluded. Naturally this setup favours Labour apparatchiks and Unite candidates, because they’re the only people paying enough attention to navigate the system. That’s the thing that’s really blocking the selection of “normal people”.

    But it’s not a problem with Unite. Unite is a political pressure group; trying to influence candidate selections is part of their job. This is a problem with the Labour Party having bad internal processes.

  31. Nah. He’s got a conflict of interest with Falkirk I believe, so it’s probably right for him to do so. Better for Labour that it happens now than in 18 months’ time.

    It will be interesting to see who gets to run their GE campaign now.

  32. Never heard of Tom Watson must admit!

  33. MrNameless – FWIW James Forsyth had a piece earlier on today suggesting Miliband might be more likely to put Douglas Alexander in change of the GE campaign rather than Tom.

    Seems full of foresight now!

  34. FWIW,

    There is a genuine issue for all parties of wanting more members and even more so activists with in many cases councillors, ex councillors plus a few relatives and friends presently dominaningt.

    In the LP the necessary centralism started by NK but carried too far by TB/GB led to a depleted membership and has left most local parties very keen to embrace Trade Unionists as full members, especially younger ones.

    On the whole when a selection process is underway union backing is a double edged sword as those within the party resenting over influence (whilst welcoming influence) coalesce around another candidate.

    There is also a growing anti SPAD/career politician feeling within the party which whilst over colourfully expressed by Len McClusky is a legitimate concern.

    Falkirk is a bit embarrassing but frankly I am not sure how successful DC’s Lenfest will prove to me (cant mention PMQs of course)

  35. Spearmint crossed posts agree with you totally.

  36. Why can’t people assigned tasks do what they are supposed to I blame the Education system.

    Future generations of UKPR posters will be post-Gove reforms Educated and their New Thread Monitors not be as unreliable as ours.

    NEW THREAD

  37. I don’t see how anyone can criticize Labour for their union links and not take an equally big swipe at the Tory party being controlled by the city of London and while the influence of the city wide boys, CBI, institute of directors and other big business pressure groups, are not public like the union influence we all know it exists. Come on do we seriously believe that money donated to the Tory party doesn’t come with a few expectations!! And the lib dems are not clean in this matter either

  38. @RiN

    (since there’s a new thread, I might get away with this)

    “Tory party being controlled by the city of London and while the influence of the city wide boys”

    For ‘Tory’, read any political party in Government, as we saw in the nougties.

  39. Stratagem

    Yes that was the ‘new’ part in New Labour!

  40. Statgeek

    Yes that was the ‘new’ part in New Labour!

  41. It’s clear that those polls with large UKIP support, also have bigger Labour leads, those with less UKIP support smaller Labour leads.
    Could that be caused by UKIP taking Tory support in some polls?

  42. From a previous thread a week ago (sorry folks, but it;s about polling/stats)…

    @Richard

    “You need a way to differentiate between old data that is needed to smooth out MOE, and old data that should be ignored because there has been an underlying event that has shifted VI.”

    Yes, it’s not easy getting a balance between data age and data value/weight, and it’s harder if we’re trying to eliminate outliers from a data set. I don’t so much look to use ‘old data’, so much as not disclude it, if that makes sense. YG have five polls per week, so I prefer multiples of five. Due to how I setup the original spreadsheet, I tend to use 30s, and while I could probably shift to 20s, there’s a certain roundness to 30 (divisible by five, six, ten and so on). What’s more, from time to time the YG poll for a Sunday rounds off a calendar month, and even better, it occasionally happens at the end of a group of thirty on the spreadsheet. Neatness, if you like.

    In fact I just had a peek, and noticed that over 30 polls, the average total samples is 53,000, which provides MoE of +/- 0.43%

    Comparing the MAD data versus the simple average for the grouped 30s for the same period, dated 31st May (Party / MAD / 30s):

    Con: 30.0 : 30.4
    Lab: 39.2 : 39.4
    Lib 10.2 : 10.2
    UKIP 14.1 : 13.6
    Green 2.0 : 2.3

    So arguably, we could just group the polls by 30s, average them, and apply the 0.43 caveat, and most of those values are within MoE (UKIP are not, but they are regionally different, and we’re into crossbreak MoE there).

    @AW

    There’s a thought. Can we even do that? If I have two polls of 1000, can I apply MoE on the average of the polls based on the total samples, or should it be the average of the samples (in this case 1000)?

    @JimJam

    Yes, I like the way it can prevent temporary blips from becoming the focus of things, but as we’ve seen recently with UKIP, blips can become trends, and it takes 2-4 weeks for the MAD method to catch up, which sometimes seems self-defeating.

    I have considered creaking them to two-week and one-week sets, but as you say, this becomes problematic. There’s a chance that in a week’s worth of data, you’ll get no outlier, where ti would be over six weeks. Or worse, there’s a big variation in a particular week and values within MoE for the previous week are now excluded, due to the MAD being calculated over a small data set.

    I have considered adding in a Margin of Error for a given set (see question posed to AW), but I would need to ensure the total samples excluded the samples for outliers (for a given party). Not easy without adding to the confusion. Ultimately we get to a point where there’a an awful lot of data manipulation, just to go from say 30.0% to 30.5%. There comes a point where it’s a pointless exercise.

1 2 3 4