House effects

A lot of the points I made in my essay on how not to report polls boiled down to not taking a poll in isolation. Not making the outlier the story, only comparing apples to apples, not cherry picking – they all boil down to similar things, especially on voting intention.

In the last couple of days I’ve watched people getting overexcited over two polls. Yesterday’s ICM poll provoked lots of Tory excitement on Twitter and comments about the Labour lead falling and it being a terrible poll for Labour and so on. ICM’s poll, of course, did not show Labour’s lead falling at all – it showed it steady for the fourth month in a row. ICM’s methodology merely produces consistently lower leads for Labour due to their methodological approach. Saturday night had the usual flurry of excitable UKIP comments on Twitter about being on the rise and being the 3rd party after the Survation poll was published, conveniently ignoring the fact that 95% of polls this year have had them in fourth – often by a very long way. There was, needless to say, no similar excitement over UKIP being on 4%, 11 points behind the Lib Dems, in the ICM poll yesterday.

Different pollsters have different approaches, on things like weighting, likelihood to vote, how they deal with don’t knows, how they prompt and so on. While all the pollsters are politically neutral, these do have some consistent partisan effects – for example, ICM’s methods tend to produce the highest levels of support for the Liberal Democrats, YouGov’s methods tend to produce the lowest levels of support for the Liberal Democrats. The graph below shows an estimate of the partisan house effects of each polling company’s voting intention methodology, calculated by comparing each company’s poll results to the rolling average of the YouGov daily poll (1)

YouGov, ICM and ComRes’s online polls tend to show the highest shares of the vote for the Conservative party. However, in the case of YouGov this is cancelled out by a tendency to also show the highest levels of support for Labour, so the result is that ICM show the lowest Labour leads while YouGov tend to show some of the highest Labour leads after Angus Reid and TNS. For the Liberal Democrats, ICM show far higher support for the party than any other company, averaging at plus 3.3 points. Next highest is Survation and ComRes’s telephone polls. At the opposite end of the spectrum YouGov tend to show significantly lower Liberal Democrat support.

It would take a much longer post to dissect the full methodology of each pollster and the partisan implications, but to pick up the general methodological factors that contribute to the house effects:

How pollsters account for likelihood to vote. Some companies like YouGov and Angus Reid do not take any account of how likely people say they are to vote away from elections(2). Companies like ICM and Populus weight by how likely people say they are to vote, so that people who say they are 10/10 certain to vote count much more than someone who says their chances of voting are only 5/10. At the opposite end of the scale from YouGov, Ipsos MORI include only those people who are 10/10 certain to vote, and exclude everyone else from their topline figures. Other twists here are ICM, who also heavily downweight anyone who says they didn’t vote in 2010, and ComRes, who use a much harsher likelihood to vote question for people voting for minor parties than for the big three. Most of the time Conservative voters say they are more likely to vote than Labour voters, so the more harshly a pollster weights or filters by likelihood to vote the better it is for the Tories.

How pollsters deal with don’t knows. Somewhere around a fifth of people normally tell pollsters they don’t know how they would vote in an election tomorrow. Some pollsters like YouGov simply ignore these respondents. Some like MORI ask them a “squeeze question”, something like “which party are you most likely to vote for?”. Others estimate how those people would vote using other information from the poll, such as party ID (ComRes) or how those people say they voted at the previous election (ICM and Populus). These adjustments tend to help parties that have lost support since the last general election – so currently ICM and Populus’s adjustment tends to help the Liberal Democrats and, to a lesser extent, the Conservatives. In past Parliaments it has helped the Labour party.

How the poll is conducted. About half the current regular pollsters do their research online, about half do it by telephone. While there is no obvious systemic difference between online and telephone polls in terms of support for the Conservatives, Labour and Liberal Democrats there is a noticable difference in support for UKIP, with polls conducted online consistently showing greater UKIP support. This may be to do with interviewer effect, with respondents being more willing to admit supporting a minor party in an online poll than to a human interviewer, or may be something to do with sampling.

How the poll is weighted. Almost all pollsters now use political weighting of some sort in their samples. In the majority of cases this means weighting the sample by how people said they voted at the last election – i.e. we know 37% of people who voted in Great Britain in 2010 voted Tory, so in a representative sample 37% of people who say they voted at the previous election. It isn’t quite as simple as that because of false recall – people tend to forget their vote, or misreport voting tactically, or claim they vote when they didn’t actually bother, or align their past behaviour with their present preferences and say how they wish they had voted with hindsight. Most pollsters estimate some level of false recall in deciding their weighting targets, Ipsos MORI reject it on principle with the effect that proportionally their samples tend to contain slightly more people who say they voted Labour at the last election, and somewhat fewer who say they voted Lib Dem.

How the poll is prompted. As discussed at the weekend, almost all companies prompt their voting intention along the lines of Conservative, Labour, Lib Dem, Scots Nats/Plaid if appropriate and Other. Survation also include UKIP in their main prompt, leading to substantially higher UKIP support in their polls.

All these factors interact with one another – so you can’t look at one in isolation. For example, MORI’s sample tends to be a bit more Labour than other parties, but their turnout filter is harsher than most other companies which disadvantages Labour and cancels out the pro-Labour effect of not weighting by past vote. ComRes’s online polls tend to find a higher level of UKIP support than many other companies, but their harsh filter on likelihood to vote for other parties cancels this out. They also change over time – so while re-allocation of don’t knows currently helps the Lib Dems, in past years it has helped Labour (and when originally introduced in the 1990s helped the Tories.)

Inevitably the question arises which polls are “right”. The question cannot be answered. Come actual elections polls using different methods all tend to cluster together and show very similar results – polls have a margin of error of plus or minus 3%, so judging which methodology is more accurate based on one single poll every five years when all the companies are within the 3% margin of error is an utter nonsense.

Realistically it a more a philosophical question than a methodological one – the reason pollster show different figures is that they are measuring different things. YouGov don’t make second guesses about don’t knows and assume everyone who says they vote will. Their figures are basically how people say they would vote tomorrow. In comparison ICM weight by how likely people say they are to vote, assume people who didn’t vote last time are less likely to do so than they say they are, and make estimates of how people who say don’t know would actually vote. Their figures are basically how ICM estimate people would actually vote tomorrow. They are two different approaches, and there is not right answer as to which one to take. Shouldn’t a pollster actually report what people say they’d do, rather than making second guesses about what they’d really do? But if a pollster has good reason to think that people wouldn’t behave how they say they will, shouldn’t they factor that in? No easy answer.

Given these differences though, when you see a poll, it is important to remember house effects and to look at the wider trends. A poll from ICM showing a smaller Labour lead than in most other companies’ polls isn’t necessarily a sign of some great collapse in Labour’s lead, it’s more likely because ICM always show a smaller Labour lead than other companies (ditto a great big Labour lead in an Angus Reid poll). That said, even a big Labour lead from ICM or a small Labour lead from Angus Reid shouldn’t get people too excited either, as any single poll can easily be an outlier. As ever, the rule remains to look at the broad trend across all the polls. Do not cherry pick the polls that tell you what you want to hear, do not try to draw trends from one company to another when they use different methods and don’t get overexcited by single outlying polls.

(1)House effects were calculated by using the daily YouGov poll as a reference point. I took a rolling 5 day average of the YouGov daily poll, and compared that to each poll from another company. This was used to calculate each company’s average difference from the YouGov daily poll. Then it was calibrated to the average for difference for each party, so that YouGov wasn’t automatically the mid-point!)

(2) YouGov do take into account likelihood to vote during election campaigns, using roughly the same approach as Populus


99 Responses to “House effects”

1 2
  1. Exceptionally helpful, and will be ignored by all political journalists as a result.

  2. Hopefully can serve as a cut and paste job for other purposes Anthony. I would hate to think you did it just for us, as I think we were already ‘there’.

  3. @Carfrew
    ‘The no-brainer response to the Tory gambit really has to be centred on ideas like:
    – you don’t pay off your debts by killing off your income.
    – If you’re in debt, you don’t cut costs by selling your car so that you can’t work any more
    – It’s not wasteful spending, it’s investment to earn more (etc)’

    Apply to EM for a consultancy job (you clearly can do it in your sleep).

  4. @AW

    I’d just add a couple of things:

    * being in the centre of the graph (Populus, ComRes, Ipsos Mori) doesn’t make them ‘better’ polling firms.
    * the variability of the position on the graph is also interesting – the extent to which a polling company is consistently different rather than randomly different

    Personally I’m always worried about stratified sampling that is done to ‘correct’ data, I know it can be necessary but I tend to prefer YouGov’s minimalist approach to some of the other methodologies.

  5. The ICM poll is probably a pretty fair poll, being bad for Labour, if Labour got 39% at the GE, I don’t think they’d consider it to be too bad.

    As for UKIP could be a wild card, it may be that UKIP will do better in some parts of the country than others, so may be difficult to plot. For instance can see UKIP doing well in the West Country and other rural areas, but not perhaps in the urban or metropolitan area.

  6. David

    The effect of parties that only stand in some regions PC, SNP, UKIP, Green can’t be ignored especially if they are more than just marginal (SNP) or target seats (UKIP) as a spoiler for what would be the winner in a PR system.

  7. @Oldnat (and others)

    I’ve addressed your concerns about regional snobbishness on the previous post on the previous post.

    @Anthony Wells

    Excellent as always, thank you.

    Regards, Martyn

  8. Splendid as ever Anthony.

    You have infinite patience.

  9. Martyn

    I complimented you on your response on the last thread.

    Anthony

    Excellent summary, and I wholly agree with Alec!

  10. @AW

    Good article as always and I have learned a lot from coming on here where you get an informed view rather than those dodgy headlines.

    However the following comment seems more like the pollsters union coming into play!!!

    “Inevitably the question arises which polls are “right”. The question cannot be answered.”

    Obviously I take on board what you say further in that paragraph that basically because there is not an election tomorrow we will never know which is right. Also by the time the election arrives the ‘don’t knows’ gradually disappear and the margin of doubt becomes less with polling organisations coming closer together.

    However taking the obvious difference of any of them which is ICM with the Lib Dems I think there are certain ways we can work out that ICM is not right. When it comes to local election results I learned on here that Lib Dems sewem to outpoll their National Election results by 50%. So if ICM says they are on 15% and they get 15% in the locals then this suggests a vote of around 10% in the National Elections. I would argue therefore that in this case ICM is not right. I am sure there are other comparisons that could be done for the other parties/polling organisations. Certain real results do not seem to have Labour on 40% (a bit lower) although the Labour vote at local compared to national elections may be harder to estimate and may vary more.

    To be honest with you my gut feeling is UGov is getting closer than the others.

  11. Personally, I think the ‘true’ lead is probably somewhere between Yougov and ICM (i.e. around 7 to 8 points).

    ICM allocates ‘don’t knows’ in a way that is perhaps less relevant at present given the Lib collapse, but Yougov doesn’t currently weight polls according to likelihood to vote until a GE/election – weighting in this way penalises Labour but makes it more realistic IMO. That’s probably why there is more of a coming together very close to GE/local elections.

  12. I’m slightly staggered by the complete lack of respect shown by Yeo to his leader and PM today. The ‘man or mouse’ bit has got the sketch writers doing cartwheels, but it’s the rest of it that is so peculiarly damaging and illuminating. Yeo seems to be saying that his own back benchers don’t know what he stands, which I would suggest is a pretty damning indictment for someone who has led them for 7 years.

    The slightly odder side of this is that Yeo is calling for Cameron to show his true self and be a man by coming out for a third runway, after making a cast iron pledge not to do this.

    And Yeo wonders why people don’t know what Cameron stands for?

  13. This was great Anthony.

    A few questions

    1. Over what time period did you do the analysis – since the last GE?
    2. If you plotted the actual differences for each pollster around each point average does it show much variation? And if not…
    3. Should we now ‘correct’ every poll into ‘average polling units’?

  14. Looking at longterm trends one needs to keep an eye out for any methodology changes.

    A small one last year (November?) which knocked maybe a point off the Labour VI on YouGov.

    Have other companies changed their method since the 2010 GE?

  15. Does the ComRes differential suggest that there is still a shy-tory factor at work in some/most polls?

    Or is the difference between their phone and online polls purely down to ‘other things’`?

  16. Alec

    Haven’t seen much of the news today, so your post has put this tweet in context.

    Jeff Breslin [email protected]
    Wee, sleekit, cow’rin, Tim Yeo beastie, O, what a panic’s in thy breastie!

  17. http://www.bbc.co.uk/news/uk-politics-19399255

    light relief.

    Man or mouse will run and run………..

  18. @PaulCroft

    It’s a pity he didn’t go for ‘man or a Muppet’ – it has a much catchier tune… (If he’s a man, he’s a Muppet of a man)

  19. Matthew – no, since Jan 2011 (as a convenient starting point after which Labour have been consistently ahead). As I said above, the effect of some methodological differences depends on political circumstance, so the house effects could have been different before Labour moved ahead and before the Lib Dem collapse. And no – don’t “correct” polls. It implies that the mid-point is probably correct, when this is not necessarily the case.

    Billy Bob – yes, in many little ways. Off the top of my head:

    * ICM have started including mobile phones in their sample and started weighting down people who did not vote in 2010.
    * ComRes have started reallocating don’t knows according to party ID, have started filtering “others” more harshly than the main parties on likelihood to vote and have switched half their fieldwork to online.
    * TNS have switched from face-to-face to online polling
    * YG have moved to weighting SNP/PC and other minor parties separately, and adjusted sampling to ensure there are more under 30s with low educational levels

    And all pollsters but MORI have essentially changed their political weights as they now weight to 2010 recall/ID rather than 2005 recall/ID – the formulas they use to work out what to weight to may be the same, but they are essentially new estimates.

    That Old Bloke – there may be a shy Tory factor at work (after all, that is part of the theory behind the reallocation of don’t knows that once again aids the Tories), but you can’t conclude it from the ComRes poll as doing the survey online brings with it far more differences than just the absence of interviewer effect, it changes the sampling, the way the data for weighting is collected, etc.

  20. Shouldn’t there be a YouGov?

  21. NICKP

    “There’ll always be a YouGov, and YouGov will be free” (though not to its clients)!

  22. There may not be a YG until tomorrow, as they don’t like doing polling on bank holidays.

  23. R HUCKLE

    You guys have ANOTHER Bank Holiday?

  24. Results from latest Sun / @YouGov survey including vote intention will be published at 7.00am tomorrow and not 10.00pm tonight
    —————–
    Unfortunately, message not with a date; so we do not know if this is today’s poll or last week’s one which was held until 7am.
    8-)

  25. Great, thanks AW.

  26. Amber:

    any ole one will do

    [finks “must kick the habit”]

  27. Excellent analysis of the LD situation from Peter Kellner.

    Really interesting.

    http://yougov.co.uk/news/2012/08/28/lib-dems-lament/

  28. oldnat:

    yes, we need regular bank hols down here – its still glorious summer in England.

  29. ALEC

    Guido has thoughts on Yeo’s real objectives.

    He is not a million miles from the truth I think.

    How Yeo gets to chair the Environmental SC is beyond me.

  30. PAULCROFT

    Ha!

    We had our summer today. Always a bit difficult to predict on which day it will happen, but always to be enjoyed. :-)

  31. What a good article. I cannot believe people really getting excited and taking to twitter and blogs over individual opinion polls (taken years from any general election) as if this was in any way earth shattering news! I think AW says it all-
    “….the rule remains to look at the broad trend across all the polls. Do not cherry pick the polls that tell you what you want to hear, do not try to draw trends from one company to another when they use different methods and don’t get overexcited by single outlying polls”.

  32. I particularly enjoyed the usee names on Guido’s site in the bit about Yeo.

    “Scum finder” stood out but all were much more inventive than my own.

  33. I guess people like to delude themselves. For example I heard various Republican bods telling us about how a poll had put them ahead today. Of course they didn’t mention the 2 which found the President slightly ahead. There is a natural tendency to draw attention – either one’s own or that of other people – to the most optimistic interpretation available. I do think that it’s about time the Guardian considered other polls rather than just the ones which they commission.

  34. BARNABY MARDER

    “I guess people like to delude themselves”

    What other explanation can there be for people being loyal to a group of professional politicians in a party – regardless of how different the philosophies and policies might be?

  35. AW- very informative with great clarity.

    “….the rule remains to look at the broad trend across all the polls. Do not cherry pick the polls that tell you what you want to hear, do not try to draw trends from one company to another when they use different methods and don’t get overexcited by single outlying polls”

    should be pinned to the start of *every* thread on ‘Political Betting’ !

  36. There were a couple of remarks following mine which tended to indicate that we did need your article. Comments like ‘seems probable that it may be correct’ type do not impress.

    It seems to me that, as the environmental balances about the choices for airline terminals have not been sufficiently investigated, let alone the basis on which such would be assessed, the decisions must be purely political. in other words they will not be based upon fact or any explanations of which facts are chosen and what weight could be given to them.

    I doubt whether such issues play heavily except locally where the hurt is to be most experienced. Time for John Mcdonnell to swing the mace I suspect.

  37. @Colin

    Thank you for the Kellner link, one of your many so offered.

    Yes it was indeed a good analysis, being based on polling fact, however sometimes unpalatable to behold.

    I suppose having played X Factor politics to save those LD seats and get government jobs, there was only one way after that – downhill. However, the voters’ collective opprobrium for this man is, i feel, as inappropriate as it was for Ted Heath. he acted in what he thought was the best interests of the state (and his party), and will be thanked with rejection by both.

    Peter Kellner points out that there are still three years, so let’s see if it stays that way.

  38. Does anyone remember a classic example during the 1997 election campaign: after a couple of years of consistently big Labour leads, there was an ICM poll with a 5% lead. Cue the headline “Labour lead in freefall” – I can’t remember which newspaper. Of course, we now know it was nothing of the sort.

    It seems little has changed.

  39. KEITHP

    “It seems little has changed.”

    I doubt that it ever will. polling companies aren’t charitable organisations. They are commercial companies who make money by asking the questions that interest groups want asked.

    Interest groups will put their own slant on any set of figures, and polls are no different.

    At least with polls by BPC members we get to see the actual numbers.

  40. Howard:

    I strongly agree with you re Ted Heath. I felt he was an honourable man who had a long record of service to this country.

    The problem many of us on the left have with Clegg is of a different nature. I’m sure he felt he was doing the right thing for the country but he has actually made it feel and look like a permanent taking of sides. I think that has been a huge political mistake.

  41. @howard

    Thank you kindly Howard, but I thought your recent comment about NC showed remarkable insight and Lprescience:
    :
    “He’ll fail, due to circumstances beyond his control (the economy), but he didn’t know that when he set out on his journey.”

  42. Although on reflection, the Lib Dems did seem rather more aware of the pitfalls of austerity prior to the election.

    Still, on the bright side, given all that’s happened, is there any indication of a shy LD effect to go along with the shy Tory thing?

  43. Love the editor’s note in this story – “Editor’s Note: England/Great Britain references clarified.”

    http://news.yahoo.com/blogs/technology-blog/meteor-explodes-over-england-violent-force-212434670.html

    Obviously the original story placed Wales in England.

    Geography, though, is probably irrelevant. This must have been God declaring either opposition or support for equal marriage laws.

  44. @Howard – ” …opprobrium for this man”

    I don’t want to be too uncharitable, but there people within the party (Steel for example over the Lords Bill – many others over his handling of AV) who have questioned his aptitude – one has to wonder whether it could ever keep pace with his ambition.

    He certainly agitated against Kennedy within months of becoming an MP… and as some have suggested, acted the ‘young cardinal electing an old Pope’ in his backing for Campbell – that is until the time came to declare his own ambition to be leader and thereby hasten Campbell’s demise.
    Maiden speech to party leader in little over two years is not bad going.

  45. US polling is likely to increasingly demonstrate AW’s points in the run up to the election.

    While I understand that pollsters tend to be partisan over there, this article interested me –

    http://www.dailykos.com/story/2012/08/28/1124880/-Daily-Kos-SEIU-State-of-the-Nation-poll-Barack-Obama-takes-six-point-lead-over-Mitt-Romney

    “You might look to the partisan composition of this survey, which is unusually blue: A very high 43 percent of voters describe themselves as Democrats, while 36 percent call themselves Republicans. That’s actually a pretty high tally for the GOP, too, but this is a good example of how fluid party ID is: That late July poll which had the race tied also featured a seven-point edge for Democrats. It’s also a good example of why you generally shouldn’t put a whole lot of stock in part self-identification. But it’s not irrelevant, either.”

    Given YG’s reliance on self-identification, could this be an issue in GB polling too?

  46. Does “pollster” mean the same thing in the USA as here?

    ““We’re not going to let our campaign be dictated by fact-checkers,” says Neil Newhouse, a Romney pollster. ”

    http://robertreich.org/#.UD1OVH4KHGk.twitter

  47. @Barnaby Marder

    I do think that it’s about time the Guardian considered other polls rather than just the ones which they commission.

    ____________________________________

    The Guardian are rather wedded to the LDs though…

  48. @ Old Nat

    “US polling is likely to increasingly demonstrate AW’s points in the run up to the election.

    While I understand that pollsters tend to be partisan over there, this article interested me –

    http://www.dailykos.com/story/2012/08/28/1124880/-Daily-Kos-SEIU-State-of-the-Nation-poll-Barack-Obama-takes-six-point-lead-over-Mitt-Romney

    “You might look to the partisan composition of this survey, which is unusually blue: A very high 43 percent of voters describe themselves as Democrats, while 36 percent call themselves Republicans. That’s actually a pretty high tally for the GOP, too, but this is a good example of how fluid party ID is: That late July poll which had the race tied also featured a seven-point edge for Democrats. It’s also a good example of why you generally shouldn’t put a whole lot of stock in part self-identification. But it’s not irrelevant, either.”

    Given YG’s reliance on self-identification, could this be an issue in GB polling too?”

    Not neccessarily. You guys don’t have the same sort of registration by party that we do here. As you guys have explained to me, you don’t register with any of your parties. You’re only a member of your party if you actually become a member and pay a membership fee. When it comes to U.S. polling, self-identification (rather than going by actual voter registration) has been proven to be far more effective. That’s because people have their political attitudes and views shift over the years but they don’t neccessarily bother to reregister because honestly it doesn’t mean that much. So to give you an example, if you had the same sort of system in the UK, when you first registered to vote in the 1960’s, you might have registered as a Liberal and even as your attitudes changed, you might not have bothered to reregister as a Nat. If you were conducting a poll based upon voter registration, you would be counted as a Lib Dem supporter of Alex Salmond. But if you were conducting a poll based upon voter self-identification, you would be counted as a Nat.

    Where the party make up of the poll becomes important is when you look at the make up of voters in exit polls from the last election. So some of these polls will vary from the norm (and they’ll be used to supply media narratives) even though their party id is way off from what will actually show up at the election.

    U.S. polls in 2010 were actually off by quite a bit. No one noticed since it didn’t really matter anyway but had polls been correct, a whole lot more Republicans would have been elected than were. These polls undersampled Democrats, undersampled younger voters, and dramatically undersampled minorities.

    That can throw off the accuracy of polls. So it’s not that self-identification is irrelevant. It’s that it only tells part of the story and it’s an estimation.

  49. @ Old Nat

    “Does “pollster” mean the same thing in the USA as here?

    ““We’re not going to let our campaign be dictated by fact-checkers,” says Neil Newhouse, a Romney pollster. ””

    I think so. What does pollster in the UK mean?

    Btw, I know I’m biased but I’m watching the RNC tonight and I have to say, this is a sh*tshow. These speeches are awful, pure dreck or as you Scots like to say, pure keech. :)

    I’m getting ot here but I have to say that Ann Romney is SO unlikeable. I don’t care that her dad was a Welsh coalminer. I mean Cindy McCain, she was likeable (I could have voted for her). She was a wealthy woman but she was down to earth. She was smart. She was ruthless. She seemed like a woman who could take care of herself and had married her husband only because she found him attractive and had a thing for military guys. Not because she needed to marry him but simply because she was into it. She was likeable.

    Even Laura Bush was likeable! She reminded me of Patty Hearst in the SLA (got caught up in a bad situation with bad people and didn’t really know better).

    But this woman……she reminds me of a stuck up, elitist, nasty, bourgie b***h who has no idea of how the real world works nor does she care to learn.

  50. YouGov –
    Lab 44, Con 32, Lib 10, UKIP 8
    I suspect it will be reported in certain circles as ‘Labour back in double figures!’ as opposed to ‘Changes within MOE’.

    Interesting that Clegg is talking about short-term wealth taxes on the rich, given that the Tory right is calling loudly for tax cuts for the rich.
    Perhaps not the best thing to do to your coalition partner – have them trapped even more between party right and junior partner.
    Would be a good plan if you were looking to have a Lib/Lab coalition in the future though.

1 2