Polls often give contrasting results. Sometimes this is because they were done at different times and public opinion has actually changed, but most of time that’s not the reason. A large part of the difference between polls showing different results is often simple random variation, good old margin of error. We’ve spoken about that a lot, but today’s post is about the other reason, systemic differences between pollsters (or “house effects”).

Pollsters use different methods, and sometimes those different choices result in consistent differences between the results they produce. One company’s polls, because of the methodological choices they make, may consistently show a higher Labour score, or a lower UKIP score, or whatever. This is not a case of deliberate bias – unlike in the USA there are not Conservative pollsters or Labour pollsters, every company is non-partisan, but the effect of their methodological decisions mean some companies do have a tendency to produce figures that are better or worse for each political party – we call these “house effects”.

2014houseffects

The graph above shows these house effects for each company, based upon all the polls published in 2014 (I’ve treated ComRes telephone and ComRes online polls as if they are separate companies, as they use different methods and have some consistent differences). To avoid any risk of bias from pollsters carrying more or less polls when a party is doing well or badly I work out the house effects by using a rolling average of the daily YouGov poll as a reference point – I see how much each poll departs from the YouGov average on the day when its fieldwork finished and take an average of those deviations over the year. Then I take the average of all those deviations and graph them relative to that (just so YouGov aren’t automatically in the middle). It’s important to note that the pollsters in the middle of the graph are not necessarily more correct, these differences are relative to one another. We can’t tell what the deviations are from the “true” figure, as we don’t know what the “true” figure is.

As you can see, the difference between the Labour and Conservative leads each company show are relatively modest. Leaving aside TNS, who tended to show substantially higher Labour leads than other companies, everyone else is within 2 points of each other. Opinium and ComRes phone polls tend to show Labour leads that are a point higher than average, MORI and ICM tend to show Labour leads that are a point lower than average. Ashcroft, YouGov, ComRes online and Populus tend to be about average. Note I’m comparing the Conservative-v-Labour gap between different pollsters, not the figures for each one. Populus, for example, consistently give Labour a higher score than Lord Ashcroft’s polls do… but they do exactly the same for the Conservatives, so when it comes to the party lead the two sets of polls tend to show much the same.

There is a much, much bigger difference when it comes to measuring the level of UKIP support. The most “UKIP friendly” pollster, Survation, tends to produce a UKIP figure that is almost 8 points higher than the most “UKIP unfriendly” pollster, ICM.

What causes the differences?

There are a lot of methodological differences between pollsters that make a difference to their end results. Some are very easy to measure and quantify, others are very difficult. Some contradict each other, so a pollster may do something that is more Tory than other pollsters, something that is less Tory than other pollsters, and end up in exactly the same place. They may interact with each other, so weighting by turnout might have a different effect on a phone poll from a telephone poll. Understanding the methodological differences is often impossibly complicated, but here are some of the key factors:

Phone or online? Whether polls get their sample from randomly dialling telephone numbers (which gives you a sample made up of the sort of people who answer cold calls and agree to take part) or from an internet panel (which gives you a sample made up of the sort of people who join internet panels) has an effect on sample make up, and sometimes that has an effect on the end result. It isn’t always the case – for example, raw phone samples tend to be more Labour inclined… but this can be corrected by weighting, so phone samples don’t necessarily produce results that are better for Labour. Where there is a very clear pattern is on UKIP support – for one reason or another, online polls show more support for UKIP than phone polls. Is this because people are happier to admit supporting UKIP when there isn’t a human interviewer? Or it is because online samples include more UKIP inclined people? We don’t know

Weighting. Pollsters weight their samples to make sure they are representative of the British population and iron out any skews and biases resulting from their sampling. All companies weight by simple demographics like age and gender, but more controversial is political weighting – using past vote or party identification to make sure the sample is politically representative of Britain. The rights and wrongs of this deserve an article in their own right, but in terms of comparing pollsters most companies weight by past vote from May 2010, YouGov weight by party ID from May 2010, Populus by current party ID, MORI and Opinium don’t use political weighting at all. This means MORI’s samples are sometimes a bit more Laboury than other phone companies (but see their likelihood to vote filter below), Opinium have speculated that their comparatively high level of UKIP support may be because they don’t weight politically and Populus tend to heavily weight down UKIP and the Greens.

Prompting. Doesn’t actually seem to make a whole lot of difference, but was endlessly accused of doing so! This is the list of options pollsters give when asking who people vote for – obviously, it doesn’t include every single party – there are hundreds – but companies draw the line in different places. The specific controversy in recent years has been UKIP and whether or not they should be prompted for in the main question. For most of this Parliament only Survation prompted for UKIP, and it was seen as a potential reason for the higher level of UKIP support that Survation found. More recently YouGov, Ashcroft and ComRes have also started including UKIP in their main prompt, but with no significant effect upon the level of UKIP support they report. Given that in the past testing found prompting was making a difference, it suggests that UKIP are now well enough established in the public mind that whether the pollster prompts for them or not no longer makes much difference.

Likelihood to vote. Most companies factor in respondents likelihood to vote somehow, but using sharply varying methods. Most of the time Conservative voters say they are more likely to vote than Labour voters, so if a pollster puts a lot of emphasis on how likely people are to actually vote it normally helps the Tories. Currently YouGov put the least emphasis on likelihood to vote (they just include everyone who gives an intention), companies like Survation, ICM and Populus weight according to likelihood to vote which is a sort of mid-way point, Ipsos MORI have a very harsh filter, taking only those people who are 10/10 certain to vote (this probably helps the Tories, but MORI’s weighting is probably quite friendly to Labour, so it evens out).

Don’t knows. Another cause of the differences between companies is how they treat people who say don’t know. YouGov and Populus just ignore those people completely. MORI and ComRes ask those people “squeeze questions”, probing to see if they’ll say who they are most likely to vote for. ICM, Lord Ashcroft and Survation go further and make some estimates about those people based on their other answers, generally assuming that a proportion of people who say don’t know will actually end up voting for the party they did last time. How this approach impacts on voting intention numbers depends on the political circumstances at the time, it tends to help any party that has lost lots of support. When ICM first pioneered it in the 1990s it helped the Tories (and was known as the “shy Tory adjustment”), these days it helps the Lib Dems, and goes a long way to explain why ICM tend to show the highest level of support for the Lib Dems.

And these are just the obvious things, there will be lots of other subtle or unusual differences (ICM weight down people who didn’t vote last time, Survation ask people to imagine all parties are standing in the seat, ComRes have a harsher turnout filter for smaller parties in their online polls, etc, etc)

Are they constant?

No. The house effects of different pollsters change over time. Part of this is because political circumstances change and the different methods have different impacts. I mentioned above that MORI have the harshest turnout filter and that most of the time this helps the Tories, but that isn’t set in stone – if Tory voters became disillusioned and less likely to vote and Labour voters became more fired up it could reverse.

It also isn’t consistent because pollsters change methodology. In 2014 TNS tended to show bigger Labour leads than other companies, but in their last poll they changed their weighting in a way that may well have stopped that. In February last year Populus changed their weights in a way that reduced Lib Dem support and increased UKIP support (and changed even more radically in 2013 when they moved from using the telephone to online). So don’t assume that because a pollster’s methods last year had a particular skew it will always be that way.

So who is right?

At the end of the day, what most people asking the question “why are those polls so different” really want to know is which one is right. Which one should they believe? There is rarely an easy answer – if there was, the pollsters who were getting it wrong would correct their methods and the differences would vanish. All pollsters are trying to get things right.

Personally speaking I obviously I think YouGov polls are right, but all the other pollsters out there will think the same thing about the polling decisions they’ve made and I’ve always tried to make UKPollingReport about explaining the differences so people can judge for themselves, rather than championing my own polls.

Occasionally you get an election when there is a really big spread across the pollsters, when some companies clearly get it right and others get it wrong, and those who are wrong change their methods or fade away. 1997 was one of those elections – ICM clearly got it right when others didn’t, and other companies mostly adopted methods like those of ICM or dropped out of political polling. These instances are rare though. Most of the time all the pollsters show about the same thing, are all within the margin of error of each other, so we never really find out who is “right” or “wrong” (as it happens, the contrast between the level of support for UKIP shown by different pollsters is so great that this may be an election where some polls end up being obviously wrong… or come the election the polls may end up converging and all showing much the same. We shall see).

In the meantime, with an impartial hat on all I can recommend is to look at a broad average of the polls. Sure, some polls may be wrong (and it’s not necessarily the outlying pollster showing something different to the rest – sometimes they’ve turned out to be the only one getting it right!) but it will at least help you steer clear of the common fallacy of assuming that the pollster showing results you like the most is the one that is most trustworthy.


231 Responses to “All about house effects”

1 2 3 5
  1. @AW

    Interesting. Now if you would only be so kind as to circulate to the media. There’s a good chap!

  2. And that folks is why we come here.

    To listen to someone who knows what they are talking about.

    Peter.

  3. Anthony – Am I right to think that YouGov grows its online panel ‘organically’ whereas a number of the new online pollsters buy them in from generic market research/survey companies? If that is the case then there has to be a big question mark re panel composition… As Mark Pack pointed out recently, the difference between political polling and other market research is that polls have the real world test of elections, whereas other market research doesn’t…

    So basically we have everyone (besides TNS pre-their changes) within a point of the average for the lead, but still huge differences on UKIP (likewise other smaller parties).

  4. @AW
    “Personally speaking I obviously think YouGov polls are right, but all the other pollsters out there will think the same thing.”

    I’m inclined to agree about YG. In an election where most pollsters are now showing a very tight contest for most popular party, YG have been steadier and more consistent in showing that to have been the case for some time than have other pollsters.

  5. NC – depends what you mean by organic. YouGov have never relied on just the sort of people who turn up at the YouGov website under their own steam, a lot of it is through advertising and recruitment campaigns targetted at whatever groups we need more of, and link ups with companies with mailing lists to recruit through them and so on.

    YouGov don’t normally buy access to third party UK panels like ResearchNow or Toluna, we use our own panel. But obviously if we were doing an international project and needed sample in countries where we don’t have panels we’d buy in third party panellists.

    I can’t speak for what other companies do, it’s not always very clear.

  6. I agree with RAF and Peter (do I get a place in the expanding debate membership, with that line?)

    An excellent summary, which sadly few journalists will read.

  7. A waste of time and effort Anthony – pity because so much of what you produce is so good.

  8. Of course no one knows which pollster will turn out to be most accurate but it does seem that Yougov is the least likely to show wild swings and swings that seem to contradict all common sense.

    For instance Ashcroft and Populus, the other two regular pollsters, often seem to throw up some quite bizarre results.

    About 6 months ago Populus was the only pollster showing frequent small Conservative leads; more recently it has tended to show Labour leads of 3-4%.

    Even given all the uncertainty over the veracity of polling in this election cycle, I think we can safely say that Labour have not significantly improved their polling position from 6 months ago.

  9. I’m a different John to the one above me – I thought it was a good article.

  10. What kind of site attracts two johns????

  11. No, I’m John!

  12. Thanks AW, very useful analysis.

  13. AW – Thanks, judging by the sort of comments I see on the YouGov website, I think it’s just as well that you don’t rely on people that turn up “under their own steam” :)

    I’m kind of curious about young voters, as there always seems to be a shortage of those in polls… Is that more down to response rates, or having too few in panels?

  14. John, John

    There’s a John B as well isn’t there? At least one of you is going to have to change your name to The Other John!

  15. And from yet another John, I can only assume that this site has a small house bias towards people whose parents were conservative (small c, non-partisan comments only please) when naming their children

    Anthony, you rock!

    YouGov my North Star, but the great thing about this site is we see all the results, although I now know not to get too excited by TNS polls unless they show the Tories in front … This also brings home how we just don’t know where UKIP really are, but based on the by-elections I wonder if the guys at the top may be closer??

  16. @OldNat

    Given that winter us beginning to bite, a long John post is only to be expected.

  17. I used to post under John P but there’s another John P.

  18. RAF

    :-)

  19. Great article AW.

    Don’t think I’ll be paying too much attention to TNS though!!

  20. @ RAF

    :)

    #humorouspoliticalwinterunderwearcomments

  21. It’s only a matter of time until we get a Sparticus…….

    Set it up for you!

    Peter.

  22. ” Given that in the past testing found prompting was making a difference, it suggests that UKIP are now well enough established in the public mind that whether the pollster prompts for them or not no longer makes much difference”
    _____

    That’s a good point but what about the Greens? Ok they have been around for a while but they are getting a lot more exposure and surely prompting for them would (at the moment) have a positive impact on their VI because unlike UKIP I don’t think they are quite as established in the public persona.

  23. I note someone thinks a journalist should read the reasoned expertise of AW. Ha Ha Ha, surely left wing geek and Bullingdon bully boy make far more stimulating political debate.

  24. spartan
    ?sp??t(?)n/
    adjective
    adjective: spartan

    showing or characterized by austerity or a lack of comfort or luxury.

    So a Conservative supporter then.

  25. @Anthony Wells

    Useful as ever, Anthony.

  26. ANTHONY
    Thanks very much

  27. Thanks AW

    I see said the blind man

  28. Statgeek

    spartan

    So a Conservative supporter then.

    Probably more like Golden Dawn as the ancient Spartans were the model for many a fascist society that came after. Oddly enough the constituency of Lakonias (capital Sparta) was the only one in Greece where Golden Dawn got more than 10%.[1] Some things never change.

    [1] Up to 20% in parts of the Mani, traditionally the most right-wing place in Greece.

  29. The irony is that with Fascism comes homophobia.

    The Spartans could not be described as homophobic ;-)

  30. Tonight’s YouGov/TheSun poll

    CON 34%, LAB 34%, LD 6%, UKIP 14%, GRN 7%

  31. The return of two party politics?

  32. Lab and Con creeping up together…..

  33. @AW

    Thanks for the article. For reference, I am not and never have been a member of the johns.

  34. Neill A

    But different two parties in the northern nations.

  35. Not much left for t’others this evening, perhaps nobody in Scotland was in.

  36. MISERABLE OLD GIT
    For reference, I am not and never have been a member of the johns.

    Another Lehrer fan? http://youtu.be/hoEVPtVk9nE

  37. Oldnat,

    You mean the Unionists and the Nationalists… And whatever the main parties are in Northern Ireland these days?

  38. @STATGEEK

    “…spartan…So a Conservative supporter then…”

    So, not a Halo or Microsoft fan then…:-)

    Anyhoo, you asked earlier today for YouGov reports from 2010/11. Could you not get them from the Wiki article?

  39. Norman I thought Crofty put it well on the last thread when he posted that the 2 biggest groupsin the Electorate are the ABTs and the ABLabs.

    In Scotland many SNP supporters manage to be both.

  40. Bill Patrick

    Jim Murphy : “I AM NOT A UNIONIST!”

  41. Main two parties media coverage looks to be working. Both been in the news a lot.

  42. @Hoof Hearted

    And as the debates are almost certainly not going to happen that’s likely to be the way it will stay. At least until the short campaign begins and Ukip get the benefit of “due weight” in England and Wales.

  43. RAF

    “Ukip get the benefit of “due weight” in England and Wales.”

    That’s going to stuff Question Time!

  44. Jim Murphy : “I AM NOT A UNIONIST!”

    Was he SHOUTING ?

  45. R&D

    Sigh. That was the point of the caps. Think of it in Ian Paisley’s dulcet tones.

  46. Grungibap reporting that hospital trusts that provide 75% of hospital care are refusing to sign new funding deals as they say they can no longer afford to apply more cuts (in terms of the money they are paid per procedure) without affecting patient safety.

    While Labour has also had it’s NHS plans savaged today, this is nicely timed for the GE. I don’t pretend to understand NHS finance and management, and I’m certain that reforms and changes are needed in some form, but any story that has ‘NHS’ and ‘cuts’ somewhere in the text is dangerous for Cameron.

  47. There is a new report out by Rob Ford on the impact of migrants on the election

    http://www.migrantsrights.org.uk/publications/policy-report/migrant-voters-2015-general-election

    Note that this is looking at migrants, not BME voters this time.

    Key findings
    Currently, migrant voters are almost as numerous as current UKIP supporters – around one voter in every ten eligible to vote in 2015 will be a migrant voter, and many more will be the children of migrants

    We estimate that just under 4 million foreign-born voters across England and Wales will be eligible to vote in the May 2015 general election.

    The migrant electorate is heavily concentrated in London – 19 of the 20 seats with the largest migrant voter shares are in Greater London.
    Migrants could constitute over a third of the electorate in around 25 seats across England and Wales in 2015, and at least a quarter of the electorate in over 50 seats.
    The migrant electorate could have decisive power in a range of key marginal seats across England and Wales: in at least 70 seats the migrant share of the electorate in 2015 is twice as large as the current majority share of the incumbent party

    So if 1 in 10 voters is a migrant voter, and migrant voters also have significantly different voting patterns to non migrant voters, I think once more this is more evidence of the need for country of birth/ ethnicity weighting in polls.

    I suspect it may even help even out some of those house effect differences!

  48. Off topic..

    Peter Hain on question time said he was delighted with Alexis Tsipras coming to power in Greece. Absolutely agree with him.

    If Labour do win the election then Peter Hain would make a great foreign secretary but I’m getting way way carried away.

  49. OLDNAT
    Tonight’s YouGov/TheSun poll
    CON 34%, LAB 34%, LD 6%, UKIP 14%, GRN 7%
    ______

    Poll-drums…

  50. @Alec

    I asked a consultant paediatrician of my acquaintance a few months ago how hospitals manage to lose money and go into debt. He said it was simple. The cash payment allocated for each specific piece of work is simply too small to pay for it to be carried out. If a procedure is “worth” £1,000 to the hospital, but because of complications or other problems it ends up costing them £2,000 in staff costs to carry it out, then they’re £1,000 in the red. And with all of the payment rates inadequate or close to the bone, there’s no way to recoup the money from “profiting” on another procedure. Hence why Circle Care pulled out of Hinchingbrooke.

    That isn’t to say that not having an internal market means that all procedures get done promptly, effectively and in-budget either. But it made more sense after he explained it. The “unit pricing” is simply unrealistic. If hospitals carry out less work, they get less money, so there’s no way out of the bind for them. The intention is to force them to find ways to do each action more cheaply. What the hospitals are now saying is that they’ve reached the limit of being able to do that without killing people.

1 2 3 5