Polls often give contrasting results. Sometimes this is because they were done at different times and public opinion has actually changed, but most of time that’s not the reason. A large part of the difference between polls showing different results is often simple random variation, good old margin of error. We’ve spoken about that a lot, but today’s post is about the other reason, systemic differences between pollsters (or “house effects”).

Pollsters use different methods, and sometimes those different choices result in consistent differences between the results they produce. One company’s polls, because of the methodological choices they make, may consistently show a higher Labour score, or a lower UKIP score, or whatever. This is not a case of deliberate bias – unlike in the USA there are not Conservative pollsters or Labour pollsters, every company is non-partisan, but the effect of their methodological decisions mean some companies do have a tendency to produce figures that are better or worse for each political party – we call these “house effects”.


The graph above shows these house effects for each company, based upon all the polls published in 2014 (I’ve treated ComRes telephone and ComRes online polls as if they are separate companies, as they use different methods and have some consistent differences). To avoid any risk of bias from pollsters carrying more or less polls when a party is doing well or badly I work out the house effects by using a rolling average of the daily YouGov poll as a reference point – I see how much each poll departs from the YouGov average on the day when its fieldwork finished and take an average of those deviations over the year. Then I take the average of all those deviations and graph them relative to that (just so YouGov aren’t automatically in the middle). It’s important to note that the pollsters in the middle of the graph are not necessarily more correct, these differences are relative to one another. We can’t tell what the deviations are from the “true” figure, as we don’t know what the “true” figure is.

As you can see, the difference between the Labour and Conservative leads each company show are relatively modest. Leaving aside TNS, who tended to show substantially higher Labour leads than other companies, everyone else is within 2 points of each other. Opinium and ComRes phone polls tend to show Labour leads that are a point higher than average, MORI and ICM tend to show Labour leads that are a point lower than average. Ashcroft, YouGov, ComRes online and Populus tend to be about average. Note I’m comparing the Conservative-v-Labour gap between different pollsters, not the figures for each one. Populus, for example, consistently give Labour a higher score than Lord Ashcroft’s polls do… but they do exactly the same for the Conservatives, so when it comes to the party lead the two sets of polls tend to show much the same.

There is a much, much bigger difference when it comes to measuring the level of UKIP support. The most “UKIP friendly” pollster, Survation, tends to produce a UKIP figure that is almost 8 points higher than the most “UKIP unfriendly” pollster, ICM.

What causes the differences?

There are a lot of methodological differences between pollsters that make a difference to their end results. Some are very easy to measure and quantify, others are very difficult. Some contradict each other, so a pollster may do something that is more Tory than other pollsters, something that is less Tory than other pollsters, and end up in exactly the same place. They may interact with each other, so weighting by turnout might have a different effect on a phone poll from a telephone poll. Understanding the methodological differences is often impossibly complicated, but here are some of the key factors:

Phone or online? Whether polls get their sample from randomly dialling telephone numbers (which gives you a sample made up of the sort of people who answer cold calls and agree to take part) or from an internet panel (which gives you a sample made up of the sort of people who join internet panels) has an effect on sample make up, and sometimes that has an effect on the end result. It isn’t always the case – for example, raw phone samples tend to be more Labour inclined… but this can be corrected by weighting, so phone samples don’t necessarily produce results that are better for Labour. Where there is a very clear pattern is on UKIP support – for one reason or another, online polls show more support for UKIP than phone polls. Is this because people are happier to admit supporting UKIP when there isn’t a human interviewer? Or it is because online samples include more UKIP inclined people? We don’t know

Weighting. Pollsters weight their samples to make sure they are representative of the British population and iron out any skews and biases resulting from their sampling. All companies weight by simple demographics like age and gender, but more controversial is political weighting – using past vote or party identification to make sure the sample is politically representative of Britain. The rights and wrongs of this deserve an article in their own right, but in terms of comparing pollsters most companies weight by past vote from May 2010, YouGov weight by party ID from May 2010, Populus by current party ID, MORI and Opinium don’t use political weighting at all. This means MORI’s samples are sometimes a bit more Laboury than other phone companies (but see their likelihood to vote filter below), Opinium have speculated that their comparatively high level of UKIP support may be because they don’t weight politically and Populus tend to heavily weight down UKIP and the Greens.

Prompting. Doesn’t actually seem to make a whole lot of difference, but was endlessly accused of doing so! This is the list of options pollsters give when asking who people vote for – obviously, it doesn’t include every single party – there are hundreds – but companies draw the line in different places. The specific controversy in recent years has been UKIP and whether or not they should be prompted for in the main question. For most of this Parliament only Survation prompted for UKIP, and it was seen as a potential reason for the higher level of UKIP support that Survation found. More recently YouGov, Ashcroft and ComRes have also started including UKIP in their main prompt, but with no significant effect upon the level of UKIP support they report. Given that in the past testing found prompting was making a difference, it suggests that UKIP are now well enough established in the public mind that whether the pollster prompts for them or not no longer makes much difference.

Likelihood to vote. Most companies factor in respondents likelihood to vote somehow, but using sharply varying methods. Most of the time Conservative voters say they are more likely to vote than Labour voters, so if a pollster puts a lot of emphasis on how likely people are to actually vote it normally helps the Tories. Currently YouGov put the least emphasis on likelihood to vote (they just include everyone who gives an intention), companies like Survation, ICM and Populus weight according to likelihood to vote which is a sort of mid-way point, Ipsos MORI have a very harsh filter, taking only those people who are 10/10 certain to vote (this probably helps the Tories, but MORI’s weighting is probably quite friendly to Labour, so it evens out).

Don’t knows. Another cause of the differences between companies is how they treat people who say don’t know. YouGov and Populus just ignore those people completely. MORI and ComRes ask those people “squeeze questions”, probing to see if they’ll say who they are most likely to vote for. ICM, Lord Ashcroft and Survation go further and make some estimates about those people based on their other answers, generally assuming that a proportion of people who say don’t know will actually end up voting for the party they did last time. How this approach impacts on voting intention numbers depends on the political circumstances at the time, it tends to help any party that has lost lots of support. When ICM first pioneered it in the 1990s it helped the Tories (and was known as the “shy Tory adjustment”), these days it helps the Lib Dems, and goes a long way to explain why ICM tend to show the highest level of support for the Lib Dems.

And these are just the obvious things, there will be lots of other subtle or unusual differences (ICM weight down people who didn’t vote last time, Survation ask people to imagine all parties are standing in the seat, ComRes have a harsher turnout filter for smaller parties in their online polls, etc, etc)

Are they constant?

No. The house effects of different pollsters change over time. Part of this is because political circumstances change and the different methods have different impacts. I mentioned above that MORI have the harshest turnout filter and that most of the time this helps the Tories, but that isn’t set in stone – if Tory voters became disillusioned and less likely to vote and Labour voters became more fired up it could reverse.

It also isn’t consistent because pollsters change methodology. In 2014 TNS tended to show bigger Labour leads than other companies, but in their last poll they changed their weighting in a way that may well have stopped that. In February last year Populus changed their weights in a way that reduced Lib Dem support and increased UKIP support (and changed even more radically in 2013 when they moved from using the telephone to online). So don’t assume that because a pollster’s methods last year had a particular skew it will always be that way.

So who is right?

At the end of the day, what most people asking the question “why are those polls so different” really want to know is which one is right. Which one should they believe? There is rarely an easy answer – if there was, the pollsters who were getting it wrong would correct their methods and the differences would vanish. All pollsters are trying to get things right.

Personally speaking I obviously I think YouGov polls are right, but all the other pollsters out there will think the same thing about the polling decisions they’ve made and I’ve always tried to make UKPollingReport about explaining the differences so people can judge for themselves, rather than championing my own polls.

Occasionally you get an election when there is a really big spread across the pollsters, when some companies clearly get it right and others get it wrong, and those who are wrong change their methods or fade away. 1997 was one of those elections – ICM clearly got it right when others didn’t, and other companies mostly adopted methods like those of ICM or dropped out of political polling. These instances are rare though. Most of the time all the pollsters show about the same thing, are all within the margin of error of each other, so we never really find out who is “right” or “wrong” (as it happens, the contrast between the level of support for UKIP shown by different pollsters is so great that this may be an election where some polls end up being obviously wrong… or come the election the polls may end up converging and all showing much the same. We shall see).

In the meantime, with an impartial hat on all I can recommend is to look at a broad average of the polls. Sure, some polls may be wrong (and it’s not necessarily the outlying pollster showing something different to the rest – sometimes they’ve turned out to be the only one getting it right!) but it will at least help you steer clear of the common fallacy of assuming that the pollster showing results you like the most is the one that is most trustworthy.

231 Responses to “All about house effects”

1 3 4 5
  1. @OldNat

    Agreed that Holyrood is the real aim for Greens. I’m sure a second list seat in Lothians will be a key aim.

    Likely that Peter McColl will be the number 2 on the list so we may see more of him in future.

    Will have to see if I can overcome my personal animosity for him. At least I no longer live in Edinburgh so it doesn’t affect my vote!

  2. Good Evening All; weekend, I always think it will last for ever.

    I agree with you, that Lib Dem figures look a bit high in the polls; I have thought that for some time.

  3. Hmmm. Not quite so good on the scarf front at the mo I’m afraid. Looks like it’s all gone a bit Pete Tong.

    A while back one of the teenagers at Morgan S came up with the idea of lending scarves to Greece. Sounded like a great idea, and the Greeks love a good scarf apparently. Greek ladies in chiffon neck scarves with huge pensions – very attractive!

    Anyway, they loved them so much we lent shed loads of them – well, when I say ‘shed loads’ I mean more like several aircraft hangars the size of Belgium (or 1.2 Wales’, if you still think in imperial).

    It all seemed fine, and we raked it in I must say, but the fly in the ointment (fly in the oil, I suppose) was that to fund the deal we borrowed vast sums of credit (‘funny money’, as Mrs A calls it) from some very obliging German banks.

    I’m a bit of a buffoon when it comes to money, so I don’t quite follow what happened next, but we used the scarves we’d lent to the Greeks (paid for with German debt) to act as a deposit on a new set of huge investments. ‘Couldn’t go wrong’ said the teenager from Morgan S – swear I remember him saying that, but I was a bit tiddly at the time I must admit.

    Anyway, back to the story. We used the ‘money’ (double leveraged collateralized debt instrument I gather is the technical term) to invest in North Sea oil. Ideal bet for the future!

    Next thing I know, I’m reading the FT (I say ‘reading’, but I generally look for the cartoons first – they can be really rather good, although I don’t always get the joke. Well, to be honest, I never get the jokes, but people tell me they are very funny) and it turns out old Uncle Saudi is being very naughty. Very naughty indeed.

    So the teenager from Morgan S calls on the mobile. He sounds a bit excitable and tells me he’s on the roof and the oil price is tanking. Said something about his girlfriend and looking after his goldfish that I didn’t really understand, but then I never really understood much of what he told me, if I’m being honest. ‘Tanking? – Good oh!’ I said – ‘that sounds excellent’. Anyway, next thing there’s a bit of a whooshing and the line goes dead – poor reception probably.

    While I’m waiting for the next cheque to arrive I get a call from an old friend at the Bank of England. He’s not on the roof, but is was 2.30am in the morning and he was also rather agitated.

    I’ve known the chap since boarding school days, and he starts telling me about some new ‘Cistercian Mix’. Couldn’t quite make it out, but I knew he was partial to classical music and once had an album of Gregorian chants, so I said ‘dear chap, did you really ring at 2.30 in the morning to discuss monastic music from the Middle Ages?’.

    Turns out I’d misheard him, and he was babbling on about something called ‘systemic risk’. Turns out the weather is too hot for scarves in Greece, so they didn’t want to pay up, but the German’s get a bit squiffy about these things and wanted their money back – all of it! Talk about ruthless efficiency!

    But when we had a look at our investment in oil, that was obviously a bit of a no go area, so everyone seemed very unhappy (except for the Greeks, who I suspect actually quite like scarves).

    Sounds like we’ve got it sorted though. The chappie from the BoE says not to worry, as he’s got some of his own funny money from HMG that he’ll let us have to pay the Germans and keep everything nice and steady. Sounds like there might be a bit to spare for us too.

    Which is just as well, as I was introduced to a new teenager at Morgan S. He’s come up with a great idea of ties to Thailand. Apparently they love the smart western look over there, and he’s calculated there is enough borrowing capacity to support 34,690 ties per head of the populations, so long as we can supply them with the credit first.

    Sounds a bit complicated, but he assures me there is absolutely no risk, and we’ve got all that money from HMG to kick the whole show off.


  4. For the small minority here interested in seat-projection models, Electionforecast have now redubbed their “Nowcast” as “Constituency Polls”, commenting in their Update notes that the previous heading was potentially misleading.

    (I, for one, certainly wondered out loud here why the Tory VI never rose above 30% and there were no doubt others that were somewhat bemused by some of the figures.)

    In their notes they state:

    It was slightly misleading to say they are estimates of what would happen in an election if held today. It is more accurate to say they are estimates of what a Lord Ashcroft constituency poll would say in the constituency if it was fielded today.

    This interesting distinction seems to imply that they feel that an Ashcroft poll may not necessarly be a good predictor of what would happen in an election. If he agrees he may just find other ways of spending his money. Against this, quite a few people think that they provide the best evidence we currently have for predicting what will happen in May.

    At the same time, the EF group seem to have ‘retired’ the old Nowcast VI table, making it difficult now to see how well the aggregate party VIs are aligned with national polling figures. A pity..

    Meanwhile, the May2015 calculator continues to produce rubbish when you type in your own percentages…

  5. Unicorn


  6. Alec,

    Did not the Greeks buy lots of scarfs from the Germans who have benefited greatly from the loans to the Greeks but now don’t see why they should accept ant responsibility.

    It reminds me of when some US banks lent money to people to buy scarfs and many of them it turned out were always likely to strgugggle to repay and the second hand value of teir scarfs was low – a sub prime scarf loan, I recall it was called.

    Kind of suggests that the German Government and some US banks have a lot in common.


  7. @CL1945

    “I agree with you, that Lib Dem figures look a bit high in the polls; I have thought that for some time.”

    You’ve never said.

  8. Addendum

    I’d be interested to hear how others interpret the EF team’s tacit scepticism about the predictive value of Ashcroft Constituency polls.

    I suspect they are hedging their bets about the near universal adoption of Ashcroft’s as-yet-unvalidated CVI measure. If this proves to be an imperfect predictor of real voting behaviour then there will be a lot of misinformation in the models that make use of these measures.

    I, weirdly, will enjoy the next thirteen weeks and six days; and the day after when government construction starts.

  10. Unicorn

    I’m not sure that they are showing “tacit scepticism” of CVI, but rather taking a guarded approach to constituency polling itself. It is a very new type of evidence, with no track record.

    Even without the Ashcroft Scottish polls, I note that they have moved O&S from a sure SNP gain to a marginal LD hold – so a bit of common sense there. :-)

  11. To be fair I recall calling Chris out on his Lib Dem height querying when they were at 11%. Those heady days!

  12. @Unicorn

    Personally I see the value of the Ashcroft Polls as a comparative rather than predictive tool.

    They are useful to tell us how certain types of seats are behaving in comparison to national trends (Con vs Lib Dem marginals say).

    They are also useful to show us seats where strong personal votes may cause seats to vary from these trends (Norman Baker in Lewes to do better than Lib Dem vs Con marginal average say).

    They are probably less valuable as a “X will definitely win this seat” predictor.

    What they should do over time is help us devise better models that say if the national swing is X these are the constituencies that will change hands.

    I think the reason everyone is so nervous about making that next level of prediction is the volatility of the UKIP vote that makes constituency predictions very hard.

  13. Amongst all you experts can one of you tell me what percentage of the You Gov sample comes from Scotland.

    Thank you

  14. Penn

    8.7% today. (though that calculation doesn’t make anyone an “expert”. :-) )

  15. ALEC

    Thank you for making the world of finance so clear. I have a son who is quite high up in an American bank that the papers are fond of quoting but I never understand or remember for more than 1/2 hour what he tells me. Now scarves make all clear. Please will you publish a book that I can buy and quote to my son.

  16. @ Northumbrianscot

    Continuing the frayed end of the thread while we all wait for Anthony’s summary, I agree that those are useful features of the polls.

    For the campaign managers, they also provide useful intelligence to guide their deployment of ground troops.

  17. BBC report on the short, frosty meeting today between Jeroen Dijsselbloem, head of the Eurogroup – the eurozone finance ministers , and Greece’s new left-wing finance minister Yanis Varoufakis .

    I thought the essence of the message from Greece was not so much in Varoufakis’ refusal to negotiate over the Greek bailout conditions with the “troika” team from the EU and IMF.-but in the picture :-

    Dijsselbloem is in suit & tie , clutching a file of papers .Varoufakis is in an open necked shirt hanging out over his belt , with left hand in his pocket . Nothing conveys more eloquently -” Things have changed around here-so f***k off back to the rest of the Bankers & Brussels suits & tell them.”


  18. The paper which cannot be linked to gives an awesome list of actions by Tsipras to role back “austerity”-from increasing the Minimum wage, through the scrapping of fees for prescriptions and hospital visits, the restoration of collective work agreements, the rehiring of workers laid off in the public sector to the granting of citizenship to migrant children born and raised in Greece & halting all asset sales.

    Greece apparently voted against new EU sanctions on Russia.

    Does the next announcement not seem obvious?-a de-facto reneging on all the Troika loan conditions-a loan from Russia to fund the Greek Budget, followed by a triumphal fraternal visit to Athens by their “best friend”-Mr. Putin.


  19. Reneging on an IMF loan in favour of a Russian one was what happened in A Very British Coup if I recall, and it worked alright for Red Harry!

    Seriously though, I’ve long thought one major problem of the radical left is how utterly ineffectual they are at campaigning, preferring to fight among themselves. But here we’re seeing a government which is not only on the radical left, but seems to have some fire in its eyes and an uncompromising approach to power.

    I really have no idea how this will turn out.

  20. It was a Libyan loan, not a Russian one.

  21. It was Iceland that got a €4bn Russian loan in 2008.

    That’s a very smart move for a country occupying a strategic geo-political location.

  22. It seems a logical step. Syriza don’t have to make a deal with Russia, but suggesting it as a possibility gives them some leverage. Been an even more promising strategy before Russia started haemorraging money on the oil price.

  23. Mr Nameless
    I do. They get gubbed.
    The German minister Gabriel said today that, unlike a few years ago, the EZ can cope with Greek exit. They have they believe built a fire-wall round the other PIGS.
    I saw the Russian option coming. This is a long term trope in Greece. I think it is the only place in the world where the political parties used to be officially called after their foreign backers so that the ancestor of the current Pasok leader was if I remember rightly the leader of the British party which competed with Russian and French parties. However I would be surprised if Russia stepped up to the plate in current circumstances.
    An interesting piece of recent research showed Greece was the only country not to have benefited from EU membership and that it had indeed lost out substantially. I fear unfolding events just might have an effect on the UK election.
    History may say that H Wilson’s greatest achievement was to keep the uk out of Vietnam but there were those at the time who wanted more demonstrative protest. In response, Wilson pointed out to the cabinet that it is never easy to kick your creditor in the balls.

  24. OLD NAT

    Thank you!

  25. @Barney Crockett

    “Wilson pointed out to the cabinet that it is never easy to kick your creditor in the balls.”

    Of course it is. You just indebt yourself so much that it’ll be more damaging to them than you I’d you can’t repay. Then you can kick them in the balls to your hearts content.

    It worked for the banking sector after all.

  26. @Colin

    That’s pretty much what happened.

    There’s a sense among the super rich that the poor have no pride and have no choice other than do what they’re told. Of course there will eventually be a deal between Greece and the EU/IMF but not before Greece teaches them a harsh lesson.

  27. I *think* they are daring the eurogroup to throw them out of the euro – so they get to leave, but not take the blame. Or get a debt write-down. Win either way?

  28. New Thread

  29. I recall saying on this site many years ago that ‘some polls favour the Tories’ which was met with a response that if I repeated such claims I would not be allowed to post on the site! Some polls do favour the Tories, some polls other parties. It isn’t suggesting bias, it’s suggesting the methodology can favour a patticular party. Maybe methodology is considered by some pollsters in a different way to others.

    Nice to see UKPR acknowledging some polls favour particular parties because of their ‘house effect’!

  30. What is it with new Greek government? Don’t any of them have neckties? Perhaps they’re waiting for the next shipment of scarves.

  31. I’ve only just read this. It’s a very interesting article. No offense to anthony’s readship figures but it deserves a wider audience.

    I have also always taken yougov for one of the more ukippy pollsters so i am surprised to see when quantified that they’re in the lower half.

1 3 4 5