Polls often give contrasting results. Sometimes this is because they were done at different times and public opinion has actually changed, but most of time that’s not the reason. A large part of the difference between polls showing different results is often simple random variation, good old margin of error. We’ve spoken about that a lot, but today’s post is about the other reason, systemic differences between pollsters (or “house effects”).

Pollsters use different methods, and sometimes those different choices result in consistent differences between the results they produce. One company’s polls, because of the methodological choices they make, may consistently show a higher Labour score, or a lower UKIP score, or whatever. This is not a case of deliberate bias – unlike in the USA there are not Conservative pollsters or Labour pollsters, every company is non-partisan, but the effect of their methodological decisions mean some companies do have a tendency to produce figures that are better or worse for each political party – we call these “house effects”.

2014houseffects

The graph above shows these house effects for each company, based upon all the polls published in 2014 (I’ve treated ComRes telephone and ComRes online polls as if they are separate companies, as they use different methods and have some consistent differences). To avoid any risk of bias from pollsters carrying more or less polls when a party is doing well or badly I work out the house effects by using a rolling average of the daily YouGov poll as a reference point – I see how much each poll departs from the YouGov average on the day when its fieldwork finished and take an average of those deviations over the year. Then I take the average of all those deviations and graph them relative to that (just so YouGov aren’t automatically in the middle). It’s important to note that the pollsters in the middle of the graph are not necessarily more correct, these differences are relative to one another. We can’t tell what the deviations are from the “true” figure, as we don’t know what the “true” figure is.

As you can see, the difference between the Labour and Conservative leads each company show are relatively modest. Leaving aside TNS, who tended to show substantially higher Labour leads than other companies, everyone else is within 2 points of each other. Opinium and ComRes phone polls tend to show Labour leads that are a point higher than average, MORI and ICM tend to show Labour leads that are a point lower than average. Ashcroft, YouGov, ComRes online and Populus tend to be about average. Note I’m comparing the Conservative-v-Labour gap between different pollsters, not the figures for each one. Populus, for example, consistently give Labour a higher score than Lord Ashcroft’s polls do… but they do exactly the same for the Conservatives, so when it comes to the party lead the two sets of polls tend to show much the same.

There is a much, much bigger difference when it comes to measuring the level of UKIP support. The most “UKIP friendly” pollster, Survation, tends to produce a UKIP figure that is almost 8 points higher than the most “UKIP unfriendly” pollster, ICM.

What causes the differences?

There are a lot of methodological differences between pollsters that make a difference to their end results. Some are very easy to measure and quantify, others are very difficult. Some contradict each other, so a pollster may do something that is more Tory than other pollsters, something that is less Tory than other pollsters, and end up in exactly the same place. They may interact with each other, so weighting by turnout might have a different effect on a phone poll from a telephone poll. Understanding the methodological differences is often impossibly complicated, but here are some of the key factors:

Phone or online? Whether polls get their sample from randomly dialling telephone numbers (which gives you a sample made up of the sort of people who answer cold calls and agree to take part) or from an internet panel (which gives you a sample made up of the sort of people who join internet panels) has an effect on sample make up, and sometimes that has an effect on the end result. It isn’t always the case – for example, raw phone samples tend to be more Labour inclined… but this can be corrected by weighting, so phone samples don’t necessarily produce results that are better for Labour. Where there is a very clear pattern is on UKIP support – for one reason or another, online polls show more support for UKIP than phone polls. Is this because people are happier to admit supporting UKIP when there isn’t a human interviewer? Or it is because online samples include more UKIP inclined people? We don’t know

Weighting. Pollsters weight their samples to make sure they are representative of the British population and iron out any skews and biases resulting from their sampling. All companies weight by simple demographics like age and gender, but more controversial is political weighting – using past vote or party identification to make sure the sample is politically representative of Britain. The rights and wrongs of this deserve an article in their own right, but in terms of comparing pollsters most companies weight by past vote from May 2010, YouGov weight by party ID from May 2010, Populus by current party ID, MORI and Opinium don’t use political weighting at all. This means MORI’s samples are sometimes a bit more Laboury than other phone companies (but see their likelihood to vote filter below), Opinium have speculated that their comparatively high level of UKIP support may be because they don’t weight politically and Populus tend to heavily weight down UKIP and the Greens.

Prompting. Doesn’t actually seem to make a whole lot of difference, but was endlessly accused of doing so! This is the list of options pollsters give when asking who people vote for – obviously, it doesn’t include every single party – there are hundreds – but companies draw the line in different places. The specific controversy in recent years has been UKIP and whether or not they should be prompted for in the main question. For most of this Parliament only Survation prompted for UKIP, and it was seen as a potential reason for the higher level of UKIP support that Survation found. More recently YouGov, Ashcroft and ComRes have also started including UKIP in their main prompt, but with no significant effect upon the level of UKIP support they report. Given that in the past testing found prompting was making a difference, it suggests that UKIP are now well enough established in the public mind that whether the pollster prompts for them or not no longer makes much difference.

Likelihood to vote. Most companies factor in respondents likelihood to vote somehow, but using sharply varying methods. Most of the time Conservative voters say they are more likely to vote than Labour voters, so if a pollster puts a lot of emphasis on how likely people are to actually vote it normally helps the Tories. Currently YouGov put the least emphasis on likelihood to vote (they just include everyone who gives an intention), companies like Survation, ICM and Populus weight according to likelihood to vote which is a sort of mid-way point, Ipsos MORI have a very harsh filter, taking only those people who are 10/10 certain to vote (this probably helps the Tories, but MORI’s weighting is probably quite friendly to Labour, so it evens out).

Don’t knows. Another cause of the differences between companies is how they treat people who say don’t know. YouGov and Populus just ignore those people completely. MORI and ComRes ask those people “squeeze questions”, probing to see if they’ll say who they are most likely to vote for. ICM, Lord Ashcroft and Survation go further and make some estimates about those people based on their other answers, generally assuming that a proportion of people who say don’t know will actually end up voting for the party they did last time. How this approach impacts on voting intention numbers depends on the political circumstances at the time, it tends to help any party that has lost lots of support. When ICM first pioneered it in the 1990s it helped the Tories (and was known as the “shy Tory adjustment”), these days it helps the Lib Dems, and goes a long way to explain why ICM tend to show the highest level of support for the Lib Dems.

And these are just the obvious things, there will be lots of other subtle or unusual differences (ICM weight down people who didn’t vote last time, Survation ask people to imagine all parties are standing in the seat, ComRes have a harsher turnout filter for smaller parties in their online polls, etc, etc)

Are they constant?

No. The house effects of different pollsters change over time. Part of this is because political circumstances change and the different methods have different impacts. I mentioned above that MORI have the harshest turnout filter and that most of the time this helps the Tories, but that isn’t set in stone – if Tory voters became disillusioned and less likely to vote and Labour voters became more fired up it could reverse.

It also isn’t consistent because pollsters change methodology. In 2014 TNS tended to show bigger Labour leads than other companies, but in their last poll they changed their weighting in a way that may well have stopped that. In February last year Populus changed their weights in a way that reduced Lib Dem support and increased UKIP support (and changed even more radically in 2013 when they moved from using the telephone to online). So don’t assume that because a pollster’s methods last year had a particular skew it will always be that way.

So who is right?

At the end of the day, what most people asking the question “why are those polls so different” really want to know is which one is right. Which one should they believe? There is rarely an easy answer – if there was, the pollsters who were getting it wrong would correct their methods and the differences would vanish. All pollsters are trying to get things right.

Personally speaking I obviously I think YouGov polls are right, but all the other pollsters out there will think the same thing about the polling decisions they’ve made and I’ve always tried to make UKPollingReport about explaining the differences so people can judge for themselves, rather than championing my own polls.

Occasionally you get an election when there is a really big spread across the pollsters, when some companies clearly get it right and others get it wrong, and those who are wrong change their methods or fade away. 1997 was one of those elections – ICM clearly got it right when others didn’t, and other companies mostly adopted methods like those of ICM or dropped out of political polling. These instances are rare though. Most of the time all the pollsters show about the same thing, are all within the margin of error of each other, so we never really find out who is “right” or “wrong” (as it happens, the contrast between the level of support for UKIP shown by different pollsters is so great that this may be an election where some polls end up being obviously wrong… or come the election the polls may end up converging and all showing much the same. We shall see).

In the meantime, with an impartial hat on all I can recommend is to look at a broad average of the polls. Sure, some polls may be wrong (and it’s not necessarily the outlying pollster showing something different to the rest – sometimes they’ve turned out to be the only one getting it right!) but it will at least help you steer clear of the common fallacy of assuming that the pollster showing results you like the most is the one that is most trustworthy.


231 Responses to “All about house effects”

1 2 3 4 5
  1. I’ve signed up to a scarf investment scheme set up by a top US bank.

    I invest a regular amount each month, tax deductable as a business expense, and get the option to lease scarves on a weekly basis.

    I then borrow money to cover the lease costs, using the scarves as collateral on my mortgage, which has enabled me to borrow £750,000 more than I would have done previously, while also keeping warm.

    My bank has then packaged up my additional mortgage with other investments from the same scheme, and sold this to other banks as a collateralized debt product, which nets me a further 6% annual return on top of the initial 15% return (including tax relief) and the use of 453 scarves per week.

    It seems to be working very well, although I recently got a letter from the IMF saying the scheme was threatening global financial stability.

    I replied, sending a nice purple mohair scarf for Christine L.

  2. Extraordinary report from the IFS showing the degree to which wages have stagnated since the recession and the extent to which living standards have fallen for the majority of the population. The findings are stark:-

    – British workers are taking home less in real terms than they were in 2001.

    – Wages are 1% lower in the third quarter of 2013 than they were in the same period 13 years ago after taking inflation into account.

    – Almost all groups have seen real wages fall since the recession.

    – Between 2008 and 2014 there is a clear pattern across the age spectrum, with larger falls in earnings at younger ages.

    What’s surprising is that been though wage declines were most pronounced between 2009-11, there has been no appreciable improvement in subsequent years even though the economy has been recovering for 18 months. It may not have been the jobless recovery some feared, but it has certainly been a “feelgood-less” recovery, certainly for the vast majority.

    When you see this sort of analysis by a wholly reputable and independent research body, it becomes obvious why there appears to be an almost total disconnect between positive headline economic statistics and a political dividend for the governing parties.

    This election may well still be all about the economy, stupid, but I don’t think it will prove to be the winning hand for the incumbents that their strategists appear so convinced that it will be. It could be a very perversely behaving double-edged sword.

    (If such a thing exists, that is. :-))

  3. The sword would go very well with a nice scarf though?

  4. Alec
    Is it right that scarves should be weaponised?

  5. The Tory leader in Slough as defected to UKIP I wonder why could it be because? She stood as the Conservative Parliamentary candidate for Slough in 2001 and 2010 – losing to Labour’s Fiona Mactaggart both times – but was overlooked for this year’s general election.

  6. OLDNAT @ RAF

    Stop rubbing it in. ;-)

  7. @CROSSBAT11

    The IFS are just telling us what we already know so it is in no way extraordinary. Following the crash there was a great deal of slack in the economy so there was little pressure on employers to increase wages. That’s how these things work. At the same time, inflation was relatively high – largely commodity driven – so real wages lost ground. Nothing was going to change until the Labour market tightened and this is starting to happen now. In addition to this, commodity prices have fallen sharply and inflation has more or less disappeared. Add to this the tax reductions in April and we are looking at a substantial general improvement in real incomes. This is all too late of course for the IFS study, but may not be too late for the election. In fact it is probably bang on time.

  8. @Crossbat11

    It’s generally recognised that the signs are all in place for significant wage rises in the near future. A more mobile, skilled and confident workforce will impose greater wage demands. Employee churn is increasing, and the IFS report is notable for its finding that employees who have mixed jobs in the last year are doing much better financially than those who didn’t.

    Furthermore – and something long anticipated by recruiters – the first graduates who have paid £27k fees leave university this summer into the kindest graduate jobs market in more than half a decade.

    Wages will increase. Naturally, Government will suddenly start paying attention to pay and claim the credit for it. Not that it will be a thing to do with anything they did.

  9. @ James Peel

    “..the fall in the labour VI in the last 10 months has been remarkable. they were averaging about 38% as late as March last year.”

    Setting aside the subjective ‘remarkable’, your facts are correct. But your further deductions are questionable. With Labour on 33% in today’s polling averages, there has been a VI drop of about 5 points since early last year (and my trend analyses show that this steady decline remain securely on track). Projection of the same trends suggest there is likely to be a further drop of 1.5 points by May 7.

    So where is this pattern likely to leave us on Election Day?

    Answer: an outcome with Labour ahead by a couple of seats and in the better position to form a visble coalition.

    If you expect anything different to happen, then you must anticipate the emergence of truly remarkable VI changes over the coming 14 weeks. Granted, this may yet happen. But there is no support to be found by referring to Labour’s VI trends over the past year (as you seem to try to do above).

  10. Populus:

    Con 34 (nc)
    Lab 35 (nc)
    LD 10 (+1)
    UKIP 14 (+1)
    Greens 4 (-2)

  11. OLD NAT
    “Trotting out the old conch shell heresy of Evald Vasilyevich Ilyenkov I see. The Gulags await you!”

    Stop, I confess, I confess. I plead also guilty of adherence to Alexadrov Vassilievich Chayanov’s revisionist theory of peasant cyclical resource management, rightly identified in @Northumbrian’s thesis on penis gourd wearing, big haired, pig swappers in the ilk of the classic Onka’s Big Moka, and in defiance of Red Ed’s proposed collectivisation of the scarf industry.as a means of preventing Alec and his ilk’s cartelisation of scarf distribution.

  12. I think we should scarf this unintentional meme of attention.

  13. Very well illustrated in David Attenborough’s programme on last night on B-C 2, where we saw the vast crowd of betusked and painted New Guinea natives bedecked in the feathers of Birds of Paradise, several in Sheffield University colours.

  14. CHRIS RILEY

    Trouble is that we have been hearing this for a long time. If the earlier predictions had been true we would have jam yesterday, not tomorrow.

  15. RNJ1

    Indeed.

    The Labour attack on this subject is essentially that-we haven’t returned to where we were.

    It always invited the response-was “where we were” a sustainable position with the benefit of hindsight.

  16. Phil – it was a very wide definition on that poll. It was everyone who did NOT say they were certain how they would vote, and did NOT say they would never vote Labour (so it included both people currently voting Labour who might change away, and people not voting Labour who might change towards them)

  17. Something strangely wrong with the May2015 seat calculator today. If you round down their GB-wide Labour VI figure from its current 32.4% to 32%, you magically lose over a hundred of the 650 parliamentary seats.

    The total number of seats accounted for by all parties drops to just 508, with the main parties only bagging about 240-250 seats each.

    Makes you wonder about the other calculations thwt come out of the model…

  18. Interesting that you get a nice linear relationship on that chart. It actually seems to reduce to one axis, Tory-UKIP, as if the pollsters all have a decent handle on the Labour vote and a reasonable idea of the total Right, but the split between tories and ‘kippers is problematic.

  19. @Chris Riley/RMJ1

    I’m not sure I share your confidence about the inevitability of a surge in wages and living standards because I think you’re assuming that today’s labour market still works in the way that it always used to do. Weaker trade union bargaining power, the exponential growth in agency employment and zero hours/short term contracts and the strange and murky world of self-employment all conspire to make wage settlements very much an employers playing field. Skill shortages may help high-tech job wages rise healthily, but most people are destined to compete for employment and pay in semi or unskilled work. That’s a very different world and I can see wage stagnation in that sort of employment living on well past recovery. Unless, of course, a whole raft of altruistic and philanthropic employers emerge! :-)

    I’m also intrigued by the theory that vast numbers of voters, having endured a fall in their living standards for many years, are going to suddenly reward a government that chucks a few pre-election tax cuts around and claims credit for a belated fall in commodity rises.

    Pavlov’s dogs don’t salivate in the way they used too.

  20. I don’t know about you guys, but I expect most voters to think exactly the same way as me by election time, because there’s no way anyone remotely rational could disagree with me.

  21. cb11

    ” I can see wage stagnation in that sort of employment living on well past recovery”

    Notably since it is in that sector that job entrants face competition from better qualified and already experienced competition from the EU. Investment in housing, for example, is investment in the skilled Polish labour force as well as in the UK construction industry and its training capacities.

  22. @Unicorn

    My current projections seem to match yours: Con 33.5, Lab 31.5, and basically dead heat on seats.

    When it comes to Ashcroft’s polling it’s encouraging that it seems to have no bias compared to everyone else. It’s just the large changes between one poll and the next. Perhaps if these polls become more frequent as polling day (hardly surprising really) then this might not be a problem.

    Anyone noticed the worsening deflation in the Eurozone? It’s bad for them, but would help an incoming government – the last thing we need is too much inflation here.

  23. CHRIS RILEY

    I saw that report on graduate pay trends-encouraging.

    OBR’s last forecast for average pay was +2% 2015 , then +3.1%, then 3.9%

    BOE’s latest comment on it was:-
    ” wage growth is likely to pick up in the near term, reflecting a gradual pickup in productivity and further improvements in labour market conditions. Wages are expected to grow a little faster than
    productivity, such that unit labour cost growth starts to pick
    up in 2015″

    CBI are forecasting average earnings growth a tad over 3% from 2015 on.

  24. Colin

    The only question relevant to polling is whether people believe that the economic forecasts on pay are accurate.

  25. I suppose scarfs could become a big trend amongst politicians, the media and almost anyone with public opinions.

    Handy for covering their brass neck and all that when they have to adopt the opposite point of view.

  26. Oldnat,

    The most striking figure on that Scottish Yougov poll average is obviously that the Tory + UKIP vote share is nearly 25%, i.e. what it was back in 1987. So much for the Scottish right dying out?

  27. @AW

    Late in the day, but thanks for the essay!

  28. (The SNP figure is interesting, but probably a blip. Scotland votes Labour, as everyone knows.)

  29. Is there any reliable data on how accurate wage growth predictions have been? Anecdotally, I feel that it’s been consistently overestimated for 5 years, but I can’t find anything to support or contradict that.

  30. Statgeek

    A scarf could benefit in hiding Eric Pickle’s neck, although it might end up looking more like a blanket.

  31. No Approval Ratings again?

    Is this intentional?

  32. @BP 11.07

    Surely you have realised by now that there are many people around who are far from being rational – or at least, do not agree with you! If only it were not so…..!

  33. LURKER

    I think it is much more personalised & nuanced than that. Voters are intelligent people.

    I would think that their personal experience in pay is added to their perception of local conditions-are companies hiring?-is better paid work appearing? .

    My guess would be that forecasts about national performance come last-perhaps in the category to do with perceived competence of alternative parties.

  34. John B,

    No, I think that the other lot are so clearly wrong that people won’t disagree with me come election time. Maybe they would in the past, but people are better informed these days and won’t succumb to the blatant propaganda of those who disagree with me on anything whatsoever.

  35. (Meaning nothing partisan, of course.)

  36. Colin

    Indeed, it is personal experience.

    To be fair to economic forecasters, I would expect that the statements of politicians come even lower in the perception-forming pecking order.

  37. Bill Patrick

    While UKIP numbers dance about (rather like Farage in a Greg Moodie cartoon) varying between 0 and 12 (Paris) this month, the average Con + UKIP this month has been 22%. Add in the LDs, the remnants of whom are probably fairly much on the right, and the percentage rises to 27%.

    Then there is the unknowable % of centre-right folk voting SNP. Add them in, and the Scottish right is very healthy.

    However, politics doesn’t seem to be ideologically split along the left/right spectrum at the moment.

  38. COLIN
    ” their personal experience in pay is added to their perception of local conditions-are companies hiring?-is better paid work appearing?

    More immediate key factors may be how marginalised are their households and personal finances . Are they having to use pay loan shops, food banks, have difficulty in paying the rent or the mortgage, or their kids having to wear hand-me-downs. Can the pay their round? Pay for a holiday?

  39. @BP

    Of course you are right. Now, just go and lie down for half an hour……

    Tories plus UKIP on 25% in Scotland. Not such a shock, I think. The Tories have been quite capable of reaching 20% on an off down the past few years, and with JM now refusing to accept the Lodge vote under any circumstances whatsoever, where else are our Orange friends to go but to UKIP or the Tories?

    Otherwise we have to assume that UKIP supporters are incomers from southern climes who have come north to get as far away as possible from our partner in the Auld Alliance – poor deluded fools!

    Seriously, however, the figures suggest that it remains an intriguing possibility that JM might not be returned to Westminster because the Tories regain Rutherglen and its hinterland.

  40. No pollster is going to admit they are partizan. But no pollster wants to upset their paymasters either. As you explained, a lot of methodology has known effects, or can be predicted.
    What I have noticed is convenient methodology changes in the run up to elections. Polling influences voters and the appearance of momentum especially so. Another factor is variation. Have volatile polling and ficus more on the most favourable. What data have you got on these factors?

  41. And as I have stated here on previous occasions, it is my firm belief that an independent Scottish Conservative Party, plainly and unequivocally uncoupled from the party down south, would do better……

  42. I agree. They could even go back to being the Unionist Party, as a counterweight to the SNP. It might have a chance at squeezing out Labour, long term.

  43. UKIP supporters in Scotland are probably disgruntled Rangers supporters who voted No and live in deprived areas worrying about Sundays result.

  44. @Alec

    Thanks for that – gave me the best laugh of the week….

  45. John B,

    I might lie down and contemplate how to improve my skills at satire!

    The Tories winning Rutherglen is unlikely, but wouldn’t hurt Murphy in particular, given that he’s the MP for East Renfrewshire. It’s possible that a swing of Labour voters to the SNP there would let the Tories jump ahead, but the swing needed to get Labour down below the Tory vote is more than is needed to win the seat for the SNP, as far as I recall. The Tories would need a significant positive swing there.

    On the Tories rebranding, calling themselves “the Scottish Unionists” might be a good idea, given that (among some Scots) Unionism has been detoxified and has had a newfound importance over the last few years.

    Oldnat,

    Agreed on all points. In fact, I don’t think that Scottish politics in particular has split on a left-right basis for a long time, perhaps as far back as 1997, because the SNP-Labour competition doesn’t fit straightforwardly onto that axis.

  46. @ KeithP

    I find myself agreeing with most of your comments, but that I have little to add beyond that. It is refreshing to have your dispassionate analysis of developments. Perhaps your geographical distance from unfolding events helps when it comes to maintaining balance.

  47. Grant Shapps rules out a Tory-UKIP coalition. So coalitions that have explicitly been ruled out so far:

    Con-SNP
    Con-UKIP

    Pragmatically ruled out:

    Lab-Con
    Lab-DUP
    Lab-UKIP
    Con-Lab
    Con-Grn

    http://www.telegraph.co.uk/news/politics/conservative/11379386/Grant-Shapps-rules-out-deal-with-Ukip.html

  48. Relevantly, I was reading Lord Ashcroft’s 2013 study of the Tories in Scotland. To summarise briefly, it seems that the Tories actually have quite a bit of potential in Scotland: Cameron is relatively popular and his social background seems to be surprisingly unproblematic for many Scots, and there are a significant proportion of potential Tory voters who would like to see him remain PM; there is also a significant proportion of people who agree with the Tories on many issues.

    The problem for the Scottish Tories is that they’re so hopeless at actually persuading their potential voters to vote for them. The main problem (aside from the old third-party problem of “You’re not going to win anyway”) seems to be that even right-wing Scots tend not to see the Tories as looking out for them in particular or Scotland in general. Unless the Tories manage to change that perception, it’s extremely unlikely that they could see a significant revival in Scotland.

  49. The Young Greens are putting out an infographic based on an age crossbreak. Our resident Greens, do please have a quiet word.

  50. CROSSBAT

    On the economy.. I think what’s happening is that the recovery is top heavy and the benefits are not filtering down to Mr & Mrs average.

    However reports also show unemployment is coming down so the people who create wealth and jobs are seeing their wealth go up but unfortunately for the onions their wages haven’t crept up.

    But I have to say compared to the mess Europe is in i think we should count ourselves lucky and I know people who are on low wages and are unemployed are finding times more difficult but that scenario is not unique just to the UK.

    The real losers today are those who are unemployed and low wage earners but the economy is growing and jobs are being created despite the horrific mess our biggest trading partner, the EU is in.

1 2 3 4 5