Donald Trump has won, so we have another round of stories about polling shortcomings, though thankfully it’s someone else’s country this time round (this is very much a personal take from across an ocean – the Yougov American and British teams are quite separate, so I have no insider angle on the YouGov American polls to offer).

A couple of weeks ago I wrote about whether there was potential for the US polls to suffer the same sort of polling mishap as Britain had experienced in 2015. It now looks as if they have. The US polling industry actually has a very good record of accuracy – they obviously have a lot more contests to poll, a lot more information to hand (and probably a lot more money!), but nevertheless – if you put aside the 2000 exit poll, you have to go back to 1948 to find a complete polling catastrophe in the US. That expectation of accuracy means they’ll probably face a lot of flak in the days ahead.

We in Britain have, shall I say, more recent experience of the art of being wrong, so here’s what insight I can offer. First the Brexit comparison. I fear this will be almost universal over the next few weeks, but when it comes to polling it is questionable:

  • In the case of Brexit, the polling picture was mixed. Put crudely, telephone polls showed a clear lead for Remain, online polls showed a tight race, with leave often ahead. Our media expected Remain to win and wrongly focused only on those polls that agreed with them, leading to a false narrative of a clear Remain lead, rather than a close run thing. Some polls were wrong, but the perception that they were all off is wrong – it was a failure of interpretation.
  • In the case of the USA, the polling picture was not really mixed. With the exception of the outlying USC Dornslife/LA Times poll all the polls tended to show a picture of Clinton leading, backed up by state polls also showing Clinton leads consistent with the national polls. People were quite right to interpret the polls as showing Clinton heading towards victory… it was the polls themselves that were wrong.

How wrong were they? As I write, it looks as if Hillary Clinton will actually get the most votes, but lose in the Electoral College. In that sense, the national polls were not wrong when they showed Clinton ahead, she really was. It’s one of the most fustrating situations to be in as a pollster, those times when statistically you are correct… but your figures have told the wrong narrative, so everyone thinks you are wrong. That doesn’t get the American pollsters off the hook though: the final polls were clustered around a 4 point lead for Clinton, when in reality it looks about 1 point. More importantly, the state polls were often way out, polls had Ohio as a tight race when Trump stomped it by 8 points. All the polls in Wisconsin had Clinton clearly ahead; Trump won. Polls in Minnesota were showing Clinton leads of 5-10 points, it ended up on a knife edge. Clearly something went deeply wrong here.

Putting aside exactly how comparable the Brexit polls and the Trump polls are, there are some potential lessons in terms of polling methodology. I am no expert in US polling, so I’ll leave it to others more knowledgable than I to dig through the entrails of the election polls. However, based on my experiences of recent mishaps in British polling, there are a couple of places I would certainly start looking.

One is turnout modelling – US pollsters often approach turnout in a very different way how British pollsters traditionally did it. We’ve always relied on weighting to the profile of the whole population and asking people if they are likely to vote. US pollsters have access to far more information on which people actually do vote, allowing they to weight their samples to the profile of actual voters in a state. This has helped the normally good record of US pollsters… but carries a potential risk if the type of people who vote changes, if there is an unexpected increase in turnout among demographics who don’t usually vote. This was one of the ways British pollsters did get burnt over Brexit. After getting the 2015 election wrong lots of British companies experimented with a more US-style approach, modelling turnout on the basis of people’s demographics. Those companies then faced problems when there was unexpectedly high turnout from more working-class, less well-educated voters at the referendum. Luckily for US pollsters, the relatively easy availability of data on who voted means they should be able to rule this in or out quite easily.

The second is sampling. The inquiry into our general election polling error in 2015 found that unrepresentative samples were the core of the problem, and I can well imagine that this is a problem that risks affecting pollsters anywhere. Across the world landline penetration is falling, response rates are falling and it seems likely that the dwindling number of people still willing to take part in polls are ever more unrepresentative. In this country our samples seemed to be skewed towards people who were too educated, who paid too much attention to politics, followed the news agenda and the political media too closely. We under-represented those with little interest in politics, and several UK pollsters have since started sampling and weighting by that to try and address the issue. Were the US pollsters to suffer a similar problem one can easily imagine how it could result in polls under-representing Donald Trump’s support. If that does end up being the case, the question will be what US pollsters do to address the issue.


1,352 Responses to “Why were the US polls wrong?”

1 23 24 25 26 27 28
  1. LibDems into double figures with YouGov (twice) and now Mori, so that looks like the new normal for them, plus the gap closing a bit for Labour but still a very long way to go…

  2. OLDNAT

    I don’t really understand what , or where, you mean by the “dashboard”.

    But thanks anyway for the info. We all see them all as Capitals I presume ?

    I suppose it is possible that name of the other UKPR contributor you mention is also a proper name. We only have it from AW’s algorithms that it isn’t. I mean they wouldn’t actually “know” would they?

  3. @TOH – “These figures do not support you post on this poll i would suggest. Have you a view on that?”

    I’d agree. There seems an odd disparity between the approval rating and headline VI, which tends to suggest a recovery from an outlier, although still odd that Labour’s VI has gone up quite sharply.

    However, I remain correct, I would argue, in saying we need more polls……

  4. “@ Thoughtful

    “I’m not sure as to why you think that I’ve either aimed my comment at you personally, nor why you think I was ‘irate’.

    Simple. It was ASSIDUOSITY in capital letters at the start of your post that led me to believe the comment was addressed to me and that its contents bore some specific relevance to myself.”

    One of those ‘bang head on wall it’s so obvious’ moments we occasionally get on UKPR.

  5. @”, we need more polls……”

    The UKPR cri de cœur. It is always true-even during a GE campaign !

    Never more so than now. There is so water of major political relevance to pass under the bridge .

  6. Laszlo

    Thanks for the link. I had never heard of the Party Members Project!

    Even the basic data about some aspects of party membership (one of the frames on the slide show on their home page) was notable –

    The contrast between Green and UKIP members is dramatic (if not unexpected).

    https://esrcpartymembersproject.org/

  7. Somerjohn,
    Not sure what the balance of posters is but I suspect the balance of posts is in favour of the retired members!

    But the number of people posting frequently is quite small and certainly within error of 50-50 on Brexit, I would say..

  8. If the consensus is that the posters on here roughly represent the views of the voters on Brexit, and that we need more polls, why don’t we do our own?

    1) Do you want hard Brexit? i.e. regain ability to control immigration regardless of effect on membership of the single market?
    2) Soft brexit?
    3) Stay in EU?

    I’ll start.

    1.

  9. Pete B

    But we would need a Full Scottish poll as well – and I suspect that might be a little less representative of Scots political opinion!

  10. ON
    You’re welcome to start your own Indyref2. I vote Leave.

  11. Pete B

    3 for me.

    (But as ever with polls, it’s not as simple as that. What I really want is full-hearted membership, but failing that, maybe hard Brexit is the way to go. For the greater good…)

  12. Colin and OldNat

    Yes, it is worth reading their various articles (there aren’t that many journal ones, so they are probably going for a book – which makes sense). The research on Conservative Party membership is also interesting. Really conflicting with the standard narrative, with polling, and with the members own perceptions.

    They use YouGov for many of their initial research, then (it seems to me) they use their own researchers.

    The great advantage of an ESRC project is that the data must be made available (eventually).

  13. @peteb

    3 for me.

  14. PETE B

    I don’t actually want any of your three choices !

    I was aware of both arguments.

    I’ve been to Luxembourg and I’ve even sat in Junkers chair, I’d like to think that my decision was as informed as it reasonably could have been, the campaigns were largely ‘noise’.

    The onus with article 50 is on the EU to present its terms and not the UK. If we go for ‘hard’ Brexit and it isn’t perfect the recriminations will be horrible.

    The problem is, that at the present time the EU hasn’t revealed any of its negotiating position.

    May has indicated – and todays unemployment figures won’t help that immigration is a major issue. Merkel is still as intransigent as ever she was and has proved a major obstacle to a soft Brexit. She has announced that she will stand again for election, although this might not be very successful.

    From the moment Article 50 is triggered the negotiations are supposed to be completed, in two years. In that time there will probably be a new French President, an adverse (for the EU) referendum result in Italy, and potentially a new German leader.

    All of this will significantly change the playing field for Brexit negotiations, to the extent that simplistic choices simply won’t do.

  15. Alec

    “However, I remain correct, I would argue, in saying we need more polls……”

    Absolutely, I think I said the same earlier.

  16. Thoughtful
    It’s only a bit of fun.

  17. Pete B

    Since we have not negotiated yet I think “hard and soft brexit” rather meaningless terms. We do not know where the balance of advantage will lie when we start negotiating so it is difficult to speculate what the result might be, if any of the negotiation.

    What I want is control of our own borders, exit from control of ECJ and no payments to the EU. Let’s sum that up as leaving the EU in the fullest sense.

  18. 3333333333333 for me (as the phone adverts say)

    BTW the most amazing thing in the Curtice paper (linked again here: http://whatukthinks.org/eu/wp-content/uploads/2016/11/Analysis-paper-9-What-do-voters-want-from-Brexit.pdf )

    is that a plurality of those surveyed were in favour of more expensive phone calls when on holiday on the continent! (although of course the question was “do you want Britain to continue to follow EU regulations on the cost of mobile phone calls made abroad) Perhaps this shows how the drip feed of straight bananas etc in the press leads people to instinctively think an EU regulation on this would be bad for them? But paradoxically the same people (including 55% of Leaver voters) appear to trust the EU more than the British government on Health and Safety! (which of course is actually the “red tape” many of the Leave leaders want to get rid of…)

    I think people tend to say yes to “balanced” looking questions. So when the question is “Allowing boats from the EU to fish in British waters in return for allowing
    British boats to fish in EU waters” most people (60%) say yes. But if the question was “Should fishing boats from the EU have free access to British fishing waters” they would, I suspect, have said resoundingly no!

    Similarly, 62% of people agreed with “No longer allowing people from EU countries who are visiting Britain to get
    NHS treatment for free”, but I bet the answer would have been very different if it had added “in return for Britons getting free access to healthcare in Europe when on holiday”

    This question was agreed with by 74% “Requiring people from the EU who want to come to live here to apply to do
    so in the same way as people from outside the EU” partly I suspect because it sounds balanced. I bet most would also have agreed with “allowing other EU citizens the right to live and work in Britain in return for the right of British citizens to live and work in Europe”

    Anyway, this makes me a very suspicious of all the questions… people adopt positions on issues (and I mean both Leavers and Remainers) on gut feeling not analysis, despite being pretty much in ignorance about what it really means. And in the end Leave won because people trusted Boris and Nigel (and the bloke next door) a bit more than Cameron and Osborne..

  19. Yes Pete sorry for being a party pooper, I just think it’s a lot more complex.

    When the accession countries joined the EU there was a limit put on numbers which Blair refused, and many many more came than they estimated.

    Suppose a situation where the EU say we can have access to the single market if we agree to free movement of people with a cap (insert appropriate number as you please).

    What then? You have control of the borders with a n upper limit.

    That I think would really drive a wedge into public opinion in the UK, and it would not be a surprise to find that actually offer this.

  20. Pete B

    I’m reminded of a comment about sexist jokes also being “a bit of fun”.

    My daughter said “Aye, ‘F U’ is ‘a bit of fun’ too!”.

    Nothing to do with your voodoo poll – your comment just reminded me of her.

  21. Andrew111

    From Curtice’s article, I got the impression that he might have been outvoted on the wording of the questions.

    While he is, of course, academically courteous in his criticism of the question wording, I suspect that his actual thoughts may have been somewhat more crudely worded in the University Staff Coffee Lounge. :-)

  22. Pete B 3 but I don’t think its on offer, then 2 but I don’t think its on offer and as for 1 i don’t want it. Abstain?

  23. Charles

    “Abstain?”

    Welcome to the Labour Party! :-)

  24. Oldnat:

    I suspect you are correct… But I also suspect that many of the mainstream polls are commissioned by people who know what they want and know how to frame the questions to get that..

    There are big lessons for political parties and campaigns here too… If you have an unpopular policy, always talk of it as balancing something that sounds much better. If you are against the unpopular policy, just hammer it repeatedly, preferably exaggerating….

  25. Andrew111

    “I also suspect that many of the mainstream polls are commissioned by people who know what they want and know how to frame the questions to get that..”

    Also agree with that. The questions, as phrased, may meet the requirements of the pollsters they are paying – but it’s a straightforward commercial transaction.

    Unless the polling companies are somehow transmogrified into the arbiters of truth, then there seems no way round that.

  26. Thoughtful: “Yes Pete sorry for being a party pooper, I just think it’s a lot more complex.”

    That’s the problem with yes/no referendums.

  27. “but I reckon we get a pretty good supply of thesis and antithesis on here, which may not always lead to synthesis”

    ——–

    Yes theres a marked lack of interest in synthesis on here, but currency doesn’t help…

  28. THOUGHTFUL

    @” it would not be a surprise to find that actually offer this.”

    That has been my feeling for a while now.

  29. It’s not long since some on here were rubbing their hands in glee at the prospect of Deutsche Bank being fined $14bn by the USA, citing it as a nail in the coffin of the eurozone.

    Now it’s RBS’s turn:

    “Royal Bank of Scotland could face a penalty of more than $12bn (£9.6bn) to settle a decades-old mis-selling scandal in the US, the body which controls the taxpayer stake in the bank has said.”

    Nail in the coffin of …?

  30. Sorry, missed a couple of intelligent replies to my comment, as I was out with Mrs A at the movies (Arrival is excellent by the way, although it wasn’t really Mrs A’s cup of tea).

    I should perhaps have been a bit clearer about what I meant by centre of gravity. I didn’t really mean in terms of Remain / Leave split, but in terms of views about the way forward for the UK in the Brexit process.

    It seems that the urge to fight, fight and fight again against Brexit is a lot stronger here than it is amongst most politicians, including Remainers, and including Labour politicians.

    That puts the 50%-60% of UKPR commenters who are pro-Remain a bit more in the SNP/LD corner than the Tory/Labour one.

    My point was that if someone like Keir Starmer or John Healey came on here under a pseudonym (who knows, maybe they do?) and expressed views similar to those they express on R4, they’d be roundly shouted down.

  31. @Somerjohn,

    Quick, let’s have Indyref2, hive off RBS to Scotland and then declare it bankrupt…

  32. I have read that trump may be interested in a transaction tax. If other world financial centres agree our treasury would benefit by £250 bn per annum.

    The benefits to the uk would be enormous. I understand that the EU is in favour but wishes such a tax to be an EU tax payable to them.

    Is the single market worth this?

  33. @Thoughtful – “The onus with article 50 is on the EU to present its terms and not the UK. If we go for ‘hard’ Brexit and it isn’t perfect the recriminations will be horrible.

    The problem is, that at the present time the EU hasn’t revealed any of its negotiating position.”

    While I generally glaze over and bypass all posts that include the term ‘article 50’, I had to pick you up on this one because you are completely wrong.

    There is no onus whatsoever on the EU to present it’s terms once we decide to formally say we wish to leave. There are perfectly adequate leaving terms available to us, that don’t require any negotiating. If we want anything else, then it’s up to us to ask for it.

    I’m not saying that it isn’t in the EU’s best interests to engage in this process, because of course it is, but many Brexiters still haven’t grasped the fact that just because the UK has decided it wants something, everyone else must do our bidding.

    That isn’t the way the world works.

  34. @Alec

    In fairness, all the talk from the UK so far has been about what we want, and all the talk from the EU so far has been about what we can’t have.

    If we have a list of 100 things we want ruled in, and they have a list of things they want ruled out that includes 99 of the things on our list, seeing their list is probably a lot more important than seeing ours.

  35. @S Thomas – “I have read that trump may be interested in a transaction tax. If other world financial centres agree our treasury would benefit by £250 bn per annum.

    The benefits to the uk would be enormous. I understand that the EU is in favour but wishes such a tax to be an EU tax payable to them.”

    No idea about Trump, but you could have readily researched the position regarding the EU.

    10 states already levy some form of Financial Transaction Tax, and the main idea behind this one was to repay taxpayers the subsidies paid out after the crash and harmonise existing taxes. As a new tax, the FTT proposals require unanimity, but the ECouncil agreed for countries wanting to pursue this to do so without the others. 11 countries agreed to do so, while the other 16 (including the UK) stayed out.

    The EU (presumably you mean the E Commission?) did not propose that the tax would go to EU budgets – it would go to the levying country. Under the current proposals, the EC estimates UK would see an additional £10b of revenues, if it chose to join the scheme.

  36. Apols – it was Euro10b, not sterling.

  37. @Neil A – “If we have a list of 100 things we want ruled in, and they have a list of things they want ruled out that includes 99 of the things on our list, seeing their list is probably a lot more important than seeing ours.”

    Again, this is to misunderstand the process.

    The task of establishing the first set of demands in a negotiation is quite difficult. The question of where to pitch your demands is a tricky one, as you ideally would want to know what the other side is thinking and what they would settle for first. The minute you have done this you have shown your hand, up to a point. Why would the EU tell us what it might be prepared to accept, when it’s better for them to keep everything in their armory an active possible issue? The minute they say to us what they would be prepared to accept, then they have lost negotiating position.

    It’s not up to the EU to be bothered about this. The UK chose to leave, and can do so without any negotiating if we want, so why should the EU do anything?

  38. Alec

    “The UK chose to leave, and can do so without any negotiating if we want”

    Although any UK decision not to negotiate at all would mean simply accepting the EU determination of how much the UK is liable to pay the EU for its future costs created during our membership – or simply defaulting on a debt.

  39. @Alec

    I agree on the best approach to negotiating, although some don’t seem to apply that to the UK side.

    My point is that putting forward UK objectives is a bit pointless until the EU is prepared to actually respond to them.

    At the moment it’s like being in a really bad restaurant.

    “I’d like the duck please”.

    “It’s off”

    “The chicken?”

    “No chicken either”.

    “How about some lamb cutlets?”

    “Sorry, all out of lamb cutlets”…

  40. @Oldnat

    My understanding is that £60bn bill related to the costs of continuing to participate in things as a transitional arrangement.

    I am not sure where the EU would stand on a unilateral demand for money from a newly-independent UK to which they were not offering anything.

  41. ALEC

    The position was stated On the PM program on radio 4 following the High Court decision on triggering article 50. The guy they had on and unfortunately I can’t remember his name stated quite clearly that the ball is in the EUs court as far as starting negotiations, and it is for them to present their terms first.

    I had no reason to doubt him, but if you know different then please point me in a direction I can research it.

    Remember that Article 218(3) of the Treaty on the Functioning of the European Union applies to the negotiations, it states:

    ” The Commission, or the High Representative of the Union for Foreign Affairs and Security Policy where the agreement envisaged relates exclusively or principally to the common foreign and security policy, shall submit recommendations to the Council, which shall adopt a decision authorising the opening of negotiations and, depending on the subject of the agreement envisaged, nominating the Union negotiator or the head of the Union’s negotiating team.”

  42. @Old Nat – ‘Welcome to the Labour Party’ – well I have joined them but I am not convinced they are doing what I want!

    @Neil A In fairness, all the talk from the UK so far has been about what we want, and all the talk from the EU so far has been about what we can’t have.

    Seems a fair description of what has been going on. Can’t we shift the conversation to what we both want and how on earth we can get it?

    Italian minister has now apparently accused Boris of insulting Italy (possibly unfairly on this occasion, although the expectation that he would do so was fair enough). Interesting to read the most popular comments on the BBC website and horribly reminiscent of the American campaign.

  43. @Charles

    It would be nice, but I think the EU are still hoping to be able to game a non-Brexit outcome, so probably not until that particular issue is put to bed.

  44. why not print £60bn and pay them.Uk will then pay no contributions

    To those who shout what a stupid idea have we not printed north of £350 billion in QE and given it to the banks?

  45. Neil A

    There are future costs that the EU will have to bear which the UK, as a member, is bound to pay – eg the pensions of EU civil servants and MEPs etc etc..

    There will no doubt be transitional costs which the UK – as an existing member – may be liable for. [1]

    My point was simply that, on such matters – as opposed to future trade deals etc – the UK can negotiate, pay up, or refuse to pay. What other options are there?

    [1] I am reminded that it was suggested in rUK that Scotland should have to bear the cost of the rUK’s negotiators! Geese and ganders come to mind. :-)

  46. @Oldnat

    I am fairly sure we could offer to pay the pensions of British Eurocrats and MEPs. Fair’s fair.

  47. Neil A

    See. You’re negotiating!

    That was my point.

  48. Neil A

    Well, I think many Labour MP’s would like to be more anti-Brexit, but they are worried about another fight with Corbyn and a bit scared to talk to people in some parts of their constituencies.

    Meanwhile quite a lot of Labour members would like to fight on and there is a petition out there to that effect.

    Speaking as a Lib Dem, I think the position of a second referendum is supported by most members, and tenable politically on the grounds that giving people an opportunity to vote cannot realistically be called “undemocratic”. I suspect it has firmed up as a result of what is being heard on the doorsteps in Richmond Park (and also by the Labour candidate announcing he would vote against A50).

    I think the effect of Brexit on politics in Britain still has a long way to go, and the consequences for all political parties are uncertain… Interesting times!

  49. Re.EU debts,

    No doubt defaulting on these these debts would leave us open to a credit downgrade and all that that entails…

    Which is why the govt will be very keen to negotiate them down… And why “just break the treaty” is not a very good idea unless we wish to pull up the drawbridge like Russia…

  50. @ANDREW111

    “Which is why the govt will be very keen to negotiate them down… And why “just break the treaty” is not a very good idea unless we wish to pull up the drawbridge like Russia…”

    Personally I think that’s where we are heading. Britain will be western Europe’s Russian equivalent. Just another rogue nationalist nation with no friends, other than possibly a 70 year old American with orange hair. God help us.

1 23 24 25 26 27 28