Donald Trump has won, so we have another round of stories about polling shortcomings, though thankfully it’s someone else’s country this time round (this is very much a personal take from across an ocean – the Yougov American and British teams are quite separate, so I have no insider angle on the YouGov American polls to offer).

A couple of weeks ago I wrote about whether there was potential for the US polls to suffer the same sort of polling mishap as Britain had experienced in 2015. It now looks as if they have. The US polling industry actually has a very good record of accuracy – they obviously have a lot more contests to poll, a lot more information to hand (and probably a lot more money!), but nevertheless – if you put aside the 2000 exit poll, you have to go back to 1948 to find a complete polling catastrophe in the US. That expectation of accuracy means they’ll probably face a lot of flak in the days ahead.

We in Britain have, shall I say, more recent experience of the art of being wrong, so here’s what insight I can offer. First the Brexit comparison. I fear this will be almost universal over the next few weeks, but when it comes to polling it is questionable:

  • In the case of Brexit, the polling picture was mixed. Put crudely, telephone polls showed a clear lead for Remain, online polls showed a tight race, with leave often ahead. Our media expected Remain to win and wrongly focused only on those polls that agreed with them, leading to a false narrative of a clear Remain lead, rather than a close run thing. Some polls were wrong, but the perception that they were all off is wrong – it was a failure of interpretation.
  • In the case of the USA, the polling picture was not really mixed. With the exception of the outlying USC Dornslife/LA Times poll all the polls tended to show a picture of Clinton leading, backed up by state polls also showing Clinton leads consistent with the national polls. People were quite right to interpret the polls as showing Clinton heading towards victory… it was the polls themselves that were wrong.

How wrong were they? As I write, it looks as if Hillary Clinton will actually get the most votes, but lose in the Electoral College. In that sense, the national polls were not wrong when they showed Clinton ahead, she really was. It’s one of the most fustrating situations to be in as a pollster, those times when statistically you are correct… but your figures have told the wrong narrative, so everyone thinks you are wrong. That doesn’t get the American pollsters off the hook though: the final polls were clustered around a 4 point lead for Clinton, when in reality it looks about 1 point. More importantly, the state polls were often way out, polls had Ohio as a tight race when Trump stomped it by 8 points. All the polls in Wisconsin had Clinton clearly ahead; Trump won. Polls in Minnesota were showing Clinton leads of 5-10 points, it ended up on a knife edge. Clearly something went deeply wrong here.

Putting aside exactly how comparable the Brexit polls and the Trump polls are, there are some potential lessons in terms of polling methodology. I am no expert in US polling, so I’ll leave it to others more knowledgable than I to dig through the entrails of the election polls. However, based on my experiences of recent mishaps in British polling, there are a couple of places I would certainly start looking.

One is turnout modelling – US pollsters often approach turnout in a very different way how British pollsters traditionally did it. We’ve always relied on weighting to the profile of the whole population and asking people if they are likely to vote. US pollsters have access to far more information on which people actually do vote, allowing they to weight their samples to the profile of actual voters in a state. This has helped the normally good record of US pollsters… but carries a potential risk if the type of people who vote changes, if there is an unexpected increase in turnout among demographics who don’t usually vote. This was one of the ways British pollsters did get burnt over Brexit. After getting the 2015 election wrong lots of British companies experimented with a more US-style approach, modelling turnout on the basis of people’s demographics. Those companies then faced problems when there was unexpectedly high turnout from more working-class, less well-educated voters at the referendum. Luckily for US pollsters, the relatively easy availability of data on who voted means they should be able to rule this in or out quite easily.

The second is sampling. The inquiry into our general election polling error in 2015 found that unrepresentative samples were the core of the problem, and I can well imagine that this is a problem that risks affecting pollsters anywhere. Across the world landline penetration is falling, response rates are falling and it seems likely that the dwindling number of people still willing to take part in polls are ever more unrepresentative. In this country our samples seemed to be skewed towards people who were too educated, who paid too much attention to politics, followed the news agenda and the political media too closely. We under-represented those with little interest in politics, and several UK pollsters have since started sampling and weighting by that to try and address the issue. Were the US pollsters to suffer a similar problem one can easily imagine how it could result in polls under-representing Donald Trump’s support. If that does end up being the case, the question will be what US pollsters do to address the issue.


Donald Trump has been citing Brexit as the model of how he could win the election despite expections, his surrogates of how there might be a shy Trump vote, like Brexit. So what, if any, lessons can we learn about the US election from recent polling experience in Britain?

In 2015 the British polls got the general election wrong. Every company had Labour and Conservative pretty much neck-and-neck, when in reality the Conservatives won by seven points. In contrast, the opinion polls as a whole were not wrong on Brexit, or at least, they were not all that wrong. Throughout the referendum campaign polls conducted by telephone generally showed Remain ahead, but polls conducted online generally showed a very tight race. Most of the online polls towards the end of the campaign showed Leave ahead, and polls by TNS and Opinium showed Leave ahead in their final eve-of-referendum polls.

That’s the first point that the parallel falls down – Brexit wasn’t a surprise because the polls were wrong. The polls were showing a race that was neck-and-neck. It was a surprise because people hadn’t believed or paid attention to that polling evidence. The media expected Remain would win, took polls showing Remain ahead more seriously and a false narrative built up that the telephone polls were more accurately reflecting the race when in the event, those online polls showing leave ahead were right. This is not the case in the US – the media don’t think Trump will lose because they are downplaying inconvenient polling evidence, they think Trump will lose because of the polling evidence consistently shows that.

In the 2015 general election however the British polls really were wrong, and while some of the polls got Brexit right, some did indeed show solid Leave victories. Do either of those have any relevance for Trump?

The first claim is the case of shy voters. Much as 1948 is the famous examples of polling failure in the US, in this country 1992 was the famous mistake, and was put down to “Shy Tories”. That is, people who intended to vote Conservative, but were unwilling to admit it to pollsters. Shy voters are extremely difficult to diagnose. If people lie to pollsters about how they’ll vote before the election but tell the truth afterwards, then it is impossible to distinguish “shy voters” from people changing their minds (in the case of recent British polls, this does not appear to be the case. In both the 2015 election and the 2016 EU referendum recontact surveys found no significant movement towards the Conservatives or towards Leave). Alternatively, if people are consistent in lying to pollsters about their intentions beforehand and lying about how they voted afterwards, it’s impossible to catch them out.

The one indirect way of diagnosing shy voters is to compare the answers given to surveys using live interviewers, and surveys conducted online (or in the US, using robocalls – something that isn’t regularly done in the UK). If people are reluctant to admit to voting a certain way, they should be less embarrassed when it isn’t an actual human being doing the interviewing. In the UK the inquiry used this approach to rule out “shy Tories” as a cause of the 2015 polling error (online polls did not have a higher level of Tory support than phone polls).

In the US election there does appear to be some prima facie evidence of “Shy Trumpers”* – online polls and robopolls have tended to produce better figures for Donald Trump than polls conducted by a human interviewer. However, when this same difference was evident during the primary season the polls without a live interviewer were not consistently more accurate (and besides, even polls conducted without a human interviewer still have Clinton reliably ahead).

The more interesting issue is sample error. It is wrong to read directly across from Brexit to Trump – while there are superficial similarities, these are different countries, very different sorts of elections, in different party systems and traditions. There will be many different drivers of support. To my mind the interesting similarity though is the demographics – the type of people who vote for Trump and voted for Brexit.

Going back to the British general election of 2015, the inquiry afterwards identified sampling error as the cause of the polling error: the sort of people who were able to be contacted by phone and agreed to take part, and the sort of people who joined online panels were unrepresentative in a way that weights and quotas were not then correcting. While the inquiry didn’t specify how the samples were wrong, my own view (and one that is shared by some other pollsters) is that the root cause was that polling samples were too engaged, too political, too educated. We disproportionately got politically-aware graduates, the sort of people who follow politics in the media and understand what is going on. We don’t get enough of the poorly educated who pay little attention to politics. Since then several British companies have adopted extra weights and quotas by education level and level of interest in politics.

The relevance for Brexit polling is that there was a strong correlation between educational qualification and how people voted. Even within age cohorts, graduates were more likely to vote to Remain, people with few or no educational qualifications were more likely to vote to Leave. People with a low level of interest in politics were also more likely to vote to Leave. These continuing sampling issues may well have contributed to some of those pollsters who did it wrong in June.

One thing that Brexit does have in common with Trump is those demographics. Trump’s support is much greater among those without a college degree. I suspect if you asked you’d find it was greater among those people who don’t normally pay much attention to politics. In the UK those are groups who we’ve had difficulty in properly representing in polling samples – if US pollsters have similar issues, then there is a potential source for error. College degree seems to be a relatively standard demographic in US polling, so I assume that is correct already. How much interest people have in politics is more nebulous, less easy to measure or control.

In Britain the root cause of polling mishaps in 2015 (and for some, but not all, companies in 2016) seems to be that the declining pool of people still willing to take part in polls under-represented certain groups, and that those groups were less likely to vote for Labour, more likely to vote for Brexit. If (and it’s a huge if – I am only reporting the British experience, not passing judgement on American polls) the sort of people who American pollsters struggle to reach in these days of declining response rates are more likely to vote for Trump, then they may experience similar problems.

Those thinking that the sort of error that affected British polls could happen in the US are indeed correct… but could happen is not the same as is happening. Saying something is possible is a long way from there being any evidence that is actually is happening. Some of the British polls got Brexit wrong, and Trump is a little bit Brexity, therefore the polls are wrong really doesn’t carry water.

xxxxxxxxxxxxxxxxxx

*This has no place in a sensible article about polling methodology, but I feel I should point out to US readers that in British schoolboy slang when I was a kid – and possibly still today – to Trump is to fart. “Shy Trump” sounds like it should refer to surreptitiously breaking wind and denying it.


-->

US Election Night

Tonight, you should hardly need telling, is US election night, so here’s a thread for overnight discussion and some things to look out for tonight.

First up a note about exit polls. Exit polls this year are only being conducted in 31 states rather than 50, they aren’t bothering with some of the safe states, including some of those states closing first (so no exit polls in Kentucky, Georgia, South Carolina and West Virginia – all safe Republican states). Exit polls are done in the same way as in the UK – people stand outside polling stations and get a proportion of people leaving after having voted to fill in surveys, although unlike in the UK they ask a full survey about their opinions and why they voted as they did, not just how they voted. Exit polls are supplemented with phone polls to take account of early voters.

Exit polls are not released until the polls in that state are closed, people will inevitably claim to have leaked exit poll data prior to that. Any claimed leaks before 10pm will be bollocks anyway – the polling data will still be under strict quarantine in a locked room. Any claimed leaks after that will probably be bollocks too, and should be ignored anyway as it won’t be probably weighted yet. Also bear in mind that in recent elections the initially released exit poll data has tended to be a bit skewed to the Democrats, becoming more accurate as actual votes come in. That may or may not be the case this time.

The initial exit polls are updated as actual votes are counted, and weighted based on the declared votes in the districts where the exit poll took place. Once the networks are certain that a party has won the seat they call it – obviously the closer a state is, the longer it takes for the networks to be certain who has won. Hence while we have a list of the time that states’ polls close and exit polls will be released, its it only the very safe states that will be called straight away. In 2008 states where the vote was relatively close (say, under 10 points) sometimes took a couple of hours after polls closed for the networks to call the race, states with the tightest races took much longer to call: Montana and Florida four hours, North Caroline a day, Missouri a week.

In a tight race, don’t expect a result in the early hours!

Looking at the timetable.

11pm. Most polls close in Indiana and Kentucky – some parts of both states are an hour behind, so the networks may not call them until the whole state has finished voting, but either way both will vote Romney.

12 midnight. Polls close in Georgia, South Carolina, Vermont & Virginia. Most polls close in Florida. Georgia and South Carolina will vote Romney, Vermont will vote Obama. Virginia and Florida are the first toss-up states. For Romney to win, he really needs to win both of these – if Obama wins either of them then it becomes difficult (but not impossible) for Romney to win. The polls in Florida are neck-and-neck, if they have been accurate it is not going to be called for many hours. Most recent polls in Virginia have shown Obama ahead, but it will probably be a few hours until it is called.

12:30 am. North Carolina, Ohio and West Virginia polls close. West Virginia will vote Romney. North Carolina and Ohio are another two key states. Recent polls have tended to show Romney ahead in North Carolina, again it is a state he really does need to win. Ohio is likely to be the key – if Romney wins Virginia, North Carolina and Florida he needs Ohio and another state (New Hampshire, Colorado, Iowa?) to win. If Obama holds Ohio he has won unless he loses something unexpected like Wisconsin or Pennsylvania. At 12:30 though this will be academic – if the race is at all close they aren’t going to be calling it yet.

1 am. A whole slew of states close: Connecticut, Delaware, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, Pennsylvania, Rhode Island, Alabama, Illinois, Mississippi, Missouri, Oklahoma, Tennessee, most of Texas and Michigan and the remaining part of Florida close. Apart from Florida the only really interesting states amongst them all are New Hampshire and Pennsylvania. Pennsylvania should be solidly Democrat, Obama won it by 10 points last time, John Kerry won it, all but one of the recent polls have shown Obama ahead, often by good margins. It should be Obama. However, both campaigns have been targeting it so it must be seen as somewhat close. In the event that Romney does win it he would probably win unless Obama had won Virginia, North Carolina or Florida instead (though frankly, the idea of the Democrats winning one of those states and not Pennsylvania is somewhat bonkers)

1:30 am. Arkansas polls close.

2:00 am. New York, Kansas, Louisana, Minnesota, Nebraska, Wisconsin, Arizona, Colorado, New Mexico, South Dakota, Wyoming polls close, the remainder of Texas and Michigan close. Again, these are mostly safe states – the two interesting ones are Wisconsin (which the polls suggest should go for Obama without too much difficulty) and Colorado, where the polls are closer, but are consistently showing Obama ahead. More interesting is that by now we’ve had a couple of hours for votes to be counted in the early-closing states, so with a bit of luck we should start to see competitive states being called. The broad picture of the night, whether Obama is probably home and dry, whether it is going to be close or whether there could be a Romney win will hopefully start to emerge around now.

3:00 am. Polls close in Iowa, Montana, Utah, Nevada and most of North Dakota. Iowa and Nevada are the last of the swing states. Nevada looks like it should go to Obama, Iowa’s polls have Obama ahead, but less convincingly.

4:00 am. California, Washington, Hawaii. Most of Idaho, Oregon. All safe states for one side or the other. The closer run states should be being called around now. Unless things have gone right down to the wire we should soon know who has won. If they have gone right down to a couple of states where the candidates are neck-and-neck it could take days. Go to bed.

5:00 am. Alaska. Go! Go to bed. Sheesh.


I haven’t written anything about the US elections yet this cycle. This is for two reasons, first because there are vast amounts of US polling to get on top of in order to say anything sensible, secondly because there are already some very good US polling sites that I couldn’t hope to better. If you want to read wise and sensible analysis of US polling don’t hang around here, go and read Mark Blumenthal and Simon Jackman.

However, since we are now within a week of the election I thought I may as well put some threads up if only for discussion.

First, all things I complain about in coverage of UK polling are the same in US polling. Most notably warnings about cherry picking, comparing like to like and being aware of methodological differences and house effects from different pollsters. For example, I keep seeing people cherry picking out Rasmussen polls and Gallup polls to claim that Romney is doing better. Rasmussen are one of the most prolific polling outfits in the US, but also tend to produce some of the most Republican results. Gallup use a very tight screen for likely voters that also tends to produce favourable figures for the republicans. Look at most other polls and Obama is doing better.

Secondly, remember that the person who gets the most votes doesn’t necessarily win, it is who wins states with enough electoral votes to win a majority (270) of the electoral college. The average picture across all the national polls in the US has Romney and Obama very much neck and neck. However, polls from the key swing states, which themselves have become very regular as the election approaches, have Obama clearly ahead in terms of electoral votes.

There are various US websites (I’ve already mentioned Pollster.com, though fivethirtyeight tends to be the best known these days) that make projections based on state polling, and these all show Barack Obama with large leads in terms of electoral votes.

This has, in turn, produced some (generally pretty poorly informed) criticism of the projection sites, normally based around what sort of weights they give to different polls, what polls they include and so on. I don’t think these criticisms carry any weight, however even if one is sceptical about the weightings, filters, trends, house effect adjustments or whatever that the various projection sites make, the bottom line is that even if one takes just a crude average of state polls, Obama is still ahead.

As I write, Obama is almost undoubtedly ahead in states worth 243 votes. He needs to pick up another 27 electoral votes to win – looking at the recent polling in states that are in play:

  • In Wisconsin (10 votes) Rasmussen has the candidates equal in their last poll, but all three polls done in the last week have Obama significantly ahead
  • There have been four Iowa (6 votes) polls in the last week, three have had Obama ahead, the other had Romney one point ahead (but had a very small sample size)
  • In Ohio (18 votes), which is very likely to be the deciding state, there have been 11 polls in the last week, ten showed Obama ahead, one had Romney ahead
  • In New Hampshire (4 votes) there have been three polls in the last week, all showing Obama ahead.
  • In Colorado (9 votes) the four polls in the last week have been evenly split, 2 showing Obama ahead, 2 Romney ahead (though the Obama polls had bigger leads)
  • In Virginia (13 votes) there have been 9 polls in the last week, 2 showed Romney ahead, 1 a tie, 6 Obama ahead.
  • Florida (28 votes) is really neck-and-neck, the last week had three Romney leads, four Obama leads, two ties

Whatever you think of complicated projections, just on the raw averaged polling numbers Obama would get in excess of 290 electoral votes and win the Presidency. If the polls are correct, then Obama is on the way to winning, with very little time indeed to turn it around.


Full results for YouGov’s weekly Sunday Times poll are now up here, mostly concentrating on next week’s budget and Britain’s relationship with the USA. On the regular leadership trackers David Cameron’s approval rating is at minus 5 (up from minus 9 last week), Miliband’s at minus 45 (from minus 38), Clegg’s at minus 46 (from minus 44).

Looking at the budget questions, even when we prompt people to consider that rises in the personal tax allowance need to be paid for through tax increases, cuts or borrowing elsewhere there is still strong support for it. Only 18% think it should remain at the current level, 16% would support an increase to £9000, 34% to the Liberal Democrat’s proposed £10,000 and 20% higher than that.

On the 50p income tax rate, only 27% support it being abolished in this budget with 60% opposed. Amongst Conservative supporters the split is very even, 45% support, 46% oppose. Labour and Liberal Democrat voters are heavily against. Opposition for it being abolished is not much less to a pledge to abolish it in a later budget – 52% would oppose this, with 29% in support.

Unsurprisingly the overwhelming majority of people (77%) would support a decrease in the level of fuel duty. There is still a substantial majority in favour when YouGov asked people to balance the competing priorities of cutting the deficit or cutting fuel duty – 59% think it is more important to cut fuel duty compared to 20% who think it is more important to cut the deficit. Opinion on the higher rate of tax relief on pension contributions (even split) and the abolition of child benefit for higher rate taxpayers (majority support) remain the same.

Turning to the USA and the special relationship, 54% think the relationship between the US & UK is very or fairly close. 20% think it should be closer, 21% weaker and 49% think it is about right. Perceptions are now that the relationship has got a little closer since David Cameron became Prime Minister – 26% think it has got closer, 13% less close and 54% no difference. This is significantly better than when YouGov asked the same question in 2010, when the figures were 15% and 20% respectively.

People still, however, don’t think we have much influence over the USA – only 11% think we have a lot or some influence, 44% think we have not a lot of influence, 39% think we have no influence at all, not much changed from 2010. On the specifics of David Cameron’s trip to the USA, 28% think the reception was over the top, but 46% think it got the balance about right.