I don’t regularly report US polls here – there are too many of them and large numbers of dedicated US sites that do it much better than I could, but the New Hampshire primary polls are worth a look: what went wrong?
Since the Iowa caucases all the polls had shown Obama leading Clinton, mostly with large, sometimes double digit leads, yet Clinton ended up winning. It’s only one primary, rather than the election itself, but in the scale of the error it’s comparable to the 1992 polls in the this country.
There are already lots of post-mortem posts going up on US sites with lots of possible explanations. From the professionals there are comments from Nancy Mathiowetz of the AAPOR, John Zogby, Scott Rasmussen, Gary Langer, the head of polling at ABC news and Frank Newport of Gallup, while the best blog analysis comes from Mark Blumenthal on Pollster.com.
Everything seems to be at an early stage at the moment, not least because there is very little data available. US pollsters aren’t compelled to release data tables in the way UK pollsters do, and many offer subscription services and charge people for more detailed information. There are also a lot more pollsters in America, and while those who are members of the AAPOR are supposed to give full details of methodology, not everyone is. I hoped yesterday to go through the tables and compare the make up of the samples with the people who actually voted according to the exit polls, but in most cases the data isn’t there to do so. At the moment therefore these are just early thoughts, no doubt pollsters will release data and have an investigation later.
What can we tell. Firstly, the polls on the Republican primary were pretty good. If you look at Charles Franklin’s graph here on www.pollster.com (second one down), the dots are pretty much on target in the Republican graph (you expect them to all be a bit down and to the left when comparing US polls with actual results because they don’t repercentage to exclude undecided as they do in this country). Given that most polls for the Democrat and Republican race were actually done together (people voting in one primary were added up in one lump, people voting in the other in another lump. The same poll, just presented in two parts) the implication here is that the sampling itself probably wasn’t the problem or the Republican race would also have been wrong.
Secondly there is the interviewer effect in polling on races with a black candidate. There is a long history in the USA of polls in races between a white candidate and a black candidate showing the black candidate doing a lot better than they actually are, theoretically because people are worried that they might come across as racist if they say they are voting against the black candidate. Could this have happened in the the New Hampshire primary? It could have, but the evidence isn’t really there – PEW published a paper last year on the subject that showed yes, there was such a phenomenon in the 1980s and 1990s, but that in more recent contests between white and black candidates in the 2006 mid-term elections there was no such discrepancy. It looks like people are now more relaxed about race and voting. It’s also worth noting that if was this all down to the interviewer effect you might expect automated polls that didn’t use an interviewer to not show the same bias, or at least, to show less of a bias. Rasmussen, who use robocalls with a computer on the end of the line instead of a person were showing great big Obama leads along with everyone else. No, it’s the sort of explanation that poll watchers find fascinating, how people’s answers are biased by various different effects, but in this case I don’t think it is the explanation.
Thirdly, the turnout model. Here we bump into a lack of information to judge the polls on. In the UK likely voters are indentified in a very straightforward manner – pollsters ask respondents if they’ll vote. We don’t always believe them, you can tell ICM that you are 6/10 likely to vote and they’ll metaphorically stroke their chin and stick you in the reject tray, probably quite rightly so, but the methodlogy is very clear. In the the models pollsters use to identify “likely voters” are more complex and often factor in things like past voting habits and so on. In many cases the information on how exactly these calculations are done isn’t easily available so it’s hard to judge, equally a lot of the cross-tabs that would give us a steer on whether they matched the make up of the people the exit polls say actually voted aren’t freely available. In the event the turnout in New Hampshire was higher than expected, particularly amongst older women who backed Clinton – if the pollsters didn’t correctly get the turnout models right, that could be the problem.
Fourthly there is late swing. A lot of people look down on this as a rather poor excuse for getting polls wrong. In my experience most pollsters are pretty scathing about it when a rival uses it to explain why they predicted an election wrongly. That doesn’t mean we can discount such a thing happening – opinion swung rapidly to Obama after the Iowa caucas, in theory it could just have easily have swung sharply away from him.
There is some evidence on this front. The exit polls said 17% of people made up their minds on the day, that’s after the opinion polls had finished. 38% decided in the last three days (including a third of Clinton’s support), when many pollsters stopped work on Monday. In his article John Zogby says he’s used to seeing around 4% to 8% of people claiming they made their minds up on the day, so this does suggest an unusually volatile electorate.He also claims that the last day and a half of his rolling 3 day poll was showing very strong figures for Clinton. Rasmussen too did some late polling into Monday night and saw the trend headed in Clinton’s direction (though obviously not that much; they were still showing a 7 point Obama lead).
Gary Langer rebuts this by pointing out that, if you exclude all the people who decided on the day, the exit polls still show Clinton would have won by 2 points. It wasn’t just those late deciders who voted for Clinton, earlier deciders were for Clinton too. The problem with that logic is that most of the 17% of people who decided on the final day who to vote for weren’t sitting on their hands up until that point. Pollsters were showing around 5-7% undecided, leaving another 10% of people giving a voting intention that they didn’t finally decide upon until the day, in other words, two days earlier that 17% of voters who decided on the day could theoretically have been telling pollsters they were intending to vote for Obama in droves, but later changed their mind and narrowly backed Clinton.
What might have caused a big late swing isn’t really my bag, I’m not a commentator on US politics. There are obvious possibilities in the natural fading of the short term boost Obama would have received from his Iowa victory or the heavy television cover of Hillary Clinton coming close to tears in responding to a question and saying why the contest mattered to her, if a significant weakness to Clinton’s candidacy is her image as an unfeeling, calculating ambition machine, you can imagine how it may have made a difference.
“Late swing” feels like a thorough cop out, so I do hope that in later analysis people find a more concrete explanation somewhere, possibility in the turnout model, at the moment though I think it really could be as mundane as there being a late swing amongst a volatile electorate who saw a sudden glimpse of humanity in a hard-faced candidate.
UPDATE: Danny Finkelstein thinks its the spiral of silence, given that the polls only underestimated Clinton and no one else. The spiral silence is quite possibly contributing to it (Clinton was on the ropes and people were embarrassed to admit to pollsters they were still backing her when they thought everyone else was leaving the sinking ship), but the explanation is too straightforward. The polls had everyone else bang on but Hillary too low? Well, it needs to add to 100% so they must have been somewhere. In fact they were the undecided, remember that US polls do not repercentage polls to exclude don’t knows so these figures included around 5-7% don’t knows. This chimes precisely with the spiral of silence, people claiming they don’t know because they are too embarrassed to say Clinton. It chimes a bit too precisely though…we’d have to accept that all the undecided ended up voting for Clinton…which is stretching credulity a bit too far. Might well be a factor, but not an open and shut case.
UPDATE 2: More comment from Jon Cohen, polling director at the Washington Post, including opinions from Peter Hart who does the NBC-Wall Street Journal poll.