I don’t regularly report US polls here – there are too many of them and large numbers of dedicated US sites that do it much better than I could, but the New Hampshire primary polls are worth a look: what went wrong?

Since the Iowa caucases all the polls had shown Obama leading Clinton, mostly with large, sometimes double digit leads, yet Clinton ended up winning. It’s only one primary, rather than the election itself, but in the scale of the error it’s comparable to the 1992 polls in the this country.

There are already lots of post-mortem posts going up on US sites with lots of possible explanations. From the professionals there are comments from Nancy Mathiowetz of the AAPOR, John Zogby, Scott Rasmussen, Gary Langer, the head of polling at ABC news and Frank Newport of Gallup, while the best blog analysis comes from Mark Blumenthal on Pollster.com.

Everything seems to be at an early stage at the moment, not least because there is very little data available. US pollsters aren’t compelled to release data tables in the way UK pollsters do, and many offer subscription services and charge people for more detailed information. There are also a lot more pollsters in America, and while those who are members of the AAPOR are supposed to give full details of methodology, not everyone is. I hoped yesterday to go through the tables and compare the make up of the samples with the people who actually voted according to the exit polls, but in most cases the data isn’t there to do so. At the moment therefore these are just early thoughts, no doubt pollsters will release data and have an investigation later.

What can we tell. Firstly, the polls on the Republican primary were pretty good. If you look at Charles Franklin’s graph here on www.pollster.com (second one down), the dots are pretty much on target in the Republican graph (you expect them to all be a bit down and to the left when comparing US polls with actual results because they don’t repercentage to exclude undecided as they do in this country). Given that most polls for the Democrat and Republican race were actually done together (people voting in one primary were added up in one lump, people voting in the other in another lump. The same poll, just presented in two parts) the implication here is that the sampling itself probably wasn’t the problem or the Republican race would also have been wrong.

Secondly there is the interviewer effect in polling on races with a black candidate. There is a long history in the USA of polls in races between a white candidate and a black candidate showing the black candidate doing a lot better than they actually are, theoretically because people are worried that they might come across as racist if they say they are voting against the black candidate. Could this have happened in the the New Hampshire primary? It could have, but the evidence isn’t really there – PEW published a paper last year on the subject that showed yes, there was such a phenomenon in the 1980s and 1990s, but that in more recent contests between white and black candidates in the 2006 mid-term elections there was no such discrepancy. It looks like people are now more relaxed about race and voting. It’s also worth noting that if was this all down to the interviewer effect you might expect automated polls that didn’t use an interviewer to not show the same bias, or at least, to show less of a bias. Rasmussen, who use robocalls with a computer on the end of the line instead of a person were showing great big Obama leads along with everyone else. No, it’s the sort of explanation that poll watchers find fascinating, how people’s answers are biased by various different effects, but in this case I don’t think it is the explanation.

Thirdly, the turnout model. Here we bump into a lack of information to judge the polls on. In the UK likely voters are indentified in a very straightforward manner – pollsters ask respondents if they’ll vote. We don’t always believe them, you can tell ICM that you are 6/10 likely to vote and they’ll metaphorically stroke their chin and stick you in the reject tray, probably quite rightly so, but the methodlogy is very clear. In the the models pollsters use to identify “likely voters” are more complex and often factor in things like past voting habits and so on. In many cases the information on how exactly these calculations are done isn’t easily available so it’s hard to judge, equally a lot of the cross-tabs that would give us a steer on whether they matched the make up of the people the exit polls say actually voted aren’t freely available. In the event the turnout in New Hampshire was higher than expected, particularly amongst older women who backed Clinton – if the pollsters didn’t correctly get the turnout models right, that could be the problem.

Fourthly there is late swing. A lot of people look down on this as a rather poor excuse for getting polls wrong. In my experience most pollsters are pretty scathing about it when a rival uses it to explain why they predicted an election wrongly. That doesn’t mean we can discount such a thing happening – opinion swung rapidly to Obama after the Iowa caucas, in theory it could just have easily have swung sharply away from him.

There is some evidence on this front. The exit polls said 17% of people made up their minds on the day, that’s after the opinion polls had finished. 38% decided in the last three days (including a third of Clinton’s support), when many pollsters stopped work on Monday. In his article John Zogby says he’s used to seeing around 4% to 8% of people claiming they made their minds up on the day, so this does suggest an unusually volatile electorate.He also claims that the last day and a half of his rolling 3 day poll was showing very strong figures for Clinton. Rasmussen too did some late polling into Monday night and saw the trend headed in Clinton’s direction (though obviously not that much; they were still showing a 7 point Obama lead).

Gary Langer rebuts this by pointing out that, if you exclude all the people who decided on the day, the exit polls still show Clinton would have won by 2 points. It wasn’t just those late deciders who voted for Clinton, earlier deciders were for Clinton too. The problem with that logic is that most of the 17% of people who decided on the final day who to vote for weren’t sitting on their hands up until that point. Pollsters were showing around 5-7% undecided, leaving another 10% of people giving a voting intention that they didn’t finally decide upon until the day, in other words, two days earlier that 17% of voters who decided on the day could theoretically have been telling pollsters they were intending to vote for Obama in droves, but later changed their mind and narrowly backed Clinton.

What might have caused a big late swing isn’t really my bag, I’m not a commentator on US politics. There are obvious possibilities in the natural fading of the short term boost Obama would have received from his Iowa victory or the heavy television cover of Hillary Clinton coming close to tears in responding to a question and saying why the contest mattered to her, if a significant weakness to Clinton’s candidacy is her image as an unfeeling, calculating ambition machine, you can imagine how it may have made a difference.

“Late swing” feels like a thorough cop out, so I do hope that in later analysis people find a more concrete explanation somewhere, possibility in the turnout model, at the moment though I think it really could be as mundane as there being a late swing amongst a volatile electorate who saw a sudden glimpse of humanity in a hard-faced candidate.

UPDATE: Danny Finkelstein thinks its the spiral of silence, given that the polls only underestimated Clinton and no one else. The spiral silence is quite possibly contributing to it (Clinton was on the ropes and people were embarrassed to admit to pollsters they were still backing her when they thought everyone else was leaving the sinking ship), but the explanation is too straightforward. The polls had everyone else bang on but Hillary too low? Well, it needs to add to 100% so they must have been somewhere. In fact they were the undecided, remember that US polls do not repercentage polls to exclude don’t knows so these figures included around 5-7% don’t knows. This chimes precisely with the spiral of silence, people claiming they don’t know because they are too embarrassed to say Clinton. It chimes a bit too precisely though…we’d have to accept that all the undecided ended up voting for Clinton…which is stretching credulity a bit too far. Might well be a factor, but not an open and shut case.

UPDATE 2: More comment from Jon Cohen, polling director at the Washington Post, including opinions from Peter Hart who does the NBC-Wall Street Journal poll.


36 Responses to “Why were the polls wrong in New Hampshire?”

  1. Don’t standard UK polls also weigh by past vote? As there obviously is no past voting behaviour for primaries, shouldn’t we expect a larger margin of error, and thus a higher likelyhood of getting it wrong?

  2. How much of it was down to the unusual primary system of New Hampshire? People are classified as “Democrat”, “Republican” or “Independent” and these independents have to decide which party race they want to vote in – they can’t vote in both but could vote in either. One analyst suggested, in the days leading up to the vote, that these independents were leaning towards Obama or McCain. If a majority decided to vote in the Republican primary rather than the Democrat this may explain some of it – if they had been asked who they’d vote for in the Democrat race before they’d decided which race to vote in

  3. Christian – margin of error is not the problem, if it was just the polls being too imprecise we would have seen some polls underestimating Clinton’s support and some polls overestimating it – they ALL underestimated it, so there’s something systemic going on.

    Could the samples all have been skewed in a way that disadvantaged Clinton – despite different pollsters using different methods, but didn’t have an effect on the Republican race at all. In theory its possible, certainly if it was something to do with a change in the electorate that pollsters haven’t caught up with (the potential problem that is allways being warned about is the gradual draft towards people who only use cellphones and therefore get left out of surveys). It might turn out to be something like that (though obviously not too few cellphone people, which would have helped Obama).

    Paul – from the data I’ve looked it it doesn’t look like it so far, the proportion of the people who voted in the Democrat Primary who were registered independents doesn’t look too far from the proportion the polls predicted. When pollsters start releasing and looking at the detailled data this will at least be the sort of thing that will be very easy to check and rule out.

  4. In previous American elections, there has been a phenomenon where white voters have given pollsters the impression that they will support a non-white candidate, but then when they get into the polling booth, they don’t actually support that candidate. That seems to me to be the most likely explanation for what happened in New Hampshire.

  5. Bob Worcester, founder of MORI, has commented in today’s Sun on the polling surrounding the New Hampshire primary. The original letter is BELOW:

    There were twenty-one polls over the five days of frenzied campaigning between the Iowa Caucus and the vote on Tuesday. The final two days there were seven polls. They had Obama ahead of Clinton by an average of seven points.

    Harold Wilson famously said, “A week is a long time in politics”. Over that week, the media elected Obama. The voters chose Clinton. In New Hampshire voters are known to be an independent bunch.

    American primaries are funny things. They are like by-elections in Britain.

    Voters know it’s not the real thing.

    American elections are popularity contests right up to 4 November when the real contest takes place. That’s when the fat lady sings.

    In New Hampshire, as happens in Iowa, the four voters in ten who describe themselves as independent of either party can choose to vote in either the Democrat or Republican primary. They are a volatile bunch.

    Some speculate that independent voters who expected a clear Obama win opted to vote in the Republican primary to support Senator McCain over Governor Romney.

    In Iowa more women supported Obama than Clinton. Women in New Hampshire voted by 4-3 for Clinton.

    The polls didn’t catch the swing back from double digit leads on the weekend.

    Hillary changed her tune in the final two days. She “found her voice”. She cut up rough.

    She nearly burst into tears at one point. She’s a tough lady. But she showed she’s human.

    Clinton was hoping that the States would fall like dominos. First Iowa, then New Hampshire. Now Michigan on the 15th, Nevada Caucuses, South Carolina’s Democratic primary, then Florida on the 29th, then superduper Tuesday on 5 February.

    It’s likely both candidates will likely be chosen then. To start the real race for the White House.

    The pollsters didn’t get it entirely wrong. On the Republican side, they called McCain over Romney. They had Huckabee, the surprise winner in Iowa, languishing in third place.

    Pundits talk about Obama as the “new Reagan”. I don’t. To me it recalls the “flower-power” days of the forgotten Senator Eugene McCarthy in 1968. He was the Obama of his day. He flowered and then faded. It was the machine candidate Hubert Humphrey who was the Democratic candidate in the 1968 Presidential election not Senator Eugene McCarthy.

  6. Isn’t it the case that the polls predicted Obama’s share quite accurately, while underestimating Clinton’s support? Who lost votes?

  7. No one lost votes, or not really significantly so. The gap is made up by undecideds. In the UK polls are repercentaged to exclude don’t knows, so the “votes” add to 100% as in a real election – in short, it’s assumed that don’t knows either won’t vote or will split along the same lines as everyone else (though Populus and ICM have an adjustment that assumes many of them will vote how they did at the last election).

    In the USA polls are repercentaged and the final polls included around 6% or so of people who didn’t give an answer. A simple explanation would be that they all voted for Clinton…but that’s probably far too simplistic, the evidence from the exit poll is that people who decided on the day did favour Hillary Clinton, so those people probably broke disproportionately for Clinton…but not every last one of them :)

  8. Looking at the actual voting figures in New Hampshire, Hillary won by 112,251 votes to 104,772 for Obama. Amazing how such a small number of voters are being regarded in such an important way, with the winning candidate polling about the same number of votes as the electorate of the Isle of Wight.

  9. I thought I read somewhere that in the US, polls are not “corrected” with methodologies we use to have in Europe (i.e., UK pllsters know that people tend not to say they vote for BNP, so they multiply the number of positive answers by 3 or 4). Is it true? And in that case it would likely be the explanation.

  10. Vonric – I think that’s unique to French polling because of the Front National, and IIRC it ended up over-representing them last time round. There is no such adjustment for the BNP in UK polls, it’s probably true that some people are embarrassed to admit supporting them and polls with an interviewer effect underestimate their true support, but given they are a fringe party it doesn’t have a great impact.

    There are cases in US polling of a similar reluctance to admit voting for extremist candidates – polls always used to underestimate support for David Duke for example, but in this case there is no obviously extremist candidate that could warrant that sort of mechanical adjustment.

    At a lower level though, yep, reluctance to admit to supporting Hillary Clinton because she is very unpopular amongst some parts, is indeed a potential reason.

  11. Anthony Wells> Ah ok, yes I might have read that on a French blog, and I thought in was general to Europe… thanks for the answer. So this means that in the UK they do not correct the results according to what they know about last polls and last election results? It looks fair to me to avoid bias…

    PS: ah ah how did you know I was thinking about the FN? I especially took the BNP to choose a UK example here ;-). Unfortunately even correcting the polls in France did not predict the result of the FN in 2002 in France, so that was not necessary but absolutely mandatory for French pollsters due to the circumstances… ouch!

  12. Anthony – this is interesting.

    Maybe you are planning to start a “World Polling Report” – looking beyond our shores ;)

  13. Vonric – if the BNP were a real factor and polls did underestimate them then I expect pollsters would start making adjustments. In reality they got 0.7% at the last election, so if anything polls over-estimated them! The reason for that is because they only stood in 1/6 of the seats, but at any rate, it’s of such marginal effect no ones ever worried about it.

  14. It is quite simply down to the fact that the media were in a frenzy about a massive 10% lead for Obama that the majority white electorate in New Hampshire panicked and came out in force to make sure that Obama did’nt win .That will keep happening in the USA for a long long time to come – in none of our lifetimes will you see a non white President of the USA . If by chance , he became leader of the Democrats – that would be a certain winner for the Republicans .

    Exactly the same as what happened in 1992 in the UK when the prospect of Neil Kinnock winning by a big majority frightened the silent “undecided” voter into urgent action to make sure he did’nt win the election .

    The “black / white” issue in western countries will always be controversial in decided the winners of elections . Most whites will say in public or to a POLLSTER that they are liberal minded and would vote for someone outside of their ethnic mix – but the reality is they would’nt enmass .

    Perhaps we should have a system like South Africa – where the candidates face is next to their name on the ballot paper – then we might see some interesting results in the UK compared to POLLS !! It certainly seems to help the ANC to increase their majority EVERY election since coming to power – in fact in South Africa there seems to be more voters at each election than actually live in the country – let’s hope we never reach that stage of democracy !

  15. Mike – we have already reached that stage of democracy, or very close to it – in certain areas since the introduction of postal balloting.

  16. There’s a theory amongst some that there’s some sort of flaw (possibly deliberate) in the electronic voting machines:

    http://www.legitgov.org/nh_machine_vs_paper.html

    Now, this might just be paranoia. I wonder if you Anthony could quickly tell us your thoughts on this, if you have any. :-)

  17. We have had Bush, then Clinton, then Bush…. the trend must be stopped. America needs change. Not more of the same.

  18. I kind of started all this on the previous blog and I have to say that I am even more confused now than I was before as to why the polls got it wrong in New Hampshire. There seem to be as many takes on this as there are forecasts about our economy! If Clinton has a more determined core base mostly women who will turn out for her come hell or high water then I fear she will beat Obama for the Democratic nomination.
    On the scarcely less fascinating Republican side it seems to me that only Fred Thompson lacks the support in the polls to stay in the race right down to the wire. No one candidate other than possibly McCain may be able to build up sufficient momentum in the national polls to help attract votes in the primaries from those Republicans desperately looking for a winner to beat a likely Clinton ticket come November. The one big hope for a Republican victory lies in the ‘anyone but Hillary’ factor but if the nomination ends up with somebody as a result of bitter disputes and dirty dealings on the convention floor they can kiss even that hope goodbye.

  19. Anthony,

    For someone who thought the Polls weren’t that wrong but there was an Obama band wagon that pulled commentators along I still think there is a strong element of people thinking it meant something that it didn’t, or making interpretations that the data didn’t show.

    For all that I accept mistakes were made, we are still dealing with a system that compared to the UK is less transparent, more commercial and highly partisan.

    I accept that the missed the late shift to HC as part of it but I am struggling to see why, so here’s one theory.

    Partly as a role model and perhaps also the “tears” thing, who HC may well have resonance with is working women, and particularly professional ones who have to work in what is still very much a “mans world”.

    Now this subset of female voters might have been more inclined to back HC but underrepresented in the polls because, They were out working, have mobiles not fixed line, and if it was evening polling weren’t the one who answered the phone.

    Also just out of interest what if anything was YouGov USA showing or indeed any on line US polster. Was there any significant difference from what we know so far between on-line, phone and face to face polling?

    Peter.

  20. Gerard Baker in The Times today offers an interesting view both on the Polls & Obama.

    If I summarise him correctly:-

    Both the pre-election polls and the exit polls showed Obama ahead-but he was never doing as well as they suggested.
    Clinton mobilised the “traditional” Democrat vote in vast numbers-ie working class/less educated/less well off/ older.Obama’s appeal is to better-off/young/well educated/Independents/some Republicans.
    In NH there were simply not enough of these “new” voters for Obama to appeal to.The Democrat turnout was up 10%-as against doubled in Iowa.

    The Pollsters did not”properly measure” this.

    Baker goes on to opine that the “traditional Democrat” usually wins the nomination over the “idealist outsider”.

  21. Peter – I don’t know of any online polling of the NH Primary. YouGov USA these days is completely unrelated to what it used to be – once we did some polls from over here in London using a small panel of Americans we’d recruited. Since then YouGov have bought a US online polling company, Polimetrix, so YouGov US polls are done by the subsidiary company over in California and I know nothing of them :)

  22. Colin –

    Be wary of what you read into the exit polls. We are used to exit polls that are straight down the wire and that don’t change. They are based only on the interviews that MORI/NOP do outside polling station and aren’t fiddled with after the event.

    Since the fuss over the exit polls in 2000 I think a lot of people realise that US exit polls are weighted as the night goes on so that by the next morning they all match the actual result, and therefore if you want actual exit polls you have to get them early before any results are known.

    Alas, sometimes the early ones may also be meddled with, but rather than weighting them to match the eventual result, they weight them to match a best estimate based on the actual exit poll info and pre-election polls, called “the prior estimate”. So the reason that the early exit polls showed Obama leading might not be because more people told pollsters they’d voted Obama than Clinton, but because the pollsters had weighted the results to show the Obama lead they expected based on pre-election polls. I don’t know for sure, I don’t know if this always happens or if it did this time…just be wary of treating the exit polls like they are entirely distinct from the pre-election polls, because I’m not sure they are.

  23. Thanks Anthony.

    The “Black Art” of Polling eh!!

  24. This happens a lot. Right before people start taking the primaries seriously, the democrats flirt with a candidate that tells them everything they’ve been longing to hear, touches each one of their tendentious g-spots. Later, when they realize that these rabid politicians can’t win a general election they usually go for the candidate that is most likely to win (kerry, the last time, rather than Dean). This, obviously, doesn’t always happen (Carter–and he ended up winning).

    Contrary to what some media say, Clinton has the best chance of winning as a democrat, because she’s likely to get more women to peel off of the republican candidate for her. Obama will not attract any more people to the democrats than usual: they already get 90% of blacks to vote democrat. I know of a few conservative women who say they’d vote for Clinton even though they don’t like her just so that a woman gets elected (this always causes a fracas!). I’m not sure Obama would benefit from a similar a-political base of support. It also helps Obama that no one is really writing about him in the US papers yet. They mostly just write what he says on stump speeches (which appeals to the media in the USA).

  25. Obama was the candidate the liberal media (including the BBC) wanted to win. So he won. Unfortunately the electorate decided otherwise.
    Perhaps he should come over here and join the Labour Party as he should fit right in with all the suits.
    PS I know it might be too much to ask for but can the BBC now tear itself away from Obama and cover the Republican election – better still revert to its coverage of the US in the1970’s and get back to covering British politics?

  26. The fact that New Hampshire has such a small electorate should surely have made the pollsters jobs much easier than in a state with millions of voters. I can’t understand the explanation that the pollsters had an incorrect idea of how many traditional Democratic voters there are in New Hampshire; surely that’s precisely their job to know those statistics. They’ve had plenty of time to ensure they had the correct demographic details in giving their results.

  27. wolf,

    The explanation is far simpler than BBC left wing bias and for me as a “lefty” more worrying.

    They went with Obama and the Democrats because it was a better story, and like the rest of the media that’s what drives the news.

    You can have virtual genocide in Kenya and it will take second place to where the McCann’s had dinner last night. That,s not left wing bias that’s a competitive news media.

    To be honest I am getting a bit tired of people here blaming political bias in the media for things they don’t like on the news.

    For years my party has been one of the worst for claiming Unionist media bias against us , almost at every turn.

    However as someone who is interested more in policy than anything else, and who tries to take a balanced view of these things, most of the attacks on the SNP for ill conceived or not properly thought through policy in the past have been by and large well founded.

    The fact that we are getting far better coverage today isn’t that we have convinced the media of the cause of independence or that they are scared to take on the party now that we are the government, it’s because we have got our act together and have improved.

    I am a great believer that the phrase ” What do they know” can be taken two ways , with an exclamation mark dismissing it or with a question mark.

    I prefer the latter, If the BBC attacks the SNP I ask first, “Is this true?”, “Are we at fault?”, “Do they have a point?”.

    To many party supporter in my own and other parties make the mistake of saying, “Well they would say that!”, “They are always attacking us!”, “We know why they do it!”.

    Just as parties shouldn’t believe their own propaganda, nor should they believe criticism of them is someone else’s.

    Peter.

  28. Anthony.ICM seem to have a strange anomaly in their reported figures- Conservative leads -Guardian +5, +6, +5,0 (in between) Other +11, +8, +7, -6. Guardian seems to be consistently under reporting the Cons lead – now rumoured that Telegraph tomorrow will show good figures for the Cons.Surely chance cannot produce these two very different series? Can it be the result of specific instructions from the Guardian?Would a polling company accept interference in their mod. operandi?

  29. Anthony. The -6 was for the Mirror ,so could be included with the Guardian series.

  30. Collin – I commented on it here. I can find no obvious difference between the method being used for ICM’s Guardian polls and their polls for other newspapers, so it does appear to be just chance (and no, no reputable pollster would fiddle with their figures in that way and ICM are a very reputable pollster).

  31. I can remember many black candidates in the US who did not live up to their poll ratings – Harvey Gantt against Jesse Helms, Tom Bradley in California, and most recently Harold Ford in the Tennessee Seante race in Nvember 2006 – the one uppe rhouse target the Democarts did not achieve. I’d be surprised if some ‘shy’ (to put it politely) effect was not part of the outcome in New Hampshire – especially as Obama had become the surprise favourite, which he wasn’t in Iowa (and may not be again).
    Arguments about why the polls are wrong are in any case less improtant unltimately than whether the US can elect a black president. The answer may be – yes, but it will be much harder. If Obama becoems the D nominee I would not put his chance of beating whoever the R is anywhere near as high as 50%.
    In fact, I’ve always believed that the first black president is more likely to be a Republican – a bit like the first woman UK PM was a Conservative.
    And very unfortunately I must add that one prominent black possible candidate that an employer of mine once worked for decided not top run partly because it was felt there was a real chance of assassination.
    I think we’d all agree that we sincerely hope that the USA has moved on in every sense in the 21st century.

  32. There are still 6% or so of people in the US who will openly admit to pollsters that they would not vote for a black candidate, presumably there are more out there who think the same but wouldn’t tell a pollster they are a racist.

    Whether that matters of course depends on whether Obama’s positive qualities are enough to outweight the negative his race will be to some voters (for some voters obviously his race will itself be a positive). And, I suppose, whether the negative effect of whatever opponent he ended up facing had an even larger effect – far more people, for example, say they could never vote for a Mormon.

  33. I notice the polls have made a mess of it again in Michigan. They were all saying the result would be something like 27%-26% for Romney and McCain. In fact the result was Romney – 39%, McCain – 30%. So Romney scored about 12% more than the polls said he would. That’s quite a big mistake. This isn’t the greatest of months for the polling companies.

  34. Andy – given that there wasn’t a race issue here, and the is no obvious reason for a very sudden swing, I’m thinking that the turnout model is could be where pollsters are falling down.

  35. I think there are other problems that pollsters face in the USA’s primaries and caucuses (apart from their endemic habit of small samples).
    One is that they cannot weight by previous voting, as there hasn’t been a previous primary with these candidates. Another is the rules, which in some states allow independents to vote in either party’s primary without prior notification.
    Finally, personal voting is likely to be more subject to late movement than party, due to the lack of the long-term identification factor.
    So there are even more reasons to be wary of pre=contest polls than usual!