Sooner or later, a pollster gets something wrong. It happens to everyone if they are in the game for long enough. There are two responses to that, there is to deny there is any problem and blame it all on a late swing, or you can go away, work out what went wrong and put it right. The good pollsters do the second one – so when all the companies got it wrong in 1992 there was an industry inquiry, and ICM in particular came up with new innovations that addressed the problem and led to many of the methods companies use today. In 2008 when MORI got the London election wrong they went away, looked at what had happened, and made changes to put it right. A pollster that gets things wrong, admits it, and then puts it right isn’t a bad thing.

Anyway, in the US election last year the most venerable of all polling companies, Gallup, managed to get things wrong, showing a small lead for Mitt Romney rather than the eventual victory for Barack Obama. They put their hands up, invited in some academics to help and went away and looked at their methods – the result is here. Gallup tested about twenty different hypotheses of things that could have gone wrong, and found that in the majority things were working okay and there was no issue to address. They ended up with four issues where they think things went wrong and caused the overestimate of Romney’s support.

Most or all of the actual problems Gallup identified aren’t directly relevant to British political polls – different system, different challenges, different methods, different solutions – but it’s still an interesting look at what can go wrong with a poll and how a company should dig through its methods if something has gone wrong.

Likelihood to Vote – In Britain pollsters have a relatively simple way of approaching likelihood to vote: they ask people how likely they are to vote, and then weight and/or filter people’s responses based upon that, either giving people’s answers more weight based on how likely they say they are to vote or excluding people below a certain threshold. The only exception to this is ICM, who also include whether people voted in the 2010 election in their likelihood to vote model. American pollsters tend to use much more complicated methods, they ask people how likely they are to vote, but also whether they voted last time, how interested they are in politics, whether they know where the polling station is and so on – there are seven questions in all, which they use to work out a likelihood to vote score and then include only those most likely to vote. Other American pollsters do much the same, but Gallup’s method put more weight on whether people voted in the past, and their adjustment ended up being more pro-Romney than some other companies. Gallup are going to go away and do more work on turnout, including whether the sort of people who take part in polls are more likely to vote anyway (something that I would certainly expect to be true).

Sampling – Most telephone pollsters in the USA get their phone numbers in a similar way to British pollsters, by using random digit dialling. This ensures that people who are ex-directory are not excluded from samples, but at the cost of getting lots of dead telephone numbers, faxes, modems, business numbers and so on. In 2011 Gallup started doing something different. Like most companies they do a fair amount of their interviews on mobile phones, and noted that the majority of ex-directory people did have mobile phones, so theorised that it was safe to randomly generate their landline sample from telephone directories, while bumping up the mobile phone sample to catch those ex-directory people on their mobiles (mobile people who reported being ex-directory were weighted up to account for the tiny percentage of ex-directory people without mobiles). In theory it should of worked. In practice, it probably didn’t – before weighting the RDD sample was more democratic, younger and more pro-Obama than the listed one, so Gallup are going back to using the more expensive RDD method.

Time zone skews – this is an interesting one. As you might expect, Gallup sample within and weight by the regions in the USA. But within some of those regions there are different time zones, and because Gallup started polling at 5pm local time, it meant that in regions that covered more than one time zone they ended up doing more interviews in the eastern part of the region. Correcting this problem would have increased Obama’s support in Gallup’s final poll by 1%. Of course, in Britain we don’t have different time zones to worry about, but it illustrates a problem that can effect any methodology design – skews within the categories you weight by. A pollster can have, for example, the correct proportion of people in the DE social class or people over the age of 55, but what if within those categories you control for people are skewed towards more affluent DEs, or people only just over the age of 55?

Race – the final problem was a rather specific one on how Gallup asked about race – instead of giving people a list of race categories and asking which applied to the respondent, they asked them one at a time and got people to say yes or no, which produced some rather odd effects like overstating the proportion of Native Americans and mixed race people.

The full Gallup review is here and if you’re interested I’d also recommend reading the verdict of Mark Blumenthal (who spotted some of the problems before Gallup did) here and who has obviously followed it infinitely more closely than me.

65 Responses to “What Gallup got wrong in the USA”

1 2
  1. BB

    “…but what of the 9% of VI who voted Tory in 2010 but now opt for UKIP… might not some of them be likely to change their VI in unpredictable ways?”

    But isn’t it likely that these are voters who HAVE found their spiritual home in UKIP? Right-wing Tories who have become disillusioned by the modernising? Doesn’t that suggest that these voters will overwhelmingly stay on the right-wing of politics, whether remaining with UKIP or returning to the fold if the Tories tack right?

    But I think you raise an interesting point. To win them by tacking right, the Tories risk forcing the 3% back to Labour. Because if my assessment of the 3% is correct (if they are a chunk of the old, post-industrial working class who have dropped through the cracks over the last 30 years) the one thing that they will NOT do is vote Tory or vote to help the Tories. OK: two things.

  2. @AW

    Please delete my posts which referenced the 1803 model of atoms colliding – any mention of cue sports and different coloured balls seems to be taboo!

  3. The Ashcroft polling looks pretty positive for Labour – a lot of ties and leads on various issues which surprise me; ie Labour is 1 point ahead on helping businesses grow!

    Expectation-wise, Labour has a 16-point lead on being in government in some form come 2015.

  4. It’s interesting that for all the Tory lurching to the right they’re now tied(!!!) with Labour on immigration in the Ashcroft poll. That’s a strong argument for Labour not to chase the Ukip defectors in and of itself, I think: the anti-immigration crowd are irreconcilables, and there is no position a mainstream party can take that will be “tough” enough to appease them.

  5. Phil Haines
    Two good results for Labour in by-elections yesterday. One gain ,one hold strong performance by UKIP in both BUT hurting the Tories much more ,differentially, than Labour.LDs losing votes to UKIP more so than Labour also.
    Seems to tie in with the Ashcroft data.

  6. Yougov poll- Labour weighted down, con and LD weighted up. It all depends on whether the weighting is right, given changed scenario (coalition in govt, demise LD, rise UKIP).
    If anyone has not yet caught it, BBC Radio4 from last Sat night ‘the longest suicide note in history’ on Labour’s 1983 campaign is essential listening. One day left on iplayer.

  7. On the basis of British Social Attitudes surveys, if Miliband is going to lead us into an egalitarian era, he’s going to do that in spite of public opinion, which is less egalitarian than it was in the 1980s. For example, the 2010 BSAS had support for redistribution of income at less than 40% of the public, in stark contrast to the past.

    “Thatcher and Attlee moved public opinion, though”, I imagine some would say. Except they didn’t: the public was already greatly in favour of the welfare state before 1945 and attitudes towards taxation & trade unions changed before the 1979 election. There is a certain “myth of rhetoric” and I remember reading an article that found no evidence that Thatcher or Reagan had done anything to change public opinion; if anything, the UK and US were more egalitarian by the late 1980s than in the early 1980s.

    There was a lot in the Economist about young people’s attitudes moving towards classical liberalism (left-wing on “social” questions and right-wing on economic issues) to a greater extent than any since sufficient polling began. Fascinatingly, one hypothesis was that this shift is a result of the welfare state becoming less generous to them, e.g. the introduction of tuition fees. Wouldn’t it be fascinating if a long-term consequence of New Labour was to erode the ideological basis of the welfare state among the young? In the long run, Blair might have done more to change public attitudes to the right than Thatcher did.

    The problem that such a shift (which I emphasise is a shift in the trend i.e. young people in the UK, on average, are not only more classically liberal than their parents, but more classicaly liberal than their parents were when their parents were their age) presents to the main parties is obvious: right now, winning elections involves ingratiating the baby-boomers, who are increasingly out-of-line with the opinion of the dominant voters of 2033. I think there are long-term problems for all the parties, but especially for the Tories and Labour, who are very fundamentally opposed to classical liberalism, and perhaps increasingly so if the Tories chase UKIP voters and Labour goes Blue on social questions.

  8. Ewen Lightfoot

    “Good results in two by-elections”.

    Do you mean local council elections or have I missed two by-elections, which wouldn’t suprise me down here in darkest Dorset.
    If they were by-elections can I have the details, many thanks.

  9. @ Bill Patrick

    I agree with every point.

  10. The upside for Labour, of course, is that social conservativism is literally dying out in the UK and it’s dying out faster than in many countries. On the downside for them, declining patriotism and social conservativism in UK are going hand-in-hand with declining collectivism in general.

    As for the Tories, I suspect that David Cameron has the right idea, but about 15 years too early. Yet, unless the Tories start changing their image now, a transition in the future may be impossible e.g. the Liberals failed to make a rapid transition to the left in the 1920s and thereby win over working-class left-wing voters, and so lost their position.

    I’m only half joking when I say that Nick Clegg may be seen as the politician who anticipated the future, just as Neville Chamberlain’s “non-socialist reformism” was central to most post-war period politics and Enoch Powell anticipated much of Thatcherism, though neither were exactly held up as influences.

  11. @ Ewen

    Like Turk- have you got the source? does not seem to be updating the local by elections any more which is a shame.

  12. Reading the Gallup paper – it seems that they think they only had technical errors (OK, some methodological ones too, but really not prominent), but nothing about the underlying assumptions. Now it is possible that the terms of reference excluded this, but it seems that there’s an attachment (understandably) to single loop learning.

  13. Now this auto-mod really escapes me – it was some technical observation Gallup’s report.

  14. LEFTY
    Am I alone in thinking that EM’s “predistribution” ideas place equal weight on raising educational and skills levels at the bottom, or in marginal groups, so that we do not have the huge and unmanageable problems of low income and ethnic minority youth unemployed or in prison or otherwise disaffected; traits which go along with poverty and income disparity, but also with costly and unacceptable social problems?

  15. A tidbit of additional info on the long term YouGov trends. Here are the rolling averages for Not Votings and Don’t Knows:

    Lib Dems DKs have declined, which probably is the source of some of their new Ukip defectors. (Except for that weird blip after the local elections- not sure what that’s about.) Tory DKs have declined a bit, although less than you might think. Labour DKs are flat as a board from March through the end of May, so whatever the cause of Labour’s decline, it’s not related to their core voters getting more or less decisive.

    Not Voting looks pretty static for everyone, which casts a slight shadow on the theory that Ukip are drawing the disenchanted back into politics. (If that idea isn’t dead already after the pitiful local election turnout.)

1 2