Mark Blumenthal over at Pollster.com has drawn my attention to this article by David Runciman in the LRB which, amongst other things, complains about quite how “rubbish” polls in the USA are with their 600-700 samples, compared to “1000-2000” samples in the UK.
I haven’t been following particularly close attention to the US polls (why would I? Obama the democrat nomination was effectively settled months ago, but the national polls will be distorted until Clinton finally accepts it and goes), but the comparison is somewhat unfair.
Firstly, he’s not comparing apples with apples. American polls normally quote as their sample size the number of likely voters, it is typical to see a poll reported as being amongst 600 “likely voters”, with the number of “unlikely voters” screened out to reach that eventual figures not made clear. In contrast, British polling companies normally quote as their sample size the number of interviews they conducted, regardless of whether those people were filtered out of voting intention questions. So, voting intentions in a UK poll with a quoted sample size of 1000, may actually be based upon 700 or so “likely voters”.
To give a couple of examples, here’s ICM’s latest poll for the Guardian. In the bumpf at the top the sample size is given as 1,008. Scroll down to page 7 though and you’ll find the voting intention figures were based on only 755 people. Here’s Ipsos-MORI’s April poll – the quoted sample size is 1,059, but the number of people involved in calculating their topline voting intention once all the unlikelies have been filtered out was only 582.
Including a lot of people isn’t necessarily a good thing anyway. Primaries are low turn participation events in the USA, not a lot of people vote in them, and the challenge for US pollsters is filtering out all those people who won’t actually take part. Getting lots of people per se can be a bad thing if those people won’t actually vote, the aim is getting the right people. Considering the rather shaky record of most British pollsters in some low turnout elections like by-elections, Scottish elections, the London mayoralty and so on, we really aren’t the experts on that front.
Of course, most of Runciman’s other points about poor media reporting of polls are music to my ears – it is misleading for newspapers to headline findings based on only 250 or so people without appropriate caveats. I first started blogging about polls because media coverage of polls was so myopic, reporting only their own polls in isolation, splashing polls that look like obvious outliers as front page sensenations and so on. Just, the US pollsters aren’t partcularly bad, nor UK pollsters particularly good. When it comes to low turnout elections like primaries, the polls produced by USA pollsters may not be that great, but it’s because low turnout elections are hard to poll, I suspect if we had similar elections over here we’d be just as bad.