“But the sheer size of the survey […] makes it of interest…”

One of the most common errors in interpreting polls and surveys is the presumption that because something has a really huge sample size it is more meaningful. Or indeed, meaningful at all. Size isn’t what makes a poll meaningful, it is how representative the sample is. Picture it this way, if you’d done an EU referendum poll of only over 60s you’d have got a result that was overwhelmingly LEAVE… even if you polled millions of them. If you did a poll and only included people under 30 you’d have got a result that was overwhelmingly REMAIN… even if you polled millions of them. What matters is that the sample accurately reflects the wider population you want them to represent, that you have the correct proportions of both young and old (and male & female, rich & poor, etc, etc). Size alone does not guarantee that.

The classic real world example of this is the 1936 Presidential Election in the USA. I’ve referred to this many times but I thought it worth reciting the story in full, if only so people can direct others to it in future.

Back in 1936 the most respected barometers of public opinion was the survey conducted by the Literary Digest, a weekly news magazine with a hefty circulation. At each Presidential election the Digest carried out a survey by mail, sending surveys to its million-plus subscriber base and to a huge list of other people, gathered from phone directories, membership organisations, subscriber lists and so on. There was no attempt at weighting or sampling, just a pure numbers grab, with literally millions of replies. This method had correctly called the winner for the 1920, 1924, 1928 and 1932 Presidential elections.

In 1936 the Digest sent out more than ten million ballots. The sample size for their final results was 2,376,523. This was, obviously, huge. One can imagine how the today’s papers would write up a poll of that size and, indeed, the Digest wrote up their results with not a little hubris. If anything, they wrote it up with huge, steaming, shovel loads of hubris. They bought all the hubris in the shop, spread it across the newsroom floor and rolled about in it cackling. Quotes included:

  • “We make no claim to infallibility. We did not coin the phrase “uncanny accuracy” which has been so freely applied to our Polls”
  • “Any sane person can not escape the implication of such a gigantic sampling of popular opinion as is embraced in THE LITERARY DIGEST straw vote.”
  • “The Poll represents the most extensive straw ballot in the field—the most experienced in view of its twenty-five years of perfecting—the most unbiased in view of its prestige—a Poll that has always previously been correct.”

digestpoll

You can presumably guess what is going to happen here. The final vote shares in the 1936 Literary Digest poll were 57% for Alf Landon (Republican) and 43% for Roosevelt (Democrat). This worked out as 151 electoral votes for Roosevelt and 380 for Landon. The actual result was 62% Roosevelt, 38% for Landon. Roosevelt received 523 in the electoral college, Landon received 8, one of the largest landslide victories in US history. Wrong does not nearly begin to describe how badly off the Literary Digest was.

At the same time George Gallup was promoting his new business, carrying out what would become proper opinion polls and using them for a syndicated newspaper column called “America Speaks”. His methods were quite far removed from modern methods – he used a mixed mode method, mail-out survey for richer respondents and face-to-face for poorer, harder to reach respondents. The sample size was also still huge by modern standards, about 40,000*. The important different from the Literary Digest poll however was that Gallup attempted to get a representative sample – the mail out surveys and sampling points for face-to-face interviews had quotas on geography and on urban and rural areas, interviewers had quotas for age, gender and socio-economic status.

pic2097election

Gallup set out to challenge and defeat the Literary Digest – a battle between a monstrously huge sample and Gallup’s smaller but more representative sample. Gallup won. His final poll predicted Roosevelt 55.7%, Landon 44.3%.* Again, by modern standards it wasn’t that accurate (the poll by his rival Elmo Roper, who was setting quotas based on the census rather than his turnout estimates was actually better, predicting Roosevelt on 61%… but he wasn’t as media savvy). Nevertheless, Gallup got the story right, the Literary Digest hideously wrong. George Gallup’s reputation was made and the Gallup organisation became the best known polling company in the US. The Literary Digest’s reputation was shattered and the magazine folded a couple of years later. The story has remained a cautionary tale of why a representative poll with a relatively small sample is more use than a large poll that makes no effort to be representative, even if it is absolutely massive.

The question of why the Digest poll was so wrong is interesting itself. Its huge error is normally explained through where the sample came from – they drew it from things like magazine subscribers, automobile association members and telephone listings. In depression era America many millions of voters didn’t have telephones and couldn’t afford cars or magazine subscriptions, creating an inbuilt bias towards wealthier Republican voters. In fact it appears to be slightly more complicated than that – Republican voters were also far more likely to return their slips than Democrat voters were. All of these factors – a skewed sampling frame, differential response rate and no attempt to combat these – combined to make the Literary Digest’s sample incredibly biased, despite its massive and impressive size.

Ultimately, it’s not the size that matters in determining if a poll is any good. It’s whether it’s representative or not. Of course, a large representative poll is better than a small representative poll (though it is a case of diminishing returns) but the representativeness is a prerequisite for it being of any use at all.

So next time you see some open-access poll shouting about having tens of thousands of responses and are tempted to think “Well, it may not be that representative, but it’s got a squillion billion replies so it must mean something, mustn’t it?” Don’t. If you want something that you can use to draw conclusions about the wider population, it really is whether it reflects that population that counts. Size alone won’t cut it.

=

* You see different sample sizes quoted for Gallup’s 1936 poll – I’ve seen people cite 50,000 as his sample size or just 3,000. The final America Speaks column before the 1936 election doesn’t include the number of responses he got (though does mention he sent out about 300,000 mailout surveys to try and get it). However, the week after (8th Nov 1936) the Boston Globe had an interview with the organisation going through the details of how they did it that says they aimed at 40,000 responses.
** If you are wondering why the headline in that thumbnail says 54% when I’ve said Gallup called the final share as 55.7%, it’s because the polls were sometimes quoted as share of the vote for all candidates, sometimes for share of the vote for just the main two parties. I’ve quoted both polls as “share of the main party vote” to keep things consistent.


475 Responses to “Size alone is not enough – the tale of the Literary Digest”

1 4 5 6 7 8 10
  1. @ LASZLO

    There’s never been a by-election in seat I’ve been living in, but I’m willing to bet that the number of uncommitted likely voters taking close note of what’s said at a candidate hustings is unlikely to make it to triple figures.

    I agree he’s a comically weak candidate, and I think Labour are going to hold on pretty comfortably, but I’m not really sure the two things are that closely related. Tho without the benefit of accurate polls we will of course never know either way…

  2. I may be missing something, but I can’t see the relevance of any of Ken’s posts to polling.

  3. The two by-elections….Labour will hold Stoke Central quite comfortably. UKIP and its maverick leader have poisoned themselves with all sorts of shenanigans.

    Up until now, I was pretty sure Labour would also hold onto Copeland but having looked at the results from the GE I didn’t realise how close the Tories and Labour were.

    Although the Lib/Dems polled a tiny 3.5% in this seat in 2015 (down 6.7%) from 2010, if they can reach their 2010 % in the seat then Labour might struggle with some of their remain voters going to the Lib/Dems. Also, Labour may lose votes from leave voters going over to UKIP and with TM visiting the seat today and showing support for the new nuclear power plant, Labour again could lose support to the Tories.

    There will certainly be a lot of churn in this by-election and the Tories may be the biggest winners.

  4. ALLAN CHRISTIE……Good idea, although the financial profile of the EU will take a significant hit when we depart into the wider world. I see that some of the EU trade agreements negotiated over many years will now have to be renegotiated in the light of the shrinking EU.
    By the way, I’m proud to be a Little Englander,( or Manchester Liberal,) defined by being a lover of free trade, anti-slavery, emancipation of women, and workers rights, in the 19th century iteration, as espoused by Adam Smith and friends, and I’m opposed to the Corn Law mentality of the EU. :-)

  5. @EDGE Of REASON “I’m willing to bet that the number of uncommitted likely voters taking close note of what’s said at a candidate hustings is unlikely to make it to triple figures.”

    Spot on. I agree with your analysis. This is Labour’s seat to lose and it’s likely the Tory vote will hold up denying the kind of tactical vote that UKIP would need to win.

    Conversely if enough Labour voters do indeed cross to UKIP to get them across the line then are we witnessing the start of a ScotLab repeat in the Midlands and the North?

  6. @Somerjohn

    I alluded to what would happen. Namely our imports and exports would rebalance away from the EU if there was a so-called punishment divorce killing a massive 50% of our EU exports. Naturally if Tariffs did that, they would also be implemented on our side too.

  7. Ken

    In the 19th century, the term “Little Englander” had nothing to do with the causes you mention. It simply meant someone opposed to the expansion of the British Empire.

    If you are using the term in a contemporary way, that also has nothing to do with those causes. Collins defines it as “a person who perceives most foreign influences on Britain’s culture and institutions as damaging or insidious”.

    While I recognise that you are doing a lot of trolling tonight, it’s probably worth remembering that effective trolling is subtle, not brash, and uses terminology correctly.

    Hope this helps.

  8. Stoke Central

    Stoke-on-Trent Central 2015
    Labour: Tristram Hunt – 12,220.
    Conservative: Liam Ascough – 7,008.
    UKIP: Mick Harold – 7,041.
    Lib Dem: Zulfiqar Ali – 1,296.

    Repeat, dark horses: Conservative.

  9. Well, well, well – the sound of ringing bells in church has been eclipsed by the sound of ringing hands in synod –

    https://www.theguardian.com/world/2017/feb/15/church-of-england-in-turmoil-as-synod-rejects-report-on-same-sex-relationships

    The men in dresses continue to try to sort out their personal problems with sex, and the report that could be summed up as saying ‘we like gays, but not that much’ gets kicked out by the synod.

    How irrelevant they all are, yet they still pretend to offer moral guidance to the nation.

  10. PATRICKBRIAN……With the best will in the world, I often visit this blog and struggle to connect the debated subject to polling. I put it down to my inability to identify subtle erudition when it stares me in the face. ;-)

  11. Established church no less Alec!!

  12. OLDNAT……Thanks for the heads up, it does help. I recognise that I’m on dangerous ground when you’re around, I was hoping you might be boning up on the latest copy of pedants monthly. ;-)

  13. Jim Jam

    Clearly, Brenda needs to “take back control” of her Church in the formerly independent (but thoroughly Establishment) Kingdom of England.

    She’ll no doubt take some good advice on the matter from Trump when he comes on his state visit to Birmingham.

    Is there a palace (or, at least, a decent restaurant) there to house the banquet?

  14. Ken

    I don’t need to “bone up” on such matters.

    They should be pretty common knowledge to anyone with a half-decent education.

  15. Completely on thread:

    The on-line poll on the Whitehaven News site has now reached 1650 votes. That is more than most proper opinion polls.

    When I first reported this poll a few days ago it was under a 100 and had Cons well ahead with Lab languishing behind UKIP and only just ahead of one of the independents.

    Now, with the benefit of greater numbers, it is clear that the LDs are going to live up to their historic track record in by-elections and come storming out of nowhere to win the seat. This is obviously down to their leader having visited from his seat next door.

    Cons will clearly be disappointed to have slipped back from favourites to third place, though Lab are now only holding on to second place by their fingertips.

    Would anyone care to nudge this Voodoo poll in a new direction? You don’t need to live in Copeland to vote.

  16. @Alec

    I think a lot of people who think that the CofE must sign up to gay marriage are not religious. Unfortunately for the CofE, that includes an awful lot of their clergy…

    (A lot can be done in Christianity to use “love thy neighbour” to obliterate Old Testament and Pauline teaching on homosexuality. But saying of a gay marriage that is part of an institution ordained by God. Of course, if someone believes that a rejection of gay marriage is irrational, then that is evidence against the existence of God. If there is a god, does not the nature of creation represent his clearest word? Many heretics have gone down similar lines of thought, e.g. when compelling Greek philosophy contradicted the Bible.)

  17. OLDNAT…..In that case I suggest you revisit your reference, or perhaps references, difficult I know for a person of such fearsome self belief. ;-)

  18. joseph1832

    “If there is a God” – why did he allow the creation of the EU?

    (That’s a freebie for Allan Christie to use).

  19. KEN
    ALLAN CHRISTIE……”Good idea, although the financial profile of the EU will take a significant hit when we depart into the wider world. I see that some of the EU trade agreements negotiated over many years will now have to be renegotiated in the light of the shrinking EU”
    _____________

    They only have themselves to blame. Personally, I would like to see the thing broken up and each of the former states say goodbye to Brussels Institutionalisation and like the UK start finding their own way in the World.

    Military alliances are fine but political and economic integration!!….Yuck!

  20. Actually, maybe the LD surge is down to them having mopped up all the remain voters while the leave majority are widely dispersed among the rest of the candidates.

    Pity Jack Lenox the Green candidate in a seat with two nuclear installations and a third planned.

  21. Old Nat @11:33

    I never had you down as a Jansenist.

    Free will means that people may choose folly. It is not a sin to be foolish, but it won’t lead to salvation.

    On the other hand, self discipline and doing the right thing will.

  22. JONESINBANGOR
    Stoke Central
    Stoke-on-Trent Central 2015
    Labour: Tristram Hunt – 12,220.
    Conservative: Liam Ascough – 7,008.
    UKIP: Mick Harold – 7,041.
    Lib Dem: Zulfiqar Ali – 1,296.
    Repeat, dark horses: Conservative
    _________

    Big ask…I can’t see it but who knows, by-elections tend to throw up some spectacular results. I’m sure Labour will hold this seat quite comfortably but with a reduced percentage.
    ……….

    OLDNAT

    Thanks, I’m always on the hunt for free metaphors. ;-)

  23. ALLAN CHRISTIE…..I agree, but be careful, OLDNAT’s trolling tonight, so make sure you don’t bang his bridge. ;-)

  24. Paul H-J

    I’m no Jensenist – not interested in classic cars.

    “It is not a sin to be foolish, but it won’t lead to salvation.”

    Try telling that to the Brexiteers (or any other partisan supporters of a particular strategy, for that matter!) :-)

  25. ALLAN CHRISTIE…..Sorry my reply was to your 11-34 post.

  26. KEN

    OLDNAT’S bridge being banged is one thing but the goals in the Champions league are certainly being banged in. Also, Aberdeen has put 7 past Motherwell tonight…that’s major sheep banging. ;-)

  27. @PAUL H-J

    Considering the content of Anthony’s article we are commenting on we should all do a public service and go over to the Whitehaven News and vote for the Green Party, just to show how ridiculous a public poll on select media truly is!

  28. Allan Christie

    Yep. Good game!

    My favourite banner at Pittodrie was “We’ll support ewe evermore”.

    Nytol.

  29. My prediction for Stoke Central:

    Labour: 7,500
    Conservative: 8,500
    UKIP: 5,000.
    Lib Dem: 3,000

  30. ALLAN CHRISTIE……Certainly a good night for neutrals, goals and incidents galore, hopefully Chelsea will contribute to the excitement next season.
    I wonder if Celtic will carry their astonishing form into the Champions League, I remember Tommy Burns, don’t cha know. :-)

  31. @Jonesonbangor
    So remortgage your house and put it on Con @41/1

  32. OLDNAT
    Allan Christie
    Yep. Good game!
    My favourite banner at Pittodrie was “We’ll support ewe evermore”.
    Nytol
    ________

    Ha! THat’s very bahh! ;-)
    ………..
    KEN

    Rodgers is building a Champions league team. If he can hold onto Moussa Dembélé after the summer then I think Celtic stand a good chance of reaching the last 16 and who knows, even have a punt at the last 8.

    The club is smashing records almost every time they take to the pitch. I’m sure Chelsea will be contributing to Champions league excitement next season and wouldn’t it be great if Celtic and Chelsea met in the final! Okay highly unlikely in Celtics part but if only!!

  33. JONESINBANGOR
    My prediction for Stoke Central:
    Labour: 7,500
    Conservative: 8,500
    UKIP: 5,000.
    Lib Dem: 3,000
    _________

    Hope your hat is edible. … ;-)

  34. @oldnat

    “However, respondents were also asked to exclude membership of UK and EU, since Brexit removes that option.

    37% : Independent Scotland in EU
    11% : Independent Scotland outwith EU
    43% : Scotland in UK, outwith EU”

    48% to 43% or if weighted to 100%, 52.7% to 47.3% in favour of Indy.

    Game changer! (had to be said)

    Of course it will come down to people choosing UK vs EU in most cases. Devil you know, or the bid scary world.

  35. @ Colin

    ‘Actually , the subject on which you touch-albeit with the flippancy of ignorance-is , for me, one of the most important topics on which UK Governments will have to expend great thought as they grapple with post their new post Brexit legal “competence”
    Replacing the CAP regime with something which rewards real environmental conservation outcomes, rather than mere possession of acreage, whilst thinking through from scratch UK’s consumption & production of Food in a Global Marketplace will test DEFRA severely.
    Civil Servants & Politicians alike will need to think very deeply ( for once) about this area before presenting voters with something they haven’t had for decades-a UK Policy for the use of Agricultural Land & Environmental Conservation on it.’

    Totally agree with you and an area which is of vital importance with regard to climate change and future trade deals, as well as food production and species diversity.

  36. Re: Universal Basic Income

    The devil is in the detail. A replacement for benefits is not the same as giving a basic income to every man woman and child.

    Child benefit was a sort of UBI which didn’t incur problems macro-economically but that can’t be said of a really comprehensive UBI.

  37. Re: Universal Basic Income

    We can’t sustainably fund our current liabilities. Where are we going to find the dosh for UBI.

    @JONESINBANGOR

    Your prediction would mean the Conservatives taking 35.4% of the vote on those figures. That would be an increase of 12.9% over the 2015 election.

    I would advise you to perhaps mortgage your garden shed and put it on the Tories at 41/1 rather than Guymonde’s suggestion!

  38. @SEA CHANGE

    “We can’t sustainably fund our current liabilities. Where are we going to find the dosh for UBI.”

    ————-

    I wonder about the costs too, but if it means more people working, or working more hours, then it could work out. Especially as this in turn might provide more stimulus.

    Another way of looking at it is that eventually over time, as growth in the economy stacks up year on year, it becomes more affordable.

    In addition, increased productivity from all these robots we’re supposed to see would make it more affordable.

    Or if the government seriously bears down on living costs, housing and bills then it reduces the benefit required.

    Or if they stimulate more growth via R&D. Or Thorium…

  39. I should add…

    Costs of the citizen’s income would also be mitigated by…

    – replacing existing benefits without the bureaucracy of all the meZns testing and admin

    – you can in effect claw back the benefit from the better off his taxation

    – if people have a guaranteed income they can plan better, be more efficient, avoid poverty traps etc.

    – maybe in the longer run if they up the value as economy grows it’ll save money via improved health outcomes, reduced crime, reduced addiction etc.

    – people can be more entrepreneurial, take a bit more risk

  40. And let’s face it, it’s a very British thing to do, given our welfare state and NHS traditions etc.

  41. I suppose there’s an outside chance it might even give Scotties an additional reason to stick with us. And we could lay the Spaceport runway across the border…

  42. @Sea Change

    Another way of looking at it is, you know how we used to have governments of whatever stripe who made sure there was full employment, where a single wage could support a family, and affordable bills, affordable housing, free tuition etc., and set up the welfare state and kept the economy growing nicely despite huge debts to pay off from the war? Well that’s the sort of thing that might make the citizen’s income more affordable. (Only with more storage and cheaper synths, hopefully. I don’t ask for much…)

  43. I’m on a roll, so why not add summat extra: the idea also finds favour among right and left. Friedman, Hayek, liked it…

  44. Jonesinbangor

    Predictions for stoke

    I hope you are wrong . My whippet has a very sad demeanour this morning.

    i tried to cheer him up by promising to read out the Ken versus Old Nat posts.

  45. S Thomas,
    “well that is an interesting though novel point of view.The logic is therefore that the EU has done such a good job there is no need for nato and american subsidy”
    Better job than the preceding 50 years regime. And compared to the levels of military investment in those previous 50 years, defence spending has fallen steadily.

  46. Re CI – how about it promoting:-
    A more cohesive society, less crime, less prisoners, less kids in poverty, less poverty related illness and disease, better performing schools due to less drag from poverty related problems etc

    Of course there is a cost benefit analysis but the simplistic idea that all spending should be cut cut cut to deal with the deficit (and, that investment in measures of long time benefit are unaffordable) is the Economics of Austerity that made 2010-13 or so worse than it needed to be and caused long time damage costing more in the long run (all imo of course).

  47. @CARFREW

    One of the difficulties with the idea of a UBI is what happens to those who don’t manage their budget well for whatever reason. Would there be another layer of welfare to help them?

    If there is, then the savings from the welfare bureaucracy would be reduced. If there isn’t then we would need a much more callous society than we currently have to accept the consequences.

    The reason UBI is liked by both edge cases of left and right is that the right assume there will be no safety net, while the left assume it will still be in place in addition to the UBI. Most arguments I read for UBI tend to fudge details like this, but they will need addressing before the idea can be taken further.

  48. Ken,
    didnt Adam Smith warn against the dangers of free trade? “People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices”

    Or maybe it depends on quite what you mean by ‘free trade’.

    Alec,
    “How irrelevant they all are”
    But from the polling perspective, elections are won by slim margins and this particular slim margin could turn out to be decisive at some point.

  49. PATRICKBRIAN
    I may be missing something, but I can’t see the relevance of any of Ken’s posts to polling.

    Ken would probably agree, but at least he brings a really positive voice to Britain going forward and his analysis in his first post was brilliant IMO.

  50. just swapped a straight jacket for a prison

1 4 5 6 7 8 10