“But the sheer size of the survey […] makes it of interest…”

One of the most common errors in interpreting polls and surveys is the presumption that because something has a really huge sample size it is more meaningful. Or indeed, meaningful at all. Size isn’t what makes a poll meaningful, it is how representative the sample is. Picture it this way, if you’d done an EU referendum poll of only over 60s you’d have got a result that was overwhelmingly LEAVE… even if you polled millions of them. If you did a poll and only included people under 30 you’d have got a result that was overwhelmingly REMAIN… even if you polled millions of them. What matters is that the sample accurately reflects the wider population you want them to represent, that you have the correct proportions of both young and old (and male & female, rich & poor, etc, etc). Size alone does not guarantee that.

The classic real world example of this is the 1936 Presidential Election in the USA. I’ve referred to this many times but I thought it worth reciting the story in full, if only so people can direct others to it in future.

Back in 1936 the most respected barometers of public opinion was the survey conducted by the Literary Digest, a weekly news magazine with a hefty circulation. At each Presidential election the Digest carried out a survey by mail, sending surveys to its million-plus subscriber base and to a huge list of other people, gathered from phone directories, membership organisations, subscriber lists and so on. There was no attempt at weighting or sampling, just a pure numbers grab, with literally millions of replies. This method had correctly called the winner for the 1920, 1924, 1928 and 1932 Presidential elections.

In 1936 the Digest sent out more than ten million ballots. The sample size for their final results was 2,376,523. This was, obviously, huge. One can imagine how the today’s papers would write up a poll of that size and, indeed, the Digest wrote up their results with not a little hubris. If anything, they wrote it up with huge, steaming, shovel loads of hubris. They bought all the hubris in the shop, spread it across the newsroom floor and rolled about in it cackling. Quotes included:

  • “We make no claim to infallibility. We did not coin the phrase “uncanny accuracy” which has been so freely applied to our Polls”
  • “Any sane person can not escape the implication of such a gigantic sampling of popular opinion as is embraced in THE LITERARY DIGEST straw vote.”
  • “The Poll represents the most extensive straw ballot in the field—the most experienced in view of its twenty-five years of perfecting—the most unbiased in view of its prestige—a Poll that has always previously been correct.”

digestpoll

You can presumably guess what is going to happen here. The final vote shares in the 1936 Literary Digest poll were 57% for Alf Landon (Republican) and 43% for Roosevelt (Democrat). This worked out as 151 electoral votes for Roosevelt and 380 for Landon. The actual result was 62% Roosevelt, 38% for Landon. Roosevelt received 523 in the electoral college, Landon received 8, one of the largest landslide victories in US history. Wrong does not nearly begin to describe how badly off the Literary Digest was.

At the same time George Gallup was promoting his new business, carrying out what would become proper opinion polls and using them for a syndicated newspaper column called “America Speaks”. His methods were quite far removed from modern methods – he used a mixed mode method, mail-out survey for richer respondents and face-to-face for poorer, harder to reach respondents. The sample size was also still huge by modern standards, about 40,000*. The important different from the Literary Digest poll however was that Gallup attempted to get a representative sample – the mail out surveys and sampling points for face-to-face interviews had quotas on geography and on urban and rural areas, interviewers had quotas for age, gender and socio-economic status.

pic2097election

Gallup set out to challenge and defeat the Literary Digest – a battle between a monstrously huge sample and Gallup’s smaller but more representative sample. Gallup won. His final poll predicted Roosevelt 55.7%, Landon 44.3%.* Again, by modern standards it wasn’t that accurate (the poll by his rival Elmo Roper, who was setting quotas based on the census rather than his turnout estimates was actually better, predicting Roosevelt on 61%… but he wasn’t as media savvy). Nevertheless, Gallup got the story right, the Literary Digest hideously wrong. George Gallup’s reputation was made and the Gallup organisation became the best known polling company in the US. The Literary Digest’s reputation was shattered and the magazine folded a couple of years later. The story has remained a cautionary tale of why a representative poll with a relatively small sample is more use than a large poll that makes no effort to be representative, even if it is absolutely massive.

The question of why the Digest poll was so wrong is interesting itself. Its huge error is normally explained through where the sample came from – they drew it from things like magazine subscribers, automobile association members and telephone listings. In depression era America many millions of voters didn’t have telephones and couldn’t afford cars or magazine subscriptions, creating an inbuilt bias towards wealthier Republican voters. In fact it appears to be slightly more complicated than that – Republican voters were also far more likely to return their slips than Democrat voters were. All of these factors – a skewed sampling frame, differential response rate and no attempt to combat these – combined to make the Literary Digest’s sample incredibly biased, despite its massive and impressive size.

Ultimately, it’s not the size that matters in determining if a poll is any good. It’s whether it’s representative or not. Of course, a large representative poll is better than a small representative poll (though it is a case of diminishing returns) but the representativeness is a prerequisite for it being of any use at all.

So next time you see some open-access poll shouting about having tens of thousands of responses and are tempted to think “Well, it may not be that representative, but it’s got a squillion billion replies so it must mean something, mustn’t it?” Don’t. If you want something that you can use to draw conclusions about the wider population, it really is whether it reflects that population that counts. Size alone won’t cut it.

=

* You see different sample sizes quoted for Gallup’s 1936 poll – I’ve seen people cite 50,000 as his sample size or just 3,000. The final America Speaks column before the 1936 election doesn’t include the number of responses he got (though does mention he sent out about 300,000 mailout surveys to try and get it). However, the week after (8th Nov 1936) the Boston Globe had an interview with the organisation going through the details of how they did it that says they aimed at 40,000 responses.
** If you are wondering why the headline in that thumbnail says 54% when I’ve said Gallup called the final share as 55.7%, it’s because the polls were sometimes quoted as share of the vote for all candidates, sometimes for share of the vote for just the main two parties. I’ve quoted both polls as “share of the main party vote” to keep things consistent.


475 Responses to “Size alone is not enough – the tale of the Literary Digest”

1 8 9 10
  1. @Neil A

    I am not saying that’s there is absolutely no one on the planet who has made the lifestyle choice to take the mick a bit.

    I am simply saying that of the set of peeps who do not appear to be exercising their agency, quite often it’s not quite as easy for them as it seems.

    And of those for whom it seems like a choice, some have decided to act like it is because they can’t really see many options.

    Sure, I’ve met people like you describe, quite often they were students who’d failed their first year and missed out on the standard career options and weren’t sure what to do. I meet them now, but these days they have options like working in bars and coffee shops, and can get tax credits etc.

    They’re often pursuing dreams, music, art, acting, writing etc.

    Some can be quite often be troubled though. If you’re not judgemental they’ll confide, and they may well have been through some rough times growing up. Or more recently. But many people probably wouldn’t know. I still come across them now…

  2. @Robin

    On what sound scientific basis do you choose to believe in the (very unlikely) least impactful model?

    Robin

    I have expressed no opinion. I have identified a climate scientist, Judy Curry, whose position is that there is disagreement and uncertainty within the climate science community. She refers in the video to which I linked the weakness of the computer model predictions – there is a growing divergence between climate model projections and observations. See it for yourself if you wish. Please don’t attribute to me any opinions. I only wish to draw attention to this video. It matters not to me what you or anyone else wishes to make of it.

  3. @Carfrew

    I think we are inching closer to mutual understanding and even, partially at least, to agreement.

    Every human trait has roots somewhere in genetics, experience, upbringing or advantage/disadvantage. Almost everything can be explained in some way, if you look deep enough.

    But in my opinion we have to allow for a large dose of free will in the equation, or we rob humanity of what makes it human. The reasons someone made a bad choice may be legion, but if it was still a choice I think to some extent people still have to live with it. We can’t design a system which levels up the outcome for every person to remove every consequence of every one of their actions.

    My starting point was that I am a big supporter of at least experimenting with Citizen’s Income, and much of my reasoning is to allow people the freedom to make those choices and to remove the stigma from doing so.

  4. Carfrew

    I am sure that your analysis that all people who drop out of university are abused children is backed by a huge amount of prejudice on your part.

    No doubt anecdotally you have spoken to a tramp in the park who has confirmed everything you previously believed

  5. JIM JAM……Have you ever been into a bank, Retail, Investment, Central, what have you got against the people who work hard every day to ensure that National and International financial needs are satisfied, in compliance with rigorous regulatory regimes. Altruistic, honest , generous, terms that describe my old colleagues in the industry, and indeed I would apply to myself..
    Do some research, every industry has its bad people, we had ours, I didn’t know any.
    You should know better, your posts are usually of a higher standard.

  6. @Neil A

    “We can’t design a system which levels up the outcome for every person to remove every consequence of every one of their actions.”

    —————

    I’m not arguing to absolve, like some left winger might!! I have an educational perspective. I wanna deal with it, not sweep it under the carpet.

    When I was teaching, I might well have some students who weren’t very motivated. I considered it my job not to make excuses for it but to figure out why in each case, and find a solution. If they wanted one, which was usually the case…

    It’s amazing how much you can turn things around if you have that capability…

  7. @S Thomas

    That’s not what I said, and your comment beggars belief

    I have already made clear that I do not consider every person who doesn’t have a job to have a hard luck story.

    I am just saying problems are more common than the more judgemental might think. Your rush to blanket mischaracterise in order to dismiss speaks volumes.

    And it isn’t just abuse, it can be all kinds of things. Recent bereavement, mental health issues, non-obvious physical illness, prejudice, and much more and combinations thereof.

  8. @S Thomas

    And as I also indicated, sometimes they’re just not sure what to do. They might try a few jobs half-heartedly and it doesn’t work out. Some people know early on what thy went to do, or don’t much care. Others it takes longer.

    There are plenty people now graduating who don’t know what they’re gonna do. These days they just go travelling for a couple of years…

  9. @Rodger

    The news story you cited is on the Higher Education Participation Rate – a weird measure that is not really used any more. The HEPR actually measures, not the proportion of young people going to university, but the likelihood of a young person going to university, which is not the same thing.

    Even if 49% of young people did go to university – which they don’t – that wouldn’t make degrees worthless.

    One thing that I have found is definitely not a reliable indicator of intelligence is a propensity to make sweeping statements on Internet messageboards on whole sections of the population based on things the speaker doesn’t understand properly but think they do.

  10. @carfrew

    Most graduates have at least 45 years of working life ahead of them. We don’t expect people to get married at 21 any more.

    We shouldn’t expect them to always have a clear and firm of their future career either, especially in as uncertain a world as this one. Most graduates are smart enough to realise now that adaptability is the only way forward.

  11. PATRICKBRIAN

    Just popped in before going out. Splendid curry which is why I didn’t give Alec a long answer, see my earlier post

    If you read his posts Alec spends a lot of time trying to rubbish others views. All Ken was doing was giving him some sound advice and I just stated the obvious.

    Alec and I go back a long way on this site. Sometimes we agree, as we do on the probability of the economy slowing this year due to higher infaltion and high consumer debt. More often we disagree as we do on the likely effects of global warming and Brexit. He’s perfectly capable of looking after himself and usually makes a good argument. I don’t actually think he needs any help.

  12. Chris Riley

    “One thing that I have found is definitely not a reliable indicator of intelligence is a propensity to make sweeping statements on Internet messageboards on whole sections of the population based on things the speaker doesn’t understand properly but think they do.”

    I certainly agree with that.

  13. @Chris Riley

    I agree. I mean, I was astonished at Oxford to meet people who on arriving already knew, indeed had known for absolutely years, that they were gonna go into advertising. Not everyone is like that. In some other countries, it’s not uncommon to do a general degree and then another more specialised degree which gives you more time to figure out what to do.

    We specialise early in this country, reducing to just a few subjects by the time of A levels. Has its advantages, but can be tricky too…

  14. SOMERJOHN

    “Indeed. Alec is one of not many posters on UKPR who still takes seriously the idea that you should produce a coherent argument backed by evidence. If that is met by “don’t take everything so seriously” are we to conclude that UKPR is just a bit of light entertainment, with serious thinkers not required?”

    I think you have that badly wrong on both counts. It would suggest it would be difficult for Ken and I to have been so successful in life without being able to think seriously. I would also suggest that Ken and I get great enjoyment from life and yes, personally I for one get great entertainment from this site.

  15. @TOH – “If you read his posts Alec spends a lot of time trying to rubbish others views. All Ken was doing was giving him some sound advice and I just stated the obvious. ”

    No he doesn’t! He pulls people up when they are factually incorrect, and expects others to pull him up when he falls foul of basic factual correctness.

    I’ll let @Ken speak for himself.

  16. THE OTHER HOWARD……..Quite, I echo your comments, certainly ALEC is more than capable of dealing with a robust interchange of views, he strikes me as being highly intelligent and of course academically assured, mind you, he is a remainer so he has his weaknesses, as do we all.
    Perhaps the cut and thrust of argument rather than interminable polite debate is what we are used to, this could explain some of the pained responses we attract. ;-)

  17. @ToH

    There are some very good rules to good netizenship that I try to abide by

    – check primary sources
    – don’t talk outside what you *know* to be your expertise.
    – if you do talk outside your expertise (which for most people is rather narrow), there is a good chance you’re wrong
    – never assume that if you say a thing and your ground is not solid, that there isn’t someone who knows more about it than you do who might be reading, and they might ask you about it
    – the real experts on an issue know that they don’t know everything. It’s possible you might know more than them on the issue you disagree with them on. However, the chances are that it is not this time, so be prepared when they politely explain their point, to listen and accept that they might be right and you might be wrong
    – some people do know more than you do
    – if you think you’re the smartest person in the room/messageboard, you almost certainly aren’t.

  18. @Colin

    “The thing is-which route is most likely to get National Governments to act on this?:-
    * A Brussels Directive & slap on the wrist , both administered by people who are unknown to most UK voters.
    or
    * A Strong Campaign inside UK, by the affected citizens of UK cities, taken up by a Political Party looking for votes?”

    Well actually, the latter can and does happen today, and is, as I speak, in London. And the campaign is ‘it’s a scandal that we’re breaking EU pollution limits’. If there were no such limits, we would be subject to a lot more of ‘I have never seen convincing evidence that NOx emissions were really harmful to people’ and, “well, those limits were set in Brussels so we need to set up a research project to see if there is really evidence to back this up. We will appoint Lord Saville to look into it throughly, as he did for the Bloody SUnday enquiry.”

  19. @Alec Co2 Warming vs Poisoning the Biosphere

    I would contend that Co2 Warming is not going to kill us and that Co2 is not a poison, in fact it is critical to life on Earth. Yes there could be hugely costly migration to higher ground and cooler climes.

    On the other hand poisoning the planet to make it uninhabitable is clearly a greater problem. A lot more money is being spent on climate change versus pollution. I question this, it does not make sense in my view.

    @BFR – On December’s Mori.

    Yes that’s true. I have always thought Mori’s methodology was suspect because it clearly under reports the UKIP VI.

  20. @Sea Change

    Contend all you want, but CO2 is a poison. Just because plant’s use it for photosynthesis doesn’t mean it isn’t dangerous for humans. Irrespective of levels of O2, An atmosphere containing 10% CO2 will kill you in 30 minutes. 3% is regarded by safety organisations as unsafe.

  21. Ken – been out.
    I thought you would understand my use of irony, stereotyping and over-simplification to highlight what I saw as an unfair dig at graduates in your earlier post.

    Perhaps my post was a tad Arid?

  22. Just to add to local by-election weirdness, the Greens have tonight taken a ward – off UKIP!

    Britain Elects [email protected] :

    Lydbrook & Ruardean (Forest of Dean) result:

    GRN: 35.3% (+27.9)
    CON: 24.3% (+6.2)
    LAB: 22.7% (-2.0)
    UKIP: 11.1% (-12.2)
    LDEM: 6.6% (+6.6)

  23. @Saffer
    If those figures are right Lab had 24.7 last time and Kipper only 23.3% yet you’re right that Britainelects shows the numbers you do that it’s a Green gain from UKIP.
    Sorry, it’s my inner struck-off accountant breaking out.

  24. ken,
    “ALEC seems unable to move on”

    A peculiar trope seems to have arisen lately, that people’s opinions ought to ‘move on’ if they find themselves on the losing side of an argument. Such a position would mean that human society was forever static and unchanging. No point at all in these polls endlessly assessing whether labour or conservative is in the lead, because once one side had established even the tiniest lead, the other would be expected to fall in line. Plainly this never happens, and all in all society believes it should not happen.

    The EU is a case in point, where the debate has raged for 50 years. Much longer if you consider the concept of a united europe, which dates back millenia.

    S Thomas,
    “Why do you find that surprising? The Tories always poll the highest amongst the oldest if you look at all the polls. The oldest always have the least qualifications as a group but this not surprising as many of them never had the opprtunity to go to university (around 6% when I went I think).

    Does this mean anything? Perhaps the oldest are the wisest,”

    Your argument seems to be that the old are tory inclined, the young labour: changes in society mean more people are educated now, so naturally there will be a higher proportion amongst the young. The determinant is age, and education per se has no effect.

    However this also works the other way around. If more education is what determines choice of labour, then you would expect younger people to be pro labour, and a group such as the older citizens who had less education to be tory.

    So does age cause party choice, or does education? labour/ education/youth correlate, as do tory/less education/age. Presumably the determinant is not innate intelligence, which I dare to presume would have a similar distribution in either age group.

  25. oops sorry, that was not S Thomas but The Other Howard.

1 8 9 10