I posted these a couple of weeks before the election, but I see more and more of them cropping up in the comments, so I think it’s worth reposting it for newcomers.

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.


675 Responses to “REPOST: Too frequently asked questions”

1 11 12 13 14
  1. Correction: I meant I think they were shown on the ITV news last night. Might be wrong though.

  2. Sky has been very biased I think so far. However, there was an interesting report tonight where they had a group of “floaters” watch the debates A. as everybody else saw them, and B. with a lot most of the body language censored (confused as to how they did this?). In situation A, the general consensus was that Clegg won, with Brown second. In B, that Brown won, with Clegg second. I was shocked! I couldn’t believe I was watching Sky!

    Dirty trick by the Guardian, albeit no surprise. As a bit of a leftie, I should like the Guardian, but I never have. The only newspaper I do like is The Independent (although only out of lack of choice).
    BBC all the way though.

  3. guys reference the 36 24 36 i posted on it earlier – it is a poll from comres with strings attachd scroll up….

  4. @Sukdor

    I’m taking this with a truckload of salt but this just got twoatted:
    ComRes shows CON-LAB-LD as 36-24-36

    __________________________________________

    Which means others at only 4%

    I don’t think so

  5. @STATTO
    Perhaps I shot the wrong man, in which case forgive me. I always knew I would go gaga when the Liberals got in.

  6. I’m guessing it wasn’t anything near a real poll then?

  7. @ Eoin

    Yep, 36 36 24 was directly after leaders debate. It is on Comres website to anyone who hasn’t seen it.

  8. Latest Comres on their website C 36 Lab29 Lib 26

  9. School students aren’t the most representative of samples! My school’s poll in 2005 (prestigious independent school in ancient university city) saw the Communist Party narrowly edging the Revolutionary Communist Party (Marxist-Leninist)

  10. It is amazing the stupidity of some people regarding polls. Last night, I was looking and Sky News’s live Facebook feed. That was when the post debate polls were being anounced. The comments regarding to the polls you wouldn’t belive! Here are a few from memory:
    1) YouGov is biased towards the Conservatives
    2) It is for the Sun so what would you expect
    3) Those polls are wrong.

    Completly gob smacking!

  11. Roland – you did, but I don’t hold grudges.

  12. The PB rumour re Yougov is 36 27 31 which may be completely wrong but does sound possible

  13. what will effect of the gruniad supporting lib dems mean ?

  14. I live in Withington and have had plenty of Libdem leaflets. The MP is a local and reasonably high profile in supporting local campaigns such as support to the extension to the Metrolink. I’d be surprised if he doesn’t increase his majority

  15. others on 6% unlikely

  16. @SimonK
    “Cricky we’re back to everyone knowing someone who is going to vote X Y Z because of A B or C…..”

    There is much speculation as to vote switching, particularly since this is the first 3 horse race for a long time, and these anecdotes help to explain different rationales.

    Experienced posters are not claiming that it is representative or predictive so, if you don’t mind, some people would like to read them to get a feel for the mood in different, albeit selective, portions of the electorate with which they may have limited contact.

    After all, you can always skip them if you are not interested.

  17. Oxo,

    Thanks for the local school vote post. Not scientific, but it all adds to the merriment.

    It does tally with the strength of LD amongst the younger vote.
    We all know that the Tory vote is v concentrated in the 55+ group.
    I do wonder if the Tory electorate is gradually ‘ageing’ out – maybe explaining why they are not >40% after 13 years opposition and economy all in a bit of a hole.

  18. Guardian readership around 1.2m – how much will this affect Lab? Comments on the Guardian site generally seem in favour of the decision but by no means everybody seems happy.

  19. Howard posted this for me earlier:

    @skudor
    ipsos-mori.com/newsevents/ca/ca.aspx?oItemId=238
    Put www. before it and google for 2005 table at bottom of Bob Worcester’s blog

    There’s a great table showing the last poll results before the 2005 election. Out of five companies no one was more than 2% out – 1% for the tories, 2% for labour and 3% for Lidbdems. Although on average they got the libdems spot on. Tories were on the whole underestimated by 1% and Lab over estimated by a range up to 2%.

    Pretty accurate.

  20. @ Polly Ticks

    You’re right of course! How can I ever hope to get over something that I’ve never got onto…

  21. @POLLY TICKS

    Yes the Guardian has come our for the Libs but also state that Libs should vote Labour in the Con/Lab marginals. Could tactical voting by Libs in there areas still make Lab the largest party on election day?

  22. I don’t think the Guardian has been supporting Labour since the last election. There have been a few fall outs

  23. Just said on Sky, “Rumours of a good poll for LD’s tonight!”

  24. ‘Guardian comes out in support of Lib Dems…’

    About as surprising as Torygraph coming out for Tories.

    The Guardian had a great reputation for backing the SDP-Liberal Allliance in the 80s. It was a disasterous commercial move for the Guardian Group. Labour voters moved to the Indie, even though it didn’t back Labour either.

    Here’s a Guardian newspaper group today virtually on the brink of bankruptcy. Compared to 1983, it has lost two-thirds of its readership. It just lost a lot more.

    Guardian closure anyone? And how ironic if it happened under a Tory govt the Guardian did so much to deliver.

    But really, would you decide your vote because of the newspaper you bought or the website you occassionally hit?

  25. @Ian Mc

    amen

  26. Sky have just said, “Rumours of a goodLD poll tonight!”

  27. The Labour vote is more Mirror/Sun than Guardian, and any Guardian reading Labour voters will be unlikely to be swayed.Tellingly the paper is advising its readers to vote Labour in Lab/Tory marginals.Next they’ll be directing bears to the nearest woods.

  28. WW

    Answer same as Sun supporting cameron -nil nil

  29. I can’t believe The Guardian have dropped Labour.

  30. @Greengrass

    No I don’t mind them, don’t get me wrong. But as I have been rounded on previously by a pack of rabid dogs it appears that these anecdotes are only selectivly allowed if they support a particulary view…

    If I can join in…

    My wife has come out for Cameron as she previously has voted Blair
    At work 1 con has switched to labour after bigotgate ( I bet he won’t actually vote )
    Down at the golf club the con vote is strong

    The street is tactically voting anti Brown.

    Actually, non of this means anything but hey ho its fun

  31. Ah – just saw the 36-24-36 on the Comres site; it’s the make up of the viewers polled.

  32. @KYLE DOWNING
    -I agree that it is a little silly to doubt a poll because you don’t agree with it’s results.
    -But since media corps have become so partisan; perhaps people believe polls reflect the political leanings of the media reporting them.

  33. “Down at the golf club the con vote is strong”

    No kidding.

  34. Mmmmm the Guardian, a state funded newspaper comes out in support of a political party further to the left than Labour. Didn’t the Soviets have their own newspaper?

    Few comments on here about what Tony Blair can bring to the party?

  35. if the guardians decision leads to a tory majority then expect them to lose readers quicker than Iose money when my wife out for a drink

  36. @CHRIS D

    Yes I was actaully, just didn’t know how to use smileys.

    Actually, our golf club has a quit a mix of people from retired chief inspectors to Tesco delivery drivers..

    But I was just been flippent

  37. On Sky, bookies estimate DC will be be 12 seats short of a majority on about 300, GB on 220 and NC around 100+

  38. Skudor

    I also have one on how Mori got it all wrong at the start of the 2005 campaign – here it is!

    ipsos-mori.com/newsevents/ca/ca.aspx?oItemId=207

    Usual drill required.

  39. my guess for yg

    35
    28
    28

  40. Official:
    Latest YouGov / Sun poll: Con: 34; Lab: 28; Lib Dem: 28. Fieldwork 29-30 April 2010; sample: 1,412

  41. @SimonK
    “as I have been rounded on previously by a pack of rabid dogs”

    Rabid dogs in cyberspace are quite amusing :-)

    “Down at the golf club the con vote is strong” – don’t expect *they’re* swingers!

  42. Eoin

    you called that a tad early, insider dealing?

  43. Roland – “What’s happened to the genius of Prime Minister Brown shineing through to victory over the vaucous little twitt “Dave”?”

    Oh don’t fret Roland, he still will.

  44. Could be good move for Grauniad. Reinstate lost readers with all those students. Primary purpose of move one suspects. Same as Murdoch used to do before son became carried away.

    Losing pronouns because of Roland’s miltary presence.

  45. So the debate had an enormous impact…..

    ..

    NOT!

    :)

  46. @Sue,

    what do you think to the Guardian editorial. very damning on Brown.

    I can’t believe it!

    Rich

  47. YYYYYYYYYYYYYEEEEEEEEHHHHHHHHHHHHAAAAAAAAAAAAAAAAA!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :)

  48. @SIMONK

    My comment would have had a smiley too if only I knew how. My only experience of a golf club is my gran’s. They don’t like the cut of my jib!

  49. If that yougov poll is correct I suspect Labour will be relieved.The “bigotgate” gaffe has had no impact.

  50. Am I the only one who sees 36-24-36 and thinks of 1970s beauty contests??? not very PC, sorry.

1 11 12 13 14