I posted these a couple of weeks before the election, but I see more and more of them cropping up in the comments, so I think it’s worth reposting it for newcomers.

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.

675 Responses to “REPOST: Too frequently asked questions”

1 12 13 14
  1. ugov 34,28, 28.. well!

  2. @Eoin

    Oh get over yourself ;)

    P.S. Historian – advocating use of Wikipedia? My teachers would have flayed you alive ;)

  3. [email protected]/28 on UNS (AW’s Swingometer)

    Lab 268
    Con 261
    LD 90
    Others 10
    NI 18

  4. I still think the Tories will fall short of an overall majority on May 6th, unless tonight indicates a lead over Labour at least 8% . All the parties are running out of time now.

  5. That’s a disappointing poll for the Tories – they would have hoped for 8% over Labour. I wonder if other polls will show a similar result?

  6. I was just about to post, before the site crashed….

    Anything less than 35% for the Tories tonight, and they will be disappointed.
    Anything more than 27% for Labour tonight, and they would be relatively happy, and that it’s not all over yet.

    So, since last Fri. night, the only movement is 1% each from Lab and Lib to ‘others’.

  7. YouGov show no notable change. Is this the election result then, bar any catastropies between now and Thursday?

    The only question now is how do election results tend to differ from opinion polls? From what I’ve heard on here, I’m lead to believe that Con and Lib can expect a bit more than predicted and Lab a bit less. Fair?

  8. Tomorrows mail has a Harris poll:


  9. @Colin Green,

    Historically, yes. But who knows.

  10. @ Johnty
    @ Ben
    Thanks both.

    Good summing up by locals, just the info I need. Will amend my site info % accordingly, but I had never thought it would go in the first place, so not included on my list.

    Have you thought about putting your local info onto the seat blog to counteract the rumours?

    Re Manchester Gorton – I too had it down only as a Lab/LD possible.

  11. In a previous post on a previous board, I said that since the second debate, the parties had settled at CON 34, LAB 27.5, LIB 29, and that movement since then had been MOE wibbling. Tonight’s YouGov poll of CON 34, LAB 28, LIB 28 tends to support that. It will be interesting to see if other polls released tonight (if there are any) also do so.

  12. @Xiby
    Solved the NE regional error on Regionals weekly cahartr at PH

    Figures should read

    North East C20 L39 LD35 0 6
    Changes last week C-5 L-6 LD11 O 0
    Changes since GE2005 C 0 L –14 LD 12 O 2

    These tie up with GE2005 data plus notional changes. Others figure was haywire
    GE2005 C20 L53 LD23 O 4

    This gives Lab/Con 7.0
    Lab/LD 13.0
    Con/LD 6

    Your latest regional YouGov is also showing NE Others too high by 3, hence total is 103.1 and not 100.1

  13. This suggests that only one thing has really made a difference in the whole campaign – the Clegg bounce following the first debate.

    It might have been different had Cameron spent the last 4 years talking about immigration. He’d have known that Clegg’s 80% statistic for EU immigration isn’t in point. Most net immigration involves non-EU citizens.

    But, then again, the only thing that doesn’t change on this site is the failure of all contributors to guess correctly the effect of stories on public opinion.

  14. Very impressed with Gordon Brown with Paxman. He is having nothing of Paxman’s interrupting nonsense and accusatory tone. He is actually running rings round Paxman on facts and figures and really showing him up. Probably won’t do him a whole lot of good but in this sort of forum he is authoritative and sure of himself.

  15. @Eoin Clarke

    Have you noticed that for NE and NW Con on Xiby’s charts is still at GE2005 % but Overall Con % is 34%. Think their percentage is being dwarfed by extra LD effect. Hence you should anticipate the Con % outside these two areas to be slightly higher than their overall 34% is actually indicating.

    The collapse of the Lab vote in the NE and NW is still allowing the ‘static’ Con vote % to take Lab/Con with swings of NE 6.5 and NW 5.0, giving them 15 Lab marginals as at the moment.

  16. new thread …
    new thread..

  17. I accept all Anthony’s about what people say on these sites about polling but i would re-interpret

    “Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support.”


    Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat GENERAL ELECTION vote level.”

    Whose to say that the local election result is not the Liberal democrat ‘support’.. that doesn’t come over to GE due to the ‘wasted vote’ argument.. or perhaps Support is ephemeral and different for different bodies

  18. Ricahrd O – You are confusing the Labour campaign with the Murdoch agenda, that’s the problem.

  19. There is an obvious missing point from the 9 above. Almost all polls ask a generic “which party will you vote for?” question. However, people are sometimes swayed by local issues and by assumptions about who could win in an FPP context.

    n a 3 horse race, “tactical voting” becomes hard to predict and the most telling poll for me is the lack of Con to LD swing in those marginals suggesting a “vote for change” may be prevalent. Therefore, despite all polls….I am predicting Conservatives to edge higher and also get more marginals than predicted with a shot at a majority…..

  20. Jimbo1972 – that isn’t missing. These are only stupid questions, your’s is a good question. :)

  21. Anthony Wells makes the interesting claim “The reality is that opinion polling is pretty much the only way of measuring public opinion. ” He dismisses various alternatives but does not even mention the best measure (at least the best for forecasting) – the bookies. His poll of polls shows the Conservatives 61 short of a majority. The Sporting Index Spread shows the Conservatives on the cusp of a remarkable overall majority. It will be interesting to see which comes closer to the actual outcome.
    Of course purists will argue that polls are just a current snapshot. This is technically correct but misses the point that people commission and read them to try to guage real outcomes and the bookies are good at anticipating those, because the ‘Wisdom of crowds’ effect includes anticipated factors wholly absent from sterile snapshot polling

  22. ” a “vote for change” may be prevalent. ”

    Don’t understand how ‘Vote for a Change” and “Vote for what the country rejected 13 years ago” acn be cofused as the same message

  23. In 1979 I was surveyed by national pollsters on 2 occasions during the campaign. What are the chances of this? I asked the pollster how many people they were interviewing in my constituency and she said 10.

  24. Anthony – polls don’t always get it right. 1992 was a classic example, and also the Al Gore/Bush US election.

    I suspect that they might be a tad inaccurate this time around and can’t wait for 10.00pm on Thursday night

  25. Much of the polling nowadays is done by the internet. How, if at all, do the pollsters adjust for possible variations of those with/without internet connections (after usual demographic weighting has been applied)?

    Apologies if this has been dealt with before, but over 600 comments are rather a lot to read through?

1 12 13 14