I posted these a couple of weeks before the election, but I see more and more of them cropping up in the comments, so I think it’s worth reposting it for newcomers.

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.


675 Responses to “REPOST: Too frequently asked questions”

1 2 3 14
  1. Good summing up AW.I’m always surprised to see people who have little faith in opinion polls posting on a site about opinion polls.

  2. hi anthony, thanks for clarifying but please be assured that the people who comment are a very unrepresentative sample. Most of us who read your blog regularly understand polling very well and appreciate your efforts

  3. Thanks Anthony – useful.

    What about the current methodologies used when weighting past votes? If the LD surge is down to a lot of people who have not voted before (young) or not voted recently – are they unrepresented?

    If so, could the polls be very wrong this time simply due to the weirdness of this election?

    Cheers,

    …Lev

  4. I read this the first time and I could not agree more. I cringe whenever I read the ‘well they have never asked me or anybody I know’ and ‘what do you expect when xxxxxxxxxxxx commissioned it’? I am going to link to this whenever I see such comments for a while.

  5. 5) Polls give the answer the people paying for it want!

    The debunking of this is also enforced by the fact that The Sun has refused to publish two polls now, one in favour of Clegg, and one suggesting that the Biggot gaffee was not that significant.

  6. Good mythbusting collection of frequent misunderstandings about polling.

    However, your answer to number 5 is idealistic to the extreme.

    The reality is that organisations always develop a ‘style’ and a flavour. I agree that the purpose of a polling organisation is not to show any bias, however I think you need to dig a little deeper into this point before dismissing it straight away like this.

    I find the belief that an organisation can be theoretically and completely non-partisan to be naive.

    Polls have flavours. I don’t claim they are ‘political’ flavours, or that they could possibly be accused of somehow supporting a political party, but they have flavours.

  7. My concern has been the use of very unrepresentative samples from certain pollsters, particularly Mori and ComRes.

    Mori doesn’t do any past vote weighting, so their polls are somewhat bizarre and volatile.

    ComRes, however, does do an adjustment, but there is no way of ensuring independence between the low density of a party’s support in the sample and the likelihood of tactical voting affecting that sample.

    For example, a poll that was overweight with voters from Con-Lab marginals, would show a much lower LD 2005 vote share than the national average. But any national swings in LD voting trends could quite easily be missed, stifled or misrepresented by the effects of tactical voting. It would be no good to simply weight up the LD vote, because LD voting trends in Con-Lab marginals would not be representative of national voting trends.

    Throughout their daily polling ComRes have been very underweight 2005 LD voters and very overweight 2005 Lab voters. 2005 Con support has been roughly inline with the 2005 election result. ComRes have adjusted for this, but have they adjusted for the complexion of constituency battles that would have produced such unrepresentative samples?

    Unless pollsters start out with representative samples, at least insofar as historic voting support is concerned, they are going to get increasingly more unstable and inaccurate projections the closer we get to polling day.

    I have a strong feeling that quite a few pollsters are going to get egg on their faces this year. And they will only have themselves to blame.

  8. Having watched the debate myself, and formed a fairly non-partisan opinion as to the debate ‘winner’ before finding out what the polls suggested, I must say the results surprised me.

    So, in the vein of opinion polls being correct, will the post debate polls have a mechanism built in to actually verify that people are saying who they thought ‘won the debate’, rather than just re-iterating their voting intention under the guise of a different question?

  9. I cant see any 2005 voting data for the yougov polls, do they publish it?

    i looked at the pdfs here h t t p://today.yougov.co.uk/archives

  10. Thanks for this posting.

    My only quibble is that I’m pretty sure pollsters claimed they’d ironed out causes of excessive inaccuracy before the 1992 general election :)

  11. The question I have asked about the reaction polls is how are the pollsters sure that the interviewees watched the programme.

    Are the MOEs higher in these instant reaction polls even if the methodolgy of ensuring a representative sample is sound.

    Have any of the polling organisations asked whether the debates have changed voters minds over how to vote or set floating voters on a definate path. Someone might say Clegg won debate one but they’re not going to vote Lib Dem.

    No doubt pollsters are scrupulous but it does seem that Guardian polls favour The centre-left and Telegraph polls favour centre-right (or am I misrembering?). Finally how many polls are sat on – I seem to have heard reports that the Sun sat on a commissioned survey that found that Bigotgate pretty much split the country 50-50 and is likely to have minimal effect on the outcome – certainly a lot less then the frenzy of the 36 hours after the event might imply.

  12. the “Political Party Identification” is the closest we get to 2005 voting in the yougov pdf.

    There, as is often the case with comres data, there seems to be less libdem supporters in the survey than the 2005 elections suggests there should be. Usually around 10%+ instead of the 14%.

    Although only 176 identified themselves as libdem in the last yougov poll, 369 said they would be voting libdem this time. Thats a massive increase, similar to when comres has figures that suggest the libdem vote has doubled.

  13. “My only quibble is that I’m pretty sure pollsters claimed they’d ironed out causes of excessive inaccuracy before the 1992 general election”

    Indeed, and blamed it on the ‘shy Tory’ voters. Because of this didn’t they go on to underestimate the scale of Labour’s 1997 victory because they factored in the shy Tories from 1992? It’ll be interesting to see if 2010 is a 2001/5 or a 1992/7 for the pollsters.

  14. One of my friend works in the field of statistics and he knows his numbers quite well :) I just haven’t the same knowledge, but I’ll post roughly what he said.

    The current pollsters work under the formula which only function properly under a 2 party system with 3% error, but once a 3 way party system come into the fold, the formula used by the pollsters are much less accurate. He said that the margins for error is more like 6%.

    I am fairly convinced that there are some truth in what he said because he is not the kind of person to talk a lot unless it’s something useful…

    well I can’t put up any counter-arguements hereafter as I only have 2nd hand info. Just thought it was interesting and I’d like to contribute :)

  15. AR polled 2443 and found 300 2005 libdem voters, which is about 12.3%, short of the 14% figure, but the closest ive seen.

    Voting intention for libdems in 2010 was 673 unweighted, and 673 weighted. No change despite the libdem voters seemingly unrepresented in the first place.

    the same poll found 554 2005 con voters, which is 22.8%, slightly higher than the 21% figure.

    voting intention for cons jumps to 735 unweighted, and 719 weighted. Weighted down slightly, but seemingly not enough.

    I know there are lots of other weighting considerations, but it looks to me that the libdem figures should be pushed up and the cons down.

    ***

    All the data ive seen shows that the surveys underrepresent the libdem voters from 2005, slightly over represent the cons, and massively over represnt the labour voters.

    Why? Is it statistically likely because the smaller the group the more likely you are to miss them?

    And why do the pollsters not seem to take that into account?

  16. Anthony, too vague a thread after such a busy night. Get down to the nitty gritty…

    Here’s my thoughts:

    A level playing field after tonight’s show.

    Brown fought back to make a mark (after yesterday’s awful gaffe).

    Cameron held his line with more confidence than shown in the previous two encounters.

    Clegg was a disappointment; waved his arms too much, got an early perspiration sheen on his top lip/forehead and struggled to answer basic questions (hence the sheen?).

    Top of his head waggled and bobbled a lot from side to side (like Vince Cable’s does) as he was making a point. Is this a necessity of the Lib Dem syllabus on public speaking?
    All he needed was a red Fez hat to complete the Tommy Cooper impersonation.

    Thank the Lord for no more ‘debates’.

  17. At end of day it doesn’t matter what polls the conservative gutter press achieve, we all know you can get whatever results you want by picking correct sample and methodology. When you are out people dont want Cameron and noone expects or wants him to win. Its Brown wanted in a crisis says ICM

  18. Having watched the 3rd debate I felt Clegg to me seemed the weakest. Brown hammered home his message well.
    Cameron looked as if he was feeling the pressure of the last few weeks.

  19. In a close race enouch voters don’t make there mind up which way to vote until the last days, it is important for the polls to get the result correct to poll right up to the last day.

  20. the last ICM poll (Fieldwork : April 23rd-25th) had weighted figures showing more libdem voters than con voters, although when ICM released the %age figures to the media it showed tories ahead of libdems by 3% points.

  21. Chris Newey,

    Can you please refrain from hijacking my name.

  22. My pet hate is definitely the “I feel” type of comment.

    You know the ones:

    “Despite the fact that NASA have visited it, taken samples, conducted experiments and so on… I have a strong feeling that the moon is made of cheese”.

    Unless or until you have regularly interviewed a subsample of the UK population and balanced your interviews with weighting and other statistical tools, the blunt fact of the matter is, most of us on here are not really interested in your “feeling”.

    So, please, think at least twice before posting it.

  23. The post debate reaction polls seem suspect:

    1) Were they really watched by partcipants?

    2) The profile of those participating might be skewed to the Tories given the alternative viewing (Coronation Street, Liverpool v Athetico Madrid, and so on ). Highly subjective point!

  24. I still think GB has Cheryl Cole lined up in a ‘Break Glass in Emergency’ type of way. This election seems likely to be won or lost on the red button of the TV remote.

  25. I think there’s been a lot of criticism of the polls because the newspapers can say they have the proof that Cameron won last night, which will legitimise him as prime minister to the millions of people that didn’t watch the debate last night.

    In truth, Cameron was pretty weak last night, but the polls show the country has stopped listening to Gordon Brown.

    If the debate had taken place when he became prime minister in 2007 then he would have probably won – there is no justification for Cameron to cut taxes for the richest people in the country at this time.

    So, for me, the polls show one clear thing: Not calling the election in 2007 is the biggest mistake of Gordon Brown’s life.

  26. PS The polls actually point to David Cameron being personally quite unpopular though, although nothing like as much as Gordon Brown.

    If Labour and the Lib Dems can somehow pull off a hung parliament with the Tories not being the majority party, perhaps by Brown agreeing that he’ll go if that’s the case, by tapping into the unpopularity of the Tories, then they could still stop Cameron being PM.

    We’ll probably have some idea of what a mammoth task that will be when tonight’s polls are published.

  27. @ Colin – “So, for me, the polls show one clear thing: Not calling the election in 2007 is the biggest mistake of Gordon Brown’s life.”

    Maybe, though it may just have delayed the inevitable for another 2-3 years.

  28. @James – I agree, although I do feel he would have increased Labour’s majority close to 100.

    Still I don’t feel Cameron came across as well as indicated in these polls and listening in numerous conversations in the way into work, I wasn’t alone in that thought.

  29. I found it quite amusing last night when dear aol Alistair Campbell said polls gave you the answer that you paid for and chuckled to myself.

    Apart fromt he fact this is a definite case of pot calling kettle as he had started spinning the result 17 minutes before the end to get the result Labour had paid for i also thought he cannot be a reader of this blog

  30. @Christopher and all the others who happen to find their own opinions [yes, OPINIONS!, not “facts”] confirmed by what they “listen to on the way to work” et cetera.

    Do us all a favor and read up on “self-fulfilling prophecies”.

    Like, for starters, read: http://en.wikipedia.org/wiki/Self-fulfilling_prophecy

    Of course most of you will still believe that what you perceive is The Truth, but maybe at least one or two could start thinking about it…

    It’s approximately the same mechanism that leads to our ignorance towards the mistakes we all make ourselves.

    cf. Tavris/Aronson “Mistakes Were Made (but Not by Me): Why We Justify Foolish Beliefs, Bad Decisions and Hurtful Acts” or pretty much every book on something called “cognitive dissonance”…

  31. @James

    Absolutely. But I get the feeling Brown will regret never having been elected as a PM, and not getting the opportunity to be PM for 5 years, until the day he dies.

    @Christopher

    I watched it with the mrs last night, who shows zero interest in politics (bar the odd right wing comment), and did fall asleep at about 9.20pm (and didnt watch the first two debates). This morning she said up until she fell asleep she felt that Cameron had been ‘slimey’ and ‘unappealing’, and had easily come across as the worst of the three.

    It makes me think that Clegg blew last night – the country has stopped listening to Brown, but Clegg didnt benefit as he was more woolly than Cameron.

  32. The LD might support Cameron if he gave them the option of a referendum on changing the voting system.

    I have been told that AV plus is the least harmful option for the Tories.

    I also believe that reducucing the number of MP’s by 60 or so and re-drawing the boundaries to even out the number of voters in each constituency reduces the built in Labour advantage in the FPTP system.

    Has anyone done the maths giving an idea of the type of result we might expect from doing both i.e AV plus and re-drawing the boundaries.

    If the result of this was not too bad for the Tories they could offer this option to the LD in return for thier support.

  33. This isn’t relevant to this particular post, sorry, but it seemed as good a place as anywhere.

    I was just looking at the ICM results (23-25 Apr), and I’m surprised more hasn’t been made of how much firmer the Tory vote appears (83% certain not to vote for another party) compared to 69% for LDs and 68% for Labour.

    Is this just not as significant as I think it is, or is it not a particularly reliable result, or have people talked about it and I’ve missed it?

  34. Neil_
    At end of day it doesn’t matter what polls the conservative gutter press achieve, we all know you can get whatever results you want by picking correct sample and methodology. When you are out people dont want Cameron and noone expects or wants him to win. Its Brown wanted in a crisis says ICM

    *****************************************************

    Have you not read any of the original post! If all posters on here posted the sort of rubbish you do, nobody would read the site.
    All we have to go on are polls from independant organisations. They currently say a majority wants Cameron, with Brown generally in last place given the debt he has left us with. end of.

  35. Morning everyone :o

    Well it’s now LESS than 1 week to go before the real poll and the only one that really matters!

    This will have been a very strange but interesting and exciting campaign which I suppose started when the LDs broke the mould in the their opinion poll ratings 2 weeks ago

    At the moment we only have the Polls to point us in any direction at all and the findings are as follows

    Conservatives have almost always been ahead in the polls for months and months and even now still have a good lead in most polls with less than a week to ge.

    Labour are coming third in most Polls now and in some cases a VERY poor third!

    LDs bob about a bit but way above what they have been used to in the past!

    The weekend polls will be the all important bench mark for the final push.

    If the Tories can edge up to 36-38% they should be able to just about form a majority government with the help of the English Marginals.

    Labour may very well plunge to a humiliating low of 24-26% and finish third.

    LDs should end up on the highest % figure they have ever enjoyed and get many more seats.

    ALL guess work and predictions of course but all great fun on this great Polling Site of run by AW.

    Anything could happen of course because NO ONE could have possibly predicted the LDs rise in the Polls OR GBs Gaff in Rochdale.

    Finally – Labour have wheeled out their Trump card today apparently in Tony Blair – pure classic – do they never listen or learn?

    :o

  36. hear hear, I used to work for IPSOS Mori, as a tele-interviewer and actually I was very impressed with their methods to ensure that polls were done as accurately as possible to reflect the make-up of the public properly

  37. Sean R
    No doubt pollsters are scrupulous but it does seem that Guardian polls favour The centre-left and Telegraph polls favour centre-right (or am I misrembering?).

    Astonishing comment in view of the thread.

    See fallacy 5 Sean.

  38. Does anyone have a summary of what the different polling companies that did poll before the 2005 election had in their final results? That would be interesting to have as a point of reference.

  39. testing

  40. It is with enormous concern that I see the amount of coverage given to the “who won the debate” polls rather then voting preference for the general election.

    The former may have some influence on how people vote but it is not itself indicative of the likely election result, some of the polls last night were effectively suggesting they were.

    Firstly these debates are entirely new to British politics, there is no established weighting, no determined margin of error and different pollsters have come up with variances that if they were for voting intention would be laughed at for obvious inaccuracy.

    Secondly this is not a presidential election.
    Like over 99.5% of the population I do not live in the constituency of any one of the three party leaders, I have no chance of voting for or against any of them.

    Thirdly even if I could vote for one of them there is no constitutional certainty they would continue in office or be replaced by someone elese I had elected, for example if this system had been in place at the last election none of these three party leaders were in place, all took over in varying degrees “mid term” after the 2005 election.

    For the record and I accept this may only be indicative only but it is from work rather than those I acquaint with personally, no one is admitting to sitting through the entire debate – either they didn’t watch at all, or they watched in part, in particular they watched at some point but then turned over or off, or they really did and are shy to admit it.
    Because of the still admittedly partly anecdotal part of this, which we are warned of in the article, I would like to see the actual viewing figures, not just of any of the debate, but also of how many actually watched the full debate or stuck it out to the end, does someone have access to these?

  41. Can people please make sure they differentiate between the personality polls that happen after the debates and the real polls.

    To say that Labour are languishing as a very poor third in the polls is wrong. They are neck and neck with the Lib Dems in most polls. The Tories have not yet stretched away.

    I though Anthony had warned that making predictions because you ‘feel’ that’s how it’s going to be is daft.

    The ratings post-debate have only really affected the Lib Dems original surge.

  42. Interesting poll of Radio 4’s undecided quartet of voters this morning.

    Following the debate three have made up their minds:

    2 will vote for Clegg, 1 for Cameron.

    The one still undecided said it is between Clegg and Brown.

    Three of the four ‘completely disagreed’ with what the polls are saying this morning.

  43. @skudor

    ipsos-mori.com/newsevents/ca/ca.aspx?oItemId=238

    Put www. before it and google for 2005 table at bottom of Bob Worcester’s blog

  44. Colin Wobbles

    Radio 4 undecideds poll

    That ‘poll’ breaks nearly all the rules above!

  45. Analysis of the ICM poll is here.

    Although Clegg came third in the ICM poll, and a poor third, he comfortably won the, perhaps, more important question, ‘if you are still undecided, who are you now most likely to vote for’.

    Cameron only got 15%.

    However, this sample is only of 23% of those polled, and it was a small poll of just 510 people

    http://www.guardian.co.uk/politics/2010/apr/30/tv-debate-cameron-brown-gaffe-poll

  46. @Howard

    4 people is a good sample!

  47. @AndyW

    Lighten up a bit Andy.

    Having a light hearted stab at predicting the next opinion poll is part of the fun of this site and people have enjoyed taking part without being too partisan or too serious!

    If you turn this site into a serious talking shop of only intellectuals and University bods most people wouldn’t even bother contributing.

    People such as EOIN Clark are indeed intellectual but also fair minded and full of interesting statistics whom I greatly respect.

    So to sum up – Lighten up Andy as we need a balance of views without being too partisan.

    Don’t be offended Andy but I had to respond.

    And above all KEEP SMILING :o

  48. William Hill has odds of just 13/8 of Brown still being PM in a month’s time

  49. Jason:

    Those figures have been brought up quite a few times.

    As always with stats you can read them anyway you want. One possibility, which fits in to my own theory, is that most conservatives have already decided and are firmly committed, but there are therefore less of them in the undecided pool.

    The “softness” of the Lab/Lib dem 5 could well be because they may vote the other way around, if it keeps the Tories out in their own constituency: it is also likely that more undecided voters will go for one of these two parties than for the Cons.

    I think an overall majority is beyond the Tories. It’s possible they may get just enough seats to decide to try and govern as a VERY minority govt.

    This will be a nightmare for them: a weak government with less than 25% of the electorate voting them in to power, unwilling to discuss changes to the FPTP system, yet bringing in many unpopular cuts and job losses and facing a buoyant Lib Dem party and a Labour party under a new leader.

1 2 3 14