As we get closer to the election there will be more and more attention paid to polls, and sadly there will be more and more comments like those below – look, for example, at the horrors that crop up in the BBC “Have Your Say” section. Clearly none of you, my dear readers, would ever leave comments like these, but just in case you come across them elsewhere…

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.

(There should be an ICM poll later on tonight as well as the YouGov poll at around 10pm – I’ll probably be in a meeting for the ICM poll at least, but will update once I am free)


182 Responses to “Too frequently asked questions”

1 2 3 4
  1. I suspect that YG’s methodology is like the met-office long range weather forecasts. Highly complicated, very scientific, and like an old piece of seaweed occasionally right.

    I believe that the problem with their model is that they have build it for a different socio-economic environment to the one in which they are currently applying it, and to the one in which they have previously tested it ….a bit like applying your UK climate model to predict the weather in Arizona.

    I wonder whether YG have become complacent with their model due to its previous success – all good modellers know that the accuracy of their models is highly uncertain when they are applied in an environment that is significantly different to that in which it has been built/tested, and the current soci-economic environment is very different to 6, 12, 18 or 24 months ago.

    Time will tell.

  2. @ AL J

    I don’t think the row with the generals will have had an effect. If it’s anything defence related, I think the trip to Afghanistan was ill timed. It looked opportunistic & like a cynical electioneering photo op.

    The row with the generals & MOD v Treasury was just more fuel to the fire of those disgusted with politics altogether rather than a vote changer, I’d think

  3. Could it be a bounce for the tories? They certainlly need it after the generally poor poll results they’ve received over the past month or so.

  4. Eoin – Browns favourite film must be ‘The Searchers’ where John Wayne’s line is ‘never apologise – its a sign of weakness’

    A great line, not sure if its true though and Wayne’s character was a racist bigot – who redeemed himself at the end.

    …”BNP’er, seeing the narrowing polls, you would hold your nose and vote Tory.” — BNP take most of their vote from blue collar labour areas.

    “want the recession to return.” No one wants the recession to return. Some may think its likely and some even point out given the manufacturing figures its an almost mathematical certainty. Its because we want to see both an end to recession and room for real future growth that some of us see the logic in making a start on cutting the deficit now.

    Lets be clear – not only do we need to cut the deficit we need to start paying back debt to cut the interest payments down. The level of debt we face now – let alone in 3 years time – will be a millstone round the neck of a drowning economy.

  5. @Amber,

    “If it’s anything defence related, I think the trip to Afghanistan was ill timed. It looked opportunistic & like a cynical electioneering photo op”.

    If it walks like a duck, and quacks like a duck…??

  6. @ GEORGE G

    Poll you are looking for is on the front page of Guardian On-Line

  7. Mark J – the BC is separate for each nation and I will leave you to look at the web sites to discover their remits.

    Primary legislation would be needed to change arrangements in the direction you seek and I suspect that a small majority Government (even) is not going to saddle itself with those arguments – we’ll see.

    This issue and the alternaives to FPTP will likely sway lots of voters – your proposals (apart from 10% reduction) would certainly make up these people’s minds which is why I do not expect DC to make a big deal about it (I hope he does naturally).

  8. @ NEIL A

    See, now I have to resist posting something partisan in response to your little duckling. So much for honesty being the best policy ;-)

  9. @TrevorsDen

    “Lets be clear – not only do we need to cut the deficit we need to start paying back debt to cut the interest payments down. The level of debt we face now – let alone in 3 years time – will be a millstone round the neck of a drowning economy.”

    Looks like that is the opinion of the EU too now – more support for the Tory position.

    For those keeping scores on Tory v Labour Economic poicy (and the false war on the difference between them)…

    Economist appear split (not surprising as 2 economists will normally have 3 opinions)

    Business is generally urging cuts sooner (for the reasons Trevorsden has outlined), and now

    The EU is uring the govt to get a grip on the deficit.

    I seriousl think that when the government says it will half the deficit in 4 years, most people think that means the debt will be cut in 4 years when in fact is only means that the rate of debt growth will only half and the debt and interes payment will continue to soar.

  10. Thanks Amber

    I’ve just had a look at ICM %’s -it seems the Tory lead has been bobbing about a bit ;-)

    24 Jan 11
    04 Feb 9
    21 Feb 7
    04 Mar 9
    11 Mar 7
    15 Mar 9

  11. You’ve resisted the temptation very well, Amber. It was mean of me to joss you partisanly (is that a word?) when you were being totally objective.

  12. Thank you cyberkarst — assuming that within a probability factor of +- 5% you are even 90% correct then your comments ought to be giving all commentators on this board huge pauses for thought.

    …. irrespective of how they weight them or estimate the level to which they are in a quandary over the whole thing.

  13. @Amber Star

    “The row with the generals & MOD v Treasury was just more fuel to the fire of those disgusted with politics altogether rather than a vote changer, I’d think2

    I strongly disagree there. i think that the disenchanted electorate is used to politicians calling one another liars etc, but when the military calls Brown a liar (in their nice diplomatic words), than is another matter. It get’s people’s attention and makes them ask themselves, what else is Brown telling us that is not true?

  14. @ AL J

    Thanks – That’s exactly the pattern I was referring to.

    Any CON gain brings out the LAB vote next time & vice versa. If they did another poll in a couple of days, I’d expect it to be back to a 7 point gap ;-)

  15. The danger of the military argument is that “our boys” is one of the few things that has the power to galvanise the public across the board. Brown must be very careful not to get caught in a half-truth, but also must remain adamant that he did all he could. How much of a tightrope that is depends entirely on what the facts of the matter are (and I won’t speculate as it would be partisan to do so). So far I haven’t seen any evidence, but if the spat were to rear up again then any evidence that might exist is quite likely to surface, expenses-style.

  16. The opinion polls out tonight, in my opinion is within the margins of error of the previous polls. Nothing has actually changed.

  17. Sky news are running the two polls as big news (tory wobble over?)- see Sky web site).. Let’s see what they do with their ‘own’ YouGov’s. The ignored them last week!

  18. TrevorsDen – for pressure groups, they only commission polls on things they think will give them answers they like. It isn’t co-incidence that the Countryside Alliance don’t commission polls asking a straight Yes or No question on hunting, nor that polls commissioned by the League Against Cruel Sports don’t ask people about a middle way option.

    Fragmeister – the one who asked for you by name would have been a proper random sample, a rare breed indeed and probably an academic survey (perhaps the British Election Study)

    Cyberkarst – nope, sorry, no idea what the typical weighting effects are or the typical effective sample size. I’d add to your points (from which I would not defer at all) that the real margins of error are also likely to be effected by either the very high non-response and non-contact rate for phone polls, and the fact that online polls are effectively quota samples rather than random samples. That’s the reason why I try not to include sample sizes on tables and reports – because the margins of error calculated from it are in reality so divorced from the theory.

    Eoin – go to the front page of the site and look at the bottom of the right sidebar.

  19. @ TONY M

    General Dannatt kind of blew the gaff on the “political impatiality” of the military IMO – so we are simply going to have to disagree on that one.

    Everybody & their granny has called Brown a liar at some point so I doubt people would react in a big way to a few generals jumping on that train. Again, we’ll need to differ. I want to avoid becoming partisan about this.

  20. @AMBER STAR

    I take your point, but without getting partisan and sticking to relevance to polling, I think that if it was just Dannat then there indeed would be no impact on the polls. However, I think that these poll results may in some small way reflect that it is not just Dannatt who have been critical of Brown – it is most of the military – from the top brass mamaging the budgets to the squaddies having to buy their own kit while on leave in the UK.

    Hopefully the above comment is a reflection on media coverage and an interpretation of potential linkage to polls rather than anything particuarly partisan.

  21. Hi Anthony,

    One issue I wonder about is the position the ‘other’ parties find themselves in, which is often neither reported or commented on. Consider UKIP which I last recall seeing at around 4%.

    If the Tories have a 15% lead, UKIP supporters can afford the luxury of a protest vote, if the Tories are on 5%, getting another 5 years of Brown seems a terrible price to pay for a vanity gesture.

    Nobody seems to be polling anti-Labour sentiments amongst the 12% of voters who don’t show up on the main three-way split. Just a thought.

  22. Sorry, but I have to disagree with AW statement that 50% of the time the given figure is within 1 point of the real figure. The difference between the results of the various pollster make this impossible to be true.

    This is a far too simplistic approach.

  23. @ Tony M

    Your point about the deficit being cut in half is an outstanding one, and something I am amazed hasn’t been picked up by the press.

    To reiterate – Labour have promised only to cut the rate of growth of the national debt, and not the level of debt itself. Without wishing to sound partisan, this does little to inspire confidence.

    I suspect that the Tories will be ramming this home during the forthcoming campaign, and so they should.

  24. @ TONY M

    I think 3 things connected to the military have resonated with the public:
    1. SAS & SBS have suffered unprecedented casualties in Afghanistan. This was a top viewed story & it shocked people. SAS are an important symbol for many people.
    2. Ordinary soldiers & their parents saying they lacked equipment.
    3. Brown’s visit to Afghanistan being perceived as cynical & opportunistic.

    I don’t think (non-Tory) folk have much time for generals; so if the military has upped CON or downed LAB, I think it’s the more ‘populist’ stories that will have been responsible.

  25. @ ANDREW MYERS

    To reiterate – Labour have promised only to cut the rate of growth of the national debt, and not the level of debt itself…. I suspect that the Tories will be ramming this home during the forthcoming campaign, and so they should.
    ————————————————————

    The CONS have exactly the same policy regarding the deficit. They have no plans to cut it further. They will start earlier & cut – rather than try to grow our way out of it.

    But they have not committed to reducing it further than the legislated amount – or if they have, it’s news to me. A specific link to an article publicly stating their policy re target deficit reduction would be welcome.

  26. I get the feeling that YG will show little change….5% gap maybe?

    Not long to wait ;-)

  27. YouGov tracker tonight: Con 37% (nc), Lab 32% (-1), LD 21% (+4)

    I took it from twitter !!

  28. Amber
    There’s a new thread ;-)

  29. @ AMBER
    “They will start earlier & cut – rather than try to grow our way out of it.”

    If I may…..I think that slightly simplifies the difference.

    Cons most certainly intend to encourage growth ( NI reduction etc) . They have just received a report on restructuring manufacturing sector from Dyson.
    .They are just less inclined to rely on forecast & unknown growth as the major factor in deficit reduction.-which I suggest Labour do.

    All will ( hopefully) be revealed in the Budget-and it is my belief that one of the most critical figures will be the GDP growth forecast for 2011 & after.

    If Darling insists on playing down spending reductions & relying on the already announced ( modest)tax increase revenues , plus growth-then the assumed rate of growth must be credible.

    At present it stands at +3.5% pa.for 2011 & after This figure is considered over optimistic by some commentators.

    We shall see if Darling sticks to it.

  30. I have not posted before, but I do like all the heated discussions!

    I would like to see a poll with 0% for all the main parties and then we might end up with politicians who actually care about the country and not themselves (please do not misinterpret this as me wanting a peripheral party, e.g. UKIP or BNP as the government). But for real political change in this country we need a massive turnout and a massive number of spoilt ballot papers. Then the political elite will have to sit up and notice. (by the way I am not really bothered about the expenses claims, it is just the way they treat the electorate as general idiots).

    Neither Cameron or Brown fill me with excitement. Unfortunately Blair did, but I was young and naive then…

  31. Anthony,

    > 4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

    I take your point, but do you have any idea why German polling companies typically poll several thousands?

    Christian

  32. Christian – I think only Emnid do. All the other German pollsters seem to have the same sort of sample sizes as those in the UK, ranging between 1000 and 2000. Forsa seem to have 2500 sometimes.

    The only things to be gained from larger sample sizes are a slightly smaller margin of error (though it is very much a case of diminishing returns) and the ability to do more detailled cross breaks.

1 2 3 4