As we get closer to the election there will be more and more attention paid to polls, and sadly there will be more and more comments like those below – look, for example, at the horrors that crop up in the BBC “Have Your Say” section. Clearly none of you, my dear readers, would ever leave comments like these, but just in case you come across them elsewhere…

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.

(There should be an ICM poll later on tonight as well as the YouGov poll at around 10pm – I’ll probably be in a meeting for the ICM poll at least, but will update once I am free)


182 Responses to “Too frequently asked questions”

1 2 3 4
  1. Good stuff Anthony. Very generous of you to assert that no such comments occur here – they’re quite frequent actually! Very good rebuttal of these all-too-frequently-stated assertions.

  2. Thank you so much, Anthony, for taking the time to set out the home truths for all to use in shooting down the doubters. And so wittily written as well. Good stuff indeed.

  3. People also regularly quote just one pollster’s figures, usually the one which is nearest to the circumstances they wish to prevail, and in the press, the one they have paid for.. Somehow, a device for including all polls, like the UKPR on this site is required.
    UNS is nearly always quoted as giving a state of play, hung or majority to one or other party, but the seats won/lost do not seem to fit UNS e.g. 2005 Labour 36%, Con 33%, but a majority out of synch with those percentages .
    I would like to see a range of both voting percentages and likely seat splits quoted across all polls rather than individual figures.

  4. Thanks for this Anthony, informative, objective and sensible.

  5. @SALTFORD MAN
    I think you are right, and especially about national swing. How many newspapers run the story”HUNG PARLIAMENT based on overall national swing”. Its almost meaningless. I have a very strong feeling backed up by some evidence that the the Labour recovery whilst very real is not anything like as useful as it looks. Lets take the well known Labour MP Fred Bloggs, member for Stalingrad in South Yorks. Last October, Fred might have expected to see his majority drop from 26,000 to 16,000. Now, it looks like he may get a majority of 28,000. All the Labour faithful are prepared to stop that little squirt from Eton. What good will this do Labour in Milton Keynes, Watford or the West Midlands where they need it. I expect very strong Labour support in the heartlands of the north. But it will not create a hung parliament and will be a long way from a majority.

  6. Great post Anthony, thanks for clearing these issues up (and nice story about the 36 election).

    And nice poll you e-mailed me today, I liked the new qualitative style.

  7. Very informative Anthony.
    I for one greatly appreciate the knowledge and immense experience you impart on this excellent forum.

  8. Roland,

    Equally likely that any big numbers that the cons get from AR or wherever are a pile up of blue votes in blue areas, surely?

    Loved “Fred Bloggs MP of Stalingrad, South Yorks” by the way.

  9. Oh thank goodness. at least that should stop a FEW of the aforementioned comments.

  10. Anthony

    Please, please enforce the deletion of comments such as those you have taken the trouble to note.

    It would be very very nice to have a political forum without such inane comments!

    Unique on the UK political front, I should think.

  11. Oh please do! I promise to be less waffly and generally partisan if there was less to respond to in the first place, just cull, cull, cull for a week or two til everyone gets the message

  12. @MITZ
    Of course in true blue areas the same thing will apply.
    Sir Bufton Tufton in Surrey Moneygrove will probably wack the LD a bit, but in the places that matter to both the big parties the Lab toTory swing will win the the GE.
    The fantastic Labour results in the heartlands won’t mean a thing.

  13. Coming in from the not so cold to this. I thought Roland was bordering on example 8 but with the point made by Mitz we have an interesting discussion on ‘regional variances’ to discuss. We do not really have a good insight of this possibility, but we are thinking that the electorate may be more irregularly distributed than the pollsters are able to keep up with. We can of course only prove this with some regional polls……….

  14. Much will depend on the turn-out. With so many disillusioned with politicians generally, it could be historically low. This would surely benefit DC, as Conservatives traditionally are more likely to vote. This could confound the pollsters as most of the electorate are unwilling to admit to staying at home. If the public’s view of Brown is similar to their view of Kinnock in 92, (helped by The Sun) then DC will be win with a small but workable majority. As AW implies, the polls are usually credibly reliable.

  15. @ Roland – but what about if there are fantastic results in the marginals too. Is there any evidence that in previous elections a narrow result on UNS has combined with a landslide-esque result in the marginals?

  16. Roland

    Humour not at all bad.

    Still far far far far far far far too obviously partisan though.

    Overall C minus – must try harder.

  17. Oh, I don’t know Sue. I mean, as the great Blackadder once said, “Civilised man seeks out good and intelligent company, so by learned discourse he may rise above the savage and closer to God. Personally, however, I like to start the day with a total d***head, to remind me I’m best”

  18. David in Frankrijk
    Unique on the UK political front, I should think

    You’ve carried out a worlwide poll to prove this of course David :-,

  19. Well Roland speaking as a Labour Party constituency chairman, and a veteran election watcher, I don’t expect marvellous results in our safest seats, certainly not increased majorities as you suggest. There may be a variation between UNS and the actual swing seen in marginals but I don’t think it will be as great as you imply.

  20. Barnaby etc

    but I don’t think it will be as great as you imply

    Here we go again – see polls Barnaby :-)

  21. @DAVID IN FRANCE
    My comment is not pro Tory I just think the situation is as I have described. Should I add Gordon Brown is a lovely man just please you?
    @ SUE
    A fantastic result in the marginals would be Tories fail to capture marginal number 10, but win marginal number 155. And it could happen.
    @ BARNABY
    Hey, I am wishing you all the best in Sheffield, Manchester, Merseyside and Tyneside.

  22. A set of very clear answers to some silly question. I do hope one or two of our regular posters ready them carefully!

    The point about these polls of course is they ask how people would vote in a general election tomorrow when the actual election is about 8 weeks away so what is of interest now are trends.

    This general election will be interesting, not least because of the televised debates and the increase in postal votes. The latter might (or might not) counter the ‘Tory turn-out’ question.

    On the subject of postal votes we recently had a postal consultative vote here. It was the third vote on the same subject. The first vote was a traditional poll, held alongside the local elections. Turn out was 22% (well below the vote in the council election). The second was again a traditional poll, held on its own in February. Turn out was 20%. The most recent poll was again in february but was postal and turn out rose to 27.5% but there was also a rise in spoilt votes, largely from people doing things like putting 2 votes in one envelope. Campaigning on both sides was vigourous in all 3. Might we see an increase in turn out by postal voters of a similar 30%? I will be fascinated to see.

  23. @ Anthony.
    Great stuff, as ever.

    @ Roland
    Its always slightly humbling to encounter a true believer

  24. Actually, a question for all you canvessers out there.

    Is your role simply to record someone’s voting intention, or do you try to change their minds?

  25. @Thans Anthony. I’m glad you clarified the questionmark over accuracy of the polls, especially in 2005.

    @Barnaby
    I think you’re right. The marginals may have a higher swing to the Cons which would give them extra seats compared to a UNS, but if the polls close further this advantage soon diminishes. Tonight’s ICM will be particularly interesting. If it shows a narrowing of the Con lead it’ll be in step with YouGov & show that the trend is continuing. ;-)

  26. ICM

    Con 40: Lab 31: Lib Dem 20. Reported by Julian Glover.

  27. Valerie, purpose of canvassing

    Answer
    Establish intention – ‘changing minds’ would, even if you ‘succeeded’ be subject to the same ‘let’s tell them anything to get rid of them’ that the genuine pollsters are so terrified of. Incidentally that’s why you should not have people with rosettes going round canvassing. If you do, then reduce results back at the office for ‘own party’ by half. Ignore probables completely or record as ‘soft’ whatever the voter tends to lean towards. Probables for the opposition are ‘hard certain’. You get my drift here.

    The only purpose of party canvassing is to drag own supporters out to polls. Liars can get free rides in this way of course – some old men like to do that.

  28. @ Howard and Valerie – Of course you can ask a question like “So would you be content to see a Tory/Labour government” to start a conversation, as this often leads to more info. Generally no persuading, but if someone really wants to engage ie on a certain issue, then no problem with answering questions

  29. Damn, really is 40/31/20 it’s on the Guardian website

  30. Guardian ICM poll on 15.03. is:-

    Con 40 + 3
    Lab31 +1
    Lib 20 nc.

    Getting nearer to what AR believe.

  31. Looks as if the polls are now swinging back to the Tories.

  32. Good stuff. One thought that DOES come to mind.

    I have always wondered about the number of people who, knowing it’s a poll, and not their actual vote, tell te pollster the opposite of what they will ultimately do. e.g. were I asked, I would probably answer UKIP, mainly because I’m hoping Call-Me-Dave will LISTEN to me, as representative of many disaffected Tories, and address issues he’s ducking. i.e. send him a wakeup call. Change or lose.

    But I’d probably be lying. But tactically!

    Because on Polling day I will probably vote Tory finally… gritting my teeth because the thought of Labour getting in is too much to bear.

    In this particular election I believe the Libdems and other parties are probably benefiting from this phenomenon.

    Can you measure/control for this in your results?

  33. @ SUE

    “Damn,”

    Calm down dear ;-)

    @ANTHONY

    THanks-very interesting !!

  34. @SUE MARSH
    Its that photo of Samcam on the front page of the Torygraph on Saturday. She looked like a film star. At least thats what my mate reckoned, I dont look at girls.

  35. Right – I may be as thick as a whale omlette, but these ICM results polled between 12th and 14th March don’t seem to me to tie in very well with the Yougov results that are posted on this site. Am I missing something? Or would it be fair to conclude that at least one of these is inaccurate?

    Following the article above, which of these is by Literary Digest and which is by George Gallup?

  36. This is very consistent with its last two polls you will have noticed (last was 14.3.10)

  37. Ok thanks…I have been culled recently and I understand why. I wanted to make my point, and I have. I’ll just watch from now on.

    I think for what it’s worth that the Guardian poll suggests what I thought all along though and that Labour will do no better than they are now, it’s just a case of maybe the Tories getting their act together before it’s too late.

  38. Sue

    It’s a long way to the election yet. There may be reverses -lets see what tonight’s YouGov brings ;-)

  39. Soooooooooooo-Labour girls & boys-

    Is this a blip?

    Is it an “8”

    Is it an AR lookalike?

    Is Brown Bounce n stalling?

    Are-I hardly dare suggest this…Cons clawing back?

    What do you all think?

    Can’t wait to see Anthony’s analysis & the subsidiary question answers.

  40. Jock

    ‘ I may be as thick as a whale omlette’

    Jock – are you actually Australian?

  41. Remind me to keep calm Colin when Labour reduces the Tory lead –Again. ;-)

  42. @AW
    Very instructive to have this factual informtion in one place Anthony. To be fair to you, these facts have been laid before us previously for one explanation or another. This list can be pinned on the wall though.

  43. Definitely seems to contradict the recent trend. I do always wonder if the sheer number of YouGov polls skews our view of them a little. But the actual headline figures aren’t so out of line with the whole “Tories high 30s, Labour low 30s, LibDems high teens” we’ve been seeing for a while.

  44. Colin – amazingly, I’m probably your best shot at a reasonable Lab analysis.

    There is no good spin to a nine, but then it is one poll (Sorry Colin, I DO discount AR, but not because I’m blindly Labour, simply because I think their methodology is flawed)

    It is increasingly hard to reconcile YouGov with “others”

    I however, don’t really believe news stories or interviews influence the polls much, I think trend is by far the best indicator of likelihood. A nine backed up by other 8s or 9s would be bad news and I’ll just have to wait and see. The peripheral questions were very good for the Tories too, so nothing to please me.

    I think we are back to waiting to see whether this is the start of a new trend towards Cons or a random nice result for them within MOE

  45. ALJ

    “It’s a long way to the election yet. There may be reverses ”

    This is what I like about you ALJ.

    None of this analytical stuff-looking for the reasons-pondering the factors-how is the campaign going-what are the change making factors?

    Nope-if its a Labour gain it’s good
    If its a Con gain-it might disappear.

    Honest & straightforward ;-)

  46. SUE

    “A nine backed up by other 8s or 9s would be bad news ”

    Actually I think it would be excellent news.

    I know-you meant to add “from my point of view”-it just slipped your mind ;-)

  47. @ Colin – there’s just no acknowledgement of a reasonable analysis from you is there?

  48. If the lefties’ hopes are upheld by YouGov, have we not already identified a cause in the ‘disloyal’ Labs? In my flippant piece on canvassing results, there was an atom of truth in the proven continual overestimating of the casual voter by party activists (except of course of Tories who were more committed than they let on – AW thinks that’s changed).

    I would like to hear from the experts on this as there is always a reason for everything. If the casual voters were fired up by something on the eve of poll, YouGov would end up smiling and we would be none the wiser.

  49. Could this poll result have anything to do with Unite, BA, and Labour donations? Ashcroft/Paul did not have an impact, but then they did not ground an airline either.

  50. @Howard – yes, the polling shows the Tories getting a swing of around 2% extra in the marginals. That’s significant, certainly, but it’s not decisive.

    Especially given that general disgust with politicians after the expenses scandal makes predicting turnout difficult – will the new non-voters draw primarily from one party or social group, or will they be demographically representative? In my LD/Labour surprisingly large amounts of the new non-voters are working class Tories, but that’s not necessarily diagnostic. Three possible reasons include: a) the areas I’ve canvassed lately being more working class and less Lib Dem than the city as a whole; b) the rarity of working class Tory voters making them more memorable and c) the possibility that they were originally Labour supporters but switched to the Tories because of frustration with Blair and Brown and that expenses just made them reject everybody.

    Switch to a different constituency, with different possible victors, and you’ll likely get an entirely different picture. With turnout so unpredictable, we’re less likely to get a regular swing than ever.

1 2 3 4