As we get closer to the election there will be more and more attention paid to polls, and sadly there will be more and more comments like those below – look, for example, at the horrors that crop up in the BBC “Have Your Say” section. Clearly none of you, my dear readers, would ever leave comments like these, but just in case you come across them elsewhere…

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!)

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, and in 2005 they were all within a couple of points of the actual result, with NOP getting it bang on.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.

(There should be an ICM poll later on tonight as well as the YouGov poll at around 10pm – I’ll probably be in a meeting for the ICM poll at least, but will update once I am free)


182 Responses to “Too frequently asked questions”

1 2 3 4
  1. @ ROLAND HAINES

    We are in complete agreement regarding concepts here Amber. The difference is the galvinisation, I think perhaps Labour were looking to close for comfort.
    ——————————————————
    Completely agree with you re: these polls. I was looking at the way it affects trends. LAB get close, waverers switch to CON. Which is where we are now.

    CONS get enough for a majority, LAB seems to attract some “anything but Tories” support. Where I think some climb-back by LAB may come from, going by past ups & downs in the polls.

  2. Cons are planning on unveiling their planned cuts after the budget.

    Their straegy is clearly to play their card close to their chest – no doubt a lesson that thay have learned from the election that never was, but also quite a ballsy one given the way their lead has been eroding and it must have been very tempting to show their hand on a few issues (e.g. drawing attention to the fact that Brown has called most military top brass ‘liars’ and asking the public do you believe brown or the military).

    The main critique of Cons seems (in my humble opinion) to have been that they do not have an alternative solution. I think that telling people now that they have a plan and will announce it will nip that in the bud without giving ammo to Lab before the election compaigning starts proper at the budget.

    The budget and the Con plans for cuts will be the next big game changer….followed by the news of the country falling back into recession.

  3. @ Eoin Clarke/Sue MARSH

    Certainly encouraging for Tories, but one poll does not mean much as we all know.

    Will be interesting to see YG tonight.

    What we can take from ICM is what I have believed and be saying for ages.

    Labour are seriously unpopular and the narrowing of the polls is more to do with incompetent Tory party as opposed to gain in Labour support. But it has to be acknowledged that core labour support has firmed there vote up.

    Also GB is seriously unpopular. This is reflected in him trailing DC in ever question. That should be the most concerning for Labour.

    GB does not help himself though. His interview on BBC R4 today in woman’s hour. When he says he will not resign even if party has less seats than Tories. If Labour do have less seats than Tories they will lose at least 75+ seats probably even more in reality. In anyone’s book surely that must be taken as a vote of no confidence. GB honestly believes he is the only person who can sort the mess out. His arrogance is quite breathtaking. This type of statement does not endear him to the swing voter.

  4. Roland – Thanks. I think my confusion has been in not realising I was the enemy. You are of course right though, that to some I am.
    I wonder if the combination of being a leftie AND being a girl is just too much for some to bear? ;)

  5. i see ICM are showing a collapse in support from others from 13% to 9%.
    This is also HALF of their share according to AR.

    This is far more a dramatic difference than any for the major parties.

    How can polls being so “scientifically” based be so far beyond the margin of error?

    Either both are well wrong, or one is spectacularly wrong.

    How can this be?

  6. Not quite sure why some Tory supporters appear to want the recession to return. I don’t think it’s at all likely, even though some of the figures suggest a rather bumpy & uncertain recovery.

  7. ROLAND HAINES

    I preferred “widows” – it conjured up much better images :-)

    Howard

    Thanks for that. Up here “Socialists” refers to the SSP or Solidarity – definitely not Labour!

  8. And – SUE MARSH – I definitely like you being a leftie & a girl. Doesn’t bother me in the slightest!

  9. “Polls give the answer the people paying for it want” —- come on, are you saying that when Greenpeace or some other pressure group commission a poll it dos not always come up with some sort of answer which suits?

    Meantime – so are LDs on 16 or 21 …. are labour on 26 or 34. ?? Thats 5 points on the LDs from a very small base. Its 6 points on Labour from a bit bigger base.

    If opinion polls are so good why the difference. In all Mr Wells’ comments he makes no comment.

    If YG are so good and are so right – then why is everyone not doing the same?

    But one point screams out from Mr wells’ comments …. “Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow”. …. is this credible – the polls invent what people are thinking. Who runs these polls – Derren Brown?

    PS – someone asked a question about Ashok Kumar. Well there was no sympathy vote for Gwyneth Dunwoody’s daughter.

  10. @ BARNABY

    Cheers, my dear :-)

    I expected this CON increase – but hoped otherwise. The wide publicity about LAB having closed enough to remain in govt as largest party has scared some of the ‘Others’. Let’s hope our complacent colleagues rise to this new challenge & get behind the govt.

  11. @Amber

    I think your right,

    a tight result is squeezing others out. This has always been why I am utterly confused at how tinky winky can credit the others at 17%.

  12. Kathryn
    Good post and you have articulated what I was pondering. People are forgetting that YouGov and the others have not altered their methods and also their methods are not hugely apart. Not only that but the trends have been somewhat divergent and one should have expected more homogeneous results on that too.

    I wonder if we will observe wildy different results this time with people doing their own thing more, as a result of internet influence, for instance, educating people as to where the marginals are. We are not getting a sense of that from the pollsters are we (sixth or otherwise).

  13. @AMBER

    Agree…17% of the electorate will not want to waste their vote. There is too much to loose.

  14. I always find ICMs question “Some people have said they would not vote in a new General Election, while others would go and vote at their polling station.
    I would like to know how certain it is that you would actually go and vote in a general election?”

    Isn’t this likely to be distorted by those significantly increased numbers who choose to exercise a postal vote – or are we expected to assume that respondents are used to dealing with stupid questions and can interpret what is actually being asked?

    After a lifetime in teaching, I’d be very wary of assuming the latter! :-)

  15. @ AL J, EOIN & SUE

    I am keeping in mind AW’s warnings about not saying polls are “wrong”. Others at 17% – Why? I cannot account for it; polls in Scotland & Wales do not see the nationalist parties soaring (sorry Old Nat & Peter), so where are all these ‘Others’ coming from?

    I struggle to believe that LAB & LD have lost so many voters to UKIP & other unmentionable parties.

  16. In my first sentence above, I meant that I find their question confusing. Memo to self “Read before pressing Submit” !

  17. @SUE MARSH
    Dont take it personally Sue, I am not a leftie or a girl, well a bit effeminate perhaps, and all this OOOH Dave trod in something nasty today, thats the Tories going down in the polls rubbish just gets on ones nerves. As I said ignore it. Practice what you preach? What do you mean Anthony?

  18. i think the truth is we can’t expect a headlong dive in the Tory numbers. There will be good weeks and bad for both sides.
    As you say, I’m sure the seeming Lab revival will have simply galvanised more Cons to decide to vote Tory, and this could well swing around – sadly to decide which party the public dislike the least it would seem.

  19. I suppose if you were a UKIPPER or BNP’er, seeing the narrowing polls, you would hold your nose and vote Tory. You might reason that in the wings the right wing will do a Geppetto on DC once elected. However, the detail from the poll should throw that up should it not? I have wondered about Greens. Strangely to my mind, the true Green is a little Englander (want to keep the £, against EU) and were of course courted by Zak and David when they used to be green. 4 to 5% are a lorra votes and not only in Liverpool.

  20. @ OLD NAT

    As an ex-teacher, I thought you were having us ‘spot the deliberate mistake’ – a favourite with my 4th year maths teacher when she got her sums ‘wrong’ ;-)

  21. Amber – on a lighthearted Labour note (to cheers us a little) that chart I directed you to last week predicted a small Tory bounce (but only for a day or two) before a collapse to a Labour lead. One chart versus all the polling evidence I know, but something to think on.

  22. Amber Star

    As far as Scotland & Wales are concerned, pollsters like ICM don’t even show Scottish sub samples, so there is no way of knowing what is happening here.

    It’s noticeable that, while the Sun asked YouGov to create “Scottish” polls from their daily data on a couple of occasions, they haven’t done that since 24 Feb (or at least haven’t published them).

  23. @ SUE MARSH

    It was a Sky chart though! Therefore their predictions of a CON ‘dead cat bounce’ are a welcome soother for the irritation that I am feeling at the moment :-)

  24. Dear all,

    I think the thing we should all bear in mind is that Yougov and ICM got darn close in 2005. the results were i beleive well within both’s margin of error. They ahve tightened their methodology even closer since. As peoples opinions hard the closer we get towards polling day i think we will see this reflected in the smaller gaps between ICM and Yougov. Populus and IPSO-Mori also included.

    I don’ t think there is anything wrong with the jury being out on the others for the time being.

    Can anyone tell me where I may find past polls on other recent elections to see how accurate they were?

  25. YouGov.
    Oh dear oh dear.
    Might be Peter Kellner’s last chance to impress.
    Or it might be head on the chopping block time.

  26. Oldnat (and others),

    Is it not time that someone in Scotland formed a Scottish polling company? Its hard to believe in these days of devolution, and presumably reduced polling costs (from the advent of the internet etc) that a Scottish poll outfit wouldn’t be viable.

  27. Amber Star

    OK Wildly off topic, but I loathed colleagues who did that! It was always demeaning to the kid who spotted your mistake. how on earth do you teach kids to learn from their mistakes if you pretend you never make them?

    Pedagogical rant over. Sorry Anthony!

  28. @ OLD NAT

    It’s noticeable that, while the Sun asked YouGov to create “Scottish” polls from their daily data on a couple of occasions, they haven’t done that since 24 Feb (or at least haven’t published them).
    ————————————————-
    They couldn’t face the barrage of e-mails you’d send querying their efforts ;-)

    I agree re: YG but I thought AR & Opinium show Scotland & Wales separately (I’m sure I checked both in the past when trying to account for their high level of Other).

  29. New Poll

    Tories 40
    labour 31
    libs 20

  30. I don’t know anyone who has been opinion polled. It’s happened to me twice. A woman actually came to my door and asked for me, which was quite flattering, then asked questions about a hypothetical GE and then questions about power tools and other products. A couple of years ago someone phoned me up and asked me questions about nother hypothetical GE. I don’t think I helped sway Brown’s decision not to go to the country but he wouldn’t have liked my opinion of him then and certainly not now. But perhaps it is time for us to unite and support the workers – those few that are left.

  31. @ OLD NAT

    Still off-topic but I sometimes wonder if GB went to a school where ‘Deliberate mistake – never admit you were wrong’ was the standard operating procedure.

    It’s one of the things I find mildly annoying about him but make allowances because nobody is perfect.

    But, like you, I prefer to simply admit I made a mistake.

  32. Amber Star

    AR do for Scotland (but there are too few polls to make anything out of the sub samples). I don’t think Opinium do, and neither now does what was our previously Scottish pollster – System 3 then swallowed up in TNS, and now in an international conglomerate.

    As far as YouGov’s Scottish samples are concerned, the average of the last 18 samples compared to 2005 shows

    SNP +4.1% : Lab +3.4% : Con +1.6% : LD -9.1%

  33. Well, well – thats even better than I thought might happen.
    The big question now is YouGov later tonight!

    38 32 19

    Thats what I predicted earlier today but now things have certainly got a little more interesting with these other 2 polls.

    Conservatives & Labour very close to my 40 30
    but the LDs must be very disappointed with these latest results.

    I know there is a long way to go yet but I am sticking to my original figures that I have always said for the GE.

    40 30 20 10

    Conservative Majority of 20 – 40 seats.

    AND NO need for another GE in October or at all until May 2015
    AND further more NO tinkering with the voting system that ONLY favours Labour ie AV
    AND a reduction in URBAN MPs by 10%

    AND no tinkering with the House of Lords any more until a full and proper debate and Referendum.

  34. @Mark

    Gee that’s a manifesto you ‘ve got there. I’ll resist the temptation to comment on the detail. :)

  35. Mark Johnson…we are in pretty much total agreement on %’s and on roughly seat numbers.

    Agree with pretty much every word….

  36. @EOIN CLARKE

    ;o

  37. @Eoin Clarke

    Sorry it should have been a :o lol

  38. Both ICM and Opinium showing small shift to the Tories tonight.

    It’ll be interesting to see if YouGov confirms the trend.

  39. Mark J – you want it all do you not? Is DC going to change the Boundary Comission’s remit then?

    Remember that YG’s survey was carried out today so post Tory leader and spouse exposure (I’m right aren’t I?).

  40. opinium poll just out

    Con 39
    Lab 28
    LD 17

    Two polls tonight 9 and 11 Con ahead

    Yougov will hopefully show the trend.

  41. Amber

    I’ve been trying to think why the Tory lead has increased. Should I dare suggest that the row with the Generals has damaged Labour? Brown said as much the other day. If this is the cause I would see it as a temporary reversal (thank you Colin), and think the trend towards Labour will continue as the campaign proper begins.

  42. @Dave in Southampton

    Thanks Dave – mustn’t be too complacent but tonight is turning out far better than I ever imagined when I was on this Morning– :o

  43. @Howard

    Why, what is the Boundary Commissions remit Howard?

    If the House of Commons votes to reduce the number of MPs by 10% then the Boundary Commission will be given a NEW remit – it’s as simple as that !

  44. I do realise I am the 54th person this evening to say the YG “will be very interesting” but like everyone else I wonder what has caused this Tory improvement. If Mr Kellners is right and it takes a while for things to filter through, then I would love to to think it is Armed Forces equipment. This is for three reasons, IMPO, its something Brown should answer for, the British public should care whether it is true that their soldiers were poorly equiped. Finally, as the father of an army officer, I would wish to see some justice for things which have happened which do not belong on this site.

  45. @ EOIN

    RE: Past GE polls – I’d bet Anthony has looked at this & will have a link somewhere if you ask him.

    You need to be fairly precise about what you are looking for. Are you asking how closely polling 8 weeks out reflected the actual at the GE, or how close their last poll before the GE was to the actual.

  46. Maybe too early to say though Mark…need to wait for YouGov really I guess….

    It has gone to and fro so much that it does need a few steady polls to get a true reflection of the past few days but I’m feeling better about it now as well and I “think” the blip may be over for the Tories now with Labour and the Budget having some tough weeks ahead I think.

  47. Al J, there’ve been quite a lot of bad news stories for Labour over the past couple of weeks, but bad news can take to time to register.

  48. @ EOIN

    Apologies for 2nd para. of last post. You are a data person – you’ll have thought of that already.

  49. ICM poll, 40/31/20 and Opinium 39/28/16.

    ICM seems the most realistic, 16 is a touch Low for the LibDems. After months of stagnation, the hills and dales now leveling back to 40/30/20. can’t be far off it??

    Thought DC was very up front and direct, giving an honest interview on the ITV/ Trevor McDonald interview last night.

    GB has certainly found himself sitting on a 3 ft board in the middle of a shark pool today re the Unite/BA dispute. No comment would have been by far the least damaging option.

    Been told of a ComRes poll due for release, 41/32/21 (????) alledgedley. Given my source, I’d choose to take that with a pinch of salt for the time being until we see some proof, cant find any trace of such a poll- we shall have to wait and see.

  50. Hello Anthony,

    One part of polling that is hardly ever discussed is the effective sample size. Take last weekend’s Guardian’s poll, for instance- looking at the tables, It was based on a sample of 1,007 respondents.

    However, the raw, unweighted figures are based only on a sample of 480.

    If this were a simple random sample, it would imply a 95% confidence interval of +/- 4.4% for the Conservatives and around +/- 3.6% for the LibDems (Labour somewhere in between).

    However, this is before the effect of the demographic and turnout weighting. We don’t know the effect of this weighting on the effective sample size, for some reason the BPC doesn’t require this information (even though it is very easy to calculate). Allow me to assume there is a weighting efficiency of 85% (given there is no disproportionate sample, this seems reasonable). This leaves an effective sample size of just over 400 – implying a CI of nearly 5 for any party with a percentage of 40%. I understand sampling theory and understand that 40% is still, in a sense, the most likely percentage, but given that conventional wisdom is that the ‘margin of error’ is +/- 3%, the actual figure is – based on my assumption – usually somewhat greater.

    I know ICM then ‘add’ some responses based on their modelling of don’t knows etc but my broader point remains.

    In the case of YouGov, I know the unweighted sample size is higher, but there is still weighting by newspaper readership and so on.

    TNS-BMRB, by doing face-to-face polling would, I imagine, also have a clustering effect to contend with.

    Do you know what the typical weighting efficiency on published polls is? Are you able to give an effective sample size for any YouGov polls?

1 2 3 4