This is an updated version of a post from way back before the 2010 election, which I felt needed another airing. Thankfully comments along these lines don’t turn up very often in the comments here, but I see them with depressing frequency on Twitter when poll results are released…

1) The polls are ALL wrong, the real position is obviously X

Er… based on what? The reality is that opinion polling is pretty much the only way of measuring public opinion. We have some straws in the wind from mid-term elections, but they tend to be low turnout protest votes, don’t tend to predict general election results and are anyway quite a long time ago now. Equally a few people point to local government by-elections, but when compared to general election results these normally grossly overestimate Liberal Democrat support. If you think the polls are wrong just because they “feel” wrong to you, it probably says more about what you would like the result to be than anything about the polls.

2) I speak to lots of people and none of them will vote for X!

Actually, so do pollsters, and unless you regularly travel around the whole country and talk to an exceptionally representative demographic spread of people, they do it better than you do. We all have a tendency to be friends with people with similar beliefs and backgrounds, so it is no surprise that many people will have a social circle with largely homogenous political views. Even if you talk to a lot of strangers about politics, you yourself are probably exerting an interviewer effect in the way you ask.

3) How come I’ve never been invited to take part?

There are about 40 million adults in the UK. Each opinion poll involves about 1,000 people. If you are talking about political voting intention polls, then probably under 100 are conducted by phone each year. You can do the sums – if there are 40,000,000 adults in the UK and 100,000 are interviewed for a political opinion poll then on average you will be interviewed once every 400 years. It may be a long wait.

4) They only interview 1000 people, you’d need to interview millions of people to make it accurate!

George Gallup used to use a marvellous analogy when people raised this point: you don’t need to eat a whole bowl of soup to tell if it is too salty, providing it is sufficently stirred a single spoonful will suffice. The same applies to polls, providing an opinion poll accurately reflects the whole electorate (e.g, it has the right balance of male and female, the right age distribution, the right income distribution, people from the different regions of Britain in the correct proportions and so on) it will also accurately reflect their opinion.

In the 1930s in the USA the Literary Digest used to do mail-in polls that really did survey millions of people, literally millions. In 1936 they sent surveys to a quarter of the entire electorate and received 2 million replies. They confidently predicted that Alf Landon would win the imminent US Presidential election with 57% of the popular vote and 370 electoral votes. George Gallup meanwhile used quota sampling to interview just a few thousand people and predicted that Landon would lose miserably to Roosevelt. In reality, Roosevelt beat Landon in a landslide, winning 61% of the vote and 523 electoral votes. Gallup was right, the Digest was wrong.

As long as it is sufficent to dampen down sample error, it isn’t the number of people that were interviewed that matters, it is how representative of the population they are. The Literary Digest interviewed millions, but they were mainly affluent people so their poll wasn’t representative. Gallup interviewed only a few thousand, but his small poll was representative, so he got it right.

5) Polls give the answer the people paying for it want

The answers that most clients are interested in are the truth – polls are very expensive, if you just wanted someone to tell you what you wanted to hear there are far cheaper sources of sycophancy. The overwhelming majority of polling is private commercial polling, not stuff for newspapers, and here clients want the truth, warts and all. Polling companies do political polling for the publicity, there is comparatively little money in it. They want to show off their accuracy to impress big money clients, so it would be downright foolish for them to sacrifice their chances with the clients from whom they make the real money to satisfy the whims of clients who don’t really pay much (not to mention that most pollsters value their own professional integrity too much!).

6) Pollsters only ask the people who they know will give them the answer they want

Responses to polls on newspaper websites and forums sometimes contain bizarre statements to the effect that all the interviews must have been done in London, the Guardian’s newsroom, Conservative Central Office etc. They aren’t, polls are sampled so they have the correct proportion of people from each region of Britain. You don’t have to trust the pollsters on this – the full tables of the polls will normally have breakdowns by demographics including region, so you can see just how many people in Scotland, Wales, the South West, etc answered the poll. You can also see from the tables that the polls contain the right proportions of young people, old people and so on.

7) There is a 3% margin of error, so if the two parties are within 3% of each other they are statistically in a dead heat

No. If a poll shows one party on 46% and one party on 45% then it is impossible to be 95% confident (the confidence interval that the 3% margin of error is based upon) that the first party isn’t actually on 43%, but it is more likely than not that the party on 46% is ahead. The 3% margin of error doesn’t mean that any percentage with that plus or minus 3 point range is equally likely, 50% of the time the “real” figure will be within 1 point of the given figure.

8 ) Polls always get it wrong

In 1992 the pollsters did get it wrong, and most of them didn’t cover themselves in glory in 1997. However, lessons have been learnt and the companies themselves have changed. Most of the companies polling today did not even exist in 1992, and the methods they use are almost unrecognisable – in 1992 everyone used face-to-face polling and there was no political weighting or reallocation of don’t knows. Today polling is either done on the phone or using internet panels, and there are various different methods of political weighting, likelihood to vote filtering and re-allocation of don’t knows. In 2001 most of the pollsters performed well, in 2005 they were all within a couple of points of the actual result, in 2010 the pollsters overestimated Lib Dem support, but were very accurate on the gap between Conservative and Labour.

9) Polls never ask about don’t knows or won’t votes

Actually they always do. The newspapers publishing them may not report the figures, but they will always be available on the pollsters’ own website. Many companies (such as ICM and Populus) not only include don’t knows in their tables, but estimate how they would actually vote if there was an election tomorrow and include a proportion of them in their topline figures.

The Guardian’s report on their poll this month has made the assumption that a drop in Conservative support is due to NHS policy, various other commentators have this morning jumped to the same conclusion.

The truth is, as ever, rather more complex than that. Firstly we should set aside the ICM poll apparently showing a 4 point drop in Tory support as the NHS row continues. This is a classic case of making the outlier the story, the media’s constant failing in reporting polls – the underlying trend in polling is ignored while the rogue poll (and the reversion to the mean afterwards) gets a headline. Looking at the broad range of polls the Conservatives have not lost four points of support: they have gone from a position in December when, after the European veto, they appeared to be slightly ahead of Labour in the polls to a position this month when they appear to be a point behind or so, a drop of one or two points.

Many commentators today have also fallen victim to a fallacy I often see in the comments here – ascribing whatever movement happens in the poll to whatever subject they personally feel strongly about. Hence the Guardian and many bloggers feel strongly about the NHS, there has been a slight drift downwards in Tory support, therefore the former is probably the cause of the latter (it’s also a sort of availability bias – the poll contained questions about the NHS, they are bad, therefore that’s the cause. What if the other questions in the poll had been about, say, crime?)

However, this ignores other possible explanations, which could actually be better evidenced. Here are a couple, though I certainly wouldn’t claim these are exhaustive – there are no doubt other possibilities, including those that are less well served with tracker polls.

The first hypothesis is the unwinding of the European veto effect. You will remember that in November Labour had a lead of four or five points. The Conservatives then pulled level after David Cameron’s veto at the European summit. We should be expecting the effect of the veto to gradually unwind and, indeed, there is evidence to suggest that is what we are seeing.

The proportion of people thinking that Europe was an important issue facing the country peaked at 38% just after the veto, but has since faded away again, and is now back down to 23%. Similarly we saw some sharp increases in perceptions of David Cameron on the back of his veto. The percentage of people thinking Cameron “sticks to what he believes in” rose from 26% to 39%, decisive went from 20% to 29%, good in a crisis went from 13% to 18%. Since then most of these figures have fallen back down a bit (“stick to what he believes in” back to 31%, decisive back to 24%, good in crisis back to 18%).

One straightforward explanation therefore is just the fading of the leadership boost that Cameron received from the veto.

A second hypothesis is less negative coverage of Ed Miliband. In January the media narrative was dominated by criticism of Ed Miliband’s leadership, and the media impression was increasingly that he was a no hoper. In February that has faded a bit, the attacks have mostly stopped coming, he’s had a few good PMQs under his belt and the media narrative has largely moved on (to a great degree to the NHS!). This is reflected in Miliband’s own ratings. In January he hit record lows in his approval ratings, reaching a nadir of minus 53 points. In February he has recovered a but to the low minus 40s. His ratings are still significantly worse than they were last year… but he has risen off the canvas.

A third hypothesis is the NHS. The recent coverage has certainly pushed the NHS up the agenda – going back to the YouGov issues tracker, the proportion of people naming the NHS as an important issue is up to 32% from 22% a month ago. We also have lots of questions that have shown the government’s policy is unpopular and that people don’t trust the Conservatives on the NHS, but these don’t necessarily indicate change – it could’ve been unpopular all along.

The ICM poll yesterday shows that people trust the Conservatives less on the NHS now than in 2006, which is an interesting finding in itself, but it doesn’t follow that the drop in trust has come in the last few weeks or is a result of the current policy. If we look at YouGov’s regular tracker on which party people trust the most on the NHS there is a distinct lack of any recent drop in the proportion of people who prefer the Conservatives on the NHS.

In fact, the Conservative trend in the last year is pretty flat – what drop there was happened in late 2010 as the government’s honeymoon faded. What is also worth noting in this graph is that the Conservative detoxification on the NHS is something of a myth – it never really worked very well, and Conservative leads on the NHS were small and transitory.

This, I suspect, is one of the reasons the NHS hasn’t damaged the Conservatives that much – most people never trusted them on it in the first place. The other reason is that most people have no clue about the reforms and what they are or are not likely to do – take this recent YouGov poll for the Sunday Times which, rather than tell people about the reforms and ask about them, just asked people to say if they supported or opposed them based on what they’d seen or heard about them. 48% of people were opposed… but over a third (34%) of people said don’t know, a comparatively high figure for any question.

That is not to say that the NHS reforms are not politically dangerous for the government – I think Tim Montgomerie’s point that any future problems with the NHS (and there will always be some problems with the NHS, even if there were not budget squeezes and reorganisation) will be blamed on the policy is well made. Equally, just because it hasn’t made a big impact so far doesn’t mean it couldn’t do so in the future.

However the claim that it is already doing significant damage the Conservatives in the polls is weak. The drop in Conservative support in the polls is small and there are alternative and perhaps better evidenced explanations for it.