Comments on: ICM show Labour recovering

By: NBeale

NBeale — Mon, 24 Dec 2007 13:46:50 +0000

Well I’ve now re-analysed the data from the other polling companies eliminating YouGov entirely. It makes essentially no difference: the retrospective errors are: ICM:0.4, IMori:-1.4, Populus:-0.4, BPIX:1.5, ComRes:0.7 vs 0.5, -1.6,-0.2,1.0,0.9 and the retrospective StDevs are 2.5, 2.6, 2.4, 2.2, 3.4 vs 2.5, 2.7, 2.6, 2.0, 3.4. Statistically therefore the conclusion is clear: even though YouGov has a different methodology they appear to be sampling from the same underlying population: they are just appreciably more accurate.

PS Anthony: shall we cross-link blogs?

By: Anthony Wells

Anthony Wells — Sun, 23 Dec 2007 19:56:22 +0000

I should interject just to encourage people to remember the difference between how volatile a poll is, and how accurate they are. A poll can be very, very consistent and still entirely wrong, so it’s probably best not to use the word accurate to describe a polling company that produces polls that have very low volatility. They are two separate things.

Incidentally, taking a sample from a smaller universe doesn’t necessarily make much difference to volatility once that universe is beyond a certain size. For example, phone pollsters theoretically take their samples from a universe of everyone with a phone (though in reality they take them from the smaller universe of people with a phone who are willing to spend 15 minutes answering a strangers questions, which is considerably smaller) so about 40 million or so adults. A sample of 1000 gives you a margin of error of 3.1%.

Draw a random sample of 1000 people from a panel of 200,000 and you’d expect a lower margin of error, right? Not really, the margin of error is 3.09%. Counter-intuitive it may be, but once the universe you are sampling gets beyond a certain size its pretty much irrelevant.

In practice of course Ipsos MORI and YouGov don’t use random sampling, so the actual calculation for margin of error doesn’t apply, and when using quota samples panel surveys can produce much lower levels of volatility because of the amount of demographic information that is readily available on panelists to construct samples.

By: Mark Senior

Mark Senior — Sun, 23 Dec 2007 17:57:30 +0000

Nick I understand this but including Yougov in your average of all polls is not necessarily a sensible thing to do as they are completely different types of polls to the other pollsters .
If you were compiling a moving average of the prices of 5 deciduous fruits 4 of which were apples and 1 was a pear , the price of the pear may remain stable and not vary as much as 1 or 2 of the varieties of apples but will tell you nothing about the change in price of apples or deciduous fruit as a whole .

By: NBeale

NBeale — Sun, 23 Dec 2007 17:42:10 +0000

i’m not talking about variations within a single polling organisation, but variations from the underlying avearages of all polls. Yougov is substantially more accurate on this basis.

By: Mark Senior

Mark Senior — Sun, 23 Dec 2007 17:01:04 +0000

Nick , Yougov is not a normal random opinion poll it is a poll taken from a relatively small compared to the population at large self selected group of people with a relatively much higher interest in politics than the average person ( it includes myself for example ) . Leaving aside the wisdom of projecting national opinion from an unrepresentative sample , you would expect a poll from a small sample to vary less than a random national poll .
I have been polled 3/4 times by Yougov in 3 or so years but never by a telephone pollster , once many many years ago by a face to face pollster I think Gallup .