I seem to spend an inordinate amount of time telling people not to look at regional splits in polls and get all excited about them, so here goes again: regional splits in individual voting intention polls rarely tell you anything at all.

The Telegraph today has looked at their Yougov poll and decided it shows the Conservatives doing badly in the North. For what it’s worth, it doesn’t even do that – it shows the Conservatives 2 points behind in the North, an aggregate of government regions in which they trailed the Labour party by 19 points in 2005 – so it actually shows a swing to the Conservatives of 8.5 points in the North, marginally better than this poll suggests they are doing in the country as a whole.

That, however, is beside the point, since even if the Telegraph had correctly interpreted what the figure meant, it would still be meaningless. The regional breaks in polls have sample sizes of only a few hundred, meaning they suffer from a much larger margin of error and are far more volatile. On top of that, they are not internally weighted. As a whole, opinion polls are demographically weighted to be representative, so we can be certain there are the correct number of men and women, young and old, working class and middle class and so on. Within regions however this may not be the case – there might, for example, be too many old people from the south and too few from the north, too few men from the midlands and too few from Scotland. The normal laws of propability mean it won’t be too skewed, and overall we know it evens out, but it all adds to the volatility.

In practice, this means the regional cross breaks jump about wildly from poll to poll. So this YouGov poll appeared to show the Conservatives doing slighty better in the North and the Midlands, hugely better in London, but worse in the rest of the South. However, if you went back to the YouGov/Telegraph poll a week before and looked at the cross-breaks there, you’d have found the Conservatives doing worse in London and the South, and best of all in the Midlands. There was another poll in April that showed the Conservatives doing best in the South, there are even a couple that have suggested they are doing better than average in Scotland. The splits are so volatile that trying to draw conclusions from the cross breaks in a single poll is pointless.

If you average out the splits over a long period of time it might give you a better idea (since you ask, this year’s YouGov figures suggest the biggest swing to the Conservatives in London, followed by the North and the Midlands. The rest of the South is actually comparatively poor, with – predictably – Scotland the worst) but even then, it’s data that’s not weighted to be properly representative of those regions. What you really need is large scale aggregration of reweighting of data from lots and lots of polls, or specially commissioned large scale polls.


20 Responses to “No, today’s poll did not show the Tories doing badly in the North”

  1. When newspapers do this, like when the Observer misunderstood data a week or so ago, you realise why so many people read blogs instead these days!

  2. Anthony,

    Thank you for explaining this so clearly and concisely !

  3. Blimey, this is poor stuff. What has gone wrong with the Telegraph in recent years? It used to be my paper of choice when Charles Moore was the editor; I found it to be something of a safe haven during the early Blair years when it was Tories against everyone else. It used to have an excellent sport section too which also seems to have gone down the pan.

  4. What on earth are the Telegraph playing at? A two per cent Labour lead in the North? That’s an amazing advance for the Conservatives. Doesn’t the Telegraph realise that Labour is usually miles ahead in the North? I think the 75% negative rating for British society should more accurately be applied to the media as far as opinion polling is concerned.

  5. Thanks Anthony.

    The British Polling Council’s website tells us that it has four “Objects”.

    Numbers 3 & 4 are :-

    * To advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.

    *Provide interested parties with advice on best practice in the conduct and reporting of polls

    Judging by the Telegraph’s “interpretation”& “reporting” of this Poll-and your need to open this thread- BPC are failing to achieve their Objects 3 & 4

    “Understanding” amongst the General Public appears not to have been “advanced” in the use of this Poll -otherwise you wouldn’t have had to say what you just said.

  6. Colin – not necessarily true. It could be that understanding of polls used to be even worse ;)

  7. Thanks Anthony and thanks for the reply in the previous thread. :)

  8. My own reading of this and general media treatment of Polls in particular but politics in general is that the rise of the web and the recession are increasingly taking their toll of newspapers.

    A combination of the fast pace of news and the need to make economise mean fewer people under more pressure to get the job done. This means that increasingly people don’t have the time or resources to do peoper reserch let alone specialise.

    Increasingly stories are cut and twisted to fit pre determined boxes because their just isn’t the time to go into depth. Whether it be just poor interpretation like this or stories where someone is trying something different being wrongly classified the quality of what we are getting is getting worse.

    Having said that i do think that it is still generally of a higher quality than that of the web where there really is some really terrible quality of output and where even the best or most popular blogs still make more basic errors than most papers.

    I’ll depress you all by predicting that in ten years we will have few if any news papers and the web will dominate but across the board content will be of a lower quality than it is now.

    U herd it hear thirst…….

    Peter.

  9. Thanks Antony,

    Always good to get the facts from an expert interpreter rather than a headline seeking Fleet St editor.

    I knew that what they were saying simply didn’t tie up with what I was hearing on the doorstep

  10. Anthony,

    This example highlights a couple of points which could usefully be addressed, both in terms of expanding understanding of polling data and enabling sensible commentary on regional trends.

    Years ago, the papers would report actual results at elections on a national basis, then broken down by various categories – inlcuding regional results. Looking back thorugh my store of papers since 1997, the regional breakdowns seem sadly lacking.

    Is there any prospect of your putting up a table of share of votes / seats on the regional pages for the Constituency section ? This would provide a better baseline for commentary on trends in each region.
    A similar table for Councils/Councillors would be a useful bonus.

    It may also be necessary to do a table showing in which area that region is included by various pollsters, as they may not all be the same.

    It would also be nice to have the comments reopened on the regional pages – maybe nearer the election ?

    Thanks,

    Paul

  11. @ Anthony

    “Colin – not necessarily true. It could be that understanding of polls used to be even worse ”

    But not this Poll Anthony. This Poll has only just been published, and it’s very explanation to the public was wrong-as you have pointed out.

  12. QUITE. Couldn’t we get all newspaper clients to sign up to a basic code of statistical literacy, or is that too much to ask?

  13. NBeale – not just newspapers, politicians too. I watched in amazement at the utter lack of statistical knowledge shown when John Prescott sat on the Westminster terrace telling us how, since 1997, cancer deaths were down x% in the south-east without providing any frame of reference for the stats (i.e. how does that compare to other regions / countries) and somehow he concluded that the NHS is brilliant.

    Just for clarification – I’m not reopening the NHS debate here (wayyyy too partisan for these comments), I’m pointing out how easy stats are to misuse.

  14. Paul H-J

    The electoral commission produced an Excel spreadsheet summary of the 2005 General Election with regional breakdowns etc.

    It can be downloaded here:

    http://www.electoralcommission.org.uk/__data/assets/excel_doc/0006/47265/

  15. Excellent points, Anthony.

    I congratulate you as one who got caught myself in posting about a recent poll repor by referring to regional data.

    I hope most of us regular contributors to your site will understand your points fully. I fear some others, perhaps given their recent writing including some journalists, may have to think out what terms like “cross breaks” mean.

    There is an important, non-statistical, point I would like to add. For political and government purposes, including not only statistical analysis but also “ordinary” comment, everybody seriously involved works on the basis of the Government and EU regions. We know exactly which constituencies and local authorities they include. These regions may not be satisfactory, particularly because the South-East region is so huge, but they are what we’ve got and understand.

    If “The Telegraph”, “The Guardian” (or “Observer”) or whoever start using terms like “The South” or “The North” we not not understand what they mean. Are the boundaries of these areas co-terminal with Government regions? Is “East Anglia” in “The Midlands” or “The South East”? Similarly, does “The North” comprise the North-West and the North-East, or does it also include Yorkshire?

    If any Editors are reading this blog (we’ve had Bob Worcester, so why not!), or Readers’ Editors in the case of “The Guardian” and “The Independent”, it might be helpful if they could issue their people with a guidance note, or Style Guide amendment, as the “Telegraph” report that is the subject of this thread (and as I am making clear the Telegraph is by no means alone in this) is using terms that are inadequately defined and therefore unclear to readers.

    A separate point: it would be nice to think that PopulusHome might have a large scale poll coming up at Conference time so that serious psephologists can compare with the similar poll last year. That would be really useful. But clearly it depends upon the commercial viability of such an exericse.

  16. Anthony,

    As always your analysis is excellent.

    having said that it would clearly be wrong of any of us to use your analysis (or indeed on any other analysis you might prepare) to make a comment on the Telegraph or other report.

    However is there not some mechanism where through your connections you could make these comments or analysis to try and “educate” these political journalists?

  17. Statistics is easily the least well understood scientific discipline out there. And the terrible danger is that most people have no idea that they have no grasp of statistics; including smart, educated people.

  18. That’s a crazy headline based on some carefully selected facts (and many others equally carefully ignored).

  19. NBeale asked, “Couldn’t we get all newspaper clients to sign up to a basic code of statistical literacy, or is that too much to ask?”

    One reason there is no basic code of statistical literacy for journalists may be that statisticians and statistical educators have not properly addressed that issue.