Years ago I slated the BBC for commissioning a poll about religion and trying to draw conclusions about people from different religious groups using pathetically small sample sizes. They are at it again in this poll.

The ComRes poll found that 63% of people agreed with the statement that “Our laws should respect and be influenced by UK religious values”, which is fair enough. The BBC report then goes onto say that “A significantly greater proportion of the Muslims and Hindus polled (albeit in relatively small numbers) supported a strong role in public life for the UK’s (essentially Christian) traditional religious values.”

If we look at the tables though these weren’t “relatively small numbers”, they were minute numbers. The poll interveiwed 21 Muslims and 9 Hindus. The “significantly greater” figures weren’t significantly different at all – 79% of Muslims agreed and 74% of Hindus agreed. The margins of error on these tiny groups are something like 23% and 32%, so they are not significantly different at all.

So, to the BBC, if you want to compare the views of different religious communities, you need to commission a bespoke poll taking representative samples from each religious community you are interested in. You can’t do it for a cutdown price on an omnibus and try to say something about British Hindus based on nine interviews. If you just want to take the views of Britain as a whole, don’t try to draw conclusions about tiny subsamples that your data cannot possibly support.

I suppose we should at least be grateful for that small caveat about sample size and that the tables are there on the BBC site to see, since the BBC story then gets picked up by other people who report it even more badly. For example, the Telegraph make the findings about Muslims – based on 21 people remember – the headline of the story. The Daily Mail surpass that by also highlighting that “three-quarters of Sikhs said…” There were 3, that’s THREE, Sikhs in the sample.


48 Responses to “BBC report findings from a sub-sample of NINE”

  1. You do despair of the media sometimes….. this is what happens when you allow people to give up maths and science at 16… :)

  2. Also, am I the only one who thinks that the smileys on this site look a bit sinister?

  3. :) :) :) :) :)

    Sorry. Trying to scare Alasdair.

    Right, on to the serious stuff. At least the BBC actually bothered to ask some Real People this time. It’s usual modus operandi seems to be to assume that if the BBC’s collective mind thinks something, the entire population must think the same (with the exception of the BNP and people who have sex with goats).

  4. Paul – confusing isn’t it? The three Sikhs were weighted up to four Sikhs. So presumably the Sikh who didn’t agree was weighted downwards for some other reason, while the two Sikhs who did agree were weighted upwards.

    The moral of this is that it’s really silly to base polls on 3 people :)

  5. The Mail and Telegraph should be really taken to task for this. It makes one wonder how much of their “news” is just (badly) rehashed stories from other sources. They’re written like something out of a secondary school exercise where you’re asked to provide a synopsis of a report (in this case one which was already bad).

  6. And the BBC should be taken to task as well; it’s all rather sikhly…

  7. “Really silly” is a very restrained verdict. Scary and scandalous in equal measure I’d have thought. Ok, so it did no harm in this instance, but there is always the danger that this kind of thing may light an unexpected spark. And it is corrosive: the cumulative effect of bad statistics (and bad science) distorts reality, trivialises subjects and degrades public discourse, especially when the perpetrator is as influential and credible as the BBC.

    Thanks for drawing it to wider attention. It really should be the subject of one of the BBC’s regular panic enquiries and lead to some more of those surplus executives departing. But I bet it won’t.

  8. I shudder to think what “conclusions” would have been drawn if all the six or three or whatever had been fundamentalists of any persuasion.

    (shakes head with disbelief)

  9. I must congratulate the BBC for 3 things: –

    1, Being prudent with my licence fee by not spending on a poll

    2, Using the results of the non-poll to give fuel to the religous argument being played out in the tabloids

    3, Its inability to think which makes us all feel better about ourselves.

    I’ll cheer for the bbc making 75% of my desk constiuency pro BBC idiocy (the other constituent is a gold fish – 100% of which gave no oppinion)

  10. It’s a consistent theme in every BBC poll – “Spin the message and damn the statistics”.

    This is just like their reporting their own unweighted poll figures because following industry practises would spoil the story they wanted to tell.

    It’s depressing. I don’t know which is worse – the BBC for distorting poll figures to support the story they want to tell, or the pollsters for taking the BBC’s shilling and delivering figures that they know the BBC will use for propaganda.

  11. Anthony,

    Any idea how ComRes got from their unweighted samples to their weighted samples ? Some of the adjustments appear to have been made for the sake of making adjustments. For example, how do you adjust for “refused” ?

    Strange that those professing no religion been boosted by more than the downward adjustment in Christians (and by more than the actual figure for Muslims/Hindus/Sikhs/Jews combined).

    Also, what are the “other” religions which purportedly exceed Muslims, Hindus, Sikhs and Jews combined by a factor of about 1.5 ?

    Funny how the BBC did not report that >70% of their sample expressed a religious affiliation in their article which was aimed to coincide with a poster campaign to promote atheism.

    So many holes to pick in this poll and its coverage !

  12. Paul, it’s weighted by normal demographics. Differences between the weighted and unweighted religious breaks are presumably just by-products of that.

    E.g, ComRes had to weight men upwards and women downwards, ABs and DEs downwards and C1C2s upwards. Under 25s upwards. Presumably therefore, the religious groups that are more numerous in the weighted sample than the unweighted one are ones that had lots of younger men and fewer older women amongst them.

  13. Are there any believable polls on Sikh attitudes to religion?

  14. A reasonable poll on Sikh views could probably be carried out in those few areas where they reside in large numbers such as Southall and Leicester, but outside those areas in would be pretty difficult to get a reliable sample.

  15. @Anthony and Paul – I think you will find that the lots of younger men were all Jedi

  16. @Paul H-J – Don’t forget Buddists, Satanists (although may not count), Agnostics, Rastafarianism, Pagamism, Unitarianism, Shinto, Greek Authodox (although Christian not sure where it would count in a BBC poll), Russioan Authodox (Ditto), Zoroastrianism, African Traditional & Diasporic (May include voodoo), Jainism, Baha’i etc etc etc

    There will also be lots of people who believe in a god but don’t know which one and would not understand the concept of agnostisism.

  17. @Andy Stidwell – I think you need to look again at your demographics as over 31% of Sikh’s live in London and the south. I myself have sikh friends who live in and around swindon.

    This is the statement from NAtional Statistics in 2001 Almost a third (31 per cent) of the Sikh population lived in the West Midlands. They were particularly concentrated in the Wolverhampton and Sandwell areas: 8 per cent and 7 per cent respectively of the populations of these local authorities were Sikh. A further 31 per cent of the Sikh population lived in London. They were especially concentrated in West London boroughs, making up almost 10 per cent of the populations of Ealing and Hounslow. They also comprised almost 10 per cent of the population of Slough in the South East.

  18. I you want to do a proper poll, Sikh and yea shall find….

    Peter.

  19. Southall is part of London as far as I know.

    I doubt there would be enough Sikhs in Swindon to carry out a reliable survey unless you specifically targeted all the Sikhs in that area which is very unlikely.

  20. @Andy – it’s quite easy to do, they tend to congregate regularly :-)

  21. http://www.bbc.co.uk/wiltshire/faith/vaisakhi_2004.shtml
    Worshippers at the Sikh temple in Swindon are in the throes of celebrating the Sikh new year and marking the birthday, on April 14th, of Sikhism. Amongst the traditional Vaisakhi (or Baisakhi) celebrations: a new flag at Swindon’s ‘place of God’ – the gurdwara.

  22. @andy – and considering the BBC can do a poll on 3 sihks, just think how good our poll would be if we just asked the ladies in the picture :-)

  23. the picture in the post that is currently being mederated for having either a website address or the word G-o-d in it :-(

  24. no comments i don’t do god and that sort of stuff as it always leads to anger.

  25. Comments made about my beloved Jesus-Mohammed like that make me so ANGRY I could KILL!!

    ;-)

  26. @Stuart – it really makes me angry when people don’t make comments in case they make people angry :-)

  27. Of course a survey conducted at a temple wouldn’t tell as about Sikh opinion in general, only about the views of those who attended temple.

    Peter.

  28. The only way to stop the sort of journalism by the BBC is to send an official complaint. I have sent one and it takes 3 mins

  29. I notice that one contributor here refer to the BBC as “respected” – well, maybe once upon a time! It is difficult to know what they were hoping to prove by this poll, other than their own ineptitude. Sadly we can also add the Telegraph to the “At one time respected” list also. As for the Mail, well what do you expect?It would appear they can’t manage fractions!

  30. Anthony,

    Over on the UK Skeptics Forum a debate on this poll has generated quite a lot of heat but little light. Perhaps you can help? Here’s the opening post:

    “You have to ask who at the BBC agreed the questions.

    “Our laws should respect and be influenced by UK religious values”
    Agree or Disagree

    How about a better question.” http://tinyurl.com/antxm8

    In simple terms the question is described as “spin”, “vague”, “unclear”…is there a problem with it?

  31. Is there such a thing as “non-practising” within the less mainstream religions?

    To what extent would one have to be “practising” in order to be counted as a voter who carries weight in a poll of religious groups?

    And what about, say a devout practising Roman Catholic who firmly rejects one or two of the tenets of Roman Catholicism?

    It’s a minefield.

  32. @Peter Cairns – there’s no such thing as a non practicing Sikh. It would be like saying you were non practicing celebate.

  33. @John T T – The Catholic one is easy although it does leave a few people cold (as they think they are Catholic when they aren’t) – it all cpomes down to the profession of faith.

    http://www.legionofmarytidewater.com/faith.htm

    We believe in one God, the Father Almighty, the maker of heaven and earth, of things visible and invisible.
    And in one Lord Jesus Christ, the Son of God, the begotten of God the Father, the Only-begotten, that is of the essence of the Father.
    God of God, Light of Light, true God of true God, begotten and not made; of the very same nature of the Father, by Whom all things came into being, in heaven and on earth, visible and invisible.
    Who for us humanity and for our salvation came down from heaven, was incarnate, was made human, was born perfectly of the holy virgin Mary by the Holy Spirit.
    By whom He took body, soul, and mind, and everything that is in man, truly and not in semblance.
    He suffered, was crucified, was buried, rose again on the third day, ascended into heaven with the same body, [and] sat at the right hand of the Father.
    He is to come with the same body and with the glory of the Father, to judge the living and the dead; of His kingdom there is no end.
    We believe in the Holy Spirit, in the uncreated and the perfect; Who spoke through the Law, prophets, and Gospels; Who came down upon the Jordan, preached through the apostles, and lived in the saints.
    We believe also in only One, Universal, Apostolic, and [Holy] Church; in one baptism in repentance, for the remission, and forgiveness of sins; and in the resurrection of the dead, in the everlasting judgement of souls and bodies, and the Kingdom of Heaven and in the everlasting life.

    Although this is only one translation from the armenian, variations exist and are in use in Churches everywhere, but the base precepts are quite clear.

  34. Cheeky. Nice one ComRes!

    Good investigating there, Polling Report.

  35. With all the anti-Brown news recently, we might have to get used to sub-samples of 9 to break down the Labour vote.

    It hasn’t been a good week for him – the FSA and Mervyn calling him out for his part in the financial crisis and Royal Mail (very unpopular among the Labour bank benches). One can see why there is so much effort going to divert attention onto Fred the Shred. I wonder if we’ll be seeing any effect in the next round of polls.

  36. John TT – I would expect there are people from every religion who identify with it culturally, but don’t actually practice it. If you wanted to do a poll of Sikhs or Muslims or whatever, I suppose it would be up to you whether you wanted to include practicising people from that religion, or people from that religious background. There is no “right” answer.

  37. Keir.

    Armenian????, it should be Nicene… Heretic….

    Peter.

  38. @Peter – it is, but I didn’t trust the latin translation :-) . You are correct though this is the Nicene Crede – Written in Turkey (Iznik – was Nicaea) in the Greek Language.

  39. Unitarians used to think of themselves as Christians, but the Christians have persuaded them otherwise, and they mostly now accept that.

    The number of non-practicing cultural Jews is very significant in relation to the others.

    In recent years huge numbers in Germany and Austria have discontinued identification with any group.

  40. In fairness the Anthony is misinterpreting what the BBC says. They’re not saying anything about the Hindu sample of 9. They’re commenting on the “Muslim and Hindu” sample of 30, and noting that they’re keener on religion in public life than the rest of the sample.

    30 is still a pathetic subsample. But it’s not as patheitc as the 9 that Anthony suggests.

  41. I think you can read it either way Nick – you took it as meaning “Muslims and Hindus collectively were more supportive”, when I read it as “Muslims and Hindus are both more supportive”.

    I think they actually meant it the way I took it – they go on to say “albeit in relatively small numbers“, not “albeit a relatively small number“. But it’s angels on the head of pin, either way is nonsense – and the Daily Telegraph and Mail have no excuse at all!

  42. Kier,

    If this is the Nicene creed, where did you get your English translation from – don’t recognise this wording from either my Roman missal or the Anglican text. Perhaps it is the English form of the Armenian ?

    If you go back to the Latin, it is unequivocally “Credo” and not “Credemus”. The first person singular is more than an Angel dancing on the head of a pin issue, it is a personal affirmation.

    Peter,

    Armenians, as opposed to Anglicans, are Schismatics not Heretics.

    Having said which, you would be hard presssed to find any heresies in the Anglican Creed whether in the BCP or more modern translations – most of which are indistinguishable from the Roman approved English.

    As I am unfamiliar with the Presbyterian or Wee Free rites, I can’t comment which versions of the Creed they use.

  43. As I said this is a translation and I carefully avoided using the standard english Catholic version (the irony)

  44. Paul H-J,

    Presbyterians (in Scotland, I don’t know about elsewhere) don’t really use any of the creeds as such, but assent to the Apostles creed. I haven’t got the text right now, but I can post it later if you wish. Looking at Kier’s translation of the Nicene creed, I can see nothing in it with which Presbyterians would disagree, except calling the virgin Mary “holy”.

    As for the use of the word “Catholic” – it simply means “universal” and the general view of it (among non Roman Catholics) is that it refers to all believers in the world and not to any particular denomination.

    I am not entirely sure wht the significance of this discussion is, even after reading the thread. Was this not supposed to be a thread where Anthony got us all to criticise the Telegraph and the Mail?

  45. @Neil – it’s “ascended” slightly higher than that and I am proud to say it’s about as none partisan as you can get :-) – so let’s not worry about the point of it…

  46. Kier,

    Non partisan…… I called someone a heretic…..

    Peter.

  47. Neil – the link is to do with identity within a sample. As a catholic (more celtic than Roman) I agree with your definition, and though I might be regarded by some as lapsed, it’s my church as much as anyone else’s, and if I decided to attend a Sikh temple every few months, I’d happily count myself as part of the Sikh sample in a poll.

    As for Anthony’s answer to me earlier (many thanks), it would seem that the pollster made no assumption about their level of commitment.