Methodology changes

An update for methodology anoraks out there. ICM tweeted earlier today that they are to start doing part of their fieldwork using mobile phones. Up to now telephone pollsters in the UK have exclusively used landlines to conduct interviews (the actual process they use is to randomly select numbers, then randomise the last digit to catch ex-directory people).

As mobile phone ownership has became widespread landline ownership has peaked and begun to fall – Ofcom’s figures are that 15% of adults live in a home that has a mobile, but no landline. This doesn’t necessarily pose a problem for phone pollsters (after all, it is possible to conduct accurate polling on the internet and only 75% of UK adults have broadband access) it depends whether mobile only people are substantially different to landline people once you have controlled for all the demographics pollsters weight by (i.e. if doesn’t matter if they are younger, or more middle class, as weighting would control for that)

Anyway, ICM at least are biting the bullet and starting to conduct 15% of their interviews over mobile phones. This has already become standard practice for many pollsters operating in other countries, but I think this may be the first of the regular pollsters in the UK to make the switch. Whether it makes any noticable difference or not I do not know.

A second change to note is ComRes, who have been experimenting with how they model likely turnout in their polls, and redistributing don’t knows/won’t says. I’m not sure of the exact details yet (Andrew Hawkins suggests it is still in flux) and Andrew tells me it is making “almost no difference to the outcome” but on their last two polls it seems to have made their figures marginally more Conservative. I’ll do a further update once all becomes clear.


86 Responses to “Methodology changes”

1 2
  1. Socal – Statgeek know what a canvesser is s/he is suggesting they never see any jut as Old Nat says he never sees an LD one.
    OLD NAT very funny re Mobiles and correlation also of course when Scotland had a good international soccer team so the reducing size of mobiles has caused Scotland’s decline as a football nation.

  2. Headbutters with menace should be prosecuted and in most circumstances this alleged sequence of events would lead to charges anywhere else
    There is a sense of politicians thinking they are above normal rules (by extension laws) from the expenses scandal and that can not be reinforced.

    Are there mitigating circumstances and should we be helping Alchoholics through the criminal justice system of course but sweeping this alleged incident under the carpet would be wrong.

    Sorry Anthony, off the polls but it isn’t a policy view I am giving.

  3. While not condoning an inarticulate drunken violent outburst, it may worthwhile to view it as a symptom. There was no provocation by all accounts – but people in such situations will take exception even to a look or a muttered word.

    It could be a clash of culture… a slight or a sneer may be a triffling matter in one context, or pistols at dawn in another.

    Alcohol destabilises rationality, the instinctual mind can make an instant connection with some small incident, and a whole trail of grievances will then boil over. However wrong an action may be, there will have been a chain of events that led to it.

  4. @ Statgeek

    What is a canvasser?

    ————————————-

    I thought it was somebody who made tents! :-)

  5. Brilliant :-

    http://www.youtube.com/watch?v=MfBMimsvTKI&feature=related

    A warning to the Coalition ( and countless others too :-) )

  6. So a Tory MEP for the East Midlands joins UKIP and is to be replaced by a soi-disant expert in UFOs, the paranormal and encounters with aliens. I’m not sure that even The Thick of It could match this for a plot line…

  7. ‘It’s extraordinary how this ex-police horse that Caneron rode is occupying the media. Several commentators have suggested it is a potent symbol of the incestuous Met/Press/Politician relationship.

    Is that media nonsense. I think so. But the press seems to think it might resonate as an image.’

    Yes it’s quite ridicuolous and I suspect Cameron will ‘ride out’ the saga.

  8. @ Lefty Lampton

    “So a Tory MEP for the East Midlands joins UKIP and is to be replaced by a soi-disant expert in UFOs, the paranormal and encounters with aliens. I’m not sure that even The Thick of It could match this for a plot line…”

    You know, I always wonder on election day, “who are the people who don’t vote?” Especially in a high turnout affair where the race seems really exciting and the outcome is not clear. You’d think everyone would be out voting (especially if you’re into politics).

    But then, I was flipping channels at the gym one day and I saw this show entitled “Ancient Aliens” on the History Channel. They extrapolate from all sorts of things to suggest ancient alien contacts and they come to this woman, who’s a member of a Native American tribe that supposedly had contact with aliens thousands of years ago. This woman, who’s a member of the tribe, is then put on screen and is just beyond weird and she’s like “well, I’m only partly descended from humans. Some of my ancestors are alien.”

    I thought to myself….this must be the type of person who doesn’t vote.

  9. @ Billy Bob

    “While not condoning an inarticulate drunken violent outburst, it may worthwhile to view it as a symptom. There was no provocation by all accounts – but people in such situations will take exception even to a look or a muttered word.

    It could be a clash of culture… a slight or a sneer may be a triffling matter in one context, or pistols at dawn in another.

    Alcohol destabilises rationality, the instinctual mind can make an instant connection with some small incident, and a whole trail of grievances will then boil over. However wrong an action may be, there will have been a chain of events that led to it.”

    I don’t condone it either. One reason I don’t drink alcohol is that it impairs judgment and tends to allow you to do and say (in my case, it’s more say) things you otherwise wouldn’t do.

    In terms of slight provocations leading to violent outbreaks because of alcohol, it can be disturbing when you’re around people who are like that. I was on a Metro train late one Saturday night and this guy, seemed college aged or mid 20’s, gets on and he is clearly hammered. And I mean HAMMERED. He can bearly stand up and at one point is dropping dollar bills on the floor. Yet another person who likes to drink yet cannot hold his liquor and doesn’t know when to stop.

    It’s clear though that this guy is a violent drunk and not a sweet drunk. Because as the train was malfunctioning (lights kept going out) and bumping along, this guy was having violent reactions towards inanimate objects on the train. He kept falling over and hurting himself. At one point I thought about going over and helping him but what I noticed was, when he’d fall over or hurt himself, he’d respond by attacking the seat, the pole, or the train door. It made me realize that if I went over to try and help him, he might take that as a provocation and attack me. So I left him alone and avoided contact with him. He finally got off the train….accidentally. But not before kicking a train door and cussing at it.

  10. leftylampton – No, no, no. The Tory MEP is joining UKIP because CCHQ wouldn’t let him be replaced by a soi-disant expert in UFOs, the paranormal and encounters with aliens.

  11. @SoCalLiberal – ” …not a sweet drunk.”

    When I was taxi driving many years ago, one passenger all of a sudden started pummeling the back of my seat like it was a boxing gym punch-bag, with a few wild swings at the headrest thrown in. My only advice is to be courteous, but straightforward and calm. Drunks are not without intelligence, you have to concentrate on continually de-escalating the situation.

    Also best to keep eye contact to a minimum in case they see something in your soul they don’t like – and keep your eyes to yourself if they are with a partner!

  12. AW.

    Apologies. I didn’t do my homework fully before committing my witterings to type. I hadn’t realised that CCHQ had kyboshed Matthews because of his eccentric opinions on external civilisations.

    Interesting though that they were quite happy to have him on the MEP list in 09 despite his musings on ConservativeHome about how the EU was darkly planning to send the Panzers in if any state dared think about seccession.

    http://conservativehome.blogs.com/platform/2009/01/rupert-matthe-1.html

    Priorities, I guess.

  13. @ Billy Bob

    “When I was taxi driving many years ago, one passenger all of a sudden started pummeling the back of my seat like it was a boxing gym punch-bag, with a few wild swings at the headrest thrown in. My only advice is to be courteous, but straightforward and calm. Drunks are not without intelligence, you have to concentrate on continually de-escalating the situation.

    Also best to keep eye contact to a minimum in case they see something in your soul they don’t like – and keep your eyes to yourself if they are with a partner!”

    What did you do in that situation? I imagine that would be quite frightening. You’re alone and are especially vulnerable.

    I’ve dealt with drunks just from my own family experience but I think it’s different when you deal with a complete stranger.

    Sweet drunks can be easier to deal with. One night I was walking home (actually, this was on the exact same night as the other incident I noticed) and I notice an interesting scene outside a restaurant. A guy is totally drunk and a woman is on the phone while stopping a taxi. She doesn’t know the guy but she knows that he is totally wasted. She’s on his phone calling his girlfriend or wife explaining that she is helping him get into a cab to go home. She needs to know the man’s address though because he’s too drunk to remember where he lives and she needs to know where to send the cab to. This guy could barely stand up either but looked friendly.

    I have a cab phobia btw (I pretty much won’t take them in most cities for a variety of reasons unless I absolutely have to) but cab drivers have an especially high stress and difficult job. Picking up and dealing with drunks, especially when they’re violent, is just one of those added high stress points. I don’t appreciate politicians who spend their time attacking cab drivers either.

    http://voices.washingtonpost.com/local-opinions/2010/09/why_cabdrivers_opposed_mayor_f.html

  14. Seen a clip of the Lib/Dem conference (counted 83 people) in Inverness and the one Annabel Goldie called a “Ginger Rodent” pocked a bit of fun at Rangers…That lot (Lib/Dems) are going from bad to potty.

  15. @ALLAN CHRISTIE

    It was Harriet Harman (Labour), not Annabell Goldie (Conservative) who called Danny Alexander a ginger rodent.

    Everyone pokes fun at Rangers.

  16. @Allan Christie

    “the Lib/Dem conference (counted 83 people) in Inverness”

    Scottish Lib Dem conference. The main one is next week in Sunderland.

  17. @Colin Green

    I thought your conference was in Gateshead/Newcastle?

  18. @ A Cairns

    Quite correct, it is actually in Gateshead but because nobody knows where Gateshead is it is always referred to as Newcastle/Gateshead. Having said that, the venue (Sage) is a magnificent centre.

    Hope nobody attending mentions Sunderland except in reference to the beating they are hopefully taking in tomorrow’s derby match.

  19. @Jim Jam, @Billy Bob

    It’s instructive to compare directly how the reallocation of DKs/refusals by ICM and Populus using assumptions compares with the reallocation by IpsosMORI and ComRes using actual responses to squeeze questions put to exactly the same type of voter. MORI use an “inclined to support” follow up question and ComRes an “if you were legally required to vote…” follow up question.

    In the tables supporting their latest polls, MORI and ComRes reveal that between them they found DK/refusal voters who voted for the main parties in 2010 in the following numbers: Con 99 Lab 69 LD100.

    ICM would simply reallocate these 268 voters as follows, based on 50% to each i.e. Con 49.5, Lab 34.5, LD 50. Populus would do the same but for a weighting of 30% to the LDs i.e.: Con 49.5, Lab 34.5, LD 30.

    Now compare those assumed responses to what the actual responses to the squeeze questions reveal, namely Con 50, Lab 55, LD 36.

    Two conclusions can be drawn. Firstly, ICM and Populus would both use such data on DKs/refusals to narrow the gap between Con and Lab even though the actual squeeze responses would call this narrowing into question. That goes some way to explaining why the current ICM and Populus headline polls show a Lab lead of only 1% and 2% respectively, compared to a Lab lead of 6% with IpsosMORI and 3% (or 5% on their pre Feb methodology – see above) with ComRes.

    Secondly, with ICM but not Populus the LD vote share would be boosted by much more than the actual squeeze responses. Because the LDs are starting with so few supporters prior to the reallocation, reallocation of DK/refusals on a squeeze question or by Populus’s 30% will still help the LDs compared to no reallocation, but by far less than ICM are assuming. This also explains why YouGov have consistently the lowest LD share – namely because there is no reallocation of DK/refusals of any kind.

    (Technical note: 1. For simplicity, I’ve aggregated the squeeze responses to ComRes and MORI. The pattern of both is similar although of the two the squeeze responses to the MORI question are slightly more favourable to Labour. 2. To keep the analysis in balance, I’ve also excluded the relatively few Con/Lab/LD respondents to the squeeze question who switched to the minor parties, because none of the companies show the number of 2010 DK/refusals who voted for the minor parties and nor do MORI and ComRes show the responses to the squeeze question for this group. 3. We can’t tell how turnout weighting affects the reallocations by all of the companies, for this group specifically.)

  20. @Phil

    Thanks very much for that clarification – you should really think about giving tutorials for people who find the tables difficult to decipher.

    Btw I like the “inclined to support” question – I’m guessing they reveal the green shoots of a few potential Con to Lab tranfers.

  21. statgeek

    @ALLAN CHRISTIE

    It was Harriet Harman (Labour), not Annabell Goldie (Conservative) who called Danny Alexander a ginger rodent.

    Everyone pokes fun at Rangers
    _______________

    I stand correct indeed it was Harriet Harman :) Mind you Goldie did say “Gleggs tend to bite” ;)

    As a Celtic fan I welcome people pocking fun at Rangers but I’m not a politician looking for votes!! :)

  22. Colin Green

    @Allan Christie

    “the Lib/Dem conference (counted 83 people) in Inverness”

    Scottish Lib Dem conference. The main one is next week in Sunderland

    ________________

    So can we expect 183 at that one? ;)

  23. @Phil

    Interesting analysis and as good an explanation as to why we’re seeing the current divergent results between different polling organisations as I’ve seen. The question it begs, however, and one that I’ve posed on these pages before, is which methodology gives the most accurate results? Of course, that’s a very different question to the one that asks which gives the results that my partisan leanings prefer!!

    I guess the only sure way of answering the key question on accuracy is to look back and see which pollster’s methodology most accurately predicted an election result on eve of poll. If I remember rightly from previous discussions, wasn’t that ICM in May 2010? If so, what method were they using in terms of re-allocating DKs, identifying shy voters etc?

    The other difficulty with polls is the + or – 3% MOE. With those statistical conditioners at play, when is a movement in a poll rating signifying genuine opinion movement and when is it just oscillating around a settled mean? The recent run of YouGov polls is lending credence to the argument that opinion is currently static and each poll is essentially telling the same story within the parameters of MOE whereas other, less frequent polls, have suggested a more significant movement of opinion in the last month.

    No doubt tonight’s YouGov will be explained away as an outlier, whoever is in the lead!!

  24. A second change to note is ComRes, who have been experimenting with how they model likely turnout in their polls, and redistributing don’t knows/won’t says. I’m not sure of the exact details yet (Andrew Hawkins suggests it is still in flux) and Andrew tells me it is making “almost no difference to the outcome” but on their last two polls it seems to have made their figures marginally more Conservative.
    —————————————
    I’m going to say the ‘unsayable’: This is b*ll*cks. IMO, Comres released a headline poll with Con & Lab the wrong way round (& the tables ComRes initially put up showed that’s what they’d done). The Indie was already running with the Con lead for 1st time since 2010 story by the time Comres realised their mistake so ComRes couldn’t retrieve the situation. Therefore they have this fudged methodology change which is still in flux or what ever blah, blah they are using to cover their blushes.

    I am being controversial because I’m bored…
    8-)

  25. Crossbatt11 – as per earlier posts Billy Bob pursuaded me that whilst ICM’s 50% DK clawback is a fair adjustment for the big 2 parties it probably isn’t for the LD.
    Populus’ 30% is more in line with the squeeze question outcome used by the other 2 pollserts mentioned.
    So ICM may well have been the most accurate just before the GE (although being closest with the last poll does not proove this in itself) but they could well be overstating LD support now in a mythical GE tomorrow.
    Anthony – my guess is that as the GE gets closer the need for squeeze or DK reallocation diminishes hence You Gov will be more accurate and their ongoing polls are good at the capture firming support as well as increasing support; is this an accurate idea?

  26. Meant to say thanks to Phil for the excellent clarity and deciphering Job.

  27. @Amber Star

    Can’t promise this will relieve the boredom:

    h
    ttp://www.ipsos-mori.com/newsevents/latestnews/1024/Can-Johann-Lamont-reverse-Labours-decline-in-Scotland.aspx

    Have to say, the more I see her on TV, the better I like her. Is she doing well in your opinion?

  28. @A Cairns

    “I thought your conference was in Gateshead/Newcastle?”

    Yes, sorry. Too many trips to Nissan in Sunderland. The spring conference is in Gateshead.

  29. CON 39%, LAB 39%, LD 8%; APP -27

  30. Oops – The YouGov is still showing Thursday’s…

  31. ROBIN.
    So still outliers for the Lib Dems

  32. Yougov

    CON 40% LAB 39% LD 9%

    From Sun’s politics page.

  33. Are we sure this isn’t a re-report of Weds? The YouGov ticker was showing Weds figures for a while, then Thurs…

  34. Well I’m certain, given I’ve got the results in front of me :)

  35. @ROBIN

    I thought that at first but on the chart it has the previous 5 polls figures for the Tories preceeding these figures.

  36. Adding mobile users accounts for unknown differences between mobile and landline users. The problem is that these differences may be the exact same differences that affect voter turnout.

    Pollsters in the last Canadian federal election low-balled the Conservatives (they thought it’d be a hung parliament instead of a Tory majority), in part because as they expanded their polling coverage, they were capturing more left-leaning but disengaged voters. I Tweeted offhandedly at the time that pollsters were capturing the public sentiment better than ballot boxes.

    http://www.theglobeandmail.com/news/politics/a-pollsters-painful-reckoning-how-could-i-have-screwed-up-so-badly/article2065573/

1 2