There are bucketloads of American websites that collate opinion polls, comment on the latest figures and try to predict election results. Rather than try to do any analysis myself, here is a round up of the some of the best links for those of you interested in following the US polls.
Polling Report is the best known US polls site, but you’ll have to pay to see any of the state level polling
Mark Blumenthal, late of mysterypollster.com, now blogs at Pollster.com…
as does Professor Charles Franklin, who also maintains his own blog at Political Arithmatik. Pollster.com also has graphs of all the recent state and district polls for the mid-term elections.
Crosstabs is another US polling blog, written by several professional pollsters
Electoral-Vote is a daily projection of the results based on the latest polling data (and like Pollster, has graphs of the latest polls for each race).
Majority Watch is a project conducted by two polling companies, Constituent Dynamics and RT Strategies, which is regularly conducted automated telephone polls in the most contested seats in the House of Representatives to create a rolling prediction of the result.
ICM/NUS - 74% of people (in general, not potential students) think that the rising cost of getting a degree will put people off applying.
ICM/Department of Education - proportion of people thinking that education is getting better has fallen over the last year, discipline is the worst problem and the overwhelming majority of people think is it getting worse. 56% think school food is getting better.
YouGov/Telegraph - the rest of YouGov’s monthly poll for the Telegraph suggests the public think there is no clear idea what we are doing in Iraq and Afghanistan, we can’t win and we should withdraw.
Populus/Channel 4 - on British Muslims and freedom of speech. 62% of people think the West should be more respectful of Islamic traditions and beliefs, but 58% say we should be allowed to say anything we like within the law even if it does offend Muslims. 53% think protests by Muslim extremists are threatening freedom of speech in this country.
Populus/BBC - 67% think Madonna should be allowed to adopt David Banda
Populus/BBC - 62% think faith schools in general are divisive, 47% think that Christian schools are fine, but Muslims schools would be a problem. 38% think all religion should be kept out of schools, with 61% disagreeing.
ICM/Observer - apparantly 96% of us are ardent recyclers…or it’s a classic example of an “are you a heartless bastard?” question. In voting polls questions are always couched in words like “At the general election in May last year, many people didn’t vote. If you did vote in the general election could you tell me which party you voted for?” to try and make it as easy as possible for people to admit to doing things that might be seen as socially irresponsible, such as not being bothered to vote. If you want realistic answers the same sort of thing should be done on questions asking people about how ethical their own behaviour is towards the environment.
There is somewhat less confusion about the actual state of the parties with YouGov’s monthly poll, which tallies with the sort of lead produced by ICM and Communicate during the past week. The topline figures with changes from YouGov’s last poll are CON 39% (+3), LAB 32% (-4), LDEM 16% (nc). YouGov’s last poll was conducted immediately after the Labour conference and was probaby the result of a “Blair boost” from his conference speech. Rather than a sudden Conservative boost, this poll looks more like a return to the sort of figures YouGov had been producing prior to the Conference season.
On a forced choice question asking, if they had to choose, whether people would prefer a Labour government led by Gordon Brown or a Conservative government led by David Cameron, Cameron now leads Gordon Brown by 46% to 33%, a thirteen point lead. This compares to a lead of 7 point lead when the same question was asked in September and a 17 point Labour lead at the last election. When YouGov release the full tables for this poll it will be interesting to see how Lib Dem voters split on the question - I’ve always viewed forced choice questions like this as a potential pointer to the way tactical voting may play in the future.
In the last week we’ve had one poll putting Labour at their lowest level since 1987, and another poll giving Labour their largest lead since April. Earlier in the week David Cameron was being asked on the radio why his party was 2 points behind, now a poll shows him 10 points ahead? That’s a 12 point difference in the lead between polls just a week apart. So, whats the real picture?
There are three basic reasons why different polls produce different figures. Firstly the actually level of party support can change, the actual picture might have changed between them. Secondly there are the methodological differences between the pollsters, the way they draw their samples, the way they weight them, the way they take account of turnout and so on. Thirdly there is sample error - the so called “margin of error”.
Underlying party support in Britain is pretty stable - if you graph opinion polls over the last couple of years it’s clear that the only events that have actually shifted political support have been David Cameron’s election as Conservative leader, which moved the Labour and Conservative parties pretty much neck and neck, and the week prior to the local elections when Labour was hit by both John Prescott’s affair and the foreign prisoner release scandal, since when the Conservatives have had a consistent lead.
However, while the underlying picture is quite stable, political events do have a short term effect. For example, Tony Blair’s conference address this year saw his personal ratings shoot up and this was reflected by several voting intention polls that showed a dramatic drop in the Conservative lead.
These short term shocks can explain part of the difference - MORI’s poll was conducted in the middle of October, and was perhaps affected by the tail end of Labour’s fading conference boost. ICM’s poll was conducted when the Conservatives were getting a lot of publicity over the report of their policy commission on taxation, which in theory could have given them a boost.
The second factor is the pollsters’ “house bias”. By that I don’t mean that any of the pollsters actually have a partisan bias, but that the differences in their methodology, the differences in the way they draw their samples, the way they weight them, the different approaches to turnout and so on inevitably have a partisan impact on the figures they produce. If you take an average of all the comparable polls since the last election (i.e. excluding polls outside the regular monthly tracker polls, and those from months when not all four companies produced polls) you get the following figures:
YouGov CON 35.8%, LAB 36.5%, LDEM 17.8% (average Labour lead 0.7%)
Ipsos-MORI CON 35.1%, LAB 36.3%, LDEM 20.8% (average Labour lead 1.2%)
Populus CON 34.5%, LAB 36.2%, LDEM 19.5% (average Labour lead 1.7%)
ICM CON 35.2%, LAB 35.5%, LDEM 20.8% (average Labour lead 0.3%)
Despite all the major differences in methodology, on average there isn’t a great difference in the results the different pollsters produce. The main difference in the reported level of Lib Dem vote, with on average 3 points difference between the lowest (YouGov) and the highest (ICM and MORI), with Populus somewhere inbetween. In terms of Conservative and Labour support, there really isn’t a vast difference between the companies.
While MORI’s last two polls have reported Labour leads in amongst polls by other companies showing Conservative leads, looking at these average figures it’s clear that MORI’s methodology does not produce figures that consistently favour Labour (or at least, hasn’t in the past. There is no guarantee that different methodologies won’t react differently to changed situations). On average ICM produce polls that show a Labour lead around 1 point lower than MORI’s polls, so while 1 point of that 12 point difference might be down to MORI’s methodology being more favour to Labour than ICM’s, it’s only a small part of the picture.
That leaves us with sample error. Opinion polls are generally quoted with a margin of error of 3% or thereabouts. In reality this a polite fiction. The formula used to calculate the margin of error is based on a genuine random sample, in reality samples are a long way from truly random. RDD phone samples used by ICM and Populus exclude people who don’t have a landline, and the 5/6 people who don’t answer or refuse to take part. MORI and YouGov polls don’t use random sampling at all, instead using variations on quota sampling. Add to that the effect of weighting, adjustments and so on. The reality of opinion polling doesn’t bear much resemblence to the mathematical theory.
In reality the margins of error are probably somewhat larger than the statistical formulas suggest*, and for whatever reason, MORI’s polls are slightly more variable than their rivals. Two polls with a margin of error of 3% are almost enough to cover the 8 point difference between ICM’s Labour score and MORI’s Labour score anyway, once you’ve considered that MORI’s real margin of error will be larger than that, potentially sample error alone could explain the whole thing.
So, looking back at the recent polls, ICM have Labour at 29%, MORI have them at 37% - a difference of 8%. Take into account the “house bias” of the two pollsters and the difference comes down to 7%. The fading of Labour’s conference boost and the Conservative publicity boost from their tax proposals could easily have shifted support by a couple of points, leaving us with 5 percentage points or so to explain using sample error. The actual levels of party support are therefore probably somewhere inbetween the two polls, but given that ICM is less volatile than MORI, they are probably somewhat closer to ICM than to MORI. Over the next few days we will have the monthly polls from both YouGov and Populus, which - touch wood - will give us a better idea how the ground really lies.
(*Can we estimate the actual margin of error based on the observed variability of the polls, rather then the theoretical formulas? We can’t take the actual deviation from the average, because polls are taken weeks apart and the actual levels of support that the polls are deviating from is moving as political support shifts. I am not a mathematician, but the best estimate I can come up with is if we adjust the figures to remove the house biases, and then measure the deviation from a rolling average of the adjusted polls (I used a rolling average of the last 8 polls for these calculations). These give a standard deviation of 1.35 for YouGov, 1.62 for ICM, 1.67 for Populus and 2.44 for MORI meaning that, assuming a normal distribution, the actual “margins of error” (the 95% confidence intervals) for the polls are about 2.7% for YouGov, 3.2% for ICM, 3.3% for Populus and 4.9% for MORI.)
ICM’s latest voting intention poll has the lowest level of Labour support since before the 1987 election (according to the Guardian it equals the lowest ever rating by ICM, back in May 1987), and puts them below the symbolic 30% level. It is the largest Conservative lead recorded by ICM since May 1992. The full topline figures, with changes from ICM’s last poll in the Sunday Telegraph, are CON 39%(+1), LAB 29%(-3), LDEM 22%(+2).
On a uniform swing this level of support would still leave the Conservatives 9 seats short of an overall majority, though in practice it is likely that such a reverse in the positions of the parties would be accompanied by changes in the pattern of tactical voting, meaning that the Conservatives might well get an overall majority.
ICM also asked a series of questions about people’s views of the NHS. 55% of people stil lthink the NHS is the envy of the world and 60% think it would give them excellent care if they were unwell. 71% of people say their family and friends have had a good experience with NHS care. Despite all this, the majority (67%) of people would go private instead if they had the money (including 61% of Labour voters). People seem to be broadly positive towards increased privatye sector involvement in the NHS - 70% of all voters think that private companies should be allowed to care for NHS patients.
Asked how well Labour have cared for the NHS, 25% of people think the NHS has improved under Labour, but 30% think it has got worse. 72% think a lot of the extra money spent on the NHS has been spent badly, with only 14% thinking it has been spent well.
A Communicate Research poll in today’s Independent gives the Conservatives an 6 point lead over Labour. The full topline figures are CON 38%, LAB 32%, LDEM 14%, Others 16%.
It is Communicate’s first voting intention poll since back in 2005, so there no changes from the last poll and it is impossible to say if there are any trends in voting intentions, although it goes without saying that the poll doesn’t appear to support MORI’s figures yesterday.
Communicate’s methodology has in the past produced figures slightly more favourable to Labour, in theory at least - in practice there wasn’t much difference between them and the other pollsters. ICM and Populus both weight by past vote, which tends to make samples “more Conservative”; Communicate don’t. In the distant past Communicate also didn’t prompt by party name, which used to favour Labour at the expence of the Lib Dems. That changed prior to the last election, so there is now no obvious methodological explanation for the low level of Liberal Democrat support.
The most obvious difference in Communicate methodology is that they use a very harsh squeeze question. Those who say they don’t know how they will vote are asked how they would vote if it was a legal requirement to do so. There has also been a slight change in the way they deal with turnout - Communicate now just filter by likelihood of voting, including those who rate themselves as 5/10 likely to vote or higher. In the past they also used to weight by likelihood, so people who said they were 5/10 likely to vote counted only half as much as someone 10/10. The likelihood weighting has been abandoned, the result of which is to slightly reduce Conservative leads.
Communicate also asked about attitudes to Iraq. 72% of people now think that the war in Iraq is unwinnable. 62% of people think the UK should withdraw as soon as possible from Iraq, regardless of whether it is completely stable. Only 28% favour staying until Iaq is stable. An ICM poll in the Guardian produced alsmost identical findings - 61% wanted troops withdrawn this year, with only 30% wanting them to stay for as long as necessary.
Still to come this week, there should be ICM voting intention figures tomorrow, and YouGov voting intention figures on Friday.
Ahead of their normal monthly poll later in the week, YouGov have separate poll in today’s Telegraph asking about taxation (presumably commissioned in response to the Conservatives’ policy commission on tax). Asked which party would handle taxation the best, the Conservatives lead Labour slightly, 25% to 20%. This is lower than at the last election, but the difference is mostly because of a huge leap in don’t knows, from 22% in 2005 to 45% now, mostly at the expense of the Liberal Democrats who are down to 8% from 19% last time. Obviously this could be lots of disillusionned supporters of a 50p top rate who are spurning the Lib Dems, but I expect the explanation is rather less exciting and rather more straight forward. The Tories haven’t put forward any plans and the Lib Dems no longer have an easily communicated plan, so most people genuinely don’t know.
Overall slightly more people (43%) agree with the statement “large scale tax cuts would make little difference to the rate of economic growth and would benefit those who are already well off at the expense of everyone else” than agree with the statement “large scale tax cuts would provide incentives for individuals and businesses and increase the rate of economic growth, thereby helping everyone” (36%). Given the choice 25% of people say they would like to see taxes cut, even if it did mean a reduction in government services, 23% of people are happy as things are and 34% would like to see government services extended even if it meant tax rises. Interestingly, even a plurality of Conservative voters (43%) would like to see the level of tax and spending stay the same or rise, with 41% of Tory voters saying they would rather both tax and spending fell.
People being positive about high taxes does not, of course, mean they are positive about paying more taxes themselves. Opinion polls invariably show that people would like people richer than themselves to pay more taxes so they can pay less. 69% of people told YouGov they thought the present tax system was unfair and 61% though they personally paid too much in tax (only 1% thought they paid too little). 65% thought that poor people paid too much in tax, 51% thought that people on middle incomes paid too much, and 54% thought that “rich people” paid too little.
Looking specifically at the Conservative party’s position on tax, more people do say they would more likely to vote Conservative if they committed themselves to cutting taxes (28%) than say it would make them less likely (14%). (As any long time readers will know, I’m not a fan of questions like this. I think people know full well the context of the question and how it will be presented in the press and use it to say whether they approve or disapprove of a policy, even if they are actually a hardcore voter who it will make no difference to. Normally YouGov minimise this problem by giving options of “no difference, would vote Tory anyway” and “no difference, wouldn’t vote Tory anyway” but this question hasn’t even got that).
The mantra of the present party leadership in saying they will put economic stability before tax cuts seems to resonate with voters - 69% of people say they would put economic stability first, as would 73% of Tory voters (though it begs the question of how many people would actally think “to hell with economic stability, give me tax cuts!” I suspect many of the 17% of people who did say that did so because they thought that there was no dichotomy, and tax cuts would not endanger economic stability).
However, the vulnerable position the party would put themselves in if they moved towards adopting Lord Forsyth’s recommends is very clear. Last weeks announcement was followed by accusations from Labour that the figures did not add up and cuts would have to be made. Asked if £21 billion could be cut from taxes without damaging the quality of public services only 36% agreed, with 45% disagreeing. Whether or not the savings could be made, it is obvious that a large proportion of the electorate would be very receptive to accusations from other parties that tax cuts would automatically mean damaging public services.
YouGov also asked about some specific tax cuts and rises. The most popular tax cut was a reduction in the basic rate of income tax, followed by stamp duty on houses and inheritance tax. There was very little support for a cut in corporation tax and the abolition of stamp duty on share transactions - cuts which would either not directly benefit individuals to a noticable extent. On transport tax there was strong opposition to increasing fuel duty for cars…but support for increased taxation of flights. 51% of people supported tax on short haul flights and 52% supported it on long-haul flights.
Apart from the airline taxes, which are an unusual case of people supporting a tax rise they themselves might have to pay, the overall picture is the same one that nearly always turns up in surveys on taxation. Most people do not think tax cuts can be made without spending cuts, people tend to favour more spending, funded by higher taxes on people richer than they themselves are, asked about themselves personally, people would like to pay less tax, presumably, since they don’t favour spending cuts, funded by higher taxes on people who are richer than themselves.
Do people prefer Gordon Brown to David Cameron? The answer you get depends on who you ask. There’s a Populus poll mentioned in the Times today (conducted for Opinion Leader Forum - who they?) that claims amongst “swing voters” Gordon Brown would be preferred as Prime Minister to David Cameron by 51% to 24%. I would look very carefully at it before drawing too many conclusions - exactly how are those “swing voters” defined? The Times says they are “people who are inclined to vote Labour at present, but say there is a fair chance of them going to another party, and those currently inclined to vote for another party or unsure about which one to vote for, but who say there is a fair chance of switching to Labour.”
In other words, assuming the Times have accurately described the sub-sample, it is skewed towards Labour - Populus’s normal definition of swing voters is those people who say they might yet change their minds. This does not seem to be that - this sub-sample would include, for example those who aren’t sure but might vote Labour, but not those who aren’t sure but might vote Conservative. It would include Lab/LD waverers, but not Con/LD waverers, etc, etc. The way the Times describes it there are more Labour sympathisers in the sub-sample than Conservative ones, so it is hardly earth-shattering to find they prefer Brown to Cameron.
That doesn’t change the fact that these are largely the people who will decide the next election. People switching between the Lib Dems and Conservatives and not voting at all matter too, but when it comes to the effect of Gordon Brown becoming PM it is the people who might move towards or away from Labour that count. Without a better understanding of the parameters of the sub-sample though it’s hard to say whether it is actually good news for Gordon Brown or not. If those 242 people are mostly current Labour supporters who might yet waver, then the finding that 41% of them think that David Cameron would be as good or better than Gordon Brown could in fact be a worrying finding for Brown. If, on the other hand, a large proportion of those people are not currently Labour supporters, but are other parties waverers (especially if they are Conservatives waverers) then it could be very good news for Gordon Brown. From the information published in the Times we simply don’t know.
(Just for the record, since it doesn’t make that much difference to a 27 point lead, the Times also says “it is statistically a small sample; the findings therefore have to allow for a margin of error of at least plus or minus 3 per cent.” 242 people is indeed a very small sample. It equates to a margin of error of at least plus or minus SIX per cent. Still, even an inaccurate caveat is a step in the right direction!)
This month’s MORI poll has also finally surfaced, this time in the FT, the full voting intentions are not there, only Conservative and Labour. With changes from MORI’s last poll in early September, they are CON 35% (nc), LAB 37% (+1). It’s an intriguing poll. The figures are pretty much unchanged from MORI’s last poll…but MORI’s last poll looked wildly out of place and was seen as a rogue poll, surrounded as it was by Populus and YouGov polls showing Conservative leads of 4 and 8 points respectively. Maybe it wasn’t a rogue poll after all and MORI’s different methodology is just producing divergent figures, or maybe the last poll was a rogue, but this one is picking up a genuine increase in Labour support - perhaps from the veil comments. ICM and YouGov’s monthly polls are due this week, so we’ll soon find out.
Despite the ongoing media focus on the story, there has been very little polling on attitudes towards Muslim women wearing the veil. Now up on MORI’s website is the most detailed questionaire we have so far, undertaken for ITV’s Tonight with Trevor MacDonald.
51% of people thought that Straw’s comments would damage race relations in Britain - but they thought he was right to say them. The poll found strong support for Jack Straw’s comments - 59% of people thought he was right to do so, and 58% thought the veil was a visible statement of separation and difference. However, as with ICM’s poll last weekend while they agreed with Straw’s comments, there was very little appetite for actually legislating against the wearing of veils - 77% thought it was womens’ right to chose to do so, with only 16% disagreeing.
While agreeing in principle with Muslim women being allowed to wear the while, when it came to specific cases there was less support. 60% of people thought that schoolchildren should be allowed to wear Islamic clothing (although the question mentioned headscarves and did not specifically mention the veil) with only 33% opposed. 52% of people said they would be happy to see a doctor who wore the veil (42% were opposed). Only 35% thought teachers should be allowed to wear a veil, with 54% opposed. 32% thought television newsreaders wearing a veil would be acceptable (59% were opposed). The strongest opposition was to the idea of a Muslim policewomen wearing a veil, only 29% of people thought this should be allowed, with 61% opposed.
Given its potential to attract hordes of nutcases to my comments section, I have been steadfastly ignoring the Lancet’s report on the increased mortality rate in Iraq since the invasion. Since Peter Cairns has asked me about it in the comments section below and I have been tagged by Matthew Turner I suppose I must reluctantly enter the fray. Thanks Peter & Matthew 
The study in Iraq was, for all intents and purposes, an opinion poll. The researchers from Johns Hopkins University sent interviewers to a sample of houses in Iraq and asked them how many people had died in that house since the invasion (they also asked how many people had died for a prior before the invasion, to act as a baseline of “normal” mortality). This allowed them to say that x% of people had died in Iraq over the period, therefore given Iraq’s total population the “excess” deaths amounted to about 655,000 people.
Obviously such a suggestion is politically charged, especially since it conflicts strongly with the Iraqi government’s official figures and the figures from third parties like Iraq Body Count. In my own view this doesn’t matter - the Iraq government has good reason to downplay casualties and is governing a country in a state approaching civil war where much of the government infrastructure has broken down. Why Iraq Body Count was ever viewed as anything other than a minimum number of casulties is beyond me - I have not the slightest reason to think that there are not vast numbers of deaths in Iraq that are not reported in two separate English language media sources (something I’ve often pondered doing is taking the UK murder statistics, and seeing if I can track down newspaper reports for every single murder in a given year. My guess is that every single killing in a media-heavy, highly developed state like the UK is reported in the local press at the very least, but it would be interesting to actually check). I wouldn’t regard official figures from a country in a state of chaos, or a count of deaths mentioned in the media as being in any sense reliable enough to use to gauge the accuracy of a study that at least attempts to be a rigorous scientific inquiry.
So, to the study itself. I am not going to explain the basic logic of extrapolating findings based on a small sample to a larger population here, though some of the dismissals have been based on the fallacy that you cannot extrapolate 629 deaths up to 655,000. The method of cluster sampling used is well established, has been successfully used in the past. In short, in the same way that we can accurately predict an election result to within a few percentage points based on interviewing just 1,000 people, we can estimate how many people have died in the whole country by seeing how many people died in a smaller group of residences and factoring up. If polls work, and they do, then so should this, despite it being a relatively small number. Unless something has gone wrong with the methodology of the survey the chances of the researchers just happening to have knocked on doors with a disproportionate number of deaths that do not reflect the country as a whole, and of there actually being fewer than 392,979 excess deaths is only 2.5% (the same chance that there have been more than 942,636 excess deaths).
So, what could have gone wrong? The more excitable fringes of the US blogosphere have come out with some interesting stuff. Let’s look at criticisms that don’t hold water first.
Firstly, the turnout is unbelievably high. The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that - incredibly - response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq - they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.
Secondly, people have been understandably confused by the mention of death certificates. Whenever possible interviewers asked if they could see the death certificate of people reported dead during the study. In 92% of cases those asked produced the certificate. This presents an apparant discrepancy - if over 80% of the deaths had been officially recorded, how come official Iraqi estimates of the dead were so low? The explanation given by the report - which seems perfectly reasonable - is that hospitals have continued to issue death certificates, but the system of collating the figures centrally has broken down to a large extent. In other words, a doctor in Iraq may still be giving out the paper certificates, but the figures are not necessarily passed on or registered with any higher authority.
Thirdly, some people have pondered whether Iraq’s mortality rate from before the invasion as determined by the study seems unfeasibly low at 5.5 per 1000. This compares to mortality figures of 10.1 for the European Union, a group of far more developed countries with better nutrition and health care. If Iraq’s pre-invasion mortality figure is artifically low, then it would wrongly inflate the number of excess deaths. However, the difference is actually because Iraq has a far younger population than the EU. Apart from countries in Southern Africa where AIDS is endemic, developed countries tend to have a higher mortality rate because they have more elderly people in proportion to young people, and an old person in a “safe” country is still more likely to die than a young fit person in an “unsafe” country. It seems that 5.3 is a perfectly reasonable figure when compared to mortality rates for similar countries like Egypt (5.2), Iran (5.6), Tunisia (5.1), Syria (4.8), Qatar (4.7), Bahrain (4.1).
Moving on, some people have raised more substantial concerns. Firstly, two of Iraq’s governates were not sampled because of errors, these areas were in the extreme North and South of the country, away from the Sunni dominated centre where the violence has been worst. It seems reasonable to assume that these two areas are likely to have reported a low casualty rate, and therefore the remaining figures are artificially high. That may be so, but they contain only 5% of the population so even if they had a very low casualty rate indeed the effect would be very small. The headline figure of 655,000 extra deaths was based on the population excluding those areas, so is still sturdy.
Secondly, the populations of the governates were estimated using 2004 figures, if there has been substantial population movements since then it could skew the figures. Again, this is a legitimate concern, but population movements would have to be very large to make a truly significant difference.
Thirdly, several people have pondered the “word of mouth” effect. The researches state in their report that having explained to the first house in a cluster their good intentions, word of mouth travelled ahead of them and made it easier to presuade the rest of the cluster of their good intentions. Some people have, quite reasonably, asked whether this could skew the result - could people with deaths in the family have become more or less likely to take part in the survey? In theory yes, they could, but given the response rate of 98% there is very little space for it to have made a difference. If it made people with deaths more likely to take part, they are 98% likely to have done so anyway. If if made them less likely to take part, it obviously didn’t have much effect.
Fourthly, interviewers had some leeway to change the area they were interviewing if the designated area was too dangerous. This could result in bias, but it seems more likely that it would lead to an underestimate in the number of deaths, as interviewers avoided the sort of area where lots of people get killed. Of course, interviewers could avoid a safer area because they had to travel through a dangerous area to get to it, but it still seems unlikely that a systemic bias of this sort could overall lead to a upwards bias in the number of deaths.
Finally, there have been questions over whether the sampling technique was biased towards urban areas. This is the most substantial problem in my view. The way the location of clusters was determined was thus - first the governate was determined, then a main road in the province was randomly selected, then a road leading off of that main road was randomly selected, and then a house on that road. It seems to be that this approach should skew the location of clusters towards urban areas, and make it less likely that rural areas and areas with informal housing like refugee camps would be selected. If violence is concentrated in urban areas this could skew the sample and give an artifically high number of casualties.
Overall, the study seems sound. There are some legitimate questions about the effect of the two missing provinces and any large population movements, but at the end of the day they would have quite minor effects on the total: they are not suddenly going to bring the figures into line with the Iraqi government figures or the Iraq Body Count figures. The possible effect of an urban bias is more worrying, potentially this could skew the figures upwards. That said, 77% of Iraq’s population live in urban areas, so even if there is a systemic bias here, in a worst case scenario of non-urban areas being entirely missed out and 23% of the country actually having a much lower mortality rate, it is not going to be a drastic change. For example - the report found the post-election death rate to be 13.2, if that actually applied only to urban areas, and the mortality rate in non-urban areas was still 5.5 (the pre-invasion figure), the overall rate would still be 11.4, which still equates to hundreds of thousands of extra deaths.
The study is based on a survey in a country in a state of near civil war and without accurate population estimates, it is not a perfect situation to be working in and obviously there are going to be question marks here and there. Surveys in developed countries aren’t perfect, let alone in war zones. The bottom line though is that the study suggests that the increase in deaths since the invasion is indeed far higher than estimates from other sources suggest.