As the NYT’s Janet Elder points out (via Andrew Gelman), journalists are once again hyping minor differences in polling while neglecting to mention the margin of error in their results.
Here’s a great example from the Des Moines Register’s article on its new poll for the Iowa Democratic caucus (my emphasis):
Barack Obama has pulled ahead in the race for Iowa’s Democratic presidential caucuses, while the party’s national frontrunner Hillary Clinton has slipped to second in the leadoff nominating state, according to The Des Moines Register’s new Iowa Poll.
…Obama, an Illinois senator, leads for the first time in the Register’s poll as the choice of 28 percent of likely caucusgoers, up from 22 percent in October. Clinton, a New York senator, was the preferred candidate of 25 percent, down from 29 percent in the previous poll.
Former North Carolina Sen. John Edwards, who led in the Register’s May poll, held steady with 23 percent, in third place, but part of the three-way battle.
…The poll shows what has continued to be a wide gap between the top three candidates and the remainder of the field. The telephone survey of 500 likely Democratic caucusgoers was conducted Nov. 25 to 28 and has a margin of error of plus or minus 4.4 percentage points.
Contrary to the article’s lede, the margin of error on the poll means that Obama and Hillary are actually tied statistically — Obama isn’t winning!
Update 12/3 12:35 PM: Let me elaborate in response to angry comments below. Yes, given the margin of error and Obama’s lead, the odds are reasonably high that he’s ahead in a naive sense. However, his lead falls short of the standard 95% confidence threshold. While I’m not a big fan of the .95 standard, which is certainly arbitrary, in my professional work, I think it’s appropriate here to to call the race a statistical tie (meaning we can’t have much confidence in who is ahead) given both the margin of error and the uncertainties of projecting who will actually turn out. Along those lines, here’s Elder:
News organizations differ on how strictly to apply the margin of sampling error. But when looking at horse race numbers in a political poll, particularly in Iowa, with its quirky caucus system, historically low turnout (5 percent of Iowans participated in the Democratic caucus in 2004) and rules that change from one year to the next — this year Iowans can register to vote at the door on caucus night — the margin of sampling error is probably best applied in its strictest sense.
If you don’t believe me, check out the new poll (via Michael Crowley) showing Hillary at 31%, Edwards at 24%, and Obama at 20% (margin of error 6%) — numbers that should make it quite clear that the Des Moines Register numbers cannot be interpreted strictly.
In the end, the best approach is to consider all of the polls. The pollster.com compilation of all Iowa Democratic polls through 11/29 makes it clear that the race is too close to call:
Update 12/3 7:47 PM: Via this comment, I learned that the poll cited above was actually in the field before the Des Moines Register poll, not after. Here’s an LA Times blog post on the timing issue:
The Hillary Clinton presidential campaign, buffeted by the Des Moines Register’s Sunday front page trumpeting its poll that gave Barack Obama a slight lead over her (28% to 25%) among likely Democratic caucus-goers, responded today with a release spotlighting two surveys that put her ahead in the Hawkeye state.
The Associated Press/Pew Research Center poll reported Clinton backed by 31% of the likely caucus-goers, followed by Obama (26%), John Edwards (19%) and Bill Richardson (10%). Iowa State University weighed in with these numbers: Clinton, 30.8%; Edwards, 24.4%; Obama 20.2%; Richardson, 11.4%.
In all three of the surveys, the advantage for the leader is within the margin of error. So basically, they all confirm one obvious point — the Democratic race in Iowa is very tight and very fluid.
A closer look at the polls, however, reveals a potentially key difference between the Register’s survey and the other two: timeliness.
The newspaper’s poll was conducted from Nov. 25 (Sunday a week ago) through last Thursday. The AP/Pew survey was conducted Nov. 7-25, while the Iowa State poll was in the field Nov. 6-18.
Pollsters like to refer to their findings as "snapshots in time." The AP/Pew and Iowa State polls strike us as a bit lengthy in the development stage.
I’ll leave the technical details of the statistics debate to the comments.