Brendan Nyhan

As the NYT’s Janet Elder points out (via Andrew Gelman), journalists are once again hyping minor differences in polling while neglecting to mention the margin of error in their results.

Here’s a great example from the Des Moines Register’s article on its new poll for the Iowa Democratic caucus (my emphasis):

Barack Obama has pulled ahead in the race for Iowa’s Democratic presidential caucuses, while the party’s national frontrunner Hillary Clinton has slipped to second in the leadoff nominating state, according to The Des Moines Register’s new Iowa Poll.

…Obama, an Illinois senator, leads for the first time in the Register’s poll as the choice of 28 percent of likely caucusgoers, up from 22 percent in October. Clinton, a New York senator, was the preferred candidate of 25 percent, down from 29 percent in the previous poll.

Former North Carolina Sen. John Edwards, who led in the Register’s May poll, held steady with 23 percent, in third place, but part of the three-way battle.

…The poll shows what has continued to be a wide gap between the top three candidates and the remainder of the field. The telephone survey of 500 likely Democratic caucusgoers was conducted Nov. 25 to 28 and has a margin of error of plus or minus 4.4 percentage points.

Contrary to the article’s lede, the margin of error on the poll means that Obama and Hillary are actually tied statistically — Obama isn’t winning!

Update 12/3 12:35 PM: Let me elaborate in response to angry comments below. Yes, given the margin of error and Obama’s lead, the odds are reasonably high that he’s ahead in a naive sense. However, his lead falls short of the standard 95% confidence threshold. While I’m not a big fan of the .95 standard, which is certainly arbitrary, in my professional work, I think it’s appropriate here to to call the race a statistical tie (meaning we can’t have much confidence in who is ahead) given both the margin of error and the uncertainties of projecting who will actually turn out. Along those lines, here’s Elder:

News organizations differ on how strictly to apply the margin of sampling error. But when looking at horse race numbers in a political poll, particularly in Iowa, with its quirky caucus system, historically low turnout (5 percent of Iowans participated in the Democratic caucus in 2004) and rules that change from one year to the next — this year Iowans can register to vote at the door on caucus night — the margin of sampling error is probably best applied in its strictest sense.

If you don’t believe me, check out the new poll (via Michael Crowley) showing Hillary at 31%, Edwards at 24%, and Obama at 20% (margin of error 6%) — numbers that should make it quite clear that the Des Moines Register numbers cannot be interpreted strictly.

In the end, the best approach is to consider all of the polls. The pollster.com compilation of all Iowa Democratic polls through 11/29 makes it clear that the race is too close to call:

Update 12/3 7:47 PM: Via this comment, I learned that the poll cited above was actually in the field before the Des Moines Register poll, not after. Here’s an LA Times blog post on the timing issue:

The Hillary Clinton presidential campaign, buffeted by the Des Moines Register’s Sunday front page trumpeting its poll that gave Barack Obama a slight lead over her (28% to 25%) among likely Democratic caucus-goers, responded today with a release spotlighting two surveys that put her ahead in the Hawkeye state.

The Associated Press/Pew Research Center poll reported Clinton backed by 31% of the likely caucus-goers, followed by Obama (26%), John Edwards (19%) and Bill Richardson (10%). Iowa State University weighed in with these numbers: Clinton, 30.8%; Edwards, 24.4%; Obama 20.2%; Richardson, 11.4%.

In all three of the surveys, the advantage for the leader is within the margin of error. So basically, they all confirm one obvious point — the Democratic race in Iowa is very tight and very fluid.

A closer look at the polls, however, reveals a potentially key difference between the Register’s survey and the other two: timeliness.

The newspaper’s poll was conducted from Nov. 25 (Sunday a week ago) through last Thursday. The AP/Pew survey was conducted Nov. 7-25, while the Iowa State poll was in the field Nov. 6-18.

Pollsters like to refer to their findings as "snapshots in time." The AP/Pew and Iowa State polls strike us as a bit lengthy in the development stage.

I’ll leave the technical details of the statistics debate to the comments.

A few weeks ago, I noted that Barack Obama had apparently taken his goo-goo tendencies to new extremes by promising that “[w]hen government officials meet with corporate lobbyists, you should be able to watch the meeting” in an email to supporters (PDF).

At the time, I questioned whether the line was vetted policy, but today’s New York Times includes an article on candidate visits to Google that confirms Obama’s proposal:

With his Google visit, however, Senator Obama succeeded in drawing attention to his plans for using technology to make government more accessible and transparent with, for example, live Internet feeds of all executive branch department and agency meetings.

While the plan sounds high-tech and provides an effective political contrast to Hillary Clinton’s closed-door health care process in 1993-1994, it makes no sense from a management perspective. Most government meetings are already a waste of time, but making them public would completely destroy any hope of real business being done. Has anyone seen footage of the ridiculous Cabinet meetings that take place in front of the cameras? They’re bad political theater where nothing of substance is accomplished. Why would we want every department and agency meeting to be stripped of the possibility for candor or frank discussion? The proposal is disastrous and should be seen as an ugly reminder of Obama’s lack of executive and management experience.

Update 12/3/07 12:52 PM: Mickey Kaus piles on:

Brendan Nyhan on the Carteresque silliness of Obama’s idea of providing (in the NYT‘s description) “live Internet feeds of all executive branch department and agency meetings.” … For a good example of how idealistic “open meeting” laws can gum up government by forcing officials to endure hours of for-show public meetings with grandstanding interest groups before actually getting down to business, read Lynn and Whitman’s terrific account of Carter’s welfare-reform failure. … P.S.: How about live internet feeds of all Obama staff meetings? The voters of Iowa deserve no less! …

Clark Hoyt, the former Knight Ridder reporter and editor (now McClatchy), is living up to his promise as public editor of the New York Times. Today he takes the paper to task for failing to fact-check candidate claims in a timely fashion — one of the reasons that McClatchy’s coverage is often so much better than that of the Times:

Last Monday’s Times reported that Rudolph Giuliani had accused Mitt Romney of having a bad record on crime while governor of Massachusetts.

“Violent crime and murder went up when he was governor,” Giuliani said of his Republican rival.

In time-honored journalistic fashion, the newspaper noted the Romney campaign’s response: No, violent crime, which includes murder, actually went down during Romney’s tenure.

If you were like me, you wondered, impatiently, why the newspaper didn’t answer a simple question: who is telling the truth? I wanted the facts, and, not for the first time, The Times let me down.

My colleague Michael McElroy came up with the facts that morning after a 10-minute check of F.B.I. statistics readily available on the Internet. Murder in Massachusetts did go up in the four years Romney was governor, from 173 in 2002, the year before he took office, to 186 in 2006, the last full year of his term. An increase of 13 murders may not seem like a crime wave in a state with a population of 6.4 million, but an increase is an increase, so Giuliani was right on that point.

But violent crime, a broader category made up of murder, rape, robbery and assault, went down in the Romney era, from 31,137 to 28,775, so Giuliani was wrong on that score and the Romney campaign was right, though it failed to mention that robberies had also increased.

After noting some effective fact-checking by Times reporters, Hoyt takes inventory of the failures:

The Times has also missed opportunities to set the record straight. When Karl Rove, on his way out of the White House as President Bush’s chief political adviser, took a shot at Hillary Clinton as “weak” on national security and said she opposed the USA Patriot Act, The Times failed to note that Senator Clinton had in fact voted twice for the act. When John Edwards said he would use his power as president to take health coverage away from Congress if it failed to pass universal coverage for all Americans, the newspaper did not note that presidents do not have that authority.

To be most useful, fact-checking needs to be timely. In October, Giuliani incorrectly claimed that the prostate cancer survival rate in England, under the “socialized medicine” he falsely implied Democrats favor, was only 44 percent, compared with 82 percent in the United States. The Times initially said the number for England was “in dispute,” though it provided all the necessary information for a reader to conclude it was wrong. It wasn’t until Friday that the newspaper declared the statistic a “false statement.”

He also calls on the Times to devote more resources to fact-checking, which would be fantastic:

But in this Internet age, with its instant news cycle, The Times is falling behind major competitors. The Washington Post started The Fact Checker on its Web site in mid-September. The fact checker is Michael Dobbs, a distinguished foreign and diplomatic correspondent who came out of retirement to start the feature…

The St. Petersburg Times and its sister publication, Congressional Quarterly, have started PolitiFact.com, a site that uses a “Truth-O-Meter”…

The pioneer online fact-checking operation, FactCheck.org, was started in 2003 by Brooks Jackson, a veteran investigative reporter, for the Annenberg Public Policy Center at the University of Pennsylvania. At its peak during the 2004 presidential election, FactCheck.org drew up to 400,000 unique visitors a day, a strong indication of the public’s appetite for help in sorting through the claims of candidates.

The Times, with its own rich Web offerings on the presidential campaign, would do well to showcase a similar fact-checking feature.

However, I do have to take issue with the claim that FactCheck.org was “[t]he pioneer online fact-checking operation” — a scrappy little independent website called Spinsanity was started back in 2001.

Via Henry Farrell, the lead article in the new American Political Science Review is Diana C. Mutz, “Effects of “In-Your-Face” Television Discourse on Perceptions of a Legitimate Opposition” (PDF):

Abstract: How do Americans acquire the impression that their political foes have some understandable basis for their views, and thus represent a legitimate opposition? How do they come to believe that reasonable people may disagree on any given political controversy? Given that few people talk regularly to those of opposing perspectives, some theorize that mass media, and television in particular, serve as an important source of exposure to the rationales for oppositional views. A series of experimental studies suggests that television does, indeed, have the capacity to encourage greater awareness of oppositional perspectives. However, common characteristics of televised political discourse—–incivility and close-up camera perspectives—–cause audiences to view oppositional perspectives as less legitimate than they would have otherwise. I discuss the broader implications of these findings for assessments of the impact of television on the political process, and for the perspective that televised political discourse provides on oppositional political views.

In her experiments, Mutz filmed actors debating from two perspectives — close-up and medium distance — and also varied interruptions and agitation, while keeping the script and debaters constant. This design allows her to show convincingly that incivility and close-up imagery, which are the defining characteristics of cable talk shows, increase subjects’ perceptions that the other side’s views are illegitimate. It’s a clever study that provides scientific confirmation of an important fear about the political consequences of shout TV. Highly recommended.

The New York Times has an impressively critical article on Rudy Giuliani’s misleading use of statistics in today’s edition. Here’s how it begins:

In almost every appearance as he campaigns for the Republican presidential nomination, Rudolph W. Giuliani cites a fusillade of statistics and facts to make his arguments about his successes in running New York City and the merits of his views.

Discussing his crime-fighting success as mayor, Mr. Giuliani told a television interviewer that New York was “the only city in America that has reduced crime every single year since 1994.” In New Hampshire this week, he told a public forum that when he became mayor in 1994, New York “had been averaging like 1,800, 1,900 murders for almost 30 years.” When a recent Republican debate turned to the question of fiscal responsibility, he boasted that “under me, spending went down by 7 percent.”

All of these statements are incomplete, exaggerated or just plain wrong. And while, to be sure, all candidates use misleading statistics from time to time, Mr. Giuliani has made statistics a central part of his candidacy as he campaigns on his record.

For instance, another major American city claims to have reduced crime every year since 1994: Chicago. New York averaged 1,514 murders a year during the three decades before Mr. Giuliani took office; it did not record more than 1,800 homicides until 1980. And Mr. Giuliani’s own memoir states that spending grew an average of 3.7 percent for most of his tenure; an aide said Mr. Giuliani had meant to say that he had proposed a 7 percent reduction in per capita spending during his time as mayor.

Where was this kind of coverage when President Bush misused statistics to sell his tax and budget plans starting during campaign 2000? (see All the President’s Spin for more) Remember, back in October 2002, Dana Milbank’s similar piece about President Bush’s dissembling in the Washington Post drew an extremely harsh reaction. The rest of the press, needless to say, did not follow his lead.

Here are a few possible explanations:

(a) Rudy’s bluster invites more critical coverage — he seems like more of a dissembler than the supposedly straight-talking Bush;
(b) Bush’s statistics were carefully parsed to be half-true, making them harder to debunk within the framework of supposedly “objective” journalism (this is the argument we make in ATPS), while Rudy’s are often just wrong;
(c) Reporters are becoming more skeptical about the use of statistics of the Bush administration;
(d) The political environment is pushing reporters toward more critical coverage of Republicans;
(e) All of the above.

What do you think?

Will Saletan tries to clean up the mess over his ill-conceived Slate series on race and intelligence, which was shredded here, here, here, and here (among others):

Last week, I wrote about the possibility of genetic IQ differences among races. I wanted to discuss whether egalitarianism could survive if this scenario, raised last month by James Watson, turned out to be true. I thought it was important to lay out the scenario’s plausibility. In doing so, I short-circuited the conversation. Most of the reaction to what I wrote has been over whether the genetic hypothesis is true, with me as an expert witness.

I don’t want this role. I’m not an expert. I think it’s misleading to dismiss the scenario, as some officials have done in response to Watson. But my attempts to characterize the evidence beyond that, even with caveats such as “partial,” “preliminary,” and “prima facie,” have backfired. I outlined the evidence primarily to illustrate the limits of the genetic hypothesis. If it turns out to be true, it will be in a less threatening form than you might imagine. As to whether it’s true, you’ll have to judge the evidence for yourself. Every responsible scholar I know says we should wait many years before drawing conclusions.

Many of you have criticized parts of the genetic argument as I related them. Others have pointed to alternative theories I truncated or left out. But the thing that has upset me most concerns a co-author of one of the articles I cited. In researching this subject, I focused on published data and relied on peer review and rebuttals to expose any relevant issue. As a result, I missed something I could have picked up from a simple glance at Wikipedia.

For the past five years, J. Philippe Rushton has been president of the Pioneer Fund, an organization dedicated to “the scientific study of heredity and human differences.” During this time, the fund has awarded at least $70,000 to the New Century Foundation. To get a flavor of what New Century stands for, check out its publications on crime (“Everyone knows that blacks are dangerous”) and heresy (“Unless whites shake off the teachings of racial orthodoxy they will cease to be a distinct people”). New Century publishes a magazine called American Renaissance, which preaches segregation. Rushton routinely speaks at its conferences.

I was negligent in failing to research and report this. I’m sorry. I owe you better than that.

In general, I’m baffled that Saletan thought he was qualified to arbitrate a scientific debate as complex as the one over race and intelligence. But his ignorance about Rushton is even more disconcerting. He didn’t even have basic biographical information about the supposed experts whose work he was drawing on, so what was he doing when he was reporting the story? Who was he talking to?

Update 12/1 1:31 PM: More from CMU’s Cosma Shalizi:

In my first post about this, I said that there were
two possible interpretations of Saletan’s actions: that he didn’t know that the
ideas he was spreading were crap, or that he did, but spread them anyway to
advance an agenda. Saying that the second interpretation was more charitable
wasn’t just a joke. Sadly, this partial mea culpa supports
the first interpretation, that of incompetence. To put it in
“shorter
William Saletan” form, what he is saying is: I am shocked — shocked!
— to discover that the people who devote their careers to providing
supposedly-scientific backing for racist ideas are, in fact, flaming racists.
And he does seem to be shocked, though it is hard
(as Yglesias
says) to see why, logically, he should strain out those gnats he
displays for our horrified inspection while swallowing the camel of group
inferiority (and telling his readers that camel is really great and the coming
thing). This indicates a level of incompetence as a reporter and researcher
that is really quite stunning —
as Brad
DeLong says, this seems like a trained incapacity.

But let me back up a minute to the bit about relying on “peer review and
rebuttals to expose any relevant issue”. There are two problems here.

One has to do with the fact that, as I said, it is
really very easy to find the rebuttals showing that Rushton’s papers, in
particular, are a tragic waste of precious trees and disk-space. For example,
in the very same
issue of the very same
journal as the paper by Rushton and Jensen which was one of Saletan’s
main sources, Richard
Nisbett, one of
the more important
psychologists of our time, takes his turn
banging his head against
this particular wall. Or, again, if Saletan had been at all curious about
the issue of head sizes, which seems to have impressed him so much, it would
have taken about five minutes with Google
Scholar to find a
demonstration that this is crap. So I really have no idea what Saletan
means when he claimed he relied on published rebuttals — did he think
they would just crawl into his lap and sit there, meowing to be read? If I had
to guess, I’d say that the most likely explanation of Saletan’s writings is
that he spent a few minutes with a search engine looking for hits on racial
differences in intelligence, took the first few blogs and papers he found that
way as The Emerging Scientific Consensus, and then stopped. But detailed
inquiry into just how he managed to screw up so badly seems
unprofitable.

Matt Yglesias also chimes in:

Saletan was busy trying to have his cake and eat it, too, and when confronted with Rushton’s rhetoric suddenly finds himself choking on it. But of course the research “proving” blacks’ genetic inferiority to whites is shot through with racism; what else would the race-science paradigm possibly be infused with? Somehow, Saletan was so busy with his counterintuitive pirouettes that he didn’t notice what side he’d landed on.

The New York Times notes an amusingly wacky Sierra Club website with tips on winning arguments about environmentalism over the holidays:

Are you likely to be the lone environmentalist at the dinner table sometime soon? Win arguments and influence people with this handy guide. You’ll find ready responses to the predictable dinner table arguments that’ll be directed at you, the nearest environmentalist. Who knows, you might even make a few converts!

• Spar a few rounds with grumpy Uncle Burt, whose personal virtues do not include conservation.

• Educate the sweet but confused Aunt Mim about the facts of global warming.

• Help wonkish Cousin Mervin get his priorities straight.

• And face off with the ever-feisty Sis who thinks you’re a lousy, no-good sell-out.

In the mail: McCain: The Myth of a Maverick by Matt Welch. More when I read it…

At Polysigh, Phil Klinkner notes the historical discrepancies between Mitt Romney’s alleged statement about Muslims in his cabinet and the history of the GOP:

Mitt Romney is in a bit of hot water over his comments that he would be unlikely to appoint a Muslim to a cabinet post. He is reported to have said:

Based on the numbers of American Muslims [as a percentage] in our population, I cannot see that a Cabinet position would be justified. But of course, I would imagine that Muslims could serve at lower levels of my administration.

Muslims currently make up anywhere from 0.5 to 2.2 percent of the population of the U.S.

In 1953, Dwight Eisenhower appointed Ezra Taft Benson as his secretary of agriculture, making the first Mormon cabinet secretary. At the time, there were approximately 1.2 million Mormons in the U.S., or approximately 0.75 percent of a total population of 160 million.

In 1969 Richard Nixon appointed George Romney, Mitt’s father as his secretary of Housing and Urban Development. At that time, there were 2.8 million Mormons, approximately 1.4 percent of the population of the time.

Does Romney believe that these appointments were not “justified” because Mormons were too small a percentage of the U.S. population? Exactly what percentage of the population do you need in order to claim a position in the Romney cabinet?

The Daily Howler’s Bob Somerby has suggested that the entire press corps now has a partisan bias against Democrats:

What has changed since 1960? At one point in his iconic first book [The Making of the President 1960], [Theodore] White painted a truly remarkable picture. He described the way the mainstream press corps was flying around the country, mocking and laughing at one of the candidates—and bonding with the other candidate, the one who was pandering to them. And wouldn’t you know it? Forty years later, during Campaign 2000, a string of major profiles painted a very similar picture! As White had done forty years before, they described the way the mainstream press corps was flying around the country, mocking and laughing at one of the candidates—and bonding with the other candidate, the one who was pandering to them. But uh-oh! In White’s account, the press corps was bonding with Candidate Kennedy—and mocking and laughing at Candidate Nixon. By the time of Campaign 2000, though, the press corps was bonding with Candidate Bush—and mocking and laughing at Candidate Gore.

In short, the press corps’s conduct was exactly the same—but the press corps’ party allegiance had changed!

Somerby previously made the same charge against Tim Russert. In both cases, it’s simplistic and unconvincing. Here’s what I said about his Russert accusations — I think the same logic applies here:

I actually agree with Somerby that Russert tends to be more aggressive in his questioning of Democrats. (Anyone remember his interview of Howard Dean during the last presidential campaign?) The problem, however, is that we can’t know Russert’s motives. More importantly, it is strange to assume that the ex-Democratic operative wants to embarrass Democrats for partisan reasons.

There’s a simpler explanation that seems more persuasive. Like most journalists, Russert is far more sensitive to the approval of his peers than to the opinion of the general public (they’re a lot like academics). So how do you win acclaim for being a tough journalist? First, you grill your subjects on alleged inconsistencies and constantly try to throw them off message (his signature style). But you must also fend off any suggestion of liberal bias, a charge that could be especially potent for Russert given his history as a Democratic operative. As a result, it makes perfect sense for him to go overboard in grilling Democrats and to treat Republicans less harshly. There’s no reason to think it has anything to do with partisan animus.