Erhan Dayi/Shutterstock
The first rule of fact-checking is that everything you read and hear is wrong.
Were you to be hired as a fact-checker, as I was in 2007, at Mother Jones—or at the other remaining bastions of fact-checking, mostly a handful of magazines known for their reporting—you would be taught that information couldn’t be trusted. It is, rather, presumed fallacious until proved otherwise. Statistics and news clips must be subjected to intense tests of verification. Don’t even think the word “Wikipedia.” In my first meeting, among new coworkers of startling cynicism and genius, the announcement that the source of some fact was a book set off a mighty wave of scoffing and eye rolling around the conference table.
In true fact-checking, literally every word of every factual statement must be traced to a primary source, whether a document or the corroborated accounts of independent experts or witnesses. “Primary source” means that if the story you’re fact-checking says some soldier was the 44th Canadian soldier to die in Afghanistan since 2002, you’re calling the Canadian Army. Once, I heard one of our fact-checkers call a bar in Mexico to ask, in Spanish, whether its floor was metal, per William T. Vollmann’s assertion.
For the record, it was. But often, it’s not. People just get things wrong. They read them wrong, or remember them wrong or the way they want to, or the information they read right was wrong in the first place. You hear the same fact a thousand times, but if you track down its origins, you find out all the repeaters are using the same source, and source zero was just guessing, or citing a highly questionable source or study. Or misciting a highly questionable source or study. Or confusing the details, so that by now, everybody’s under the erroneous impression that a shot of espresso contains more caffeine than a cup of coffee.
So I wasn’t a week into my job before I, too, had undergone the completely life-changing installation of an irrevocable—and warranted—skepticism of everything I heard or read. So it’s life-ruining, also. Which is why I did something book writers rarely do—which is subject their manuscripts to a Mother Jones-style fact-check—and that they rarely do for really good reason—which is that holy fucking shit is it hard.
Enter former MoJo research editor Leigh Ferrara, a fact-checking and multitasking wizard, and the hardest-working and most charming person you could hope to be stuck in a studio apartment with for 12-hour fact-review marathons. The book, For Us Surrender is Out of the Question, and the MoJo piece of the same name, was about Burmese refugees who snuck back into their country to document human rights atrocities. And, you know, the whole history of Burma. The manuscript I gave Leigh had 1,240 footnotes, plus piles of sources noted haphazardly within the text, plus a bunch of sentences with no sourcing at all. So: I’d read and subsequently written that Burma had the fourth-highest child mortality rate in the world, and Leigh had to figure out whether that was true or not. It’s not. Burma was 36th on that list, actually, which Leigh tracked down in UNICEF’s “State of the World’s Children 2009.” That took care of 11 words, out of more than 100,000. And that was a pretty easy one. (BTW, it is now 44th.)
Buy the book here.For historical details, we unearthed and paged through colonialists’ reports and missionaries’ diaries, or cross-checked information with other historical accounts we’d made sure weren’t all using the same one original source—as is often the case—and/or vetted the minutiae and main ideas with scholars and specialists. We tracked down witnesses to and experts on subjects way outside the spotlight of popular scrutiny. We then evaluated those sources, trying to determine if they were reliable and where they were getting their information. One of Leigh’s experts helpfully eliminated a handful of questions from her long list of outstanding facts; then she realized the source of his expertise was a book we’d already determined to be mistake-tastic.
Further complicating the fact-checking process was the inconvenience that there’s often no such thing as “fact.” Another figure I’d cited was that trade between China and Burma was up to $2.6 billion in 2008, from $630 million in 2001. That turned out to be “true” (shout-out to the Wall Street Journal for its totally solid Burma info. Ditto The Washington Post. And The Irrawaddy, the Burmese exile paper, does work that, in addition to filling a critical reporting void, is incredibly reliable.), based on data Leigh uncovered in the United Nations Commodity Trade Statistics Database. Mexicali’s 13 Negro bar either does or does not have a metal floor, but the trouble with this trade statistic is perfectly summed up immediately following it in the WSJ article where it appeared: “Analysts say the official numbers vastly understate the full extent of China’s investments in Myanmar.” Mmmm, analysts do say that. Knowing that—however meticulously we sourced our facts, and though every estimate we were working so hard to find and confirm was at least the best possible estimate in existence—”true” is often still kind of a relative concept…it’s demoralizing.
We both had dark moments while trying to keep a million little pieces needing verification up in the air. I had nightmares about working on the final edits and not being able to write any of the words I wanted because I didn’t have sourcing for them and there was no time for further fact-checking. Leigh started inadvertently holding her breath when opening emails from sources, because they might say that she was out of luck, or an idiot, because the assertion she was asking them to confirm—which I’d pulled from non-fact-checked books or articles—was absurd. We each went through a period of extreme temporomandibular pain, at which point we realized we’d started clenching our teeth furiously. We took turns psyching each other up, holding up opposite sides of motivational conversation:
“Why are we doing this?”
“It’s fine. We’re doing a great job. People are going to give us trophies when they realize how thorough we’ve been.”
“Yeah, because anyone will ever even notice that we did this, and they totally give trophies for fact-checking.”
“You’re doing a great job. It’ll all be worth it when we get the trophies.”
It was worth it, actually. I wrote, for example, a long and exciting description of some freaky shit a certain Burmese hill tribe did during a certain world war. I won’t go into who the source was or the possible sourcing mistakes made, but the upshot is that if I hadn’t omitted and you had repeated this story at a cocktail party, at worst a scholar of any of several disciplines would’ve recognized it as a complete load of crap. At the very least, you’d have been going around spreading freaky lies. And that’s how Leigh and I comforted ourselves during the aforementioned marathon reviews, comparing three hundred pages of single-spaced notes from more than 700 paper and electronic and human sources while I made changes on more pages than not.
“I can’t believe how much money and how much of our lives this has cost.”
“You can’t put a price on truth.”
“You’re right. We’re heroes.”
In one chapter, I riffed, based on what I’d read, on how the KMT (a.k.a. Kuomintang or Chinese Nationalist Party) used Dodge and Ford trucks—Dodge and Ford trucks!—and how that was further evidence of how ridiculous the United States’ denial of assisting the Chinese in Burma was. But you won’t find that in the book, because a foreign-relations scholar pointed out to me that anyone could get mass-produced American vehicles anywhere. There were parades of them on the Ho Chi Minh Trail during Vietnam; did I think the United States was also assisting the Viet Cong? I constructed, and then had to thoroughly de- and reconstruct, a story about ancient Burmese race relations after reading a mountain of history books—books that apparently every archaeologist and Southeast Asian history specialist (we consulted five) knows are based on long-discredited colonial theory riddled with “sheer fantasy” and “Orientalist cliché.” I wrote an entire chapter based on a first-rate historical account of some stupid and misguided action on the part of the United States that oh, man, did I want to be true. One expert corroborated that it was. Three others said it wasn’t, exactly. A declassified State Department memo settled the dispute in favor of the nays. It’s cool; there was plenty of equally stupid and misguided action to take its place. See chapter six.
“No matter how hard we work, we’re going to miss something. All this work and we’ll still know that there are mistakes we didn’t catch. It’s so futile.”
“No! Trophies!”
We terrorized the United States Department of Homeland Security, the World Health Organization, the United Nations Office on Drugs and Crime, the United Nations High Commissioner for Refugees, the history and epidemiology departments of several universities, the authors of dozens of books, the staffs of countless NGOs, lawyers and doctors and soldiers and refugees and multinational corporations and activists. And so on. I’d be swimming in debt without the research support that was provided by the Investigative Fund at The Nation Institute and the Fund for Investigative Journalism. But no matter how much time and money went into reporting this story, and digging up obscure and untold details (see the stats comparing eastern Burma with Darfur, chapter 12), and then making sure they were right—no matter how many months Leigh worked (eight, if you really want to know, so you can imagine how long it would take to fact-check a book in which half the narrative isn’t recounted conversations about socialism and blow jobs), she still could’ve misinterpreted information herself, and the information could still have flaws, and though the changes we made were hundreds, the mistakes we caught were certainly not every one, and never could be, even if we had unlimited resources and lived in the Library of Congress.
Nevertheless, I’d feel pretty confident breaking any tidbit of the book out at a cocktail party full of Southeast Asia wonks. And were a scholar or skeptic I was drinking with to question my source, I’d be able to say, “Report E/CN.4/1995/65 of the United Nations Economic and Social Council, Commission on Human Rights.” Or “Knowles’ 1829 Memoir of Mrs. Ann H. Judson, Late Missionary to Burmah; Including a History of the American Baptist Mission in the Burman Empire. No, no, the second edition.”
This story originally appeared under the headline: “What is Good Fact-Checking? Amid PolitiFact’s “lie of the year” controversy, a shout-out to what it takes to get to the facts of the matter.” For background on that controversy, see Round 1 and Round 2 and subsequent commentary here, here, here, here, and here.
