Inside Science Magazine's 'Sting' of Open Access Journals

On May 21, Addai W. Ahshean of Burkina Faso's Cotarsch University received a rejection email from an editor at the Open Bioactive Compounds Journal. After a nearly six month review process, Ahshean's study, "Allo-protolichesterinic acid inhibits the growth of murine SV40 transformed autonomic ganglia blastoma cells in vitro," was rejected because the journal deemed it was "out of its scope."

Editors at Bentham, one of the world's largest publishers of science journals, suggested he resubmit it to another journal under its umbrella.

Three months later, with essentially no revisions, the paper—which claimed that a chemical derived from a type of lichen could stop the growth of cancer—was accepted, by the same journal he'd originally submitted it to.

The only problem? Ahshean doesn't exist, and neither does Burkina Faso's Cotarsch University. The metabolite, the lichen, even the types of cancer it treats, were all generated at random by a computer program. The science was complete nonsense.

It was so bad, in fact, that "any reviewer with more than a high-school knowledge of chemistry … should have spotted the paper's shortcomings immediately. Its experiments are so hopelessly flawed that the results are meaningless." Those are the words of the paper's true author, science journalist John Bohannon, who discusses the ploy in the latest issue of Science.

The fact that one journal decided to publish his paper isn't terribly alarming. But the fact that 157 open access journals of the 304 he submitted to decided to publish it is. Bohannon says the "sting operation" helps to expose the "wild west" of open access science journals—one where there is little oversight, mysterious staff members, and the money flows through sketchy offshore accounts.

"I really was totally shocked. I thought the acceptance would be something like 10-20 percent," he told me. "From the start, it was very quickly above 50 percent acceptance rate and it just never went down."

Open access journals' collective credibility has come under fire before: Critics say that many journals are out to make a buck and don't care about the science. Others say that their review processes simply aren't up to the standards of subscription journals like Science or Nature. Since publishing the results of his experiment Thursday, Bohannon has gotten plenty of hate mail from people who say that Science, a subscription journal (and one of the most prestigious in the world) simply has it out for open access. That's not the case, he says.

"I'm a freelance journalist. I started this thing on my own and I did it on my own. Science only agreed to publish it about a month ago," he said. "Everyone's really worked up about me not including subscription journals in this experiment, and that's a legit argument. In an ideal world, they'd be included."

In truth, he's a bit more than a freelance journalist: Bohannon has a PhD in molecular biology from Oxford, has taught at Harvard, done TED talks about making PhD students perform their theses as interpretive dances, was the first journalist to get the military to release civilian casualty data, and once got Stephen Colbert to eat cat food.

He decided to take on this sting operation last year, when a Nigerian scientist friend of his told him about the problems she'd experienced with the open access publishing process. He set to work writing Python scripts that would generate random African names and institution names, something he did so that he wouldn't have to set up shell websites for fake universities. Many research institutions in developing nations don't have established web presences, so their lack of one wouldn't raise any red flags. He wanted to make sure a paper was accepted or rejected based on the science alone, not a school's reputation. "If a paper comes in, it shouldn't matter who it's from. It should be accepted or rejected on its own merits," he said.

Bohannon started with open access publications because he wanted to recreate his friend's experience. Soon, he was submitting 10 papers a week. The original plan was to write five papers in each area of science, but once he got started he realized the whole thing "was taking friggin forever."

"I didn't have any grand plans to exclude subscription journals, but I just had to get started," he said. "In a lot of cases, the editing process took weeks and they wanted me to fix the most irritating things—change the reference format or the margins. It was almost never about the science."

The target of open access journals was probably a good one. Many people have criticized even the most well-known open access journals, such as PLOS One, as having a "publish first, judge later" mantra. That journal has, in just a few years, become the world's largest peer reviewed journal.

Earlier this year, ecologist Andrew Tredennick said he'd probably stop publishing in PLOS One because "there is still too much negative bias against the journal and against people that have 'too many' PLOS One papers on their CV." A conversation on Twitter earlier this year called publishing in PLOS One "career suicide" for a scientist and Ted Hart, an ecologist and open data proponent tweeted that "for better or ill most of my peers look at PLOS One as the dumping ground for papers rejected at 'real' journals."

@ethanwhite but it seems for better or ill most of my peers look at @plosone as the dumping ground for papers rejected at "real" journals.

— Ted Hart (@DistribEcology) January 15, 2013

Bohannon said he'd "never been so happy to receive a rejection letter" as when one of PLOS One's editors told Dr. Masabla Ohyeanorrow that his paper had been rejected because "the data provided does not fully support the conclusions."

"PLOS represents a lot. They're the leaders of the open-access movement and they do a lot of good things. I didn't want them to have an unlucky day and have something slip through," Bohannon said. "I think they were completely vindicated by this thing."

PLOS co-founder Michael Eisen didn't feel quite the same way. In a response to Bohannon's story, Eisen fires at Science for publishing a now-infamous paper about bacteria incorporating arsenic into their DNA. "The story is an interesting exploration of the ways peer review is, and isn’t, implemented in today’s biomedical publishing industry," Eisen wrote. "Sadly, but predictably, Science spins this as a problem with open access."

"Nowhere do I claim that open access is worse than traditional subscription," Bohannon said.

In the end, Bohannon ended up with a treasure trove of information. Science has posted each and every paper he submitted, along with every email he and the editors of the journals exchanged during the process. He calls his experience with Bentham "the tip of the iceberg."

"There is just such bizarre sad stuff in there," he said. "I don't want to say this is of the same importance as Wikileaks because it's not, but it's similar because there's a big data dump that people will be going through for a while."

In the case of Bentham, they asked Bohannon's "Ahshean" to provide a resume and a letter of recommendation from his advisor, a step that's unprecedented in science journal publishing. He says their "verification department" was likely formed after the company published several false papers over the past several years. In the end, despite asking him to submit to another journal—and the fact that one reviewer said the "results are controversial" and another said "the paper is confused and ambiguous in some parts" and should be "deeply changed"—they accepted it without serious revision anyway. Emails to Bentham's editors from Motherboard went unreturned.

So what can be taken from all of this? It's hard to say.

Science journalists like me rely on the credibility of journals when reporting the news, and though some studies can seem fishy, not many of us can quickly review a study and know when to call bullshit. The publishing process might be better at subscription journals, but perhaps not.

It comes down to two things: First, there are a whole lot of journals out there, and many of them have less than sterling reputations. Second, and perhaps more importantly, is that cutting-edge research can be beyond the experience of even the most skilled peer reviewers. In other words, no single paper should be treated as gospel, which has always been the case, open-access or not.

"Honestly most scientists don't have all the expertise you would need to see through these things," Bohannon said. "Even the ones who do, there's a lot of trust and there's so much you have to assume authors are being honest about."