Anonymous Internet Vigilantes Are Taking Peer Review Into Their Own Hands
PubPeer users went from discussing papers to hunting down fraud—and have embroiled the site in the most important internet privacy case you've never heard of.
Illustration: Shaye Anderson
Since 2012, the message board PubPeer has served as a sort of 4chan for science, allowing anyone to post anonymous comments on scientific studies. Originally intended as a forum for the discussion of methods and results, PubPeer has perhaps become best known as a clearinghouse for accusations of scientific error, fraud, and misconduct—forcing journals to issue corrections and retractions, damaging careers, and eventually embroiling the site in a court case in which it's advised by Edward Snowden's legal team at the American Civil Liberties Union.
In the view of its critics, PubPeer enables an unchecked stream of accusations with no accountability. But to its supporters, PubPeer is maybe the only consistently effective way to expose fraud and error in the current scientific system. It exists at a time of quiet crisis for science and science journals, when the community is concerned about an inability to replicate past results—the so-called "reproducibility crisis"—and the number of papers retracted is on the rise. The traditional system of peer review seems unable to address these problems.
"We started it because we wanted more detailed arguments about science, and we were really shocked at how many fundamental problems there are with papers, involving very questionable research practices and rather obvious misconduct," said Brandon Stell, a neuroscientist at the Centre National de la Recherche Scientifique in Paris and the creator of PubPeer.
There's certainly no denying its effect. According to Retraction Watch, a blog that monitors scientific corrections, errors, and fraud, at least three high-profile scientists in the past few months have had their studies retracted by journals after their data was questioned by anonymous commenters on PubPeer.
The most frightening words a researcher could read on PubPeer are 'There are concerns'
One of the scientists, Fazlul Sarkar, is currently suing several of the commenters. His lawyers argue the site must reveal the identities of the users that have done damage to Sarkar's career, after he lost a tenured position at the University of Mississippi. PubPeer has refused to release the information. Both Google and Twitter have filed a court brief in support of the site, which is currently being defended pro-bono by lawyers from the ACLU.
It's perhaps the most interesting case about internet privacy you've never heard of, and it all stems from a frustration among scientists with the shadowy politics of publishing and peer review.
At its base, PubPeer is a site that allows anyone to post comments on any scientific paper listed on the federally-funded PubMed database, either anonymously or under their own name. It's functionally very simple, but the built-in anonymity makes it a safe outlet for scientists—especially young, early-career scientists—to discuss and criticize research without fear of repercussion. And that's something they're apparently eager to do: The site has logged over 55,000 mostly anonymous comments since its launch.
Back in October 2013, someone on the PubPeer site started threads for about 20 previously published papers on which Fazlul Sarkar, a cancer researcher then at Wayne State University in Michigan, was an author. The papers span over a decade and involve a variety of complex molecular signalling pathways involved in cancer. The issues raised by the comments, though, were relatively straightforward: They claimed that images in these studies appeared to have been changed, duplicated, and re-used across papers, suggesting that the experiments they appeared in may have never actually happened, or could have produced different results.
Stell noted that, in an effort to keep the discussion civil (and legal), PubPeer specifically requests that users do not accuse authors outright of misrepresentation or fraud. Comments are moderated in case they break these guidelines, so any discussion of such allegations tends to have a muted tone.
That doesn't make this group of self-appointed watchdogs any less effective, though. The most frightening words a researcher could read on PubPeer are "There are concerns."
Discussion over "concerns" surrounding Sarkar's work expanded rapidly as it became clear the commenters had found a rich vein to mine: According to the NIH funding database and PubMed, Sarkar has received more than $12 million in NIH funding and authored over 500 research papers over his career. The community is nothing if not meticulous—PubPeer commenters have been known to pull up decades-old PhD theses looking for dirt—and a search of the message board shows that eventually 77 papers with Sarkar on the author list were presented for scrutiny. By checking the papers against each other patterns began to emerge; for example, one user claims a single set of images were duplicated up to 54 times in 13 papers, across three years.
What the community appeared to have identified was a shady scientific photoshopping spree that had gone undiscovered by grant committees and the editors and peer reviewers at prestigious journals for over a decade.
Many of the site's visitors embrace this kind of investigative approach.
"I came to PubPeer partly out of frustration," Elisabeth Bik, a microbiology researcher at Stanford University and regular PubPeer user, told me. Bik said she posts under her own name when discussing the scientific merits of a paper, but prefers to remain anonymous when pursuing her other major interest: identifying manipulated or fraudulent images in scientific figures. (Bik said that while she has used PubPeer to discuss image manipulation on several occasions, she was not involved in the Sarkar threads.)
"For some of these cases I was reporting them to the [journal] editors," she said. "This seemed to be the most honest way, and it also gives the authors a chance to reply, but it was very frustrating to realise that many of these editors would never write me back or they would write me back like, 'Oh we're going to investigate,' and then crickets after that."
Other scientists I spoke to described similar experiences of sharing concerns with journals and being ignored or strung along, all the while convinced that the targets of their criticism were being shielded.
When asked about these concerns from scientists, a spokesperson for Nature Research, which publishes multiple journals including Nature, provided a statement to Motherboard by email. They said that submitted papers "undergo rigorous peer review," and are always assumed to be "provided in good faith." The papers are automatically checked for plagiarism and image-manipulation, and editors and peer reviewers can be engaged to correct or retract a paper if concerns arise.
"We take all concerns about papers we have published seriously, whether raised by identified individuals or reported anonymously, and consider each one carefully on a case-by-case basis," they said.
A representative for Cell Press similarly affirmed their editors' commitment to correcting the scientific record, noting that "because we consider the investigation process confidential we don't report back in detail to the person who submitted the concern."
Many observers though, aren't convinced that journals and other gatekeepers are doing enough on their own. "For decades, researchers and editors were ignoring anonymous and critical comments until you see some sort of accountability," said Ivan Oransky of Retraction Watch. He said that his site is regularly contacted by would-be critics and whistleblowers, but the volume is simply too high for the small staff of journalists to pursue.
PubPeer, he said, offers an outlet for them.
Accusing someone of fraud isn't a game, least of all to the person being accused. And the allegations against Sarkar weren't just an academic exercise: As often happens with popular PubPeer threads, someone had been sending the findings to any official channel that might be able to do something about it, including the journals that printed Sarkar's research, and his employers.
Unfortunately for Sarkar, this happened at a critical time in his career. According to court documents obtained by Retraction Watch, in June 2014 he had just resigned from Wayne State University and accepted a prestigious offer of a tenured position at the University of Mississippi, including a lab start-up bonus and a salary of $350,000. But just 11 days before he was supposed to start that position—his attorney notes he had just made an offer on a house in Oxford, Mississippi—the offer was rescinded. All because of the PubPeer investigation.
As reported in the court documents, a letter dated June 19, 2014 from Larry Walker, director of the National Center for Natural Products Research at the University of Mississippi Cancer Institute and Sarkar's main contact about the job, "cited PubPeer as the reason, stating in relevant part that he had received a series of emails forwarded anonymously from (sic?) PubPeer.com, containing several posts regarding papers from your lab." Emails were allegedly also sent anonymously to other researchers at the university, effectively making all of Sarkar's new co-workers aware of the controversy before he even arrived on campus.
It's perhaps the most interesting case about internet privacy you've never heard of
Sarkar responded in 2014 by suing both the university for terminating his offer and several anonymous PubPeer commenters for defamation. In January 2016, a judge dismissed the case against the University of Mississippi, but the case against PubPeer is ongoing. Sarkar's lawyers argue that the commenters weren't simply discussing his work, but attacking his character. They sent PubPeer a subpoena to release the names and IP addresses of the commenters in question.
As a hobby project run by an early-career research scientist and his friends, PubPeer wasn't exactly well-equipped to defend itself from a legal challenge. But its appeal as an anonymous whistleblowing platform placed it in the vicinity of debates on privacy and disclosure that were still raging less than a year after Snowden's first NSA document release.
"I get this email from somebody, basically they had been in contact with Ben Wizner, and Ben Wizner would be interested in talking to us about this," said Stell.
Ben Wizner, director of the ACLU's Speech, Privacy, and Technology Project, is one of Edward Snowden's lawyers.
"They said he's willing to help us and I'm like, wow, this is amazing!"
The ACLU quickly arranged a legal team to work on PubPeer's behalf, at no charge.
"People aren't exactly being thrown in jail, but scientific discourse suffers when scientists can't speak their mind free from the threat that their current or future superiors would know that it's them lodging the criticism," said Alexander Abdo, a senior attorney at the ACLU and one of several working on PubPeer's case. "You have to burn all of your bridges to go public with the criticisms you have. Anonymity offers some breathing space."
"I think PubPeer is trying to take advantage of anonymity for the very reason it is constitutionally protected. Anonymity offers you a shield from those who would persecute you for your views," he added.
Stell, the founder of PubPeer, agreed. "If we [stick] our neck out and [say] something about our colleagues' work, then that could have a serious impact on our own careers," he said.
But Nicholas Roumel, a partner at NachtLaw in Michigan who is representing Sarkar in the case, argues that the anonymous commenters have been given greater protection than their target, who can't even address his accusers by name.
"Of course anonymous speakers should be given First Amendment protection," he told Motherboard via email. "The issue is whether anonymous speech should be given greater protection under the First Amendment than non-anonymous speech. The cowards who post anonymously claim they deserve greater protection because they fear persecution—but they have no problem destroying careers like Dr. Sarkar's from beneath their rocks. That's hypocrisy."
Roumel, who pointed out that he is a member of the ACLU and chairs his local ACLU lawyers' committee, argues that his client is being held to a higher legal standard than his accusers. He said that by filing briefs in support of PubPeer, social media giants like Google and Twitter are agreeing with an idea that he finds troubling.
"They argue that it should be more difficult for victims like Dr. Sarkar to file and prove their cases against anonymous defendants," he said.
Sarkar did not respond to requests for comment for this story.
As it stands, the case is effectively deadlocked. In March 2015, a judge denied the subpoena against the majority of the PubPeer commenters, save one, whose contributions, Sarkar's attorneys argued, were especially and egregiously damaging. PubPeer was ordered to hand over all available information they had on the user, including the IP address they used to access the site.
The ACLU has appealed that decision, while Sarkar's attorneys have appealed the decision to dismiss the other requests. The appeal hearings are set to take place on October 4th.
In perhaps the strangest twist to emerge, a court brief submitted by Sarkar's attorneys shows that the PubPeer user whose identity Sarkar's lawyers are most interested in used an email address belonging to "Clare Francis." Francis is a pseudonym used by science publishing's greatest gadfly, a person or group that has sent "hundreds" (possibly now thousands) of anonymous tips to journal editors, beginning sometime around 2010. She is so prolific that the publishing giant Elsevier previously revealed that they have special directions to editors in place for evaluating tips originating from Francis—helped along, no doubt, by Francis's propensity to CC the tips to the New York Times.
As such, the latest legal order has the potential to clear up one of the longest-running mysteries in science publishing, by unmasking the person or persons responsible for the Clare Francis phenomenon.
Although the case is stalled, the damage has already been done to Sarkar's career. As of the time of this writing thirteen of his high profile papers were retracted by the journals that published them—six in just one month—and, as one astute commenter on Retraction Watch noted, Sarkar's name appears on a list of staff retiring from Wayne State University this year. He is a scientist without a lab, and the first results that appear on his PubMed listing are now a string of retractions.
There are many within the scientific community who are wary of putting too much stock in the investigative power of an anonymous mob. The editors of two large journals have made arguments against anonymous criticism, and Michael Blatt, the editor of Plant Physiology, has addressed PubPeer directly.
In an editorial, Blatt agreed that there must be a way to address "bona-fide fraud" in his own field of plant biology, but maintained that "anonymity is not the answer, however, not if due process is to ensure civil society and protect the innocent from denouncement or worse."
Many PubPeer commenters who spoke with Motherboard had worries about people using anonymous posts to carry out personal or misplaced attacks.
"There are definitely cases where it seems like someone has an agenda," one said. Another suggested there could be cases where an ex-student or employee could make unfounded allegations about a colleague with whom they had a poor working relationship.
Even when motivations are pure, there is also the chance of erroneous accusations. Bik, the Stanford microbiologist, said that when she first started flagging possible image manipulations on PubPeer, she occasionally got it wrong. Most erroneous or easily explained concerns posted on the site are either cleared up quickly by the commenters or the authors of the paper in question. Still, the threads are a public record most scientists wouldn't want associated with their work. Bik said she has been much more careful since.
But scientific fraud may be far more widespread than a few bad apples. Bik has turned her interest in image manipulation into an official parallel research stream, and she recently released a paper about this on a pre-print server with Arturo Casadevall and Ferric Fang, influential scientists and advocates for journal reform. They showed that almost four percent of images in a survey of 20,000 papers from top journals were "problematic," with many showing signs of direct image manipulation. Authors with a single manipulated image were likely to have more, they write, and some journals were worse offenders than others.
And as it stands, the best way to get those cases addressed often still appears to be raising an anonymous alarm. As Oransky, of Retraction Watch, pointed out, PubMed itself recently introduced a comment system linked to scientists' institutional email addresses, allowing them to comment on papers under their real names. But it doesn't quite deliver the same results. "There are no cases of retractions coming out of named comments, as far as I'm aware," he said. "Anonymity is the key here," he added.
Stell said that while he hopes to encourage people to use their own names and foster more basic scientific discussion, he also plans to keep PubPeer functioning as an effective whistleblowing platform. He says the site is both beefing up its anonymity—holding as little user data as possible, and making it easy for people to set up to connect through Tor—and working on a variety of ways to encourage people to post discussion and criticism under their real names.
The idea is that when all scientists—not just the few that are appointed for peer-review—are free to evaluate research in a public forum, fraud and bad data would become less of an issue. But while PubPeer's founders attempt to steer users away from the rigid hierarchy of scientific journals and toward better scientific discussion, the community is free to continue functioning as a sort of unruly conscience to the powers that be, using the tools of anonymity and the internet to clean up science in their own way.
Correction: An earlier version of the story incorrectly quoted journalist Leonid Schneider saying PubPeer users are "trolling peer review to promote its ethics." The quote did not refer to PubPeer users, so it has been removed from the story.
Get six of our favorite Motherboard stories every day by signing up for our newsletter.