How the project was initiated. In 1978, one of us (SJC) was a newly hired assistant professor of psychology at the University of North Dakota, and the other one us (DPP) was in his fourth year as an assistant professor, one year away from the all-important tenure decision. One day at lunch with several colleagues, a discussion took place about bias in publishing. A senior colleague opined that a recent publication in a prestigious psychology journal would never have been published if the author had not been an eminent researcher from Harvard. He argued that if he had submitted the same manuscript for publication, it would have been rejected in a New York minute. We were not so sure. This individual's own difficulties getting his work published colored his view of science and prompted his conspiracy feelings.
Later, in discussing his accusation of bias in psychological science, we asked ourselves how we could test this claim. Eventually, we decided to do a natural experiment based on a literary example we had read about. The writer Jerzy Kosinski, author of the 1969 award-winning novel Steps, allowed a freelance writer named Chuck Ross resubmit a typed manuscript of Steps to fourteen well-known publishing houses, including Random House, the original publisher of the novel. Ross slightly disguised the manuscript to avoid immediate detection-changing only the title and Kosinski's name, nothing else. Remarkably, none of the editorial consultants at these fourteen publishers detected the ruse. All went on to review the manuscript of Steps and all rejected it, despite having won the 1969 National Book Award. Humorously, one publisher's letter stated "while the manuscript was vaguely reminiscent of Kosinski, it lacked his dramatic intensity!"
Would a similar ruse work in the world of scientific journal reviews? Would editors and reviewers fail to recognize already-published articles in their own journals by top authors and go on to reject them? To find out, we selected a number of recently published articles in prestigious psychology journals, had our secretary retype them verbatim, with a few changes to lightly disguise them--replacing the author's name with an unknown name, and paraphrasing the title, so it would not be mechanically detected. Then we would send these disguised articles back to the same journals that had recently published them and request the editor to consider the manuscript for publication in his (the editors were all men) journal, never hinting that his journal had already published it. Would they repeat their prior decision and accept the article for publication? Or would they reject it, citing various reasons? If the latter, it would give some credence to our colleague who complained that his work would be rejected even if it was as good as a colleague's from a prestigious university.
We got permission from the original authors to use their articles in this way; assuring them we had no intention of actually republishing their work and would debrief the editor of the ruse after the experiment. In addition to asking whether editors and reviewers would repeat their previous publication decision when the authors were no longer eminent scholars from top universities like Harvard and Stanford, we wondered if the reverse might occur: would they accept a previously rejected manuscript by an unknown author when it was resubmitted to them with the name of an eminent person on it as the author?
We set about preparing the materials to do this experiment. We got permission of eminent authors at top universities to use their recently-published articles, we had a printer create stationary for a fake research center that connoted low status (the Northern Plains Center for Research), and we rented a post office box in the neighboring state of Minnesota to use for correspondence so that editors and reviewers would not connect the manuscript to the University of North Dakota in the event they detected the ruse and told other editors to beware of manuscripts sent to their journals from persons at our university.
In the first wave of the experiment, we sent thirteen manuscripts back to their same journal with cover letters written on our fake stationary and asked the editors (all but one of whom were the same editors who had made the decision to publish them previously) that "our" papers be considered for publication, without hinting that they had already been published or were part of an experiment looking into claims of editorial bias against authors at lower-status institutions. The second part of the experiment would look at previously-rejected papers to see if they would be accepted when the author was changed from an unknown person to an eminent scholar.
To our delight, almost none of the reviewers and editors recognized the manuscripts as ones recently published by their journals. They went on to review them and in most cases, recommended their rejection. They pointed to flaws in the designs, analyses, theorizing, etc. that they felt invalidated the study. They left no door open for revision! However, we ran into a problem: one editor, whose reviewers had detected the ruse, alerted the association of journal editors. He warned them that someone was sending previously-published articles back to the same journals that had published them, with a few cosmetic changes, to see if they would be re-accepted. Actually, this individual claimed we were actually trying to publish these articles under "our" name, which was, of course, crazy. As already noted, we had promised the authors of these manuscripts that we had no intention of actually publishing them and the names we used were fakes that did not exist in psychology and with a little attention would have been revealed as joke name (e.g., Beulah L. Ardass; Frank Lee Manure). So, the second aspect of the experiment-would previously-rejected manuscripts be accepted when a high-status author's name was substituted for the original low-status author-was derailed.
We tried to publish the results of the first part of the study, reporting the details of the editorial decisions to reject the papers, describing the reviewers' concerns, etc. Somewhat ironically, the first two journals we submitted our results to rejected our paper. It was not clear what the reason was because in both cases the reviews contained both positive and negative comments, mostly the former. It did not help our cause that by this time a campaign to punish us had taken on national dimensions. Below, we describe an unfortunate aspect of this campaign
Obstacles encountered in publishing the findings as well as afterwards.
One angry editor figured out the post office we used in Minnesota was not far from the University of North Dakota, and he surmised that the culprits must be from that institution. He notified the chairman of the psychology department, asking him if he or the ethics committee had approved such a study, threatening that it was perhaps fitting that the entire department suffer the punishment of not having their future work published. At that time (1978), there was no ethics approval that social scientists needed before doing their studies, something that was to be mandated later. Before launching the study, we had given the chair a description of our study, which made clear that we would obtain the approval of the original authors and had no intention to republish the papers. We explained that we were not obtaining the approval of the journal editors because doing so might compromise the study.
Long story short, we were immediately reprimanded. The secretary was told not to type any further manuscripts for us (recall that half the study was incomplete) until we signed an agreement to cease and desist further work on the study. This was the least punitive reaction, it turned out. DPP was coming up for his 5th year tenure review at this time, and the editor's reaction assumed a central part of that process. DPP was roundly criticized for exercising poor judgment in doing a study that could jeopardize the ability of colleagues to publish. (SJC was treated more leniently, under the assumption that he was too new to realize the poor decision he made to do this study, whereas DPP should have known better.) DPP was denied tenure. This led to a lengthy appeals process with affidavits by most members of the psychology department, numerous internal reviews at the college and university level. Interestingly, these reviews recommended that DPP be given tenure, given that his prior annual reviews had been positive. While the appeals process was wending its way through the system, something truly fortuitous occurred. All clinically-accredited programs must be re-accredited every 5 years. The process involves sending out the records of all faculty to an external team of reviewers appointed by the American Psychological Association (APA). Then, this team visits the department and meets with faculty and makes its recommendation with regard to reaccreditation.
DPP had his bags packed. He had resigned himself to a very lengthy legal process that would take years, and his contract ended in one month. Unbeknownst to us, the APA team had written in its report that they were very concerned about his tenure denial, regarding him as one of the top researchers in the department and chiding the department for making such a denial without prior warning. Coupled with the internal reviews that also recommended a reversal of the tenure denial, the president of the university intervened and reversed the department's decision. Our belief is that he was told by the university counsel that the tenure denial would be overturned in court, and the university stood to lose on numerous bases. So, DPP was retroactively granted tenure and given an extended sabbatical leave. Within five years, he was promoted to full professor. SJC left the University of North Dakota for a faculty position at Cornell University, in no small part because of his behavior during this period.
Why was the work so highly discussed?
Our study was highly discussed, ultimately accruing 652 citations (Google Scholar). The reason is obvious: peer reviewed publications are the coin of the realm among academics. Our success in getting our work into journals, particularly top journals, is integral to all that is valued in the academy-hiring, reappointment, tenure, promotion, merit pay raises, professional recognition/awards. Our study, even though it was incomplete, raises doubts about the reliability of the peer review process and hinted at bias in favor of high-status authors and their institutions. To make this insinuation stronger required the very data that we were unable to collect-the publication outcome of previously rejected papers that were written by unknown authors from lower-status institutions when they were resubmitted with high-status names on them. But even without this component, our results raised the specter of an unreliable process in which the fate of a given manuscript being published was unreliable. When it was published in the Behavioral and Brain Sciences in 1982, it was accompanied by approximately 50 commentaries, invited reactions from journal editors in all fields, not just psychology. Most of these reactions were positive, stating that our results should be embraced by the peer review system and lead to the abandonment of non-blind reviewing in which reviewers are told the identity of the authors before making their recommendations. Within a few years of our study's publication*, the APA changed from non-blinded reviews to blinded ones, citing our study as the impetus.
This article and its reviews are distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and redistribution in any medium, provided that the original author and source are credited.