A brief survey on peer review in scholarly communication

  1. 1.  The Winnower, New York, NY 10009
  2. 2.  Simon Fraser University

Intro

Peer review, the process by which subject experts evaluate a piece of work, is an integral part of scholarly publishing.   In most cases peer review is coordinated by an editor at a scholarly journal and is performed privately amongst two to three peers.  There is no standard report that peer reviewers complete, however, they are generally asked to make a decision if the paper is acceptable for publication or not based on the merits of the paper.  The traditional process of peer review while widely regarded as being crucial in scholarly publishing has become increasingly criticized for its shortcomings.  Strikingly, tests on the veracity of peer review have shown that most major errors are not detected during peer review (Smith 2010, Godlee, Gale, and Martyn 1998, Baxt et al. 1998).  These findings have called for new ways of improving peer review to be explored.  One suggestion is to make peer review more transparent  (i.e. the contents can be read by the public) and for it to occur post-publication (Smith 2010).  Post-publication peer review is organized similarly to pre-publication peer review except that it occurs after a paper has been published or made public.  Post-publication peer review has also been used to describe blogging, journal clubs, and other forms of review that occur on the Internet or in private.  To better understand how researchers view traditional peer review versus open peer review, and other new forms of peer review we surveyed the scholarly community via an informal survey.

Results & Discussion

We invited scholars to take part in our survey by invitation through Twitter and The Winnower emailing list.  We acknowledge that our methods of recruitment may represent a biased population of researchers towards new models of publishing and scholarly communication, given The Winnower’s model of publication and review.  We received seventy-nine responses over the course of a few weeks of solicitation.  Of these seventy-nine, 70% (55) were faculty, librarian, or researcher, including early career researcher with the remainder made up of PhD students, Master students, and other, at 18% (14), 4% (3), and 8% (6) respectively.  Those identifying as “other” in the survey included publishers, independent scholars, and retired professors.  In agreement with this, 61% of respondents identified as being at University.

Figure 1: Employment Level of 79 respondents. 70% (55) were faculty, librarian, or researcher, including early career researcher.  The remainder was made up of PhD students, Master students and other, 18% (14), 4% (3), and 8% (6) respectively. 

Figure 2: Employer/Organization of 79 respondents.

The respondents polled came from a diverse background of study with Biology and the Life sciences making up the largest percentage of the group (Table 1).  Most respondents had previously had their own work peer reviewed (92%) and had performed peer review of others work before (94%). 

  Table 1. Fields of study

Field of Study

% of respondents

Agriculture and Food sciences

0%

Arts and Architecture

1%

Biology and Life sciences

30%

Chemistry

0%

Earth and environmental Sciences

8%

Health Sciences

12%

History and Archeology

1%

Languages and Literature

3%

Law and Political science

0%

Mathematics and Statistics

1%

Physics and Astronomy

7%

Social Sciences

16%

Technology and Engineering

11%

None of these

10%

In general the survey showed that most believe peer review was beneficial, both to give and to receive.  While the merit of reviews was recognized amongst researchers the poll showed most believed that peer review did not aide greatly in the advancement of their career. (Table 2).

Table 2. Benefits of peer review

Question

Strongly disagree

Somewhat disagree

Neither agree nor disagree

Somewhat agree

Strongly agree

I benefit from being peer reviewed

0.0

6.3

3.1

34.4

56.3

Others benefit from my reviews of their work

1.5

3.1

4.6

40.0

50.8

Reviews DO count for career advancement in my current position (salary reviews and/or tenure and promotion)

36.9

16.9

24.6

18.5

3.1

Reviews should count for career advancement in my current unit (salary reviews and/or tenure and promotion)

4.6

4.6

21.5

38.5

30.8

Because peer reviews are currently private and thus hard to measure as an output by promotion and tenure committees and others, it is possible that open reviews may rectify this.  Moreover, would making peer reviews open change the content or tone of them? When asked if peer reviews were made public, would they change? Most respondents answered yes, but to varying degrees (Figure 3).  When asked how, many seemed to suggest that reviews may improve in various aspects from tone to better justification of critiques (see appendix).

Figure 3. Do you think the review would have been different if it had been published for others to see?

Most respondents believe that making their reviews open would not require a significant amount of more work to be undertaken (Figure 4).  If peer reviews were open and could consequently be counted by various evaluators from grant agencies to promotion and tenure where would researchers list their reviews.  The majority responded that they would list them on their curriculum vitae under “professional service” (Figure 5)

Figure 4. Assuming you wanted to: how much more time/effort do you think it would take to make your peer review something that could be posted publicly?

Figure 5. If published, researchers would largely consider reviews as professional service.

Thus, based on our survey researchers see value in peer review, believe that open review would generally improve reviews, and that peer reviews should count for career advancement.  However, most reviews remain private.  When asked what change would need to happen for reviewers to make their reviews public, most said if tenure and promotion committees explicitly valued them and if their peers also published theirs (Figure 6).

Figure 6. Reasons that would incentivize scholars to make their peer reviews publicly available.  Respondents selected two reasons that would make them more likely to take the effort to make their reviews public.

Given the value of openly reviewing work amongst peers, many labs participate in a semi-regular meeting called “journal club.”  In our survey 56% of respondents participate in journal clubs at varying frequencies (Figure 7).  Although the majority of those we surveyed participate in journal clubs, the large majority of them do not publish them (67%).  Again, when polled what would change this most pointed their promotion and tenure committees explicitly valuing them and their peers also publishing theirs (Figure 8). 

Figure 7.  Do you or any of your peers participate in a "journal club" (meet informally to discuss published academic papers)?

Figure 8.  Reasons that would incentivize scholars to make their journal club discussions publicly available.

References

Baxt, William G., Joseph F. Waeckerle, Jesse A. Berlin, and Michael L. Callaham. 1998.\n "Who Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Peer Reviewer Performance." Annals of Emergency Medicine no. 32 (3):310-317. doi: http://dx.doi.org/10.1016/S0196-0644(98)70006-X.

Godlee, F., C. R. Gale, and C. N. Martyn. 1998. "Effect on the quality of peer review \nof blinding reviewers and asking them to sign their reports: A randomized controlled trial." JAMA no. 280 (3):237-240. doi: 10.1001/jama.280.3.237.

Smith, R. 2010. "Classical peer review: an empty gun." Breast Cancer Res no. 12 Suppl \n4:S13. doi: 10.1186/bcr2742.

Appendix

“Because different person has different behavior and skills.”

"the review makes the work  recoverable improving the content and scientific critics are very important!"

“Most reviews asked for clarification of the methodology”

"I can't say how it is in other fields, but in business, I think there is a tendency for reviewers to inflate their own citation counts by requiring researchers to cite their works regardless of how directly they relate to the work in question. Also, I think it is unreasonable for reviewers to ‘rewrite’ the study in question by saying, ‘you should have tested the question by doing X.’ The study is already done. It's too late to conduct the research or redesign how the research should be conducted. Requiring someone to collect more data, etc., is unethical. To do another analysis is OK, but a minor pain.”

“If you want a study run a certain way, write your own paper."

“I've generally found commenters, if anything, a bit repressed when it comes to pointing out problems.  Then again, I've a degree in Analytic Philosophy from the UK, which treats discourse as a blood sport, so maybe I'm biased.”

“reviewers may be less prone to make strong criticisms if they can be recognized.”

“I think it would have been written more as a commentary on the paper rather than just a list of criticisms.”

“I wrote a paper about how Semantic Web Technologies are not a disruptive innovation, but incremental in various fields of application. And I sent it to a conference on Semantic Web Technologies. Reviews were very mixed - one wrote it's a nice contribution, another that it doesn't add any value. Having reviews published could have led to a discussion on whether meta analyses like mine are valued and welcome - of course, provided that anyone reads the reviews, which would be my primary concern.”

“It depends more on whether review is anonymous. May be harder to be critical of a colleague's paper if you are identified.”

“It depends on the question whether the report has been signed by the referee or not. If yes, the referee will take more effort to complete the review because he knows that his identity will be disclosed.”

“I think reviewers in my particular sub-field commonly overlook their personal biases. If reviews were open for others to see and comment, this would become better recognized over time, and reviewers would at least adapt their tone.”

"It would have been better written and more thoughtful in its comments.”

“I have been lucky enough to only experience constructive criticism”

“Reviewers made good points that led to improvements to the manuscript, but the delivery was terse and could have been more respectful.”

“Because it was done by others in my field who knew me or knew of me, I think it may have taken on a more professional, distant tone than it did.”

“Peer review has given some valuable comments.”

“I think some reviewers might have put a bit more effort in if the reviews were published, and given more specific comments.”

“I think some unreasonable requests for endless experiments would not have been made if reviews open. Legitimate criticisms would remain.”

“Critiques would have to be justified by fact and argument”

“Most of my reviews have been fair and helpful. I think the claims and demands made by reviewers would have been a bit less harsh if they'd had to publish them.”

“They would have been forced to elaborate and inform”

“I'm an editor at two medical journals: The BMJ and BMJ Open, both of which use open peer review. The BMJ conducted two randomised controlled trials showing that a) signed review and b) open peer review, with prepublication histories published next to each paper did not lower the quality or depth of content in peer reviewers' reports. The tone was, however, slightly more constructive”

“I have had a solid peer-review experience thus far. I may not agree but comment has been thought-provoking and well-intended.”

“I think it would be more thorough and the critics would be more constructing. I believe that if you are obligated to write a review but you are not really motivated/interested/competent in the very specific topic, your review is less useful or less relevant, than if it would have been done by volunteer experts. However, maybe we need both type of reviewers for the sake of objectivity.”

"1. reviewers would be hesitant to use trashy or insulting remarks - kindness matter and when your name is attached to your trashy report, that is not a cool thing.

2. some reviewers would not accept the review invitation because obviously the paper is not within their expertise. that means better quality of review with better reviewers

3. reviewers would be constructive; after all that is really cool and what I would expect from myself as a reviewer"

“I think that if people have to sign their name to a document, they take more care and effort to review carefully and thoughtfully.”

If it had been published the remarks that are exceptionally rude and/or stupid would probably have been avoided by the reviewers.”

I think that reviewers can be caustic and mean.”

“More careful interpretation and checking of statements. More thoughtful alternative explanations. More justification of opinion re merit of piece”

Reviews

Showing 2 Reviews

  • Placeholder
    Sven Kochmann
    Confidence in paper
    Quality of figures
    Quality of writing
    Originality of work
    1

    I aggree with Jonathan that it should be more emphasized that there are "only" 79 total responses and there should be a concluding section.

    Additionally, maybe you can improve the Figures a little bit: I find the description of the groups on the x-axes hard to read - maybe use patterns and/or colors and a legend instead? Also, how about adding the exact value at the top of the corresponding bar? How about scaling not every figure to 100% on the y-axis but instead to the corresponding maximum? Is there a reason why Figure 6 is a pie chart?

    My suggestion is to use a horizontal bar diagram for everything, so that the bars go from left to right. This way you can easily write the description into the bar (or to the right of it). I think that would be much easier to read and compare!

    Another little idea: You have this nice Table 1 with all these areas. It would be maybe nice to group them together in 3 or 4 larger groups (e.g. Nature science, Humanities, Others...) and then see for certain questions the differences. Of course, since n=79 is a relatively small sample set, generalization is difficult, but maybe it gives some indication? This might also be an option if you have a larger sample set.

  • Peelle headshot 1
    Jonathan Peelle
    Confidence in paper
    Quality of figures
    Quality of writing
    Originality of work
    0

    I have no competing interests.

    This is an interesting and novel contribution looking researchers’ opinions about peer review, particularly regarding open peer review and other potential innovations. An anonymous online survey was conducted asking about various aspects of peer review. The paper reports the results. One interesting finding is the importance of a scholar’s peers publishing their reviews, and the promotion/tenure committee valuing published reviews, in determining reviewers’ likelihood of making their reviews open.

    At the outset it’s worth noting that there were 79 total responses, which is really my only concern about the results. This seems small for a survey, particularly given the heterogeneity of the respondents, and makes me wonder about how the findings would generalize to a larger sample.

    I was surprised to see that most people did not think reviews counted towards career advancement. I agree with this, but always have the sense that one reason junior scholars perform review is a perceived career benefit. Juxtaposed to this is that most of the respondents thought peer review should count more towards career advancement.

    One thing that would have been nice would have been to make the specific questions, and the data/analysis, publicly available. For example, using Open Science Framework (http://osf.io) or GitHub. See also http://centerforopenscience.org/top/. If this is not possible, it would be nice to include the reason why in the paper so it becomes part of the scientific record.

    I found the paper well-written and easy to read. I would have appreciated a short conclusions section to tie together some main themes from all of the results (8 figures worth).

    All in all I found this to be a useful survey. I would be interested to see a version with a larger number of respondents, which might also allow for looking at differences across career stage, discipline, etc.

License

This article and its reviews are distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and redistribution in any medium, provided that the original author and source are credited.