An author, peer reviewer, and a journal editor answer questions about peer review
It’s always interesting to view the same topic from different perspectives, isn’t it? On the first day of Peer Review Week 2021, we had a live panel discussion where our speakers shared their perspectives as an author, reviewer, and publisher. The panel was moderated by @chris leonard (Moderator – Director of Products and Strategy, Cactus Communications) and included Michael Willis (Researcher Advocate, Wiley), @Raj sundaram (Staff Research Scientist, National Institute of Advanced Industrial Science and Technology, Japan), and Dr. @Yufita Chinta (Environmental Science Researcher, Hokkaido University, Japan). Christopher, Michael, Raj, and Yufita talked about how they each view peer review differently and what they expect from each other. The discussion saw a lot of questions from the researchers who had tuned in. Since the panelists could not answer all of the questions during the session, they shared their responses here below.
And don’t forget to catch the recording of the session here.
Is it right for a co-author of a paper to be selected a reviewer? If yes, why?
Michael Willis: If you mean is it right for the individual to review the paper on which the reviewer is a co-author, definitely not! If you mean is it right for a co-author to be selected as a reviewer of another paper, yes, that's perfectly fine if the reviewer has the expertise and skills to review a paper.
Rajyashree Sundaram: Being chosen as a reviewer for the same paper in which the person is a co-author is a conflict of interest, and is NOT right. I don’t think this happens/can happen in general. Authors/the journal putting forward a former co-author as a reviewer for the current paper seems to be a gray area. It is a gray area in terms of ethics (in my opinion) as one cannot guarantee impartiality, especially if it is a system where the reviewer can see the author list. If the journal/editor chooses a former co-author as a reviewer, authors cannot do much about it. When the authors put forward a former co-author, some journals have no issues with it (anyway it is up to the editor to choose the actual reviewers). However, some journals/publishing houses have a rule that authors should not propose former co-authors who have published together in the past X (3-5) years as reviewers. This is reasonable, in my opinion.
Yufita Chinta: I am not sure about this. As far as I know, a co-author is not eligible to be the reviewer of her/his own manuscript through the journal review system. This is to make a fair process of the peer review.
I am a Ghanaian on a PhD program in India, what I love about my supervisor is he always passes on his reviews to me to handle them for him but will go through with me afterwards before he submit his report. It is about being proactive.
Michael Willis: You have a great supervisor – you have the opportunity to learn and be mentored. Be sure that you get the credit for your contribution - the journal should know that you are doing the bulk of the work.
Raj Sundaram: Good that you feel that way. It is good training initially as far as later you get your own papers to review, are initiated into academic circles/network as a reviewer in your own right and get credit for your reviewing activity – with your effort and time recognized by your institution/department/on platforms like Publons/by journals (certificates given in your name), etc.
Yufita Chinta: I feel your positive thinking about it, which is good. Handling the review becomes a big practice for me on publishing, so I think about it positively too. For your case, I think as long as you communicate (i.e., checking the responses of the comments each other) well with your supervisor, that will be great and fine.
I was once invited to review an article from an international journal. Can I request the editor to issue a certificate for peer review as an evidence for the service of peer review?
Michael Willis: Yes, you can certainly request this but it will be the journal's decision as to whether or not it can issue the certificate. More and more journals do this now.
Raj Sundaram: Some journals already have this system of certificates automatically issued after the review leads to a final decision. You can ask the editor – but unlikely they might issue a certificate if there is no such system already in place. In this case, you can register your review activity on platforms like Publons.
Yufita Chinta: I have the similar experience. Luckily, they mentioned about providing the certificate at the beginning. It means that receiving the certificate for giving peer review service is possible. I think you can ask the editor. Asking is free.
A lot of ECRs feel like they're not ready to start reviewing. Is there any specific point in your career where you felt "READY"? What led to the switch?
Raj Sundaram: In my case, opportunities to be a reviewer came by after I started publishing my own papers. Therefore, I had already had a glimpse of the publishing system. Before the first review, I took an online course and read up the "how to review" section of the journal to understand what is expected from a reviewer. For journals I review for the first time, I read the "how to review" section anyway. This is important I feel, how much ever we may think ourselves/seem to be "seasoned reviewers" as the publishing industry is a rapidly (ever)changing landscape. I suppose, I tried to be prepared for the role as best as I can. I don’t remember if I felt "ready" or not. In hindsight, I seem to make decisions not on the basis of if I feel ready, but on the basis of practicalities. Being a reviewer was important for my career – to form and grow networks indirectly. For instance, if the paper was interesting, sometimes, I would get in touch with authors after publication. Being a reviewer was important to also grow as a scientist – to learn more about publishing, writing, critical thinking, and as a source for new ideas. When we review a paper, new ideas pop up because we are forced to look at the loopholes of a work, see knowledge gaps, learn about a related but new area, etc. Last but not least, I also felt it was my duty to give back to the community my two cents as a reviewer – just like how reviewers of my previous published papers (and papers we read and build our work on) had volunteered their services.
Yufita Chinta: This question makes me smile, since I too am a beginner in reviewing papers. The specific points I have to ensure myself that I am READY to review are:
(1) I can allocate some time.
(2) I have enough knowledge about the subject.
(3) I dedicate myself to support the papers to be published.
I think my experiences on writing my own papers lead to the switch. When I receive an offer to review a paper, I firstly feel how hard the authors work for it. Thus, I check whether I have those three points to make me READY at that time.
What would be the reviewer expectation from editor?
Michael Willis: If you mean, "What should the reviewer expect from the editor?" I think reviewers expect the editor to provide them with everything they need to assess the manuscript fairly and robustly. That means they will be given sufficient time to review, and they'll be provided with all the materials (manuscript, supporting information, data, etc.) that are needed.
Raj Sundaram: As a reviewer, my expectations from the editor are:
(1) Act as a bridge between reviewers and authors
(2) Take immediate and fair action when there are issues (miscommunications, when poorly written rebuttals and major issues in science/data/misconduct are flagged, etc.)
Yufita Chinta: I don’t have many experiences yet, but I would expect the editor to do two things:
(1) Input my background to the reviewer database, so that the editor won't send me papers from a different subject
(2) A balanced speed of response (i.e., sometimes it happens when the editor asks me to respond quickly, but the editor responds to me slowly).
Do editors, like referees, read articles in detail? Or do they merely reflect upon the comments of the reviewer to the authors?
Michael Willis: Editors should read the article but not necessarily with an eye for the detail. They are most interested in knowing whether the article is in scope for the journal, and whether it seems - at face value - scientifically robust. They entrust the reviewers with the task of providing a detailed critique of the article. When they receive the reviewers' comments, they are likely to read them in conjunction with the article, to see if they think the reviewers are fair in their critique and to decide whether or not they should accept or reject the article.
Raj Sundaram: Overall, editors seem to read to the depth necessary to evaluate if the article can pass the desk stage into the review stage – essentially to the extent necessary to form first impressions (novelty, journal fit, impact, overall scientific quality, etc.). In case there are issues flagged by authors/reviewers during the review process, editors do seem to read papers in-depth.
Is an interactive peer review platform where all reviewers anonymously (if possible) and collectively share ideas on a paper possible? Because when they do this individually, they sometimes end up confusing the author(s) with their differing views.
Michael Willis: Interactive peer review is most commonly a collective platform for reviews – and yes, there is a risk that authors may receive widely diverging comments. This is where editorial oversight is really powerful, because it helps guide the author in knowing how to respond and revise the manuscript. Without editorial oversight, the author can pick and choose what they want.
Raj Sundaram: Yes – some publishing houses use an interactive peer-review platform. I feel there are upsides to interactive review if the reviewer number is limited (say to 2-3 reviewers + comments from editor and desk editors) and the platform is structured well. The platform structure and ease of use can make a night-ord-day difference in enabling clear communication (in my opinion). MS-Word like comments/comment section under blog-type of "interactive" platform from many people can end up confusing everyone. But if the interactive review platform is built appropriately, confusions can be mitigated.
Yufita Chinta: I love this idea. Hopefully I will get the experience of this interactive peer review someday. Thank you for bringing this question up here.
Usually famous researchers do not get harsh comments. Why? Is it due to their reputation?
Michael Willis: I don't know that there is hard evidence for this. But if it is true, it does make a strong case for double-anonymous peer review.
Raj Sundaram: Anecdotally, this seems to happen sometimes. Perhaps, reputations, wide networks and power, may have a role to play. Famous researchers are well connected and collaborate widely with fellow researchers/former students – who may go on to be editors/be absorbed in the reviewer pool. Halo-effect (in any case) cannot be ruled out, I guess.
Is there any bias regarding country from which manuscript is submitted?
Michael Willis: There should not be, but unfortunately even the best editor and reviewer can have unconscious bias. Double-anonymous peer review can mitigate against this risk of bias.
Raj Sundaram: More than the country, biases (including Halo effect) may happen based on how well-known/powerful the labs from which the manuscripts come. Typically, due to more resources, labs in certain countries have higher probability to become more powerful/well-known/well-connected. Some of these labs also tend to produce higher quality work due to more resources. At least some biases (related to country of origin, race, gender, etc.) can be reduced by anonymizing authors, in my opinion. In addition, bias is inherent human behavior (sub-conscious bias). Awareness and training of reviewers, editors, etc. could help reduce bias creeping in and negatively affecting decisions.
How long should one wait to get response from reviewers after having revised and submitted to the journal to know what is happening?
Michael Willis: Journals should give authors an indication of how long the process can take. If only minor revisions have been requested, the editor may be in a position to make a decision on the article within one to two weeks (depending on the subject discipline), but if major revisions have been requested the editor will probably send the article back to the original reviewers and allow them another few weeks to look at the manuscript. Note that timelines will vary from field to field, and time of year (public holidays in some parts of the world such as Christmas can delay the process).
Raj Sundaram: Depends on the journal, subject area, timing (year-end, festivals, natural disasters, personal problems, sickness), level of revision required, etc. In my opinion, a reasonable maximum wait time before inquiring is 1 month.
Yufita Chinta: I think this is depends on the journal and the level of the revision (i.e., minor, moderate, or major). In my case, I waited for around a month and 1-2 weeks after resubmitting my papers with a major and minor revisions, respectively. Some journals have an online and open submission system showing the status of the manuscript, e.g., 'with editor', 'under review', 'need revision', 'accepted', and so on. This online system is really helpful for me personally. Unfortunately, some journals do not have a similar system which makes me panic sometimes. For this case, I predict what might happen and how long at least until I receive the first review. For example, 'with editor' and 'reviewers assigned' status (i.e., when the editor reads the cover letter, manuscript, and data presentation and finds the appropriate reviewers) need 1-2 week; 'under review' status (i.e., when the reviewers agree to review the manuscript and start to review) needs 1-2 months (for the first time review). Based on this simple prediction, I can be a bit relaxed during the waiting time.
What can we do as researchers when we feel that the reviews are based on the biases of the peer reviewers which can significantly change the direction of the paper?
Michael Willis: There is no harm in appealing to the editor, stating your concerns and asking that the editor reconsider how the paper should be revised.
Raj Sundaram: In the current system
(1) Try to negotiate with peer-reviewers during the review process rationally (but politely) based on evidence - literature and data, try to understand the reviewer viewpoint, mitigate direction change if necessary and reach a middle ground. Sometimes, a direction change (even if we don’t like it as authors) may be necessary to improve paper quality. I try to see if this is the case as an author when I get reviews that change the direction of the paper.
(2) If reviews are obviously biased, flag with the editor with evidence (data, communication record between authors and reviewers, etc.).
Yufita Chinta: This is a good question. As an early career researcher, I am confused on making a decision about whether I need to follow all of the reviewer's suggestions or not. Personally, I sometimes have an insecure feeling if I do not follow the suggestions. I worry if my manuscript will be rejected if I do not follow the suggestions. However, I am learning to scientifically defense, especially because the manuscript is mine and has been constructed with its story and direction based on many discussions before landing to the reviewers' tables. The reviewers may give their perspective and suggestion from different points of view, as what we expect from the expert. But, the direction has been decided. So, it should be the authors who control the direction by following or not following the suggestions. Because the manuscript is a scientific paper, the agreement and disagreement, especially the disagreement, need the proper scientific reasons. I think the reviewers will receive the defense then. This is one of the expectations of a reviewer to the authors that have been mentioned by Raj during the live panel discussion.
How can peer review be done more constructively?
Michael Willis: Peer review is a social phenomenon – it's about how individuals interact with and behave towards each other. Therefore, to be more constructive, it's necessary for researchers to think more constructively about their relationships with other researchers. That will mean they should be fair, they should seek to improve the work rather than destroy it, they should try to help science progress rather than slow it down, and they should put themselves in the author's shoes - 'What would I want this author to say about my work?'
Raj Sundaram: Some suggestions:
(1) Improve transparency – open reviews and/or double anonymized, provide space to share raw data (open data).
(2) Make review process more structured – give out templates for both reviews (to reviewers) and rebuttals (to authors).
(3) Have avenues for support for both reviewers and authors – e.g., have a desk editor to help. authors and reviewers understand each other, bridge gaps in communications/misunderstandings
(4) Create avenues for redressal – Have a formal process to flag issues of misconduct on the part of authors and reviewers with editors/journals.
(5) Create a formal structure for training, feedback, re-training reviewers – e.g., have graded courses (on various aspects ranging from ethics to critical thinking) for reviewers to choose and maintain a qualified quality reviewer pool, provide feedback on review quality, re-train on courses as may be required.
Yufita Chinta: I am not sure, but the interactive peer review system that has been questioned earlier is a good idea for obtaining the constructive peer review process.
I have had a bad experience with a journal. They had my paper for more than 8 months and they got one reviewer who sent minor comments. The editor then sent me a rejection. How could I complain about that?
Michael Willis: You are at liberty to appeal to the editor and ask for another opinion, unless the editor has made it clear that the decision is final and cannot be appealed. Note that the COPE Guide to 'Ethical editing for new editors' includes this: 'You should ensure that peer review is undertaken in a timely fashion so that authors do not experience undue delays.' It is possible that the editor had difficulty in finding another reviewer so decided to make the decision based on one review, and also took his/her own interpretation of the paper into account. In this case the editor should have provided an explanation in the decision letter.
Raj Sundaram: If I were in your shoes, I would not fight it. Instead, would take the rejection as a chance to publish elsewhere in another journal. Also, it is not normal to keep a paper for 8 months. Please check the turnaround times of journals before submitting. In future experiences, might be better to inquire 1 month after submission if there is no response/progress. Also, better withdraw the paper if it takes > 2 months (after submission) with no response even after inquiring and submit elsewhere.
Yufita Chinta: First of all, I am sorry to hear this bad experience of yours. I am not sure if my comment can help you. I think you can write to the editor to clarify the rejection, while you get a minor revision status from the reviewer. The clarification can include whether you have a possibility to resubmit after revising the manuscript based on the comments or you are not allowed to resubmit. If you get a delayed response for your clarification, then I think you can move on to a new journal.