A plea for reasoned reviewers – advice for writing a review

How to be a good reviewer (a view from an editor)

“A reviewer at the American National Institutes of Health evaluates a grant proposal.” From a good overview on peer review. http://en.wikipedia.org/wiki/Peer_review

I’m on the editorial board of two journals and over the years have seen many reviews come in. What I have been impressed with is the overall quality of the reviews. Many of you are excellent and constructive reviewers. I have consistently learned from you on how to be a better reviewer (and editor) myself. This post is aimed more for the smaller fraction of reviews that I have seen – as well as a reminder for the rest of us.

Here, I thought I would summarize my frustrations with some of the reviews that I have seen.  While still relatively rare, unfortunately, these seem to be increasingly more common.  This list is by no means complete and there are likely some items I’ve forgotten. Further, there are several good guides out there on how to be a good reviewer (see here, here, and here) including a good one over at Dynamic Ecology.

Remember, the goal of the reviewer is to: (i) offer a clear path for for the authors for how the science could be improved; (ii) help educate the editor on why the paper is potentially important and where it stands/fits in with the literature; (iii) provide thoughtful, scholarly, and constructive advice for how the science could (if needed) be improved. As an editor I rely on my reviewers to offer a clear perspective on the novelty and importance of the paper. Further, I want to know how the paper fits into the bigger picture of the field – a scholarly view of the contribution. I look to you as a reviewer to provide this sagely view. . . .

My reviewer pet peeves (or a guide to providing constructive scholarly reviews)

  1. I know you as a reviewer are busy but it does not give you an excuse to be cranky and not provide any constructive comments. It may be the worst paper in the world – that does not mean you have liberty to trash talk authors, question, approach etc. The goal of a reviewer is to be constructive no matter how much you dislike the paper. Our science will only succeed if we help out our fellow researchers. Its your duty.
  2. PLEASE organize your review!  You would be surprised with the ‘stream of unedited consciousness’ that sometimes I get back from reviewers. If I have to take time to reread your review over and over again so that I can understand your rational argument it probably means you have issues that go beyond the scope of the paper you are reviewing. Reviews should provide a clear roadmap for the authors and the editor for the importance of the contribution and how the science could be improved.  I want to see a clear rational argument why a paper should be rejected or modified. A laundry list that is unorganized only makes my job as an editor more difficult and says that you really don’t care.  See here for a good bit of advice for how to organize your review .
  3. Don’t say  ‘this is well known in the literature’, ‘we already knew this’, ‘ the results have already been shown’ etc. without providing citations or justification. Anyone can type these words – its more difficult to backup your rational. As an editor I am looking to you to provide clear logic. You would be surprised how many reviewers do not back up their assertions/claims without citations!  Explain just why ‘it is well known’ and show me who actually ‘showed these same results’ with providing a citation and/or quantitative reasoning.
  4. Please do not use ‘The paper is not novel enough for this journal’ as your main critique. Do not use ‘not novel enough’ as an excuse for rejection  I see this comment all the time. It is really just code or short hand – its a sneaky way to shoot down a paper that you don’t like for whatever reason without providing solid justification. Its a cheap shot. Don’t do it. If you use the *not novel enough* card you should at least be able to point to very clear rational reasons (note plural) why as well as provide citations.  You would be surprised how many reviewers resort to such a gameplan for shooting down papers.
  5. Don’t move the goalpost in your reviews. Yes, new issues may come up in the review but let’s be reasonable here. If you keep changing the rules of the game it just slows everyone down. Unless the issues are fatal to the central conclusions moving the goalpost comes across as another tactic for rejecting papers you don’t like.
  6. Don’t get overly bogged down in the weeds of statistical minutia, writing style, figure clarity etc. etc.  As an editor (and as an author) I want to hear if the goals of the paper and the science are good. Step back and focus on the Science not the weeds. . . Focus on the conclusions of the paper. Do your suggested changes or the issues you may have really impact the conclusions of the paper?  Is your issue a fatal flaw in the conclusions or are you just recommend how you would do the analyses? I want to know if the conclusions will change not how you would have done it.
  7. Bad organization is not necessarily a reason to reject a paper. Sure the writing may be crap or you may organize the paper differently but its your job to see the forest for the trees.
  8. I know you think the author should cite your work (or your favorite papers) but that is not an excuse to reject the paper.
  9. Please no personal negative comments, hearsay, hyperbole etc. A review is no place for water-cooler talk, gossip or for cheap shots.  While I thankfully don’t see many reviews like this I have seen enough reviews that refer to “discussions I have had with others that did not like this work” or “the opinions of my colleagues and lab group” or “things I have heard about so and so” etc.
  10. If the manuscript is clearly from an undergraduate or graduate student try to go out of your way to be extra constructive. There is nothing more demoralizing to a young scientist than a terse, sloppy, bad review. One of the best experiences that I recently observed was when one of my newer graduate students received back a set of helpful and nice (yes, they were actually nice) reviews. You could see the light go off -‘ ah! this is how Science works! It is a peer self improving endeavor’.  Those helpful reviews did more than I ever could do as an advisor.
  11. Be on time . . . mea culpa! This is one that I constantly struggle with as a reviewer myself. I certainly have come in late on reviews.  Good science relies on all of us to be timely and we should all remind ourselves that publishing papers is one side of the publishing coin. We need to assist our colleagues with providing our reviews and doing so in a timely fashion is part of the science contract we all abide to.

10 Comments Add yours

  1. Hi Brian, amazing and very useful post. Thanks for that. I hope that I’m doing a good job as a reviewer 😉 I will anyway try to use your tips for my subsequent reviews. I do have one question for you, is there some guidelines or reasons for signing or not a review? I must admit that at the beginning I always remained anonymous in my reviews but recently I did sign two of my reviews maybe because I was influenced by others doing it… I actually don’t really know what to think about that. As an author I actually like when people sign their reviews but I wonder what the editors think about this way of doing. What is your own opinion about that? Shall we always sign our review for transparency and maybe also for visibility?

    1. Thanks for the note Jonathan. Folks generally have widely divergent views on the pluses and minuses of signing your reviews. I don’t have strong opinions here but think that there are more benefits to signing reviews and lean toward more openness on review signing but there may be situations where it is likely best to remain anonymous.

  2. schadtc says:

    While some of these points are clearly valid, as a reviewer (and also an editor) I have to disagree on a few of them. I am seeing more and more papers submitted that have no business wasting a reviewers time, let alone journal space. I would say roughly a 20% of the papers probably should have never made it off of the authors desk, or out of the editors que. In my opinion editorial rejections should be used more often! Maybe this has made me bitter, but so be it.

    #2. Constructive comments, yes, definately. However I often can find myself cranky as a reviewer. Especially if it generally is a very bad paper. In reality my time is VERY limited and precious. If you send me a crappy product to review and waste hours of this time, don’t expect me to be nice.

    #3 and #4. I dont see it as my job as a reviewer to provide a relevant bibliography for the science area. The authors of the paper should be doing that. Sure, a reviewer should give them a couple of citation hints if they have missed one or two. However often as not, many papers I see have not bothered to really study and cite properly the work in their area. Sometimes it should be enough from to hear from another expert in the area (whom you have chosen as an editor) that the authors have not cited the relevant work, or that their work is not novel. If the authors can’t figure that out on their own… not my problem as a reviewer. It is not code for anything.

    #6. If we are not getting into the weeds than what is point of the exercise? There are many, often small, flaws that can kill the usefulness, conclusions, or relevance of a study.

    #7. If the writing is crap I will reject it every time. If you can’t understand the author’s point, then what is the point of publishing the paper? Does it really improve the state of our science? Organization and clarity of presentation are not optional, it’s a must!

  3. Hi Christopher, (https://schadtlab.wordpress.com/people/),

    Thanks for your candid thoughts here. I certainly understand that a reviewers time is previous. I do agree with you on the point that if a paper does not reach a minimum bar more rejections should occur at the editorial level so as not to waste everyones time – that is perhaps the topic of a later post.

    If a paper does make it through the editorial filter we are looking to our reviewers to help us assess the quality of the science and importance of the contribution. As an author I am looking to the review process to provide critical and constructive input so that the science is improved.

    I guess we disagree on a few points here. I do see it as our job as reviewers to provide a relevant bibliography for the science area if it is needed. The literature is big and none of us can master it all. As an editor, I’m looking toward a certain reviewer because of her or his scholarly expertise in a given area so that the contributing science is improved. Its our job to do this. Also, reviews should primarily focus on the rational behind the central conclusions and discussion points of the paper (the forest so to speak). Sure there may be analyses that could be better, different, redone, assumptions tested, alternative models fit, alternative statistical paths to take, etc. but in the end if a clear case cannot be made that these suggestions could/will change the central conclusions then these such a review is not productive and in the end is just wasting everyones time. Regarding the writing, I do in principle agree that working through a poorly written paper is challenging if not painful. If it is impossible to even understand what was done or the methods used then full stop – this is grounds for rejection as one cannot follow the path to the conclusions. If the science is sound or mostly sound but the writing needs improvement or if the reviewer would rewrite or reorganize differently then I don’t see these points as grounds for rejection.

    To be clear our reviews should keep the following three points in mind

    (i) offer a clear path for for the authors for how the science could be improved;

    (ii) help educate the editor on why the paper is potentially important and where it stands/fits in with the literature;

    (iii) provide thoughtful, scholarly, and constructive advice for how the science could (if needed) be improved.

    Again, as an editor I rely on my reviewers to offer a clear scholarly perspective on the novelty and importance of the paper. Further, I want to know how the paper fits into the bigger picture of the field – a scholarly view of the contribution.

  4. agnerob says:

    Thanks for this interesting discussion on reviewing manuscripts. May I add to the discussion by referencing a couple of articles from the social sciences that address this very issue (Dutta, 2009) and the challenges reviewers face in rejecting manuscripts (Robinson & Agne, 2010).

    Dutta, M. J. (2006). The ten commandments of reviewing: The promise of a kinder, gentler discipline. Health Communication, 197-200.

    Robinson, J. D., & Agne, R. R. (2010). Kindness, gentility, and rejection: An analysis of 99 manuscripts. Health Communication, 504-511.

  5. fergdoug says:

    I agree with much of this. I, too, was an editor and even had problems with members of the editorial board, some of whom refused to review articles because they were “too busy.” I hardly ever say ‘no’ to a chance to review, because finding a little time to see a manuscript seems a privilege. My pet peeve is the length of time reviewers are given by editors, often much too long. I get a request about once a month, but I simply set a reminder on my calendar a few days before the deadline and forget about it. (I figure that the other reviewer(s) won’t do the review early, so why should I?). Thus, the manuscript can simmer for months, because of the whole “too busy” bias. I think four weeks is more than enough time to turn around a manuscript. If not, just say no. When I was editor, I sent reminders to everyone (and removed board members who were perpetually late).

  6. Amaefule Julia Ifeoma says:

    It was a thoughtful suggestion. I like it

  7. Hi Brian et al.,
    Your points and the papers mentioned are very useful, especially as none of us ever had a proper course in reviewing.

    – May I add 1 point to your list: do not underestimate the time it costs you to properly review a ms. The whole review process costs me generally 4-7 hours per ms. I always print out the ms, as you’ll have a better overview of its structure on paper than on the screen, stapling all text together, and separately all figures and graphs. I start with reading through the full ms, marking the points where I am not sure of or have doubts.
    I’ll subsequently let the ms simmer for 2 nights and then read it thoroughly again from beginning to end. Whereas I am often positively impressed at first reading, the critical issues often only come up after the 2nd time…. Properly formulating these issues in a way that is useful for the authors takes another considerable chunk of time.
    Now, reviews can be amazingly different in their quality and the critical issues they overlook or uncover. It would be interesting to know how much others think that is necessary for a good-quality review…

    – Related to that may be the question whether it is fair both towards your colleagues who spend so much time in doing the research and writing the ms as to the editor to accept a review request if you know you actually cannot spend the proper time for it? It is true that we all do this work for free, but if the professor actually is so busy, I think the job could often be done better by a post-doc who has more time to spend.

    – Having said so, as an author, please have your ms always read by an outsider before you send it in for review. So many unclarity that costs a lot of time of reviewers and editor could have been easily spotted and removed if a fresh mind of a student or colleague had checked the ms..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s