50度灰

How to peer review

Author tutorials 

Evaluating manuscripts

When you first receive the manuscript it is recommended that you read it through once and focus on the wider context of the research.

Ask questions such as:

  • What research question(s) do the authors address? Do they make a good argument for why a question is important?
  • What methods do the authors use to answer the question? Are the methods the most current available or is there a newer more powerful method available? Does their overall strategy seem like a good one, or are there major problems with their methods? Are there other experiments that would greatly improve the quality of the manuscript? If so, are they necessary to make the work publishable? Would any different data help confirm the presented results and strengthen the paper?
  • Were the results analyzed and interpreted correctly? Does the evidence support the authors鈥 conclusions?
  • Will the results advance your field in some way? If so, how much? Does the importance of the advance match the standards of the journal?
  • Will other researchers be interested in reading the study? If so, what types of researchers? Do they match the journal鈥檚 audience? Does the manuscript take account of international and cultural differences? Is there an alternative readership that the paper would be more suitable for? For example, a study about renal disease in children might be suitable for either a pediatrics-centric journal or one that is targeted at nephrologists.
  • Does the manuscript fit together well? Does it clearly describe what was done, why it was done, and what the results mean?
  • Is the manuscript written well and easy to read? If the manuscript has many mistakes, you can suggest that the authors have it checked by a native English speaker. If the language quality is so poor that it is difficult to understand, you can ask that the manuscript be corrected before you review it.

After your first reading, write one or two paragraphs summarizing what the manuscript is about and how it adds to current knowledge in your field. Mention the strengths of the manuscript, but also any problems that make you believe it should not be published, or that would need to be corrected to make it publishable. These summary paragraphs are the start of your review, and they will demonstrate to the editor and authors that you have read the manuscript carefully. They will also help the editor, who may not be a specialist in this particular field, understand the wider context of the research. Finally, these paragraphs will highlight the manuscript鈥檚 main messages that will be taken away by readers.You can then proceed in evaluating the individual sections of the paper.

Title, abstract, and keywords

The title, abstract and keywords are items that will help other researchers to find the published paper and decide if they will read further. Abstracts must be a clear, short summary of the full manuscript. Researchers want their work to be read, so it is important that their abstract be interesting and hold the reader鈥檚 attention. Some questions to ask yourself about the title, abstract and keywords are:

  • Does the title accurately say what the study was about? If not, can you suggest a different title?
  • Does the abstract effectively summarize the manuscript? Could the abstract be understood by a researcher outside your specialty?
  • Does it include enough information to stand alone? Does the abstract contain information that is unnecessary?
  • Is there any information in the abstract that is not in the main text of the manuscript?
  • If present, will the key words help readers to find the article? Are they specific, and do they represent the manuscript content?

Introduction

Like the title and abstract, the Introduction tells the reader what the manuscript will be about. However, unlike the abstract, the Introduction gives the background for the research question.While reviewing the Introduction, ask the following questions:

  • Does it explain the background well enough that researchers outside your specialty can understand it?
  • Does it accurately describe current knowledge related to the research question?Does the Introduction contain unnecessary information? Can it be made more concise?
  • Are the reasons for performing the study clear?
  • Are the aims of the study clearly defined and consistent with the rest of the manuscript? Does it introduce the research鈥檚 theoretical framework? 
  • Have the authors missed any key references that would be important for a reader to access? Make suggestions for additional, relevant references if necessary.

Materials and methods

The study鈥檚 methods are one of the most important parts used to judge the overall quality of the paper. In addition the Methods section should give readers enough information so that they can repeat the experiments. Reviewers should look for potential sources of bias in the way the study was designed and carried out, and for places where more explanation is needed.

The specific types of information in a Methods section will vary from field to field and from study to study. However, some general rules for Methods sections are:

  • It should be clear from the Methods section how all of the data in the Results section were obtained.
  • The study system should be clearly described. In medicine, for example, researchers need to specify the number of study subjects; how, when, and where the subjects were recruited, and that the study obtained appropriate 鈥榠nformed consent鈥 documents; and what criteria subjects had to meet to be included in the study.
  • In most cases, the experiments should include appropriate controls or comparators. The conditions of the controls should be specified.
  • The outcomes of the study should be defined, and the outcome measures should be objectively validated.
  • The methods used to analyze the data must be statistically sound.
  • For qualitative studies, an established qualitative research method (e.g. grounded theory is often used in sociology) must be used as appropriate for the study question.
  • If the authors used a technique from a published study, they should include a citation and a summary of the procedure in the text. The method also needs to be appropriate to the present experiment.
  • All materials and instruments should be identified, including the supplier鈥檚 name and location. For example, 鈥淭ests were conducted with a Vulcanizer 2.0 (XYZ Instruments, Mumbai, India).鈥
  • The Methods section should not have information that belongs in another section (such as the Introduction or Results)

You may suggest if additional experiments would greatly improve the quality of the manuscript. Your suggestions should be in line with the study鈥檚 aims. Remember that almost any study could be strengthened by further experiments, so only suggest further work if you believe that the manuscript is not publishable without it.

Results and figures

Readers will usually first look at a manuscript鈥檚 title, abstract and results. Therefore the results section including any figures and tables are some of the most important parts of the manuscript. You should carefully examine the figures and tables to check they accurately describe the results. If you think it necessary, you can suggest any changes that would make the results easier to understand.

  • For figures, check that the plotted parameters are clearly defined. Figures and tables should include measures of uncertainty, such as standard error or confidence intervals, as well as the sample size.
  • Table headings and figure legends should be detailed enough that readers can understand the data without reading the main text.
  • Look for places where data are unnecessarily repeated in figures, tables or main text. The text should point out key findings or trends, not repeat data presented elsewhere. Similarly figures that present a very small amount of data can often be combined with another or deleted and replaced with an explanation in the manuscript text.
  • If a result is not central to the study鈥檚 aims, it is often acceptable to summarize it but not present the data. However, failing to show important data, or too many instances of 鈥渄ata not shown,鈥 are unacceptable and you can recommend that it be added into the main manuscript.
  • Interesting data that are not needed to support the study鈥檚 major conclusions might be better presented as supplementary material rather than the main text of the paper; feel free to point out such data in your comments.
  • Feedback on whether the data are presented in the most appropriate manner; for example, is a table being used where a graph would give increased clarity? Do the figures appear to be genuine, i.e. without evidence of manipulation, and of a high enough quality to be published in their present form?
  • Watch for places where the authors have included interpretations in the Results section. This section should simply state what the results were, not what they might mean. Interpretations and inferences belong in the Discussion section. (However, for journals that combine the Results and Discussion sections, results and interpretations do not need to be separated.)

Statistics

Most scientific manuscripts include statistical analysis, and a study鈥檚 conclusions depend on the results of these analyses. If the data are analyzed or reported incorrectly, the manuscript will mislead readers. Therefore, as a scientist, and as a peer reviewer, it is important to have a solid understanding of statistics, and to carefully examine the statistical methods and reporting in manuscripts you review. If you do not feel qualified to fully evaluate the statistics, tell the editor this in your comments so that they know to ask someone else to review them.Some questions to ask as you review statistical analyses and results are:

  • Was the sample size appropriate and/or justified? Did the authors perform a power analysis as part of their study design?
  • Did the data meet the assumptions of the tests used? (e.g., many statistical tests can only be used for data with a normal distribution. Data such as proportions or counts of the number of events are generally not normally distributed and have to be either transformed or, preferably, analyzed with statistical models suitable for these data types). Were the tests used appropriate?
  • Are the individual data points statistically independent? If there were repeated measurements (for instance, multiple measurements on the same patient), have appropriate statistical models been used?
  • Have potential sources of bias (e.g. confounding variables) been considered and accounted for in the analysis?
  • When percentages are presented, are the numerator and denominator clear? E.g., 鈥淥f the 500 bee colonies, 200 (40%) were affected by the virus,鈥 or, 鈥淔orty percent (200/500) of the bee colonies were affected by the virus.鈥
  • Are p-values reported where appropriate? Generally, a p-value should accompany all statistical comparisons mentioned in the text, figures and tables. The actual p-value should be stated (e.g. p = 0.049 and p = 0.0021 rather than p 鈥 0.05 or p 鈥 0.01). However, it is acceptable to state p 鈥 0.0001 if the value is below this threshold. The Statistical Analysis section should also state the threshold for accepting significance, such as "Values of P 鈥 0.05 were considered statistically significant"

Common problems with methods and statistics

There are a number of common problems you might consider when reviewing the methods and statistical analysis of a study. These include:

  • Replication that is absent or inadequate. Replication is essential in order to minimize sampling error. If a study does not have the right number of replicates, general inferences cannot be made from it and the power of statistical analyses done on the data would be too low. The result of low statistical power is that real differences or treatment effect cannot be detected.
  • Confounding. The problem of confounding means that differences due to experimental treatments cannot be separated from other factors that might be causing the observed difference. Confounding can be avoided by careful experimental design, such as proper replication, controls and randomization.
  • Poor sampling methods. In observational studies, random sampling is needed to make sure that the experimental sample is representative of the whole population. If random sampling has not been used, check that the authors justify their sampling methods.
  • Lack of randomization. In experimental studies, 鈥渢reatments鈥 must be randomly allocated to experimental units (or vice versa), to make sure that the groups being compared are similar and factors that could confound interpretation of treatment effects are minimized.
  • Pseudoreplication. The sample size should reflect the number of different times that the effect of interest was independently tested. For instance, if there are repeated measurements on the same set of subjects, as might occur when measuring individuals repeatedly over a period of time, individual data points are not independent. In these cases, averages per individual, or appropriate statistical models that account for repeated measures (e.g. mixed effects models), should be used to analyze the data. If the statistics are not explained, pseudoreplication can often be spotted by looking at the degrees of freedom (essentially, the number of independent pieces of information) of the statistical tests.

Reviewing review articles

A review article is written to summarize the current state of understanding on a topic, and peer reviewing these types of articles requires a slightly different set of criteria compared with empirical articles. Unless it is a systematic review/meta-analysis methods are not important or reported. The quality of a review article can be judged on aspects such as timeliness, the breadth and accuracy of the discussion, and if it indicates the best avenues for future research. The review article should present an unbiased summary of the current understanding of the topic, and therefore the peer reviewer must assess the selection of studies that are cited by the paper. As review article contains a large amount of detailed information, its structure and flow are also important.

Next: Evaluating the discussion and conclusions

Tip: If you suspect image manipulation or believe it would be beneficial to see the uncropped and unedited versions of the images inform the editor in the 鈥榗onfidential comments鈥 to the editors section. They can then request the original figure files from the authors.

For further support

We hope that with this tutorial you have a clearer idea of how the peer review process works and feel confident in becoming a peer reviewer.

If you feel that you would like some further support with writing, reviewing, and publishing, 50度灰 offer some services which may be of help.

  • offers high quality  English language and scientific editing. During , Editors will improve the English in your manuscript to ensure the meaning is clear and identify problems that require your review. With experienced development editors will improve the scientific presentation of your research in your manuscript and cover letter, if supplied. They will also provide you with a report containing feedback on the most important issues identified during the edit, as well as journal recommendations.
  • Our affiliates also provide English language editing* as well as other author services that may support you in preparing your manuscript.
  • We provide both and training for researchers on all aspects of the manuscript writing process.

* Please note, using an editing service is neither a requirement nor a guarantee of acceptance for publication. 

Stay up to date