AI technology was supposed to streamline scientific peer review. Instead, it’s proving easy to fool.
For roughly a decade, new research papers have piled up faster than scientists can rigorously review them. Some researchers have resorted to AI tools to lighten their reviewing load, cutting time spent on paper reviews from days or weeks to minutes. But scientists can easily manipulate their papers to trick an AI peer review tool into rating them as stronger or more publishable than they really are, computer scientist Joachim Baumann and colleagues report. What’s more, AI-generated reviews often sound the same, losing the nuance and diversity of human evaluation.
“We are being swamped with more papers than we have the capacity to review, so we do need some solutions, and automation can help for some parts of it,” says Baumann, of Stanford University. But thorough experiments and evaluation are needed before such tools enter the peer review process, he says. Otherwise, AI tools might inadvertently perpetuate the biases they’re known to carry and reduce the variety of opinions weighing in on new science.
The team will present their findings July 8 at the International Conference on Machine Learning in Seoul, South Korea.
Many researchers have already adopted AI tools in their work. Of the nearly 20,000 papers submitted to the 2026 International Conference on Learning Representations, or ICLR, about 1 in 5 were fully AI-generated, according to a case study from November by the company Pangram. AI is becoming common in peer review, too. A December survey of 1,600 scientists in 111 countries found that more than half had used AI tools to help review papers, including summarizing studies and assessing the strength of a paper’s arguments.
“AI tools are inherently opaque and dilute responsibilities and accountabilities,” says Mohammad Hosseini, a bioethicist from Northwestern University Feinberg School of Medicine in Chicago who was not involved in either study. “When you introduce a nontransparent actor like AI within a system that for a long time was trying to become more transparent, it is a step backward, and there can be unforeseen consequences.”
One consequence could be a loss in the diversity of feedback on manuscripts. Diversity of opinions is important in paper reviews “because a lot of these decisions of whether to accept the paper or not, whether something is novel enough or not, whether a certain limitation is enough to have a paper rejected, these are often very subjective decisions,” Baumann says. “It seems natural that we also want a diverse set of opinions to be represented whenever we automate things with AI.”
In the study, Baumann and his colleagues analyzed AI-generated and human-written reviews of papers submitted to ICLR 2026. The team examined the semantic and linguistic patterns in the reviews and found that those generated using AI tools were much more similar to one another than human or human-assisted reviews.
The researchers also randomly selected 60 ICLR papers and prompted AI models to generate detailed reviews in the manner of a human reviewer at ICLR. Then they asked two large language models to rewrite the papers to obtain higher scores based on the feedback in the AI-generated reviews. In most cases, the scores given by three AI reviewer models after the rewrite were higher than from the AI-generated reviews before.
Most of the modifications made during the rewrite were stylistic, such as the use of hedging words like “may” and “suggests” and emphasis words like “strong” and “robust.” Some of these changes might have made things clearer, but there were also obvious cases of scientific misconduct, Baumann says. Models added findings from experiments that weren’t actually run, in essence making up results, he says. AI-generated reviews of these 60 papers were also far more similar to one other than the human reviews were, both before and after the rewrite, for the same paper.
Many conferences now prohibit the use of AI tools for peer review. Others are experimenting with and evaluating the quality of AI-generated reviews to determine whether AI should be officially integrated into the review process. But while performance on some tasks like checking for hallucinated references and formatting errors can be easily tested, subjective questions about whether a paper’s contribution is meaningful to a research community are much harder to evaluate, Baumann says.
He and other researchers wonder whether AI reviewers would be able to judge new research that goes against prior work or introduces something novel, such as a new experimental setup or a new model architecture. “There just might be certain topics that get a low score from AI reviewers, even though they could be incredibly valuable contributions to the community,” Baumann says.
Their research also found that the 60 rewritten papers were much more similar to each other than the original papers were. There’s a concern that this could lead to an “intellectual monoculture,” the researchers write. If, for instance, many researchers use the same large language model to help them write the paper, there would be more similar papers, and “scientific writing will converge toward whatever style the AI reviewer rewards,” the team writes.
While this is a serious risk, it might not be one exclusive to AI reviewing scientific papers, says Graham Neubig of Carnegie Mellon University in Pittsburgh. “Paper authors have long considered ‘what will reviewers think’ when they write papers, and this can cause them to go for ‘safer,’ more incremental ideas and noncontroversial topics,” he says. “In a way, AI-enhanced review processes may even provide a way to push back against this, by explicitly encouraging AI reviewers to reward more creative ideas.”
Read the full article here
