Gary King is Wrong About Peer-Review – But Not the Way You Think

It is neigh impossible to evade LLM-based uproar these days. Only a few days ago, Gary King made the headlines with a(n unpublished) paper titled “AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices.” In the paper, King and his co-author Steven Worthington claim that LLMs now surpass essentially all humans in terms of quality of peer review. The claim is simple: why keep scholars in the miserable peer-review system if AI can do it for us?

Now, there have been three responses to it that you would probably expect: First the apologetics who praise this as the salvation from peer review hell, where scholars perform unpaid review work that chips away at precious time not spent on further research, and instead designate an AI for this sometimes tedious work. Second are the “purists” who reject this idea on the normative basis of a conviction that peer review must remain a human undertaking. And third are people who dissected the actual research presented, took a close look at the paper and its appendix, and uncovered a whole barrage of issues and fallacies that decimate the claims of the paper to rubble.

I personally believe, groups two and three are right, but not necessarily for the reasons you may think of right now. But one thing after another.

What are King and Worthington’s Claims?

King and Worthington essentially drag-raced human reviews against AI-generated paper reviews and tried to see which ones would win. They took a sample of a bit more than 100 papers that had received reviews via Harvard University’s review platform “RapidPeer.” Then, they had an AI agent (Claude Opus 4.8) write additional reviews to all papers, and finally used GPT 5.5 as a judge to see whether the human reviews or the LLM-reviews were better. And it turns out, in almost all cases, the LLM-generated review won against the human reviews, achieving an almost 100% win-rate.

This sounds odd. For research, it is highly unusual to see such a success rate. Typically, we’re happy if we find effects in the range of $r^2 \approx 0.3$, and therefore an almost 100% definitive result should raise our suspicion. And indeed, people have thankfully combed through the linked Online Appendix and found a bunch of issues. I believe the most important issues to flag and keep in mind for the remainder of this article are the following:

• The corpus consisted almost exclusively of quantitative/empirical papers. There was little qualitative work in there.
• The dimensions to compare humans and LLM-generated reviews have been generated by an LLM, and were not checked against human-derived dimensions, so there is no Cohen’s kappa or Cronbach’s alpha (or an equivalent) to see how much humans and LLMs would agree.
• In other words, an LLM generated the dimensions on which it would then later on judge paper reviews. King and Worthington did, however, consult expert reviewers to double-check these findings.
• But as other commentators have pointed out, there are strong indications that these expert reviewers were not domain experts, and as such were potentially lacking important domain context to properly compare the reviews.

(Thanks to Per Engzell and Lauren Donohue as well as some others for their work dissecting the paper and appendix.)

Despite all of these flagged issues, I want to emphasize that King and Worthington did for the most part do their due diligence:

1. They properly requested an IRB review and discuss potential legal issues with their methods.
2. They acknowledged the problem of providing proprietary information to cloud-based LLM models, and have taken steps to ensure the papers don’t end up in some training corpus of Anthropic or OpenAI.
3. They include a large array of robustness checks that check a wide array of potential issues with the paper (though not all).
4. They address many limitations and potential concerns of the paper and its wider implications in the discussion faithfully and comprehensively.

So let the record show that I am generally quite content with the research integrity-side of this paper, as the authors did go a long way to ensure they conducted proper research.

But this also makes the way in which they are wrong all the more interesting. Because I do believe them that LLMs are excellent at reviewing papers, and I have been using LLMs for friendly reviews for a few years now. Before my papers get to any colleague or a journal for review, I typically use an LLM to find some final remaining issues that I overlooked to ensure the paper is in the best-possible state before I make other people look over it.

The Issue Are Neither Their Methods, Nor Their Claim

Now, where do I think King and Worthington are wrong? And why do I think that? Well, the implication of the paper is quite obvious: why make humans do peer-review if an LLM can do it much more efficiently? And that is a question that few have raised so far. Most of the criticism towards the paper focuses either on issues with its methodology (which is not wrong either), or just outright rejects the possibility that LLMs could even perform proper review, while failing to bring compelling arguments for this standpoint.

So let me return to the question: why should we not let LLMs do that work for us? The question this implies is simple: who do we do research for? What is the purpose of research, and, by extension, writing and reviewing papers?

I argue that research is a fundamental human endeavor, and that we write papers not to document some findings, but to discuss these findings and contextualize them in the broader academic discourse.

As my colleague Sabrina Mai has correctly identified, we could, in theory, compress every paper to a list of bullet points. But that would completely remove the story that is an integral part of every paper, a discussion of its implications, and, by extension, the invitation to other researchers to respond to it, either with their own research, or a rebuttal.

But this view on the research we do, and more importantly, on the output we produce, the papers we write and subsequently review for each other, is not just a normative statement on how I would like academia to work. This is merely the only logical conclusion from King and Worthington’s paper, and I can demonstrate this with a small thought experiment.

Imagine the top x % of journals in every discipline officially abolished human peer-review, and instead set up an automated LLM-based review pipeline where every submission gets commented on by some language model. Then, it would no longer be frowned upon to actually make an LLM write the paper in the first place. When we know that a human is going to read a paper, we will still put effort into actually writing it,¹ but if we already know that not a single human soul is going to read it, why would we bother writing it ourselves either?

This would then lead to some weird LLM-automation pipeline, where two LLMs would talk back and forth for a few minutes until the paper gets the stamp of acceptance, maybe pass a final editor before being published. This practice will quickly proliferate, and journals with little money to pay for some Anthropic or OpenAI subscription will have a lot of issues dealing with the flood of AI generated papers. At the same time, researchers who lack the money to pay the potentially hundreds of Dollars for subscriptions will not be able to compete with labs with enough funds to publish 20–30 papers per month this way.

This will then in turn lead to one of two outcomes. Either, if hiring committees continue to trust the amount of publications as a measure of academic excellence, this will lead to a strong social stratification where Matthew-effects will flourish like never before. Affiliates from richer institutes can quite literally “pay to win,” getting tenure track positions or grants merely by virtue of having the resources to bolster their citation count, with anybody from poorer universities losing out. Or, alternatively, if hiring committees stop using publication counts as a metric for academic excellence, what will they replace it with? In addition, why would academics then even consider publishing papers, if they become worthless by virtue of sheer quantity?

Peer-Review is Broken, but Technical Solutions Won’t Fix a Social Problem

In short, peer-review is certainly broken, but throwing LLMs, and, by extension, money and GPU compute onto the problem will only exacerbate the issues we have. The question academics have to answer today is not one of whether to abolish peer-review, but a more fundamental one: what is science even about?

That is why I argue that the reason we write and review papers is not because we just need to check some claims and analytical results of papers. We write papers because we want to change how we see our world. We write papers because we believe that we found some data that supports a viewpoint that the respective field didn’t yet consider, or data that refutes a well-regarded viewpoint. We write papers because we are in a constant exchange with our peers on what is going on in the world. And this is the same process in anthropology as in astrophysics. If even physicists have arguments on whether Einstein was right or whether we have to revisit special relativity, this is a strong indication that the paper writing process is more than a mere documentation and publication of results. It is a discussion where we take (oftentimes normative positions), but supported with citations and data.

The problems of peer-review are not that we just have to somehow adjust the workload of reviewers, or that we have to somehow remove the human from the equation. Rather, we have to figure out what the incentive structure for academics is, improve recognition for doing peer-review, and making sure that academics do have the time and mental energy to actually perform peer-review. I’d argue that one big cause of the issues we face with peer-review is that academics have to spend more and more time fighting over less and less money, all the while bearing excruciating teaching-loads that leave little time for actually thinking and developing those competing viewpoints that we write papers for.

LLMs are great tools to find issues with your papers before you make your colleagues, friends, and peer-reviewers read through it, but they cannot perform the thinking, reflection, and lively debating that is so integral to science.