AI judges agree on known bugs and split on which code review is better

Quickchat AI tested whether AI judges can grade code reviews. Judges agreed when they checked reviews against a list of known bugs, and disagreed when asked which of two reviews was better.

For the bug checks, a judge reads one review and marks each bug on the answer key as found, partly found, or missed. Judges from three model families (GPT-6 Sol, Claude Opus 5.5, and DeepSeek V4.1 Flash) gave the same verdict on about 99% of checks. Almost all the disagreements were over partial credit. DeepSeek got there with no repository access, working from the prompt alone.

Picking the better review was a different story. Two judges, GPT-6.1 Sol and Claude Opus 5.5, read the same 298 head-to-head matchups and named the same winner in 167 of them (56%). Chance would give about 50%. Each judge also crowned a different reviewer. In many splits, both judges agreed one review found more bugs and the other wasted less of the author's time, then weighed those two things differently.

That matters when benchmark scores bunch up. Quickchat's top three code review setups scored 43.9, 43.2 and 43.1 out of 100 on Martian's Code Review Bench, and rerunning the scoring can move that benchmark by about three points. Neither the score nor a judge could say which was better.

Quickchat lists its own limits. The bug answer key was built by agents and hasn't been checked by a human. The matchups cover only 20 pull requests, and Quickchat builds one of the reviewers being judged. The key, the reviews, and every judge verdict are in a public Hugging Face dataset, so you can recompute the counts.