Same model, six code reviewers, 38% to 81% of the bugs

Quickchat published a benchmark of six AI code reviewers that all ran on the same model, GPT-6.1 Sol. Across 145 real open-source pull requests, they found between 38% and 81% of the same 394 bugs. The top reviewer is Quickchat's own QuickReview, and the author designed and ran the test.

The five third-party tools are Codex CLI, OpenCode, pi-review's three-reviewer panel, open-code-review and PR-Agent. Without QuickReview, those five still ranged from 38% to 69%. Claude agents built the answer key, partly from QuickReview's earlier reviews. No human has checked the key or the AI judge that scores it. QuickReview also cost the most, at $1.41 a PR against 7 to 59 cents for the others, and most of its lead came from medium- and low-severity bugs. On the 56 merge-blocking bugs, four of the six reviewers were close.

Quickchat also moved all six reviewers from GPT-6 Sol to GPT-6.1 Sol on 40 of the PRs. None found more than 4 points more bugs, and OpenCode found 10 points fewer. The newer model did make the third-party tools' comments more accurate. The share the judge graded correct went up for all five, for example Codex CLI from 78% to 94%.

The post says two settings were worth 10 to 11 points each in its trial. One is giving the reviewer the pull request title and description, which lifted Codex CLI and OpenCode by 10 points at no extra cost. The other is the reasoning effort setting. Pi-review and open-code-review gained only a point or two from the description.

The key, every review and every verdict are public as a dataset on Hugging Face (DOI 10.57967/hf/11598), so you can recount the results yourself.