My AI code reviewer kept missing bugs in the very lines it commented on

Quickchat published a breakdown of where its open-source AI reviewer, QuickReview, misses bugs. Many misses sit under a comment the reviewer had already left on those lines, about something else.

The test ran the September build against 185 GitHub pull requests from AACR-Bench, which has an answer key of 520 known bugs. QuickReview missed 80 of them outright. For 49 of those 80, one of its comments started inside the bug's own lines. Moving the same bugs to random changed lines gives about 37. The gap held on the 145 PRs the build wasn't tuned on (31 against about 24) and on a re-run (41 against about 31). On the older frozen build it was only slightly above chance (27 against about 21).

In all 49 cases the comment was about something else. 26 were about another real bug, 19 were true but minor, and 4 were wrong. One example is Ollama's PR #10135. The reviewer flagged a blocker about cursor movement on the line that prints the oldest progress lines instead of the newest. It never mentioned the actual bug there.

The misses were mostly low severity (35 low, 12 medium, 2 high). A comment also didn't mean the rest of the block was safe. Bugs under an existing comment were missed 20% of the time, against 15% overall. Quickchat's advice is to look for a second bug under every comment an AI reviewer leaves.

The answer key and the judge are both LLM-based, and no human has checked either. Quickchat also says the result shows where misses are, not that a comment caused one. The reviews, answer key and judge verdicts are public on Hugging Face.