Skip to main content
Post

GPT-6.2 Astra in my state-machine review finds more state bugs and comments far outside state

Quickchat's QuickReview team ran its state-machine review on OpenAI's GPT-6.2 Astra and published the results. It used the same 34 AACR-Bench pull requests and the same judge as its earlier GPT-6.1 Sol runs.

The state-machine review reads a change through eight narrow reviewers, each looking for one kind of state bug. A validator then replays every claimed sequence of events against the code before anything is posted. On Astra it found 52 of the 55 state bugs in the answer key. The same review on Sol found 44, and the default review on Sol found 45. Astra found every bug the Sol state review found, plus 8 more. It missed three, and the Sol state review missed the same three.

The review is meant to post state findings only, but Astra left 448 inline comments against 159 on Sol and 397 for the default review. Of the 152 answer-key defects those comments found, 100 aren't state bugs. On Sol, 48 of 92 weren't. The default review on Sol still found more defects across all 186 in the key, 161 against 152, and it hasn't been run on Astra yet.

It also costs more. At list price, a review ran $21.06 per pull request against $2.78 on Sol, about 7.6 times as much. The run went through a ChatGPT login, so nothing was billed. Reviews averaged 15.1 minutes on Astra and 12.0 on Sol. Astra's lens passes are slower, so if you run the lenses on it, raise lens_pass_deadline_min under [engine] above the default 8 minutes. This run used 20.

The state-machine review is in QuickReview 1.5.0 and later, and the raw comments, judge verdicts, and costs are in the benchmark dataset on Hugging Face. This is one run per model, so treat the numbers as a single data point.

Loading replies