AI code review: measure useful findings, not the number of comments

The recent article and the question it raises

Michelle Zhou and Alejandro Carderera de Diego published GitHub’s ReviewBench article ↗ on 2026-10-05. It introduces an evaluation of AI reviewers using real pull requests and validated findings. The useful learning question is simple: which comments help you prevent a defect?

The exercise below is our own design, not a reproduction of that benchmark or a claim about a particular model’s score. You can carry it out on a small command-line utility rather than a large production repository.

Give a finding an evidence trail

Imagine a CSV expense parser that accepts a date, an amount and a category. Before asking an AI to review it, write the behavior you want: reject malformed amounts, preserve valid negative adjustments, and report the line that cannot be parsed. Choose these requirements yourself so you can judge the answer independently.

For every review comment, ask for the exact location, the input that exposes the problem, the expected behavior and the observed behavior. “This might fail” is a hypothesis. A minimal input that demonstrates an incorrect result is evidence. If the comment requests a refactor, ask which maintenance problem the refactor addresses; do not quietly count a preference as a confirmed defect.

Keep the original patch while investigating. Applying every proposed edit first makes it harder to learn which comment was useful and which introduced another problem. A reviewer’s confidence is not the reproduction step.

Measure three separate things

Prepare three deliberately flawed variants of the parser: one rejects valid negative values, one loses the input line number, and one misinterprets an empty amount. Also prepare an unchanged control that follows your stated requirements. These are invented examples, not security exploits or vendor test cases.

Record confirmed findings, known issues the reviewer missed, and invalid or irrelevant comments. A reviewer that writes many comments might still miss the one behavior you care about. A quiet review might produce a valuable warning, or might simply overlook the defect. Keep those possibilities separate rather than choosing a winner from comment count alone.

Repeat the same review twice with the same patch and settings. Record which findings remain reproducible. This small exercise does not establish broad reliability, but it can expose sensitivity to phrasing or randomness before you rely on the tool for larger changes.

Understand what the product’s review does

GitHub’s current usage documentation ↗ explains how to request and revisit a Copilot review. A comment review does not count as approval by default; optional approvals are a separate configuration. This is a living reference checked on 2026-10-11, not another newly published blog post.

For the exercise, keep a human decision after reproduction and before accepting a fix. Re-run the examples against the modified parser and inspect the diff. Passing the three cases says something useful about those cases, while leaving unrelated behaviors unproven.

A small deliverable you can inspect

Make a one-page table with a row for each comment: location, reproduction input, verdict, severity and follow-up. Add a final row for each seeded defect that received no comment. Finish by writing one reason to keep the reviewer and one situation in which you would need additional checks. This produces a concrete review habit without pretending you have reproduced a public leaderboard.