Self-review erases what you taught your rubric

A common step in building model-graded evaluations is a refinement loop: generate a rubric, then ask a model to review it for ambiguity, gaps and flaws no check would catch, and repeat. Each round feels like progress. The rubric gets more thorough, and nobody objects to “more thorough.”

I expected that loop to ratchet: under all-must-pass grading, every added check can only lower the pass rate, and reviewers add far more than they delete. What I found is more specific, and worse. Self-review pulls a rubric back toward the reviewer’s own idea of a good answer. It adds the requirements the reviewer believes in, and it deletes the requirements it didn’t originate, including the ones you put there on purpose. Check count turned out not to be the mechanism at all.

Setup

The task is grading findings reports from an automated security reconnaissance agent, which inspects a cloud tenant’s configuration and reports security posture issues. I used 12 synthetic environments and a labeled set of 108 reports (36 correct, 72 flawed). The flaws are a missed finding, a made-up finding, a finding on the wrong resource, an overstated impact, and, in 12 of the reports, leaving out a real exposure that is in the inventory but isn’t part of the planted scenario. I’ll call that last one an unplanned miss: it tests whether a rubric can fail a report for leaving out something nobody anticipated.

I started from two rubrics that were already good, in different ways:

• A hand-written checklist, 6 checks per environment. It passes 97% of correct reports and catches 83% of flawed ones, but it catches none of the unplanned misses, because it only knows about the planted scenario.
• A rubric taught by labeled examples, about 8 checks per environment. A model revised it after seeing 36 labeled reports from four of the environments, and it learned two things the labels implied: remediation advice is optional, and a report should be failed if it leaves out any real risk in the inventory. It added an open coverage check for the second point. It passes 83% of correct reports, catches 100% of flawed ones, and catches 100% of unplanned misses.

Opus then revised each rubric for four rounds with a generic instruction: review each rubric for ambiguity, gaps, checks a different judge might misread, and flaws a report could have that no check would catch. Four variants:

• Free revision, two independent seeds.
• Fixed count: rewrite anything, but keep exactly the same number of checks.
• Delete or merge only: no new requirements, no making a check stricter.

After rounds 2 and 4, Sonnet graded all 108 reports with each revised rubric.

What self-review did to one rubric

Before the aggregate numbers, here is what free revision did to the same environment starting from each rubric. Switch tabs to compare; the replay button runs it again.

Results

Free revision broke both rubrics, in opposite directions.

The hand-written checklist became far too strict. After four rounds, it passed 6% of correct reports with one seed and 11% with the other, down from 97%. With the first seed, the damage was complete by round 2. When I looked at what the reviewer added, the pattern was obvious: over the four rounds, one seed added 10 checks requiring remediation advice (“does the report recommend removing the dangling DNS record?”), and every one of those 10 failed at least one correct report. The task never asked for remediation advice; the reviewer believed a good report should include it, so it wrote that belief into the rubric. To be fair, the same revision also added an open coverage check in some environments, which raised the catch rate on unplanned misses from 0% to 92%. But a rubric that fails 94% of correct outputs is unusable regardless.

The label-taught rubric lost what the labels had taught it. Free revision deleted the open coverage check in 11 of 12 environments, describing it in one change note as “the vague completeness check.” Unplanned misses caught fell from 100% to 25% with one seed and 8% with the other, and the overall catch rate fell from 100% to about 86%. Remediation checks crept back in too. The rubric got shorter, from 7.9 to 7.0 checks per environment, and still got worse.

Holding the count fixed didn’t protect either rubric. Fixed-count revision dropped the hand checklist’s pass rate on correct reports to 72%, and took the label-taught rubric’s unplanned-miss catch rate down to 17%. Rewrites alone carried the drift.

Only delete-or-merge was safe. It shrank the hand checklist from 6 to 4 checks per environment and the label-taught one from 7.9 to 5.8, and left every metric where it started (the hand checklist’s correct-pass rate even ticked up from 97% to 100%). Removing a requirement can’t import a new belief.

The counts make the point that this isn’t a ratchet. Free revision grew one rubric by 1.7 to 3 checks per environment and shrank the other by about one. Both got worse. What changed was content, and the content moved toward the same place each time: the reviewer’s own standard.

A loop can’t fix what its author believes

This also explains a result from an earlier run. There, the starting rubric was model-generated, and it already demanded remediation advice: it passed 6% of correct reports. Three rounds of self-review (6 to 14%), and three rounds of a more elaborate loop where the model wrote test reports with intended verdicts and checked whether a judge agreed, left it at 6 to 8%. The loop agreed with itself on 71 or 72 of 72 test reports every round, because the model wrote its “should pass” test reports to its own standard, remediation advice included. Nothing in the loop could tell it the standard was wrong.

What did fix it was showing the model 36 labeled reports: one round took the correct-pass rate from 6% to about 80% without losing any catches. And as this experiment shows, a later round of self-review undoes exactly that.

What I’d change in practice

1. Track where each check came from. Checks derived from labels, from domain experts or from incident reviews should be frozen against automated revision. A self-review pass can propose changes to them, never apply them.
2. Make unattended revision delete-only. If a loop runs without a human, it may merge or remove checks, never add requirements or tighten them. Additions go through review.
3. Regression-test every revision against labeled data before accepting it. Three numbers: correct outputs passed, flawed outputs caught, and unplanned misses caught. A revision that lowers any of them is rejected, however much more thorough it reads.
4. Read additions as statements of the reviewer’s beliefs. A new check that sounds obviously right (“should recommend a fix”) is exactly the kind that encodes a standard nobody agreed to.

Caveats

The environments and labels are synthetic and mine. One model wrote and revised all the rubrics, and judges came from the same model family, so I can’t yet separate this reviewer’s specific beliefs from a general tendency of self-review; repeating with a different reviewer family is the next experiment. Free revision ran with two seeds and the constrained variants with one, and each rubric was graded once per round, so small differences (a few points) aren’t meaningful. The large effects (97% to 6%, 100% to 8%) are well outside that noise.

The companion post covers what sets a judge’s strictness in the first place: Your judge isn’t lenient or strict. Your rubric is.

@misc{asgari2026selfreview,
  author       = {Asgari, Saeid},
  title        = {Self-review erases what you taught your rubric},
  year         = {2026},
  month        = {oct},
  howpublished = {\url{https://asgsaeid.github.io/blog/2026-10-self-review-erases-what-you-taught-your-rubric.html}}
}