Your judge isn’t lenient or strict. Your rubric is.

Large language model (LLM) judges have a reputation for leniency: they pass outputs a careful human would reject. Checklist rubrics are the usual fix. Break “is this good?” into atomic yes/no checks, require all of them to pass, and the judge can no longer wave things through.

When I generated checklists this way for a security task, the judge went from lenient to failing almost everything, including outputs that were correct. That made me suspect leniency and strictness aren’t properties of the judge at all. So I held the judge and the outputs fixed and varied only the rubric.

Short version: what the rubric asks for decides strictness. Swapping the judge model barely matters, and so does rendering the same checks as a checklist, a paragraph or a 1 to 5 scale. The way most people validate judges, a single accuracy number on a defect-heavy test set, can’t see any of this.

Setup

The task is grading findings reports from an automated security reconnaissance agent: an agent that inspects a cloud tenant’s configuration and writes a short report of security posture issues. I built 12 synthetic environments, each with a ground-truth inventory and two real findings. For each environment I wrote 2 correct reports and 4 reports with exactly one planted flaw: a missed finding, a made-up finding, a finding placed on the wrong resource, or an overstated impact. That gives 72 reports, 24 correct and 48 flawed, with labels known by construction.

The judge sees the inventory and one report at a time (one report per environment per batch, so it never compares versions of the same report). Sonnet is the main judge, every condition was run twice, and a subset was repeated with Haiku and Opus. I report two numbers per rubric instead of accuracy:

• Correct reports passed: how often the judge lets a good report through.
• Flawed reports caught: how often it fails a report with a planted flaw.

The rubrics varied along three axes:

• Content: no checks at all (a free-form “is this report correct?”), a generic 1 to 5 scale (groundedness, completeness, actionability), 6 checks I wrote by hand per environment, or 12 checks per environment generated by Opus from the inventory and ranked by importance.
• Count: the first 2, 4 or 6 hand checks; the first 4, 8 or 12 generated checks. These are nested, so a shorter rubric is a prefix of a longer one.
• Rendering: the same propositions as a yes/no checklist (all must pass), as one paragraph the judge reads holistically, or each scored 1 to 5 (pass if every score is at least 4).

Content decides, rendering doesn’t

The figure below plots every rubric by the two rates. Step through the views to compare one factor at a time.

There are two clusters. The hand-written checks sit in the top right whichever way I render them: about 96 to 98% of correct reports pass and every flawed report is caught. The generated checks sit in the top left whichever way I render them: every flaw is caught, but only 2 to 8% of correct reports pass. Rendering the 12 generated checks as a paragraph didn’t soften them at all (4% either way), and neither did scoring them 1 to 5 (8%).

So what makes the generated checks strict? I looked at which checks failed the correct reports. Almost all of them asked for things the task never required: “does the report recommend disabling anonymous access?”, or a specific secondary detail about who manages a group. The correct reports were correct; they just didn’t contain remediation advice. Earlier I measured this directly on a different set of generated rubrics: 16 of 92 checks failed at least three quarters of all reports, good or bad, and those 16 checks alone explained 31 of the 34 correct reports that were wrongly failed.

Count only hurts when the checks are wrong. The nested hand checks show the trade you’d hope for: going from 2 to 4 to 6 checks raised the catch rate from 52% to 74% to 100%, and correct reports kept passing at 96% throughout. The nested generated checks show the opposite: 4 checks passed 21% of correct reports, 8 checks passed 2%. Adding a check costs you exactly that check’s false-fail rate. It’s a cheap addition when that rate is zero and an expensive one when it isn’t.

That cost compounds multiplicatively. Under all-must-pass aggregation, the share of correct reports that pass equals the product of each check’s pass rate on correct reports. I checked this on 12 rubric versions from a related experiment, and the product predicted the observed pass rate to within 1 point every time. A rubric of 10 checks that each wrongly fail a good report 5% of the time passes only 60% of good reports, and nobody reading the checks one at a time would spot a problem.

The judge model barely matters

With the rubric held fixed, Haiku, Sonnet and Opus mostly land within a few points of each other. The free-form judge passed every correct report with all three models. The generated 12-check rubric passed 0 to 4% with all three. The hand checklist passed 96 to 100%.

The one exception is informative. With a generic 1 to 5 scale and a pass threshold of 5, the judges split wide: Haiku passed 12% of correct reports, Sonnet 56%, Opus 79%. A top-of-scale threshold measures how readily each model hands out a perfect score, which differs between models in a way checks don’t. If you use a scale, the threshold is a strictness dial, and it’s calibrated differently for every judge: at 3, Sonnet passed 96% of reports with a missing finding; at 4, 58%; at 5, none of them, but also only 56% of correct ones.

The leniency the literature reports is real, but it belongs to the free-form setting (switch the figure to its second tab). With no rubric content, Sonnet passed every report that left out a real finding, and Haiku and Opus passed 75 to 83% of them. The judge isn’t checking completeness because nothing asked it to.

Why this usually goes unnoticed

Most judge validation sets are built by planting defects, so most of their items are flawed. On such a set, a rubric that fails everything looks good. Drag the slider to change the mix.

The first-draft generated rubric from my earlier experiment passes 6% of correct reports and catches 99% of flawed ones. Its accuracy is 89% when 90% of the set is flawed, 68% at two thirds (my set), 52% at half and 36% at one third. Its balanced accuracy, the mean of the two rates, is 52% everywhere: a coin flip. A hand-written checklist that actually works looks worse than it on a 90%-flawed set.

The aggregation rule hides in the same way. With the per-check answers held fixed, switching from “all checks must pass” to “at most one check may fail” moved that rubric’s pass rate on correct reports from 6% to 69%.

Three rounds of rewording its checks moved the same number from 6% to 8%. The aggregator is a bigger lever than the wording, and it’s rarely reported.

With human labels

My recon labels come from defects I planted, so I repeated the count experiment on a set with human labels: 146 summaries from the SummEdits benchmark, half judged consistent with their source by human annotators. Here the checks were generic (one ranked list of 10 consistency checks, written once and applied to every item), so I’d expect a much milder effect than with scenario-specific generated checks.

The direction holds: each added check trades correct passes for catches. Free-form passed 91% of consistent summaries and caught 76% of inconsistent ones. Ten checks passed 77% and caught 84%. The magnitude is far smaller, because generic checks rarely demand something a correct summary lacks. One difference from the recon task: the same 10 checks rendered as a paragraph were somewhat more lenient here (83% versus 77% passed), so rendering isn’t always irrelevant, just second-order.

What I’d change in practice

1. Report two numbers, never accuracy alone. Correct outputs passed and flawed outputs caught. Accuracy and raw agreement on a defect-heavy set will flatter a rubric that fails everything.
2. Audit every check’s fail rate on real, unlabeled outputs. A check that fails nearly all outputs is almost always asking for something the task doesn’t require. You don’t need labels to find it.
3. Pick the aggregation rule deliberately and report it. All-must-pass compounds every small false-fail rate.
4. Give the rubric author the output contract. Generated checks invent requirements (remediation advice, in my case) when the author doesn’t know what the agent was told to produce.
5. If you use a scale, treat the threshold as a calibration parameter per judge model, not a fixed convention.

Caveats

The recon environments and their labels are synthetic and mine, which is why I added the human-labeled replication. All three judges and the rubric author are Claude models, so “the judge barely matters” is shown within one model family. Each condition ran twice; run-to-run differences were within a few points. The set has 24 correct reports, so differences under about 5 points shouldn’t be read as real.

A companion post looks at what happens when a model is asked to improve a rubric: Self-review erases what you taught your rubric.

@misc{asgari2026judge,
  author       = {Asgari, Saeid},
  title        = {Your judge isn't lenient or strict. Your rubric is.},
  year         = {2026},
  month        = {oct},
  howpublished = {\url{https://asgsaeid.github.io/blog/2026-10-your-judge-isnt-lenient-or-strict-your-rubric-is.html}}
}