Experiments on evaluating and improving tool-using agents: what makes a large language model (LLM) judge strict or lenient, why rubrics drift when a model revises them, and how an agent can learn without a hidden test.
Holding the judge and the outputs fixed, what the rubric asks for moved the pass rate on correct outputs from 2% to 98%. Swapping the judge model barely mattered.
Four rounds of model self-review pulled two good rubrics toward the reviewer’s own standard. Only delete-only revision was safe.
Five findings from a matched, five-repeat benchmark of self-designed evaluators, ACE and a ten-label expert calibration on tau2-bench and AppWorld.