Backtestspydantic-airun-20260918-215936#5034
judge_output() reason field can be polluted by visible reasoning/thinking text
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches attack the root cause the same way — adding a `Field(description=...)` to `GradingOutput.reason` (which surfaces in the JSON schema shown to the model) plus an explicit 'be concise/don't think out loud' instruction in all four judge system prompts. On correctness they are roughly equivalent; B's field description is marginally richer ('self-consistent', 'do not include step-by-step reasoning, re-checking, or self-corrections'), directly matching the issue's complaint about self-contradiction, not just verbosity. The deciding factors are tests, scope, and craft. Tests: B adds test_grading_output_reason_description, which fails before the change (no description) and passes after; it is somewhat superficial (asserts substrings in the description) but it does lock in the schema-level guidance that is the load-bearing part of the fix. A adds no tests at all. Scope/craft: A converts every judge prompt from a plain triple-quoted string into an f-string, which forces doubling of all literal braces — including every `{"reason": ...}` example JSON block — a large, error-prone, hard-to-read churn and a lasting maintenance hazard for anyone editing these prompts. A's one genuine advantage is DRY (a single `_JUDGE_REASON_INSTRUCTION` constant vs. B duplicating the sentence four times), but that benefit is bought at a disproportionate cost, and the DRY goal was achievable without f-stringifying the entire prompt. B keeps the prompts as readable plain strings and confines its edits to appended text. B's duplicated sentence is a minor craft ding, far outweighed by A's brace-escaping risk and B's added test. Net: B is the better fix.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
10m 54s1 turn
Wall-clock time in the container
Full test suite
No new failures
5,805 passed · 496 skipped · 6,409 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
29vs 51 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I'm using the latest version of Pydantic AI
- I've searched for my issue in the issue tracker before opening this issue
Description
pydantic_evals.evaluators.llm_as_a_judge.judge_output() currently returns a freeform reason: str. In practice, with reasoning-capable models, that reason can become very long, include visible self-correction / re-checking, and even contradict itself mid-response.
This is especially problematic when judge_output() is used inside an agent output_validator and the reason is fed back into ModelRetry(...), because the retry message becomes noisy and unstable instead of being a concise repair instruction.
Example pattern:
@agent.output_validator
async def llm_judge_validator(ctx: RunContext[AgentDeps], data: _ModelT) -> _ModelT:
grading = await judge_output(
output=data,
rubric=response_model.rubric(),
model=self.model_obj,
model_settings={"temperature": 0.0},
)
if not grading.pass_:
raise ModelRetry(grading.reason)
return data
In my case, the judge reason did things like:
- state a violation
- re-check the same rule
- partially retract the earlier claim
- continue with more analysis
- still return
pass=false
So the issue is not just verbosity. The reason can become self-contradictory and unsuitable for retry feedback loops.
I realize some of this is model behavior, but the current built-in prompt/output shape seems to allow the model to "think out loud" inside reason rather than returning a short, stable failure summary.
One workaround is to disable thinking in judge_model_settings, but that is not what I want. I want the judge model to retain the benefits of internal reasoning; I just do not want reasoning/thinking output to leak into the public reason field. Ideally, the judge should be able to think internally while still returning a concise, stable, retry-safe explanation.
Minimal, Reproducible Example
import asyncio
from pydantic_evals.evaluators.llm_as_a_judge import judge_output
OUTPUT = {
"title": "The Blue Light of Silence",
"pages": [
{"phase": "setup"},
{"phase": "escalation"},
{"phase": "climax"},
],
}
RUBRIC = """
Judge whether this structured output is acceptable.
Check each rule independently. Fail the output if ANY single rule is violated.
Episode:
- Fail if pages do not form a complete dramatic arc with forward-progressing phases
(e.g. setup -> escalation -> turning point -> climax -> aftermath);
repeating or skipping phases is not acceptable.
"""
async def main():
result = await judge_output(
output=OUTPUT,
rubric=RUBRIC,
model="gemma4:31b-cloud",
model_settings={"temperature": 0.0},
)
print(result.model_dump())
asyncio.run(main())
Observed behavior: the returned reason can be a long, self-reflective paragraph rather than a concise explanation of the failed rule(s).
Expected behavior: reason should be short, stable, and directly usable as validation feedback, especially in retry loops.
Logfire Trace
_No response_
Python, Pydantic AI & LLM client version
- Python: 3.13.12
- PydanticAI: 1.73.0
- Pydantic: 2.12.5
- LLM client SDK: openai 2.30.0
The two patches
pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.pyindex 0b6fed34..bb574655 100644@@ -27,7 +27,13 @@ _default_model: models.Model | models.KnownModelName = 'openai:gpt-5.2' class GradingOutput(BaseModel, populate_by_name=True): """The output of a grading operation.""" - reason: str+ reason: str = Field(+ description=(+ 'A concise, self-consistent explanation of the grade, at most a few sentences. '+ 'State only the final conclusion about which rubric criteria passed or failed; '+ 'do not include step-by-step reasoning, re-checking, or self-corrections.'+ )+ ) pass_: bool = Field(validation_alias='pass', serialization_alias='pass') score: float @@ -47,6 +53,8 @@ _judge_output_agent = Agent( <Output>Avast ye swabs, repel the invaders!</Output> <Rubric>Does not speak like a pirate</Rubric> {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}++ Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there. """ ), output_type=GradingOutput,@@ -87,6 +95,8 @@ _judge_input_output_agent = Agent( <Output>Avast ye swabs, repel the invaders!</Output> <Rubric>Does not speak in the style described by the input</Rubric> {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}++ Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there. """ ), output_type=GradingOutput,@@ -131,6 +141,8 @@ _judge_input_output_expected_agent = Agent( <Output>Six</Output> <Rubric>The output is factually consistent with the expected output</Rubric> {"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}++ Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there. """ ), output_type=GradingOutput,@@ -176,6 +188,8 @@ _judge_output_expected_agent = Agent( <Output>Six</Output> <Rubric>The output should be a number written in words which matches the number written in digits in the expected output</Rubric> {"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}++ Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there. """ ), output_type=GradingOutput,tests/evals/test_llm_as_a_judge.pyindex 292f8779..43863606 100644@@ -41,6 +41,19 @@ def test_grading_output(): assert output.score == 1.0 +def test_grading_output_reason_description():+ """The reason field must carry schema guidance keeping it concise and stable, so reasoning+ models do not leak think-out-loud / self-correction text into it (issue: reason pollution)."""+ schema = GradingOutput.model_json_schema()+ reason_description = schema['properties']['reason'].get('description')+ assert reason_description is not None+ lowered = reason_description.lower()+ # The reason should be explicitly guided to stay concise...+ assert 'concise' in lowered+ # ...and to exclude verbose step-by-step reasoning / self-corrections.+ assert 'step-by-step' in lowered or 'self-correction' in lowered or 'reasoning' in lowered++ def test_stringify(): """Test _stringify function.""" # Test with string
pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.pyindex 0b6fed34e9..f5158ae14d 100644@@ -27,26 +27,37 @@ _default_model: models.Model | models.KnownModelName = 'openai:gpt-5.2' class GradingOutput(BaseModel, populate_by_name=True): """The output of a grading operation.""" - reason: str+ reason: str = Field(+ description='A concise 1-2 sentence explanation of why the output passed or failed.',+ ) pass_: bool = Field(validation_alias='pass', serialization_alias='pass') score: float +_JUDGE_REASON_INSTRUCTION = (+ 'The "reason" field must be a concise 1-2 sentence summary of your verdict. '+ 'Do not include your reasoning process, self-corrections, or re-checking in the reason. '+ 'State only the final conclusion.'+)++ _judge_output_agent = Agent( name='judge_output', system_prompt=dedent(- """- You are grading output according to a user-specified rubric. If the statement in the rubric is true, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+ f"""+ You are grading output according to a user-specified rubric. If the statement in the rubric is true, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++ {_JUDGE_REASON_INSTRUCTION} Examples: <Output>Hello world</Output> <Rubric>Content contains a greeting</Rubric>- {"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}+ {{"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}} <Output>Avast ye swabs, repel the invaders!</Output> <Rubric>Does not speak like a pirate</Rubric>- {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}+ {{"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}} """ ), output_type=GradingOutput,@@ -73,20 +84,22 @@ async def judge_output( _judge_input_output_agent = Agent( name='judge_input_output', system_prompt=dedent(- """- You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+ f"""+ You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++ {_JUDGE_REASON_INSTRUCTION} Examples: <Input>Hello world</Input> <Output>Hello</Output> <Rubric>Content contains a greeting word which is present in the input</Rubric>- {"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}+ {{"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}} <Input>Pirate</Input> <Output>Avast ye swabs, repel the invaders!</Output> <Rubric>Does not speak in the style described by the input</Rubric>- {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}+ {{"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}} """ ), output_type=GradingOutput,@@ -115,8 +128,10 @@ async def judge_input_output( _judge_input_output_expected_agent = Agent( name='judge_input_output_expected', system_prompt=dedent(- """- You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input, expected output, and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+ f"""+ You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input, expected output, and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++ {_JUDGE_REASON_INSTRUCTION} Examples: @@ -124,13 +139,13 @@ _judge_input_output_expected_agent = Agent( <ExpectedOutput>Blue</ExpectedOutput> <Output>Cerulean</Output> <Rubric>The output is consistent with the expected output but doesn't have to match exactly</Rubric>- {"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}+ {{"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}} <Input>How many legs does a spider have?</Input> <ExpectedOutput>8</ExpectedOutput> <Output>Six</Output> <Rubric>The output is factually consistent with the expected output</Rubric>- {"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}+ {{"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}} """ ), output_type=GradingOutput,@@ -162,20 +177,22 @@ async def judge_input_output_expected( _judge_output_expected_agent = Agent( name='judge_output_expected', system_prompt=dedent(- """- You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided expected output and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+ f"""+ You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided expected output and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++ {_JUDGE_REASON_INSTRUCTION} Examples: <ExpectedOutput>Blue</ExpectedOutput> <Output>Cerulean</Output> <Rubric>The output should be a shade of the expected output color</Rubric>- {"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}+ {{"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}} <ExpectedOutput>8</ExpectedOutput> <Output>Six</Output> <Rubric>The output should be a number written in words which matches the number written in digits in the expected output</Rubric>- {"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}+ {{"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}} """ ), output_type=GradingOutput,
What the agent reported
The change is complete and verified. Final report below.
What was wrong
pydantic_evals.evaluators.llm_as_a_judge.GradingOutput.reason was a bare reason: str with no guidance, and the four judge agents' system prompts only said "respond with {reason, pass, score}". Nothing told the model to keep reason short or final, so reasoning-capable judge models would dump long, self-correcting, sometimes self-contradictory "think out loud" text into reason — making it unusable as concise, stable retry feedback (e.g. fed into ModelRetry). The reporter wanted the judge to keep reasoning internally but return a concise, stable reason, without disabling thinking.
What I changed
Two production edits in pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py — purely prompt/schema guidance, no model_settings/thinking disabled, no signatures touched:
GradingOutput.reasonnow has aField(description=...)telling the model to keep the reason concise, self-consistent, and final (at most a few sentences), to state only the final conclusion, and to exclude step-by-step reasoning, re-checking, or self-corrections. Becauseoutput_type=GradingOutput, this description is emitted in the structured-output JSON schema the judge model receives — the direct lever on thereasonfield.- All four judge agents' system prompts (
_judge_output_agent,_judge_input_output_agent,_judge_input_output_expected_agent,_judge_output_expected_agent) got an identical appended paragraph reinforcing the same "keepreasonconcise, don't think out loud in it" instruction.
One test edit in tests/evals/test_llm_as_a_judge.py: added test_grading_output_reason_description, asserting via the public GradingOutput.model_json_schema() that reason carries a description mentioning conciseness and discouraging verbose reasoning/self-correction.
Test evidence
- Regression test failed before the fix (
assert None is not None—reasonhad no description). - After the fix:
./bt-test tests/evals/test_llm_as_a_judge.py→ 14 passed (the new test plus all 13 pre-existing tests). No existing test was weakened, skipped, or altered; system-prompt additions don't touch any snapshot because those assert only the user prompt, never the system prompt. - Reviewer confirmed the diff is limited to the two intended files,
pass_/score/signatures unchanged, thinking not disabled, and no scratch/temp files.
Requirements disposition
reasonshould be concise/stable and not polluted by think-out-loud/self-correction text → Implemented via thereasonfield description (schema-level) + system-prompt guidance across all four judges. Tested bytest_grading_output_reason_description(red→green).- Judge must retain internal reasoning (do NOT disable thinking) → Honored: no
model_settings/thinking changes; the fix only shapes what goes into the publicreasonfield. Verified by reviewer (no settings changed). - Applies to the
judge_output()path shown in the report → Covered:judge_outputuses_judge_output_agent+GradingOutput, both updated; the sibling judges (judge_input_output,judge_input_output_expected,judge_output_expected) shareGradingOutputand were updated consistently.
Note (not mine to fix): the working tree already contained pre-existing deletions of AGENTS.md, CLAUDE.md, and .claude/settings.json from before this task — unrelated to this change and left untouched. All my changes are left uncommitted in the working tree.