Backtestspydantic-airun-20260918-215936#5034

Issue #5034 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

judge_output() reason field can be polluted by visible reasoning/thinking text

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches attack the root cause the same way — adding a `Field(description=...)` to `GradingOutput.reason` (which surfaces in the JSON schema shown to the model) plus an explicit 'be concise/don't think out loud' instruction in all four judge system prompts. On correctness they are roughly equivalent; B's field description is marginally richer ('self-consistent', 'do not include step-by-step reasoning, re-checking, or self-corrections'), directly matching the issue's complaint about self-contradiction, not just verbosity. The deciding factors are tests, scope, and craft. Tests: B adds test_grading_output_reason_description, which fails before the change (no description) and passes after; it is somewhat superficial (asserts substrings in the description) but it does lock in the schema-level guidance that is the load-bearing part of the fix. A adds no tests at all. Scope/craft: A converts every judge prompt from a plain triple-quoted string into an f-string, which forces doubling of all literal braces — including every `{"reason": ...}` example JSON block — a large, error-prone, hard-to-read churn and a lasting maintenance hazard for anyone editing these prompts. A's one genuine advantage is DRY (a single `_JUDGE_REASON_INSTRUCTION` constant vs. B duplicating the sentence four times), but that benefit is bought at a disproportionate cost, and the DRY goal was achievable without f-stringifying the entire prompt. B keeps the prompts as readable plain strings and confines its edits to appended text. B's duplicated sentence is a minor craft ding, far outweighed by A's brace-escaping risk and B's added test. Net: B is the better fix.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

10m 54s1 turn

Wall-clock time in the container

Full test suite

No new failures

5,805 passed · 496 skipped · 6,409 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

29vs 51 merged

Changed lines, ours vs merged

Initial Checks

Description

pydantic_evals.evaluators.llm_as_a_judge.judge_output() currently returns a freeform reason: str. In practice, with reasoning-capable models, that reason can become very long, include visible self-correction / re-checking, and even contradict itself mid-response.

This is especially problematic when judge_output() is used inside an agent output_validator and the reason is fed back into ModelRetry(...), because the retry message becomes noisy and unstable instead of being a concise repair instruction.

Example pattern:

@agent.output_validator
async def llm_judge_validator(ctx: RunContext[AgentDeps], data: _ModelT) -> _ModelT:
    grading = await judge_output(
        output=data,
        rubric=response_model.rubric(),
        model=self.model_obj,
        model_settings={"temperature": 0.0},
    )
    if not grading.pass_:
        raise ModelRetry(grading.reason)
    return data

In my case, the judge reason did things like:

  • state a violation
  • re-check the same rule
  • partially retract the earlier claim
  • continue with more analysis
  • still return pass=false

So the issue is not just verbosity. The reason can become self-contradictory and unsuitable for retry feedback loops.

I realize some of this is model behavior, but the current built-in prompt/output shape seems to allow the model to "think out loud" inside reason rather than returning a short, stable failure summary.

One workaround is to disable thinking in judge_model_settings, but that is not what I want. I want the judge model to retain the benefits of internal reasoning; I just do not want reasoning/thinking output to leak into the public reason field. Ideally, the judge should be able to think internally while still returning a concise, stable, retry-safe explanation.

Minimal, Reproducible Example

import asyncio

from pydantic_evals.evaluators.llm_as_a_judge import judge_output

OUTPUT = {
    "title": "The Blue Light of Silence",
    "pages": [
        {"phase": "setup"},
        {"phase": "escalation"},
        {"phase": "climax"},
    ],
}

RUBRIC = """
Judge whether this structured output is acceptable.
Check each rule independently. Fail the output if ANY single rule is violated.

Episode:
- Fail if pages do not form a complete dramatic arc with forward-progressing phases
  (e.g. setup -> escalation -> turning point -> climax -> aftermath);
  repeating or skipping phases is not acceptable.
"""

async def main():
    result = await judge_output(
        output=OUTPUT,
        rubric=RUBRIC,
        model="gemma4:31b-cloud",
        model_settings={"temperature": 0.0},
    )
    print(result.model_dump())

asyncio.run(main())

Observed behavior: the returned reason can be a long, self-reflective paragraph rather than a concise explanation of the failed rule(s).

Expected behavior: reason should be short, stable, and directly usable as validation feedback, especially in retry loops.

Logfire Trace

_No response_

Python, Pydantic AI & LLM client version

  • Python: 3.13.12
  • PydanticAI: 1.73.0
  • Pydantic: 2.12.5
  • LLM client SDK: openai 2.30.0

The two patches

Our patch+28 −1
pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.pyindex 0b6fed34..bb574655 100644@@ -27,7 +27,13 @@ _default_model: models.Model | models.KnownModelName = 'openai:gpt-5.2' class GradingOutput(BaseModel, populate_by_name=True):     """The output of a grading operation.""" -    reason: str+    reason: str = Field(+        description=(+            'A concise, self-consistent explanation of the grade, at most a few sentences. '+            'State only the final conclusion about which rubric criteria passed or failed; '+            'do not include step-by-step reasoning, re-checking, or self-corrections.'+        )+    )     pass_: bool = Field(validation_alias='pass', serialization_alias='pass')     score: float @@ -47,6 +53,8 @@ _judge_output_agent = Agent(         <Output>Avast ye swabs, repel the invaders!</Output>         <Rubric>Does not speak like a pirate</Rubric>         {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}++        Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there.         """     ),     output_type=GradingOutput,@@ -87,6 +95,8 @@ _judge_input_output_agent = Agent(         <Output>Avast ye swabs, repel the invaders!</Output>         <Rubric>Does not speak in the style described by the input</Rubric>         {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}++        Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there.         """     ),     output_type=GradingOutput,@@ -131,6 +141,8 @@ _judge_input_output_expected_agent = Agent(         <Output>Six</Output>         <Rubric>The output is factually consistent with the expected output</Rubric>         {"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}++        Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there.         """     ),     output_type=GradingOutput,@@ -176,6 +188,8 @@ _judge_output_expected_agent = Agent(         <Output>Six</Output>         <Rubric>The output should be a number written in words which matches the number written in digits in the expected output</Rubric>         {"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}++        Keep the `reason` concise, self-consistent, and final: briefly explain why the output does or does not satisfy the rubric. Do not think out loud, re-check, or retract earlier statements in the `reason` field; include only your final conclusion there.         """     ),     output_type=GradingOutput,tests/evals/test_llm_as_a_judge.pyindex 292f8779..43863606 100644@@ -41,6 +41,19 @@ def test_grading_output():     assert output.score == 1.0  +def test_grading_output_reason_description():+    """The reason field must carry schema guidance keeping it concise and stable, so reasoning+    models do not leak think-out-loud / self-correction text into it (issue: reason pollution)."""+    schema = GradingOutput.model_json_schema()+    reason_description = schema['properties']['reason'].get('description')+    assert reason_description is not None+    lowered = reason_description.lower()+    # The reason should be explicitly guided to stay concise...+    assert 'concise' in lowered+    # ...and to exclude verbose step-by-step reasoning / self-corrections.+    assert 'step-by-step' in lowered or 'self-correction' in lowered or 'reasoning' in lowered++ def test_stringify():     """Test _stringify function."""     # Test with string 
Merged fix · PR #+34 −17
pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.pyindex 0b6fed34e9..f5158ae14d 100644@@ -27,26 +27,37 @@ _default_model: models.Model | models.KnownModelName = 'openai:gpt-5.2' class GradingOutput(BaseModel, populate_by_name=True):     """The output of a grading operation.""" -    reason: str+    reason: str = Field(+        description='A concise 1-2 sentence explanation of why the output passed or failed.',+    )     pass_: bool = Field(validation_alias='pass', serialization_alias='pass')     score: float  +_JUDGE_REASON_INSTRUCTION = (+    'The "reason" field must be a concise 1-2 sentence summary of your verdict. '+    'Do not include your reasoning process, self-corrections, or re-checking in the reason. '+    'State only the final conclusion.'+)++ _judge_output_agent = Agent(     name='judge_output',     system_prompt=dedent(-        """-        You are grading output according to a user-specified rubric. If the statement in the rubric is true, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+        f"""+        You are grading output according to a user-specified rubric. If the statement in the rubric is true, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++        {_JUDGE_REASON_INSTRUCTION}          Examples:          <Output>Hello world</Output>         <Rubric>Content contains a greeting</Rubric>-        {"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}+        {{"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}}          <Output>Avast ye swabs, repel the invaders!</Output>         <Rubric>Does not speak like a pirate</Rubric>-        {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}+        {{"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}}         """     ),     output_type=GradingOutput,@@ -73,20 +84,22 @@ async def judge_output( _judge_input_output_agent = Agent(     name='judge_input_output',     system_prompt=dedent(-        """-        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+        f"""+        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++        {_JUDGE_REASON_INSTRUCTION}          Examples:          <Input>Hello world</Input>         <Output>Hello</Output>         <Rubric>Content contains a greeting word which is present in the input</Rubric>-        {"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}+        {{"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}}          <Input>Pirate</Input>         <Output>Avast ye swabs, repel the invaders!</Output>         <Rubric>Does not speak in the style described by the input</Rubric>-        {"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}+        {{"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}}         """     ),     output_type=GradingOutput,@@ -115,8 +128,10 @@ async def judge_input_output( _judge_input_output_expected_agent = Agent(     name='judge_input_output_expected',     system_prompt=dedent(-        """-        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input, expected output, and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+        f"""+        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided input, expected output, and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++        {_JUDGE_REASON_INSTRUCTION}          Examples: @@ -124,13 +139,13 @@ _judge_input_output_expected_agent = Agent(         <ExpectedOutput>Blue</ExpectedOutput>         <Output>Cerulean</Output>         <Rubric>The output is consistent with the expected output but doesn't have to match exactly</Rubric>-        {"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}+        {{"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}}          <Input>How many legs does a spider have?</Input>         <ExpectedOutput>8</ExpectedOutput>         <Output>Six</Output>         <Rubric>The output is factually consistent with the expected output</Rubric>-        {"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}+        {{"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}}         """     ),     output_type=GradingOutput,@@ -162,20 +177,22 @@ async def judge_input_output_expected( _judge_output_expected_agent = Agent(     name='judge_output_expected',     system_prompt=dedent(-        """-        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided expected output and output, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}+        f"""+        You are grading output according to a user-specified rubric. If the statement in the rubric is true for the provided expected output and output, then the output passes the test. You respond with a JSON object with this structure: {{reason: string, pass: boolean, score: number}}++        {_JUDGE_REASON_INSTRUCTION}          Examples:          <ExpectedOutput>Blue</ExpectedOutput>         <Output>Cerulean</Output>         <Rubric>The output should be a shade of the expected output color</Rubric>-        {"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}+        {{"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}}          <ExpectedOutput>8</ExpectedOutput>         <Output>Six</Output>         <Rubric>The output should be a number written in words which matches the number written in digits in the expected output</Rubric>-        {"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}+        {{"reason": "The output is 'Six' which is a different number than 8", "pass": false, "score": 0.0}}         """     ),     output_type=GradingOutput, 

What the agent reported

The change is complete and verified. Final report below.

What was wrong

pydantic_evals.evaluators.llm_as_a_judge.GradingOutput.reason was a bare reason: str with no guidance, and the four judge agents' system prompts only said "respond with {reason, pass, score}". Nothing told the model to keep reason short or final, so reasoning-capable judge models would dump long, self-correcting, sometimes self-contradictory "think out loud" text into reason — making it unusable as concise, stable retry feedback (e.g. fed into ModelRetry). The reporter wanted the judge to keep reasoning internally but return a concise, stable reason, without disabling thinking.

What I changed

Two production edits in pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py — purely prompt/schema guidance, no model_settings/thinking disabled, no signatures touched:

  1. GradingOutput.reason now has a Field(description=...) telling the model to keep the reason concise, self-consistent, and final (at most a few sentences), to state only the final conclusion, and to exclude step-by-step reasoning, re-checking, or self-corrections. Because output_type=GradingOutput, this description is emitted in the structured-output JSON schema the judge model receives — the direct lever on the reason field.
  2. All four judge agents' system prompts (_judge_output_agent, _judge_input_output_agent, _judge_input_output_expected_agent, _judge_output_expected_agent) got an identical appended paragraph reinforcing the same "keep reason concise, don't think out loud in it" instruction.

One test edit in tests/evals/test_llm_as_a_judge.py: added test_grading_output_reason_description, asserting via the public GradingOutput.model_json_schema() that reason carries a description mentioning conciseness and discouraging verbose reasoning/self-correction.

Test evidence

  • Regression test failed before the fix (assert None is not None — reason had no description).
  • After the fix: ./bt-test tests/evals/test_llm_as_a_judge.py → 14 passed (the new test plus all 13 pre-existing tests). No existing test was weakened, skipped, or altered; system-prompt additions don't touch any snapshot because those assert only the user prompt, never the system prompt.
  • Reviewer confirmed the diff is limited to the two intended files, pass_/score/signatures unchanged, thinking not disabled, and no scratch/temp files.

Requirements disposition

  • reason should be concise/stable and not polluted by think-out-loud/self-correction text → Implemented via the reason field description (schema-level) + system-prompt guidance across all four judges. Tested by test_grading_output_reason_description (red→green).
  • Judge must retain internal reasoning (do NOT disable thinking) → Honored: no model_settings/thinking changes; the fix only shapes what goes into the public reason field. Verified by reviewer (no settings changed).
  • Applies to the judge_output() path shown in the report → Covered: judge_output uses _judge_output_agent + GradingOutput, both updated; the sibling judges (judge_input_output, judge_input_output_expected, judge_output_expected) share GradingOutput and were updated consistently.

Note (not mine to fix): the working tree already contained pre-existing deletions of AGENTS.md, CLAUDE.md, and .claude/settings.json from before this task — unrelated to this change and left untouched. All my changes are left uncommitted in the working tree.