Backtestspydantic-airun-20260918-215936#6376

Issue #6376 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

Bug: ToolOutput accepts negative max_retries without validation

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches resolve the root cause identically — raising UserError for negative max_retries in ToolOutput.__init__, consistent with the referenced Tool fix (#6375), and both correctly guard against None. Functionally they are equivalent (A validates before assigning attributes, B after; neither difference matters since the exception aborts construction either way). The decisive differences are tests and scope. Patch A adds three targeted tests: a negative value raising UserError (fails before the fix, passes after), plus zero and None acceptance cases that pin the boundary and guard against over-eager validation — non-tautological and directly exercising the change. Patch B adds no tests at all. On scope, Patch B is worse: it introduces a stray, empty, accidentally-created file named 'finally\357\200\272' (a garbage artifact, clear incidental churn that shouldn't be committed), whereas Patch A's only non-source change is the necessary test additions. Craft slightly favors A as well — validating inputs before mutating instance state is marginally cleaner than assigning then checking. With equal correctness, A wins decisively on tests and scope.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

9m 25s1 turn

Wall-clock time in the container

Full test suite

No new failures

7,934 passed · 541 skipped · 8,554 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

21vs 2 merged

Changed lines, ours vs merged

Description

ToolOutput accepts max_retries without validation. Negative values cause undefined retry behavior. Same pattern as Tool.max_retries which was fixed in #6375.

Expected behavior

max_retries should raise UserError for negative values, consistent with the fix for Tool.

The two patches

Our patch+21 −0
pydantic_ai_slim/pydantic_ai/output.pyindex af915d15..87f13daf 100644@@ -139,6 +139,8 @@ class ToolOutput(Generic[OutputDataT]):         strict: bool | None = None,         sequential: bool = False,     ):+        if max_retries is not None and max_retries < 0:+            raise exceptions.UserError(f'max_retries must be >= 0, got {max_retries}')         self.output = type_         self.name = name         self.description = descriptiontests/test_agent_output_schemas.pyindex 80f94cf8..cca41823 100644@@ -13,6 +13,7 @@ from pydantic_ai import (     StructuredDict,     TextOutput,     ToolOutput,+    UserError, ) from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart from pydantic_ai.models.function import AgentInfo, FunctionModel@@ -751,3 +752,21 @@ async def test_output_type_description(output_type: type, expected_schema: dict[ async def test_nested_output_type_description(output_type: type, expected_schema: dict[str, object]):     agent: Agent[object, str] = Agent('test', output_type=output_type)     assert remove_schema_descriptions(agent.output_json_schema()) == expected_schema+++# --- ToolOutput parameter validation -------------------------------------------+++def test_tool_output_rejects_negative_max_retries():+    with pytest.raises(UserError, match='max_retries must be >= 0'):+        ToolOutput(str, max_retries=-1)+++def test_tool_output_accepts_zero_max_retries():+    output = ToolOutput(str, max_retries=0)+    assert output.max_retries == 0+++def test_tool_output_accepts_none_max_retries():+    output = ToolOutput(str, max_retries=None)+    assert output.max_retries is None 
Merged fix · PR #+2 −0
diff --git "a/finally\357\200\272" "b/finally\357\200\272"new file mode 100644index 0000000000..e69de29bb2pydantic_ai_slim/pydantic_ai/output.pyindex af915d15f3..77a937bdc2 100644@@ -143,6 +143,8 @@ class ToolOutput(Generic[OutputDataT]):         self.name = name         self.description = description         self.max_retries = max_retries+        if self.max_retries is not None and self.max_retries < 0:+            raise exceptions.UserError(f'max_retries must be >= 0, got {self.max_retries}')         self.strict = strict         self.sequential = sequential  

What the agent reported

The fix is complete and verified. Here is my final report.

What was wrong

ToolOutput.__init__ in pydantic_ai_slim/pydantic_ai/output.py accepted max_retries and stored it directly with no validation. Negative values would silently produce undefined retry behavior — inconsistent with Tool.max_retries, which was fixed in #6375 to reject negatives with a UserError.

What I changed

pydantic_ai_slim/pydantic_ai/output.py — added a 2-line guard at the top of ToolOutput.__init__, before any attribute assignment:

if max_retries is not None and max_retries < 0:
    raise exceptions.UserError(f'max_retries must be >= 0, got {max_retries}')

This mirrors #6375 exactly (same condition, same message string). It uses exceptions.UserError because that file already references UserError via the exceptions module rather than a bare import — keeping the change minimal and localized. None is still accepted (preserves the "use agent-level default" behavior).

tests/test_agent_output_schemas.py — added UserError to the existing from pydantic_ai import (...) block and appended a "ToolOutput parameter validation" region with three tests mirroring the #6375 test set: test_tool_output_rejects_negative_max_retries (the reported scenario → UserError), plus zero and None boundary cases.

Test evidence

  • Before the fix: test_tool_output_rejects_negative_max_retries failed with DID NOT RAISE UserError; the zero/none tests passed.
  • After the fix: all three new tests pass.
  • No regressions: the full tests/test_agent_output_schemas.py file — 18 passed.
  • Reviewer confirmed the diff matches the #6375 convention, asserts the user-visible outcome, and changes nothing unrelated.

Working tree contains only my two edits (the three pre-existing file deletions from the session-start snapshot are unrelated and untouched by me). No commits made.

Requirements checklist

  1. max_retries raises UserError for negative values on ToolOutput — ✅ Implemented; verified by test_tool_output_rejects_negative_max_retries (fails before, passes after).
  2. Consistent with the Tool fix (#6375) — ✅ Same condition (is not None and < 0), identical message 'max_retries must be >= 0, got {max_retries}', same UserError type; test set structurally mirrors test_tools.py.

Non-blocking note from review: #6375 introduced a reusable _validate_max_retries helper in tools.py; I intentionally inlined the equivalent check in output.py rather than importing across modules, to keep the fix minimal and avoid a new cross-module dependency. Behavior and message are identical.