Backtestspydantic-airun-20260918-215936#6376
Bug: ToolOutput accepts negative max_retries without validation
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches resolve the root cause identically — raising UserError for negative max_retries in ToolOutput.__init__, consistent with the referenced Tool fix (#6375), and both correctly guard against None. Functionally they are equivalent (A validates before assigning attributes, B after; neither difference matters since the exception aborts construction either way). The decisive differences are tests and scope. Patch A adds three targeted tests: a negative value raising UserError (fails before the fix, passes after), plus zero and None acceptance cases that pin the boundary and guard against over-eager validation — non-tautological and directly exercising the change. Patch B adds no tests at all. On scope, Patch B is worse: it introduces a stray, empty, accidentally-created file named 'finally\357\200\272' (a garbage artifact, clear incidental churn that shouldn't be committed), whereas Patch A's only non-source change is the necessary test additions. Craft slightly favors A as well — validating inputs before mutating instance state is marginally cleaner than assigning then checking. With equal correctness, A wins decisively on tests and scope.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
9m 25s1 turn
Wall-clock time in the container
Full test suite
No new failures
7,934 passed · 541 skipped · 8,554 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
21vs 2 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
ToolOutput accepts max_retries without validation. Negative values cause undefined retry behavior. Same pattern as Tool.max_retries which was fixed in #6375.
Expected behavior
max_retries should raise UserError for negative values, consistent with the fix for Tool.
The two patches
pydantic_ai_slim/pydantic_ai/output.pyindex af915d15..87f13daf 100644@@ -139,6 +139,8 @@ class ToolOutput(Generic[OutputDataT]): strict: bool | None = None, sequential: bool = False, ):+ if max_retries is not None and max_retries < 0:+ raise exceptions.UserError(f'max_retries must be >= 0, got {max_retries}') self.output = type_ self.name = name self.description = descriptiontests/test_agent_output_schemas.pyindex 80f94cf8..cca41823 100644@@ -13,6 +13,7 @@ from pydantic_ai import ( StructuredDict, TextOutput, ToolOutput,+ UserError, ) from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart from pydantic_ai.models.function import AgentInfo, FunctionModel@@ -751,3 +752,21 @@ async def test_output_type_description(output_type: type, expected_schema: dict[ async def test_nested_output_type_description(output_type: type, expected_schema: dict[str, object]): agent: Agent[object, str] = Agent('test', output_type=output_type) assert remove_schema_descriptions(agent.output_json_schema()) == expected_schema+++# --- ToolOutput parameter validation -------------------------------------------+++def test_tool_output_rejects_negative_max_retries():+ with pytest.raises(UserError, match='max_retries must be >= 0'):+ ToolOutput(str, max_retries=-1)+++def test_tool_output_accepts_zero_max_retries():+ output = ToolOutput(str, max_retries=0)+ assert output.max_retries == 0+++def test_tool_output_accepts_none_max_retries():+ output = ToolOutput(str, max_retries=None)+ assert output.max_retries is None
diff --git "a/finally\357\200\272" "b/finally\357\200\272"new file mode 100644index 0000000000..e69de29bb2pydantic_ai_slim/pydantic_ai/output.pyindex af915d15f3..77a937bdc2 100644@@ -143,6 +143,8 @@ class ToolOutput(Generic[OutputDataT]): self.name = name self.description = description self.max_retries = max_retries+ if self.max_retries is not None and self.max_retries < 0:+ raise exceptions.UserError(f'max_retries must be >= 0, got {self.max_retries}') self.strict = strict self.sequential = sequential
What the agent reported
The fix is complete and verified. Here is my final report.
What was wrong
ToolOutput.__init__ in pydantic_ai_slim/pydantic_ai/output.py accepted max_retries and stored it directly with no validation. Negative values would silently produce undefined retry behavior — inconsistent with Tool.max_retries, which was fixed in #6375 to reject negatives with a UserError.
What I changed
pydantic_ai_slim/pydantic_ai/output.py — added a 2-line guard at the top of ToolOutput.__init__, before any attribute assignment:
if max_retries is not None and max_retries < 0:
raise exceptions.UserError(f'max_retries must be >= 0, got {max_retries}')
This mirrors #6375 exactly (same condition, same message string). It uses exceptions.UserError because that file already references UserError via the exceptions module rather than a bare import — keeping the change minimal and localized. None is still accepted (preserves the "use agent-level default" behavior).
tests/test_agent_output_schemas.py — added UserError to the existing from pydantic_ai import (...) block and appended a "ToolOutput parameter validation" region with three tests mirroring the #6375 test set: test_tool_output_rejects_negative_max_retries (the reported scenario → UserError), plus zero and None boundary cases.
Test evidence
- Before the fix:
test_tool_output_rejects_negative_max_retriesfailed withDID NOT RAISE UserError; the zero/none tests passed. - After the fix: all three new tests pass.
- No regressions: the full
tests/test_agent_output_schemas.pyfile — 18 passed. - Reviewer confirmed the diff matches the #6375 convention, asserts the user-visible outcome, and changes nothing unrelated.
Working tree contains only my two edits (the three pre-existing file deletions from the session-start snapshot are unrelated and untouched by me). No commits made.
Requirements checklist
max_retriesraisesUserErrorfor negative values onToolOutput— ✅ Implemented; verified bytest_tool_output_rejects_negative_max_retries(fails before, passes after).- Consistent with the
Toolfix (#6375) — ✅ Same condition (is not None and < 0), identical message'max_retries must be >= 0, got {max_retries}', sameUserErrortype; test set structurally mirrorstest_tools.py.
Non-blocking note from review: #6375 introduced a reusable _validate_max_retries helper in tools.py; I intentionally inlined the equivalent check in output.py rather than importing across modules, to keep the fix minimal and avoid a new cross-module dependency. Behavior and message are identical.