Backtestspydantic-airun-20260918-215936#4733
AGUIEventStream emits TOOL_CALL_ARGS after TOOL_CALL_END for same tool_call_id (stray delta), violating AG-UI protocol
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches apply the identical root-cause fix: they add an `_ended_tool_call_ids` set, populate it in both `handle_tool_call_end` and `handle_builtin_tool_call_end` (correctly recording the mapped builtin id, not the raw one), and short-circuit `handle_tool_call_delta` when the id has already ended. This addresses the actual issue — a stray delta arriving after TOOL_CALL_END — rather than masking a symptom, and it handles the builtin-tool scenario the issue reports (MCPServerTool via Responses API), since the drop check runs after the `_builtin_tool_call_ids` remapping. So correctness is a wash. The tiebreakers both favor B. Tests: B adds a targeted regression test that reproduces the exact scenario (PartStart/PartDelta/PartEnd/stray-PartDelta for a builtin id), asserts the full ordered event snapshot, AND explicitly checks no TOOL_CALL_ARGS follows TOOL_CALL_END — this fails before the fix and passes after; A ships no test at all. Craft: B's `default_factory=set[str]` matches the surrounding `_builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])` convention exactly, whereas A's bare `set` is slightly less consistent, and B's local var name `tool_call_id` in the builtin handler reads more naturally than A's `builtin_id`. Scope for both is minimal and confined to the fix (plus B's test, which is appropriate, not churn). B is better on tests and craft with equal correctness.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
14m 53s2 turns
Wall-clock time in the container
Full test suite
No new failures
5,129 passed · 273 skipped · 5,480 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
113vs 10 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I'm using the latest version of Pydantic AI
- I've searched for my issue in the issue tracker before opening this issue
Description
Bug Description
AGUIEventStream.handle_tool_call_delta receives a ToolCallPartDelta after handle_builtin_tool_call_end has already been called for the same tool_call_id. This causes a TOOL_CALL_ARGS SSE event to be emitted after TOOL_CALL_END for the same tool call, which violates the AG-UI protocol and crashes the @ag-ui/client verifier.
Observed SSE Output (from curling the backend)
TOOL_CALL_START (id: mcp_22b...)
TOOL_CALL_ARGS (id: mcp_22b..., delta: '{"action":"call_tool",...,"tool_args":')
TOOL_CALL_ARGS (id: mcp_22b..., delta: '}')
TOOL_CALL_END (id: mcp_22b...)
TOOL_CALL_ARGS (id: mcp_22b..., delta: '}') <- stray delta AFTER end
The final TOOL_CALL_ARGS event arrives after TOOL_CALL_END, which is invalid per the AG-UI protocol spec.
Steps to Reproduce
- Use
MCPServerToolwith the OpenAI Responses API model in streaming mode via the AG-UI endpoint. - Trigger a tool call that emits multiple argument deltas.
- Inspect the raw SSE stream (e.g. with
curl). - Observe a
TOOL_CALL_ARGSevent emitted afterTOOL_CALL_ENDfor the sametool_call_id.
Expected Behavior
No TOOL_CALL_ARGS events should be emitted for a tool_call_id after TOOL_CALL_END has been sent for that ID. The stream should be ordered: TOOL_CALL_START → one or more TOOL_CALL_ARGS → TOOL_CALL_END.
Actual Behavior
A stray TOOL_CALL_ARGS delta is emitted after TOOL_CALL_END, violating the AG-UI protocol and causing the @ag-ui/client verifier to crash/error.
Related Issues
- #4623 — builtin tool results dropped on follow-up AG-UI requests
- ag-ui-protocol/ag-ui#1275 — TOOL_CALL_START events suppressed after tool result
Neither of the above covers this exact scenario (stray post-end delta).
Environment
- Model: Internal OpenAI Responses API-compatible endpoint routing to custom inference (streaming mode); not the official OpenAI API
- Tool type:
MCPServerTool - Component:
AGUIEventStream(AG-UI integration)
Minimal, Reproducible Example
Logfire Trace
_No response_
Python, Pydantic AI & LLM client version
- Python: latest
- Pydantic AI: latest
- LLM provider SDK: openai (Responses API, streaming mode)
The two patches
pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.pyindex 5ae697b1..747117e9 100644@@ -76,6 +76,7 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output _thinking_text: bool = False _builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])+ _ended_tool_call_ids: set[str] = field(default_factory=set[str]) _error: bool = False @property@@ -203,16 +204,24 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output assert tool_call_id, '`ToolCallPartDelta.tool_call_id` must be set' if tool_call_id in self._builtin_tool_call_ids: tool_call_id = self._builtin_tool_call_ids[tool_call_id]+ # A `ToolCallPartDelta` can arrive after the part's `PartEndEvent` (i.e. after we've already+ # emitted `TOOL_CALL_END`). Emitting args after `TOOL_CALL_END` violates the AG-UI protocol,+ # so drop the stray delta.+ if tool_call_id in self._ended_tool_call_ids:+ return yield ToolCallArgsEvent( tool_call_id=tool_call_id, delta=delta.args_delta if isinstance(delta.args_delta, str) else json.dumps(delta.args_delta), ) async def handle_tool_call_end(self, part: ToolCallPart) -> AsyncIterator[BaseEvent]:+ self._ended_tool_call_ids.add(part.tool_call_id) yield ToolCallEndEvent(tool_call_id=part.tool_call_id) async def handle_builtin_tool_call_end(self, part: BuiltinToolCallPart) -> AsyncIterator[BaseEvent]:- yield ToolCallEndEvent(tool_call_id=self._builtin_tool_call_ids[part.tool_call_id])+ tool_call_id = self._builtin_tool_call_ids[part.tool_call_id]+ self._ended_tool_call_ids.add(tool_call_id)+ yield ToolCallEndEvent(tool_call_id=tool_call_id) async def handle_builtin_tool_return(self, part: BuiltinToolReturnPart) -> AsyncIterator[BaseEvent]: tool_call_id = self._builtin_tool_call_ids[part.tool_call_id]tests/test_ag_ui.pyindex 5a1e6c11..52ed56da 100644@@ -2402,6 +2402,108 @@ async def test_event_stream_multiple_responses_with_tool_calls(): assert result_message_id != new_message_id +async def test_event_stream_no_tool_call_args_after_end():+ """Regression test: no stray TOOL_CALL_ARGS after TOOL_CALL_END.++ The model stream can emit a `ToolCallPartDelta` (as a `PartDeltaEvent`) for a+ builtin tool call AFTER the `PartEndEvent` for that same part. The buggy+ `handle_tool_call_delta` still finds the id in `_builtin_tool_call_ids` and+ re-emits a `TOOL_CALL_ARGS` (ToolCallArgsEvent) event after the+ `TOOL_CALL_END` (ToolCallEndEvent) for the same builtin `tool_call_id`,+ violating the AG-UI protocol ordering+ (TOOL_CALL_START -> TOOL_CALL_ARGS* -> TOOL_CALL_END).+ """++ async def event_generator():+ yield PartStartEvent(+ index=0,+ part=BuiltinToolCallPart(+ tool_name='mcp',+ args='{"action":"call_tool"',+ tool_call_id='mcp_1',+ provider_name='mcp',+ ),+ )+ yield PartDeltaEvent(+ index=0,+ delta=ToolCallPartDelta(args_delta=',"tool_args":', tool_call_id='mcp_1'),+ )+ yield PartEndEvent(+ index=0,+ part=BuiltinToolCallPart(+ tool_name='mcp',+ args='{"action":"call_tool","tool_args":}',+ tool_call_id='mcp_1',+ provider_name='mcp',+ ),+ )+ # STRAY delta after the part has already ended - this is the bug trigger.+ yield PartDeltaEvent(+ index=0,+ delta=ToolCallPartDelta(args_delta='}', tool_call_id='mcp_1'),+ )++ run_input = create_input(+ UserMessage(+ id='msg_1',+ content='Call the tool',+ ),+ )+ event_stream = AGUIEventStream(run_input=run_input)+ events = [+ json.loads(event.removeprefix('data: '))+ async for event in event_stream.encode_stream(event_stream.transform_stream(event_generator()))+ ]++ assert events == snapshot(+ [+ {+ 'type': 'RUN_STARTED',+ 'timestamp': IsInt(),+ 'threadId': (thread_id := IsSameStr()),+ 'runId': (run_id := IsSameStr()),+ },+ {+ 'type': 'TOOL_CALL_START',+ 'timestamp': IsInt(),+ 'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+ 'toolCallName': 'mcp',+ 'parentMessageId': IsStr(),+ },+ {+ 'type': 'TOOL_CALL_ARGS',+ 'timestamp': IsInt(),+ 'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+ 'delta': '{"action":"call_tool"',+ },+ {+ 'type': 'TOOL_CALL_ARGS',+ 'timestamp': IsInt(),+ 'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+ 'delta': ',"tool_args":',+ },+ {'type': 'TOOL_CALL_END', 'timestamp': IsInt(), 'toolCallId': 'pyd_ai_builtin|mcp|mcp_1'},+ {+ 'type': 'RUN_FINISHED',+ 'timestamp': IsInt(),+ 'threadId': thread_id,+ 'runId': run_id,+ },+ ]+ )++ # Explicitly verify no TOOL_CALL_ARGS is emitted after TOOL_CALL_END for this id.+ end_index = next(+ i+ for i, event in enumerate(events)+ if event['type'] == 'TOOL_CALL_END' and event['toolCallId'] == 'pyd_ai_builtin|mcp|mcp_1'+ )+ assert not any(+ event['type'] == 'TOOL_CALL_ARGS' and event.get('toolCallId') == 'pyd_ai_builtin|mcp|mcp_1'+ for event in events[end_index + 1 :]+ )++ async def test_timestamps_are_set(): """Test that all AG-UI events have timestamps set.""" agent = Agent(
pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.pyindex 5ae697b1e2..c2b49276e7 100644@@ -76,6 +76,7 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output _thinking_text: bool = False _builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])+ _ended_tool_call_ids: set[str] = field(default_factory=set) _error: bool = False @property@@ -203,16 +204,23 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output assert tool_call_id, '`ToolCallPartDelta.tool_call_id` must be set' if tool_call_id in self._builtin_tool_call_ids: tool_call_id = self._builtin_tool_call_ids[tool_call_id]+ # Drop stray deltas that arrive after TOOL_CALL_END has already been emitted for this ID.+ # See https://github.com/pydantic/pydantic-ai/issues/4733+ if tool_call_id in self._ended_tool_call_ids:+ return yield ToolCallArgsEvent( tool_call_id=tool_call_id, delta=delta.args_delta if isinstance(delta.args_delta, str) else json.dumps(delta.args_delta), ) async def handle_tool_call_end(self, part: ToolCallPart) -> AsyncIterator[BaseEvent]:+ self._ended_tool_call_ids.add(part.tool_call_id) yield ToolCallEndEvent(tool_call_id=part.tool_call_id) async def handle_builtin_tool_call_end(self, part: BuiltinToolCallPart) -> AsyncIterator[BaseEvent]:- yield ToolCallEndEvent(tool_call_id=self._builtin_tool_call_ids[part.tool_call_id])+ builtin_id = self._builtin_tool_call_ids[part.tool_call_id]+ self._ended_tool_call_ids.add(builtin_id)+ yield ToolCallEndEvent(tool_call_id=builtin_id) async def handle_builtin_tool_return(self, part: BuiltinToolReturnPart) -> AsyncIterator[BaseEvent]: tool_call_id = self._builtin_tool_call_ids[part.tool_call_id]
What the agent reported
Everything checks out. The only source changes from this work are the two intended files; the three deleted config/doc files were already deleted before I began (per the initial git status snapshot) and are unrelated to this fix. No scratch/temp files.
Summary
What was wrong
AGUIEventStream (pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.py) could emit a TOOL_CALL_ARGS SSE event after TOOL_CALL_END for the same tool_call_id, violating the AG-UI protocol. The model stream can deliver a ToolCallPartDelta (dispatched to handle_tool_call_delta) after the part's PartEndEvent (dispatched to handle_builtin_tool_call_end/handle_tool_call_end). Nothing tracked which tool-call IDs had already ended, so the stray delta re-emitted args — and for builtin calls the ID still lived in _builtin_tool_call_ids, so it was even remapped and re-emitted.
What I changed (production, _event_stream.py, +11 lines)
- Added
_ended_tool_call_ids: set[str]field to track IDs whoseTOOL_CALL_ENDwas already emitted. handle_tool_call_endandhandle_builtin_tool_call_endnow record the emitted (protocol-resolved) ID into that set.handle_tool_call_deltaresolves the ID (including the builtinpyd_ai_builtin|…remap) and returns early — suppressing theTOOL_CALL_ARGS— if that ID has already ended. The stored ID is the same resolved ID the delta handler compares against, so suppression works for both regular and builtin tool calls.
Test evidence
- Added
test_event_stream_no_tool_call_args_after_endintests/test_ag_ui.py, reproducing the exact scenario (builtin tool-call start → delta → end → stray delta after end). It asserts both a full snapshot ending START → ARGS → ARGS → END → RUN_FINISHED and an explicit check that noTOOL_CALL_ARGSfor that ID appears afterTOOL_CALL_END. - The test failed before the fix (stray
TOOL_CALL_ARGS {'delta': '}'}afterTOOL_CALL_END) and passes after. - Gate:
./bt-test tests/test_ag_ui.py tests/test_ui.py→ 50 passed, no regressions.
Requirements disposition
- No
TOOL_CALL_ARGSafterTOOL_CALL_ENDfor atool_call_id; ordering must be START → ARGS → END* — Implemented, covered by the new regression test (builtin path) and verified by the full AG-UI/UI suite. - Reported scenario is builtin (
MCPServerTool/ OpenAI Responses streaming) — Implemented and tested on the builtin path withprovider_name='mcp'and the prefixedpyd_ai_builtin|mcp|mcp_1ID.
Reviewer note (non-blocking): the fix also guards the regular (non-builtin) tool-call path, which is correct by inspection but not directly covered by a dedicated test. I kept the change minimal and did not add a second test, since the reported and reproduced scenario is the builtin path; the regular path shares the identical suppression logic.