Backtestspydantic-airun-20260918-215936#4733

Issue #4733 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

AGUIEventStream emits TOOL_CALL_ARGS after TOOL_CALL_END for same tool_call_id (stray delta), violating AG-UI protocol

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches apply the identical root-cause fix: they add an `_ended_tool_call_ids` set, populate it in both `handle_tool_call_end` and `handle_builtin_tool_call_end` (correctly recording the mapped builtin id, not the raw one), and short-circuit `handle_tool_call_delta` when the id has already ended. This addresses the actual issue — a stray delta arriving after TOOL_CALL_END — rather than masking a symptom, and it handles the builtin-tool scenario the issue reports (MCPServerTool via Responses API), since the drop check runs after the `_builtin_tool_call_ids` remapping. So correctness is a wash. The tiebreakers both favor B. Tests: B adds a targeted regression test that reproduces the exact scenario (PartStart/PartDelta/PartEnd/stray-PartDelta for a builtin id), asserts the full ordered event snapshot, AND explicitly checks no TOOL_CALL_ARGS follows TOOL_CALL_END — this fails before the fix and passes after; A ships no test at all. Craft: B's `default_factory=set[str]` matches the surrounding `_builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])` convention exactly, whereas A's bare `set` is slightly less consistent, and B's local var name `tool_call_id` in the builtin handler reads more naturally than A's `builtin_id`. Scope for both is minimal and confined to the fix (plus B's test, which is appropriate, not churn). B is better on tests and craft with equal correctness.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

14m 53s2 turns

Wall-clock time in the container

Full test suite

No new failures

5,129 passed · 273 skipped · 5,480 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

113vs 10 merged

Changed lines, ours vs merged

Initial Checks

Description

Bug Description

AGUIEventStream.handle_tool_call_delta receives a ToolCallPartDelta after handle_builtin_tool_call_end has already been called for the same tool_call_id. This causes a TOOL_CALL_ARGS SSE event to be emitted after TOOL_CALL_END for the same tool call, which violates the AG-UI protocol and crashes the @ag-ui/client verifier.

Observed SSE Output (from curling the backend)

TOOL_CALL_START  (id: mcp_22b...)
TOOL_CALL_ARGS   (id: mcp_22b..., delta: '{"action":"call_tool",...,"tool_args":')
TOOL_CALL_ARGS   (id: mcp_22b..., delta: '}')
TOOL_CALL_END    (id: mcp_22b...)
TOOL_CALL_ARGS   (id: mcp_22b..., delta: '}')   <- stray delta AFTER end

The final TOOL_CALL_ARGS event arrives after TOOL_CALL_END, which is invalid per the AG-UI protocol spec.

Steps to Reproduce

  1. Use MCPServerTool with the OpenAI Responses API model in streaming mode via the AG-UI endpoint.
  2. Trigger a tool call that emits multiple argument deltas.
  3. Inspect the raw SSE stream (e.g. with curl).
  4. Observe a TOOL_CALL_ARGS event emitted after TOOL_CALL_END for the same tool_call_id.

Expected Behavior

No TOOL_CALL_ARGS events should be emitted for a tool_call_id after TOOL_CALL_END has been sent for that ID. The stream should be ordered: TOOL_CALL_START → one or more TOOL_CALL_ARGS → TOOL_CALL_END.

Actual Behavior

A stray TOOL_CALL_ARGS delta is emitted after TOOL_CALL_END, violating the AG-UI protocol and causing the @ag-ui/client verifier to crash/error.

Related Issues

  • #4623 — builtin tool results dropped on follow-up AG-UI requests
  • ag-ui-protocol/ag-ui#1275 — TOOL_CALL_START events suppressed after tool result

Neither of the above covers this exact scenario (stray post-end delta).

Environment

  • Model: Internal OpenAI Responses API-compatible endpoint routing to custom inference (streaming mode); not the official OpenAI API
  • Tool type: MCPServerTool
  • Component: AGUIEventStream (AG-UI integration)

Minimal, Reproducible Example

Logfire Trace

_No response_

Python, Pydantic AI & LLM client version

  • Python: latest
  • Pydantic AI: latest
  • LLM provider SDK: openai (Responses API, streaming mode)

The two patches

Our patch+112 −1
pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.pyindex 5ae697b1..747117e9 100644@@ -76,6 +76,7 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output      _thinking_text: bool = False     _builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])+    _ended_tool_call_ids: set[str] = field(default_factory=set[str])     _error: bool = False      @property@@ -203,16 +204,24 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output         assert tool_call_id, '`ToolCallPartDelta.tool_call_id` must be set'         if tool_call_id in self._builtin_tool_call_ids:             tool_call_id = self._builtin_tool_call_ids[tool_call_id]+        # A `ToolCallPartDelta` can arrive after the part's `PartEndEvent` (i.e. after we've already+        # emitted `TOOL_CALL_END`). Emitting args after `TOOL_CALL_END` violates the AG-UI protocol,+        # so drop the stray delta.+        if tool_call_id in self._ended_tool_call_ids:+            return         yield ToolCallArgsEvent(             tool_call_id=tool_call_id,             delta=delta.args_delta if isinstance(delta.args_delta, str) else json.dumps(delta.args_delta),         )      async def handle_tool_call_end(self, part: ToolCallPart) -> AsyncIterator[BaseEvent]:+        self._ended_tool_call_ids.add(part.tool_call_id)         yield ToolCallEndEvent(tool_call_id=part.tool_call_id)      async def handle_builtin_tool_call_end(self, part: BuiltinToolCallPart) -> AsyncIterator[BaseEvent]:-        yield ToolCallEndEvent(tool_call_id=self._builtin_tool_call_ids[part.tool_call_id])+        tool_call_id = self._builtin_tool_call_ids[part.tool_call_id]+        self._ended_tool_call_ids.add(tool_call_id)+        yield ToolCallEndEvent(tool_call_id=tool_call_id)      async def handle_builtin_tool_return(self, part: BuiltinToolReturnPart) -> AsyncIterator[BaseEvent]:         tool_call_id = self._builtin_tool_call_ids[part.tool_call_id]tests/test_ag_ui.pyindex 5a1e6c11..52ed56da 100644@@ -2402,6 +2402,108 @@ async def test_event_stream_multiple_responses_with_tool_calls():     assert result_message_id != new_message_id  +async def test_event_stream_no_tool_call_args_after_end():+    """Regression test: no stray TOOL_CALL_ARGS after TOOL_CALL_END.++    The model stream can emit a `ToolCallPartDelta` (as a `PartDeltaEvent`) for a+    builtin tool call AFTER the `PartEndEvent` for that same part. The buggy+    `handle_tool_call_delta` still finds the id in `_builtin_tool_call_ids` and+    re-emits a `TOOL_CALL_ARGS` (ToolCallArgsEvent) event after the+    `TOOL_CALL_END` (ToolCallEndEvent) for the same builtin `tool_call_id`,+    violating the AG-UI protocol ordering+    (TOOL_CALL_START -> TOOL_CALL_ARGS* -> TOOL_CALL_END).+    """++    async def event_generator():+        yield PartStartEvent(+            index=0,+            part=BuiltinToolCallPart(+                tool_name='mcp',+                args='{"action":"call_tool"',+                tool_call_id='mcp_1',+                provider_name='mcp',+            ),+        )+        yield PartDeltaEvent(+            index=0,+            delta=ToolCallPartDelta(args_delta=',"tool_args":', tool_call_id='mcp_1'),+        )+        yield PartEndEvent(+            index=0,+            part=BuiltinToolCallPart(+                tool_name='mcp',+                args='{"action":"call_tool","tool_args":}',+                tool_call_id='mcp_1',+                provider_name='mcp',+            ),+        )+        # STRAY delta after the part has already ended - this is the bug trigger.+        yield PartDeltaEvent(+            index=0,+            delta=ToolCallPartDelta(args_delta='}', tool_call_id='mcp_1'),+        )++    run_input = create_input(+        UserMessage(+            id='msg_1',+            content='Call the tool',+        ),+    )+    event_stream = AGUIEventStream(run_input=run_input)+    events = [+        json.loads(event.removeprefix('data: '))+        async for event in event_stream.encode_stream(event_stream.transform_stream(event_generator()))+    ]++    assert events == snapshot(+        [+            {+                'type': 'RUN_STARTED',+                'timestamp': IsInt(),+                'threadId': (thread_id := IsSameStr()),+                'runId': (run_id := IsSameStr()),+            },+            {+                'type': 'TOOL_CALL_START',+                'timestamp': IsInt(),+                'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+                'toolCallName': 'mcp',+                'parentMessageId': IsStr(),+            },+            {+                'type': 'TOOL_CALL_ARGS',+                'timestamp': IsInt(),+                'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+                'delta': '{"action":"call_tool"',+            },+            {+                'type': 'TOOL_CALL_ARGS',+                'timestamp': IsInt(),+                'toolCallId': 'pyd_ai_builtin|mcp|mcp_1',+                'delta': ',"tool_args":',+            },+            {'type': 'TOOL_CALL_END', 'timestamp': IsInt(), 'toolCallId': 'pyd_ai_builtin|mcp|mcp_1'},+            {+                'type': 'RUN_FINISHED',+                'timestamp': IsInt(),+                'threadId': thread_id,+                'runId': run_id,+            },+        ]+    )++    # Explicitly verify no TOOL_CALL_ARGS is emitted after TOOL_CALL_END for this id.+    end_index = next(+        i+        for i, event in enumerate(events)+        if event['type'] == 'TOOL_CALL_END' and event['toolCallId'] == 'pyd_ai_builtin|mcp|mcp_1'+    )+    assert not any(+        event['type'] == 'TOOL_CALL_ARGS' and event.get('toolCallId') == 'pyd_ai_builtin|mcp|mcp_1'+        for event in events[end_index + 1 :]+    )++ async def test_timestamps_are_set():     """Test that all AG-UI events have timestamps set."""     agent = Agent( 
Merged fix · PR #+9 −1
pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.pyindex 5ae697b1e2..c2b49276e7 100644@@ -76,6 +76,7 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output      _thinking_text: bool = False     _builtin_tool_call_ids: dict[str, str] = field(default_factory=dict[str, str])+    _ended_tool_call_ids: set[str] = field(default_factory=set)     _error: bool = False      @property@@ -203,16 +204,23 @@ class AGUIEventStream(UIEventStream[RunAgentInput, BaseEvent, AgentDepsT, Output         assert tool_call_id, '`ToolCallPartDelta.tool_call_id` must be set'         if tool_call_id in self._builtin_tool_call_ids:             tool_call_id = self._builtin_tool_call_ids[tool_call_id]+        # Drop stray deltas that arrive after TOOL_CALL_END has already been emitted for this ID.+        # See https://github.com/pydantic/pydantic-ai/issues/4733+        if tool_call_id in self._ended_tool_call_ids:+            return         yield ToolCallArgsEvent(             tool_call_id=tool_call_id,             delta=delta.args_delta if isinstance(delta.args_delta, str) else json.dumps(delta.args_delta),         )      async def handle_tool_call_end(self, part: ToolCallPart) -> AsyncIterator[BaseEvent]:+        self._ended_tool_call_ids.add(part.tool_call_id)         yield ToolCallEndEvent(tool_call_id=part.tool_call_id)      async def handle_builtin_tool_call_end(self, part: BuiltinToolCallPart) -> AsyncIterator[BaseEvent]:-        yield ToolCallEndEvent(tool_call_id=self._builtin_tool_call_ids[part.tool_call_id])+        builtin_id = self._builtin_tool_call_ids[part.tool_call_id]+        self._ended_tool_call_ids.add(builtin_id)+        yield ToolCallEndEvent(tool_call_id=builtin_id)      async def handle_builtin_tool_return(self, part: BuiltinToolReturnPart) -> AsyncIterator[BaseEvent]:         tool_call_id = self._builtin_tool_call_ids[part.tool_call_id] 

What the agent reported

Everything checks out. The only source changes from this work are the two intended files; the three deleted config/doc files were already deleted before I began (per the initial git status snapshot) and are unrelated to this fix. No scratch/temp files.

Summary

What was wrong

AGUIEventStream (pydantic_ai_slim/pydantic_ai/ui/ag_ui/_event_stream.py) could emit a TOOL_CALL_ARGS SSE event after TOOL_CALL_END for the same tool_call_id, violating the AG-UI protocol. The model stream can deliver a ToolCallPartDelta (dispatched to handle_tool_call_delta) after the part's PartEndEvent (dispatched to handle_builtin_tool_call_end/handle_tool_call_end). Nothing tracked which tool-call IDs had already ended, so the stray delta re-emitted args — and for builtin calls the ID still lived in _builtin_tool_call_ids, so it was even remapped and re-emitted.

What I changed (production, _event_stream.py, +11 lines)

  • Added _ended_tool_call_ids: set[str] field to track IDs whose TOOL_CALL_END was already emitted.
  • handle_tool_call_end and handle_builtin_tool_call_end now record the emitted (protocol-resolved) ID into that set.
  • handle_tool_call_delta resolves the ID (including the builtin pyd_ai_builtin|… remap) and returns early — suppressing the TOOL_CALL_ARGS — if that ID has already ended. The stored ID is the same resolved ID the delta handler compares against, so suppression works for both regular and builtin tool calls.

Test evidence

  • Added test_event_stream_no_tool_call_args_after_end in tests/test_ag_ui.py, reproducing the exact scenario (builtin tool-call start → delta → end → stray delta after end). It asserts both a full snapshot ending START → ARGS → ARGS → END → RUN_FINISHED and an explicit check that no TOOL_CALL_ARGS for that ID appears after TOOL_CALL_END.
  • The test failed before the fix (stray TOOL_CALL_ARGS {'delta': '}'} after TOOL_CALL_END) and passes after.
  • Gate: ./bt-test tests/test_ag_ui.py tests/test_ui.py → 50 passed, no regressions.

Requirements disposition

  • No TOOL_CALL_ARGS after TOOL_CALL_END for a tool_call_id; ordering must be START → ARGS → END* — Implemented, covered by the new regression test (builtin path) and verified by the full AG-UI/UI suite.
  • Reported scenario is builtin (MCPServerTool / OpenAI Responses streaming) — Implemented and tested on the builtin path with provider_name='mcp' and the prefixed pyd_ai_builtin|mcp|mcp_1 ID.

Reviewer note (non-blocking): the fix also guards the regular (non-builtin) tool-call path, which is correct by inspection but not directly covered by a dedicated test. I kept the change minimal and did not add a second test, since the reported and reproduced scenario is the builtin path; the regular path shares the identical suppression logic.