Backtestsopenai-agents-pythonrun-20261024-184640#5339

Issue #5339 Merged fix PR #5366 Run 24 Oct 2026 · run-20261024-184640 openai/openai-agents-python

RunState.add_input input is silently dropped under stop_at_tool_names

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches make the same core change: in RunState.add_input they widen the stop_at_tool_names check from the interrupted tools only to every function call in the last processed response (adding both name and qualified_name), which is exactly the root cause the issue identifies and matches the set resolve_interrupted_turn evaluates. The independent tests confirm both fix the reported scenario identically (base FAIL on the rejection and silent-drop tests, both patches PASS, control case passes everywhere). The meaningful difference is B's additional condition that skips the completed-sibling names when the interrupted response also contains a pending handoff. In the runner, handoffs are resolved before the tool_use_behavior final-output check, so a response with a handoff continues with the receiving agent and does call the model again; in that case A would wrongly refuse valid input (a false rejection against the contract the issue states: raise only when the model will not be called again), while B accepts it and its parametrized test asserts the input reaches the delegate's model call. B's test also covers the qualified-name case via tool_namespace, the unrelated-stop-name control, streamed and serialized variants, in one table. A's tests are also solid (4 parametrizations, a nested agent-as-tool stop case, and a control case) and A's code is simpler and conservative, but it over-rejects in the handoff case. Minor craft nit against B: the handoff carve-out applies only to the completed siblings, not to the pre-existing interrupted-tool-name set, which is slightly inconsistent, and both patches' test helpers (from_json/from_string, namespace kwarg, model.extend) could not be verified here. On balance B models the resume semantics more precisely without adding unrelated churn, so B is the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

9m 28s

Wall-clock time in the container

Full test suite

No new failures

2,429 passed · 2 skipped · 2,432 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

146vs 101 merged

Changed lines, ours vs merged

Please read this first

  • Have you read the docs? Yes, the human-in-the-loop and agents pages.
  • Have you searched for related issues? Yes. #5239 (fixed by #5240) is the resume behaviour this builds on; I found nothing about add_input in this state.

Describe the bug

When one model response contains a tool listed in stop_at_tool_names that needs no approval and a sibling call that needs approval, the stop tool runs and the turn is interrupted. RunState.add_input(...) accepts new input on that state. After the approval, the resumed run ends on the stop tool's output without calling the model again (the behaviour #5240 introduced), so the staged input is never sent to the model and does not appear in to_input_list(). No error or warning is raised. With tool_use_behavior="stop_on_first_tool" the same add_input call is refused with UserError: Cannot add input to an interrupted RunState whose tool result may end the run.

On 0.22.3 the input did reach the model, because the resume ignored the completed stop tool (#5239). Since 0.23.0 the input is lost.

Debug information

  • Agents SDK version: 0.23.1 (also checked the code on main at 125efa02, unchanged)
  • Related library versions (optional, e.g. any-llm, litellm, or pydantic): none needed
  • Python version: 3.12.3
  • Operating system: Linux (Pop!_OS 24.04)
  • Model and model provider: none, agents.testing.ScriptedModel, no network
  • Does the issue reproduce with the latest Agents SDK release? Yes, 0.23.1
  • Does the issue occur consistently or intermittently? Consistently

Output of the script below on 0.23.1:

python: 3.12.3
openai-agents: 0.23.1
  stop_on_first_tool                 add_input=rejected (UserError) final_output='finish:x' model_calls=1 input_sent_to_model=False input_in_history=False
  stop_at_tool_names=['finish']      add_input=accepted             final_output='finish:x' model_calls=1 input_sent_to_model=False input_in_history=False
expected: with stop_at_tool_names=['finish'] add_input is rejected the same way as under stop_on_first_tool, because the resumed run ends on the completed 'finish' result and never calls the model again

Repro steps

"""RunState.add_input accepts input that is then silently dropped when a completed
stop_at_tool_names sibling ends the resumed run."""

import asyncio
import importlib.metadata
import os
import platform

os.environ.setdefault("OPENAI_API_KEY", "sk-test")
os.environ.setdefault("OPENAI_AGENTS_DISABLE_TRACING", "1")

from agents import Agent, Runner, function_tool
from agents.testing import ScriptedModel, assistant_message, function_call

print(f"python: {platform.python_version()}")
print(f"openai-agents: {importlib.metadata.version('openai-agents')}")


@function_tool
def finish(x: str) -> str:
    """Stop tool."""
    return f"finish:{x}"


@function_tool(needs_approval=True)
def protected(y: str) -> str:
    """Approval-gated tool."""
    return f"protected:{y}"


async def run_case(label: str, behavior) -> None:
    model = ScriptedModel(
        steps=[
            [
                function_call("finish", {"x": "x"}, call_id="f1"),
                function_call("protected", {"y": "y"}, call_id="p1"),
            ],
            [assistant_message("model-ran-again")],
        ]
    )
    agent = Agent(name="a", model=model, tools=[finish, protected], tool_use_behavior=behavior)
    first = await Runner.run(agent, "go")
    state = first.to_state()
    try:
        state.add_input("please also do Z")
        add_input = "accepted"
    except Exception as exc:
        add_input = f"rejected ({type(exc).__name__})"
    for item in first.interruptions:
        state.approve(item)
    resumed = await Runner.run(agent, state)
    seen_by_model = any("please also do Z" in str(call.input) for call in model.calls)
    in_history = "please also do Z" in str(resumed.to_input_list())
    print(
        f"  {label:34} add_input={add_input:20} final_output={resumed.final_output!r} "
        f"model_calls={len(model.calls)} "
        f"input_sent_to_model={seen_by_model} input_in_history={in_history}"
    )


async def main() -> None:
    await run_case("stop_on_first_tool", "stop_on_first_tool")
    await run_case(
        "stop_at_tool_names=['finish']", {"stop_at_tool_names": ["finish"]}
    )
    print(
        "expected: with stop_at_tool_names=['finish'] add_input is rejected the same way as "
        "under stop_on_first_tool, because the resumed run ends on the completed 'finish' "
        "result and never calls the model again"
    )


asyncio.run(main())

Expected behavior

With stop_at_tool_names=["finish"], add_input should raise the same UserError it raises under stop_on_first_tool, because the resumed run is going to end on the completed finish result and never call the model again. Input that the state accepted should not disappear without an error.

Cause. RunState.add_input (src/agents/run_state.py 1072-1087 at 125efa02) decides whether the resume can end the run by intersecting stop_at_tool_names with the names of the interrupted tools only. Since #5240, resolve_interrupted_turn (src/agents/run_internal/turn_resolution.py 2869-2904) also feeds the outputs of siblings that already completed in the same response into the tool_use_behavior check, so a stop tool that already ran ends the run, and add_input does not account for that.

Possible fix. In add_input, compare stop_at_tool_names with every function call in the interrupted response (the completed siblings as well as the pending ones), the same set that resolve_interrupted_turn now evaluates, and add a test with a stop tool plus an approval-gated sibling that asserts add_input raises.

The two patches

Our patch+142 −4
src/agents/run_state.pyindex 490a8c1b..21f3f989 100644@@ -1076,16 +1076,20 @@ class RunState(Generic[TContext, TAgent]):             if self._current_agent is None:                 raise UserError("Cannot add input to a RunState without a current agent")             tool_use_behavior = self._current_agent.tool_use_behavior-            interrupted_tool_names = {+            turn_tool_names = {                 item.tool_name                 for item in self._current_step.interruptions                 if item.tool_name is not None             }+            # Resuming also feeds completed sibling function outputs from the interrupted response+            # into the tool_use_behavior check, so a stop tool that already ran ends the run too.+            if self._last_processed_response is not None:+                for function_run in self._last_processed_response.functions:+                    turn_tool_names.add(function_run.function_tool.name)+                    turn_tool_names.add(function_run.function_tool.qualified_name)             stops_before_next_model = tool_use_behavior == "stop_on_first_tool" or (                 isinstance(tool_use_behavior, dict)-                and bool(-                    interrupted_tool_names & set(tool_use_behavior.get("stop_at_tool_names", []))-                )+                and bool(turn_tool_names & set(tool_use_behavior.get("stop_at_tool_names", [])))             )             if stops_before_next_model or callable(tool_use_behavior):                 raise UserError(tests/test_run_state_pending_input.pyindex fdf1d2cd..d3f3c6b1 100644@@ -400,6 +400,140 @@ async def test_interruption_without_guaranteed_next_model_rejects_input(     assert state.to_json() == before  +@pytest.mark.asyncio+@pytest.mark.parametrize("streamed", [False, True])+@pytest.mark.parametrize("round_trip", [False, True])+async def test_completed_stop_tool_sibling_rejects_input(streamed: bool, round_trip: bool) -> None:+    @function_tool+    def finish(value: str) -> str:+        return f"finish:{value}"++    @function_tool(needs_approval=True)+    def protected_tool(value: str) -> str:+        return f"protected:{value}"++    model = ScriptedModel(+        steps=[+            [+                get_function_tool_call("finish", '{"value":"x"}', call_id="call-finish"),+                get_function_tool_call("protected_tool", '{"value":"y"}', call_id="call-protected"),+            ],+            [get_text_message("Model ran again")],+        ]+    )+    agent = Agent(+        name="assistant",+        model=model,+        tools=[finish, protected_tool],+        tool_use_behavior={"stop_at_tool_names": ["finish"]},+    )+    if streamed:+        interrupted_stream = Runner.run_streamed(agent, "Initial request")+        async for _event in interrupted_stream.stream_events():+            pass+        interruptions = interrupted_stream.interruptions+        state = interrupted_stream.to_state()+    else:+        interrupted = await Runner.run(agent, "Initial request")+        interruptions = interrupted.interruptions+        state = interrupted.to_state()+    assert len(interruptions) == 1+    if round_trip:+        state = await RunState.from_json(agent, state.to_json())++    before = state.to_json()+    with pytest.raises(UserError, match="tool result may end the run"):+        state.add_input("Late input")+    assert state.to_json() == before+    assert state.pending_input == []++    for item in state.get_interruptions():+        state.approve(item)+    resumed = await Runner.run(agent, state)+    assert resumed.final_output == "finish:x"+    assert len(model.calls) == 1+++@pytest.mark.asyncio+@pytest.mark.parametrize("round_trip", [False, True])+async def test_stop_agent_tool_with_nested_approval_rejects_input(round_trip: bool) -> None:+    @function_tool(needs_approval=True)+    def inner_tool(value: str) -> str:+        return f"inner:{value}"++    inner_model = ScriptedModel(+        steps=[+            [get_function_tool_call("inner_tool", '{"value":"x"}', call_id="call-inner")],+            [get_text_message("Sub done")],+        ]+    )+    sub_agent = Agent(name="sub", model=inner_model, tools=[inner_tool])+    outer_model = ScriptedModel(+        steps=[+            [get_function_tool_call("sub_tool", '{"input":"go"}', call_id="call-sub")],+            [get_text_message("Outer ran again")],+        ]+    )+    agent = Agent(+        name="assistant",+        model=outer_model,+        tools=[sub_agent.as_tool(tool_name="sub_tool", tool_description="Sub agent")],+        tool_use_behavior={"stop_at_tool_names": ["sub_tool"]},+    )+    interrupted = await Runner.run(agent, "Initial request")+    assert [item.tool_name for item in interrupted.interruptions] == ["inner_tool"]+    state = interrupted.to_state()+    if round_trip:+        state = await RunState.from_json(agent, state.to_json())++    before = state.to_json()+    with pytest.raises(UserError, match="tool result may end the run"):+        state.add_input("Late input")+    assert state.to_json() == before+    assert state.pending_input == []+++@pytest.mark.asyncio+async def test_stop_tool_not_in_interrupted_response_still_accepts_input() -> None:+    @function_tool+    def finish(value: str) -> str:+        return f"finish:{value}"++    @function_tool+    def helper(value: str) -> str:+        return f"helper:{value}"++    @function_tool(needs_approval=True)+    def protected_tool(value: str) -> str:+        return f"protected:{value}"++    model = ScriptedModel(+        steps=[+            [+                get_function_tool_call("helper", '{"value":"x"}', call_id="call-helper"),+                get_function_tool_call("protected_tool", '{"value":"y"}', call_id="call-protected"),+            ],+            [get_text_message("Done")],+        ]+    )+    agent = Agent(+        name="assistant",+        model=model,+        tools=[finish, helper, protected_tool],+        tool_use_behavior={"stop_at_tool_names": ["finish"]},+    )+    interrupted = await Runner.run(agent, "Initial request")+    state = interrupted.to_state()+    state.add_input("Late input")+    for item in interrupted.interruptions:+        state.approve(item)++    resumed = await Runner.run(agent, state)+    assert resumed.final_output == "Done"+    model_input = cast(list[TResponseInputItem], model.calls[-1].input)+    assert [_message_text(item) for item in model_input].count("Late input") == 1++ @pytest.mark.asyncio async def test_pending_input_guardrail_trip_keeps_input_recoverable() -> None:     model, agent, state, _calls = await _make_after_turn_state() 
Merged fix · PR #5366+95 −6
src/agents/run_state.pyindex 490a8c1b8..a6cfead2d 100644@@ -1076,16 +1076,23 @@ class RunState(Generic[TContext, TAgent]):             if self._current_agent is None:                 raise UserError("Cannot add input to a RunState without a current agent")             tool_use_behavior = self._current_agent.tool_use_behavior-            interrupted_tool_names = {+            response_tool_names = {                 item.tool_name                 for item in self._current_step.interruptions                 if item.tool_name is not None             }+            # Completed siblings also participate in the resumed turn's final-output decision.+            # A pending handoff takes precedence and continues with the receiving agent instead.+            if (+                self._last_processed_response is not None+                and not self._last_processed_response.handoffs+            ):+                for function_run in self._last_processed_response.functions:+                    response_tool_names.add(function_run.function_tool.name)+                    response_tool_names.add(function_run.function_tool.qualified_name)             stops_before_next_model = tool_use_behavior == "stop_on_first_tool" or (                 isinstance(tool_use_behavior, dict)-                and bool(-                    interrupted_tool_names & set(tool_use_behavior.get("stop_at_tool_names", []))-                )+                and bool(response_tool_names & set(tool_use_behavior.get("stop_at_tool_names", [])))             )             if stops_before_next_model or callable(tool_use_behavior):                 raise UserError(tests/test_run_state_pending_input.pyindex fdf1d2cd7..9519909d2 100644@@ -9,7 +9,7 @@ from openai.types.responses.response_computer_tool_call import (     ResponseComputerToolCall, ) -from agents import Agent, ComputerTool, InputItem, RunConfig, Runner, function_tool+from agents import Agent, ComputerTool, InputItem, RunConfig, Runner, function_tool, tool_namespace from agents.exceptions import InputGuardrailTripwireTriggered, ModelBehaviorError, UserError from agents.guardrail import GuardrailFunctionOutput, InputGuardrail from agents.items import ModelResponse, TResponseInputItem@@ -25,7 +25,7 @@ from agents.usage import Usage  from .model_test_helpers import get_exact_output_stream_step from .test_computer_tool_lifecycle import FakeComputer-from .test_responses import get_function_tool_call, get_text_message+from .test_responses import get_function_tool_call, get_handoff_tool_call, get_text_message from .utils.simple_session import SimpleListSession  @@ -400,6 +400,88 @@ async def test_interruption_without_guaranteed_next_model_rejects_input(     assert state.to_json() == before  +@pytest.mark.asyncio+@pytest.mark.parametrize("streamed", [False, True])+@pytest.mark.parametrize("serialized", [False, True])+@pytest.mark.parametrize(+    "stop_name,has_handoff",+    [("finish", False), ("workflow.finish", False), ("unrelated", False), ("finish", True)],+)+async def test_completed_stop_sibling_controls_pending_input(+    streamed: bool, serialized: bool, stop_name: str, has_handoff: bool+) -> None:+    calls: list[str] = []++    @function_tool+    def finish() -> str:+        calls.append("finish")+        return "Finished"++    @function_tool(needs_approval=True)+    def protected_tool() -> str:+        calls.append("protected")+        return "Approved"++    model = ScriptedModel()+    delegate = Agent(name="delegate", model=model)+    model.extend(+        [+            [+                get_function_tool_call("finish", "{}", "finish-call", namespace="workflow"),+                get_function_tool_call("protected_tool", "{}", "protected-call"),+                *([get_handoff_tool_call(delegate)] if has_handoff else []),+            ],+            [get_text_message("Continued")],+        ]+    )+    agent = Agent(+        name="assistant",+        model=model,+        tools=[+            *tool_namespace(name="workflow", description="Workflow tools", tools=[finish]),+            protected_tool,+        ],+        tool_use_behavior={"stop_at_tool_names": [stop_name]},+        handoffs=[delegate] if has_handoff else [],+    )++    async def run(input: str | RunState[Any]) -> Any:+        if streamed:+            result = Runner.run_streamed(agent, input)+            async for _event in result.stream_events():+                pass+            return result+        return await Runner.run(agent, input)++    interrupted = await run("Initial request")+    state = interrupted.to_state()+    if serialized:+        state = await RunState.from_string(agent, state.to_string())+    assert calls == ["finish"]+    before = state.to_json()+    continues = stop_name == "unrelated" or has_handoff+    if continues:+        state.add_input("Late input")+    else:+        with pytest.raises(UserError, match="tool result may end the run"):+            state.add_input("Late input")+        assert state.to_json() == before++    state.approve(state.get_interruptions()[0])+    resumed = await run(state)+    assert calls == ["finish", "protected"]+    assert state.pending_input == []+    if continues:+        assert resumed.final_output == "Continued"+        assert resumed.last_agent is (delegate if has_handoff else agent)+        assert len(model.calls) == 2+        model_input = cast(list[TResponseInputItem], model.calls[-1].input)+        assert [_message_text(item) for item in model_input].count("Late input") == 1+    else:+        assert resumed.final_output == "Finished"+        assert len(model.calls) == 1++ @pytest.mark.asyncio async def test_pending_input_guardrail_trip_keeps_input_recoverable() -> None:     model, agent, state, _calls = await _make_after_turn_state()