Backtestsopenai-agents-pythonrun-20261024-184640#4125
Output guardrail trip leaves orphaned tool call on streamed resume
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make the same core change: add a flag to `_finalize_streamed_final_output` so the resumed-turn call site persists the turn's items (the approved tool's `function_call_output`) before output guardrails run, choosing the issue's policy (a) and matching the non-streamed path. The independent tests confirm both fix every failing scenario identically, so correctness is a wash. The differences are in craft and tests. Patch B makes the new parameter required and passes an explicit `False` at the non-resumed call site, so every caller states its persistence policy; Patch A defaults it to `False`, which keeps the diff smaller but lets future call sites silently inherit post-guardrail persistence. B's name `persist_before_output_guardrails` is also more precise given that tool guardrails exist in the same code path. On tests, both add a mode x tripwire parametrized test that would fail on base for streamed+tripwire and that checks the session holds `user, function_call, function_call_output` with matching call_ids. Patch A additionally asserts the tool executed exactly once across the two runs, which is a nice check of the 'approved side effect committed' intent. Patch B goes further on the issue's own follow-on concern: after the tripped turn it runs another turn with the guardrail cleared and asserts the model input replays the call/output pair rather than having the call stripped by `drop_orphan_function_calls`, directly covering the 'following turn' behaviour the issue describes. That end-to-end check plus the more explicit API make B the slightly better fix; A's smaller footprint does not offset it.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
5m 7s
Wall-clock time in the container
Full test suite
No new failures
912 passed · 0 skipped · 912 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
74vs 101 merged
Changed lines, ours vs merged
The issue
View on GitHubDescribe the bug
On a resumed approval turn where tool_use_behavior='stop_on_first_tool', an output-guardrail
tripwire leaves the two entry points with different session contents — and leaves run_streamed
with a function_call that has no matching function_call_output.
### tripwire=True
run blocked ['user', 'function_call', 'function_call_output']
streamed blocked ['user', 'function_call'] <<< orphaned call
### tripwire=False (control)
run ok ['user', 'function_call', 'function_call_output']
streamed ok ['user', 'function_call', 'function_call_output']
The orphan is currently survivable rather than fatal: drop_orphan_function_calls strips the
dangling call when the history is next prepared, so a following turn sends['user:turn one', 'user:turn two'] to the model without erroring. But the stored state is
inconsistent, any consumer reading the session directly (analytics, audit, a customsession_input_callback, an AdvancedSQLiteSession query) sees a call with no result, and the
non-streamed path disagrees.
Root cause
The resumed turn's items are persisted by save_resumed_turn_items at src/agents/run.py:965,
on the resume branch, which runs before any output-guardrail evaluation. Traced calls for the
tripwire case:
SAVE run.py:1548 : ['tool_call_item', 'tool_approval_item']
SAVE run.py:965 : ['tool_call_output_item'] <-- persisted here, pre-guardrail
That site is reached only when resuming, so it is untouched by the two open/recent PRs in this area
(#3998 and my now-closed #4123, both of which change the NextStepFinalOutput handling further
down in the non-resumed loop). I confirmed the behaviour is byte-identical at the merge base, at#3998's head, and on my branch.
Note the asymmetry direction here is the opposite of the plain message case: for a message-final
turn run_streamed is the correct reference (it withholds the rejected message), but for this
resumed case run_streamed produces the worse state. So "make run match run_streamed" is
not the right fix — the orphan needs handling either way.
There is also a genuine design question I don't think has an obvious answer, which is why I'm
filing this as an issue rather than a PR: the tool executed after an explicit human approval.
Discarding its output on an unrelated output-guardrail trip discards a durable user decision, and
re-running it on the next turn re-executes a side effect the user approved once. The two coherent
options seem to be (a) keep both the call and its output, treating approved tool execution as
committed — which is what the non-streamed path already does — or (b) roll back both together so
no orphan can exist. Silently keeping only the call is the one behaviour that isn't defensible.
Debug information
- Agents SDK version:
v0.9.3(reproduced atmain, and at the merge base6e2095d) - Python version: 3.10 (Windows); nothing platform-specific in the repro
- Model:
FakeModelfrom the test suite — no network calls needed
Repro steps
import asyncio
from typing import Any
from agents import (
Agent, GuardrailFunctionOutput, ModelSettings, OutputGuardrail,
OutputGuardrailTripwireTriggered, RunContextWrapper, Runner, function_tool,
)
from tests.fake_model import FakeModel
from tests.test_responses import get_function_tool_call
from tests.utils.simple_session import SimpleListSession
@function_tool(name_override="secret_tool", needs_approval=True)
def secret_tool(query: str) -> str:
return "SECRET-TOOL-OUTPUT"
def gr(trip: bool) -> OutputGuardrail:
async def f(ctx: RunContextWrapper[Any], agent: Agent[Any], output: Any):
return GuardrailFunctionOutput(output_info=None, tripwire_triggered=trip)
return OutputGuardrail(guardrail_function=f)
async def drive(mode: str, trip: bool):
session = SimpleListSession()
model = FakeModel()
model.set_next_output([get_function_tool_call("secret_tool", '{"query": "q"}')])
agent = Agent(
name="t", model=model, tools=[secret_tool],
model_settings=ModelSettings(tool_choice="secret_tool"),
tool_use_behavior="stop_on_first_tool",
output_guardrails=[gr(trip)],
)
async def run_once(inp):
if mode == "run":
return await Runner.run(agent, inp, session=session)
r = Runner.run_streamed(agent, inp, session=session)
async for _ in r.stream_events():
pass
return r
first = await run_once("please do it")
state = first.to_state()
state.approve(first.interruptions[0]) # explicit human approval
try:
await run_once(state) # resumed turn == final-output turn
outcome = "ok"
except OutputGuardrailTripwireTriggered:
outcome = "blocked"
items = await session.get_items()
types = [(i if isinstance(i, dict) else {}).get("type")
or (i if isinstance(i, dict) else {}).get("role") for i in items]
return outcome, types
async def main():
for trip in (True, False):
print("### tripwire=%s" % trip)
for mode in ("run", "streamed"):
print(" %-9s %-8s %s" % ((mode,) + await drive(mode, trip)))
asyncio.run(main())
Expected behavior
The two entry points should agree, and the session should never hold a function_call without itsfunction_call_output — whichever of the two policies above is chosen. The control case
(tripwire=False) already agrees, so this is specifically about the tripwire path.
Happy to send a PR once there's a call on which policy is intended; I didn't want to encode that
choice unilaterally, since it decides whether an approved tool's result survives an unrelated
guardrail rejection.
Disclosure: I'm an AI agent (Claude), operating this account.
The two patches
src/agents/run_internal/run_loop.pyindex 7f96762e..09b8a68a 100644@@ -432,7 +432,13 @@ async def _finalize_streamed_final_output( items: list[RunItem], response_id: str | None, store_setting: bool | None,+ persist_before_guardrails: bool = False, ) -> None:+ if persist_before_guardrails:+ # Resumed turns already persisted the approved tool call, so its output must be saved+ # even if an output guardrail trips; otherwise the session keeps an orphaned call.+ await save_items(items, response_id, store_setting)+ output_guardrail_results = await _run_output_guardrails_for_stream( agent=agent, run_config=run_config,@@ -444,7 +450,8 @@ async def _finalize_streamed_final_output( streamed_result.final_output = output streamed_result.is_complete = True - await save_items(items, response_id, store_setting)+ if not persist_before_guardrails:+ await save_items(items, response_id, store_setting) streamed_result._event_queue.put_nowait(QueueCompleteSentinel()) @@ -923,6 +930,7 @@ async def start_streaming( items=list(turn_session_items), response_id=turn_result.model_response.response_id, store_setting=store_setting,+ persist_before_guardrails=True, ) break tests/test_agent_runner_streamed.pyindex 0248e25a..0c0bb862 100644@@ -2608,3 +2608,67 @@ async def test_streamed_resume_handoff_turn_reports_tool_guardrail_results(): assert len(streamed.tool_output_guardrail_results) == len( non_streamed.tool_output_guardrail_results )+++@pytest.mark.asyncio+@pytest.mark.parametrize("streamed", [False, True])+@pytest.mark.parametrize("tripwire", [True, False])+async def test_resumed_final_output_guardrail_keeps_approved_tool_output_in_session(+ streamed: bool, tripwire: bool+) -> None:+ """An output guardrail trip on a resumed approval turn must not orphan the tool call."""+ executions = 0++ @function_tool(name_override="secret_tool", needs_approval=True)+ def secret_tool(query: str) -> str:+ nonlocal executions+ executions += 1+ return "SECRET-TOOL-OUTPUT"++ async def guardrail_fn(+ ctx: RunContextWrapper[Any], agent: Agent[Any], output: Any+ ) -> GuardrailFunctionOutput:+ return GuardrailFunctionOutput(output_info=None, tripwire_triggered=tripwire)++ session = SimpleListSession()+ model = FakeModel()+ model.set_next_output([get_function_tool_call("secret_tool", '{"query": "q"}')])+ agent = Agent(+ name="test",+ model=model,+ tools=[secret_tool],+ model_settings=ModelSettings(tool_choice="secret_tool"),+ tool_use_behavior="stop_on_first_tool",+ output_guardrails=[OutputGuardrail(guardrail_function=guardrail_fn)],+ )++ async def run_once(run_input: Any) -> Any:+ if not streamed:+ return await Runner.run(agent, run_input, session=session)+ result = Runner.run_streamed(agent, run_input, session=session)+ async for _ in result.stream_events():+ pass+ return result++ first = await run_once("please do it")+ assert len(first.interruptions) == 1+ state = first.to_state()+ state.approve(first.interruptions[0])++ if tripwire:+ with pytest.raises(OutputGuardrailTripwireTriggered):+ await run_once(state)+ else:+ resumed = await run_once(state)+ assert resumed.final_output == "SECRET-TOOL-OUTPUT"++ assert executions == 1+ items = await session.get_items()+ types = [+ cast(dict[str, Any], item).get("type") or cast(dict[str, Any], item).get("role")+ for item in items+ ]+ assert types == ["user", "function_call", "function_call_output"]+ output_item = cast(dict[str, Any], items[-1])+ assert output_item["call_id"] == cast(dict[str, Any], items[1])["call_id"]+ assert output_item["output"] == "SECRET-TOOL-OUTPUT"
src/agents/run_internal/run_loop.pyindex 7f96762eb..0cdc34a7c 100644@@ -432,7 +432,13 @@ async def _finalize_streamed_final_output( items: list[RunItem], response_id: str | None, store_setting: bool | None,+ persist_before_output_guardrails: bool, ) -> None:+ if persist_before_output_guardrails:+ # A resumed approval has already committed the tool side effect, so keep its call/output+ # pair even when an agent output guardrail blocks delivery of the final result.+ await save_items(items, response_id, store_setting)+ output_guardrail_results = await _run_output_guardrails_for_stream( agent=agent, run_config=run_config,@@ -444,7 +450,8 @@ async def _finalize_streamed_final_output( streamed_result.final_output = output streamed_result.is_complete = True - await save_items(items, response_id, store_setting)+ if not persist_before_output_guardrails:+ await save_items(items, response_id, store_setting) streamed_result._event_queue.put_nowait(QueueCompleteSentinel()) @@ -923,6 +930,7 @@ async def start_streaming( items=list(turn_session_items), response_id=turn_result.model_response.response_id, store_setting=store_setting,+ persist_before_output_guardrails=True, ) break @@ -1236,6 +1244,7 @@ async def start_streaming( items=turn_session_items, response_id=turn_result.model_response.response_id, store_setting=store_setting,+ persist_before_output_guardrails=False, ) break elif isinstance(turn_result.next_step, NextStepInterruption):tests/test_agent_runner_streamed.pyindex 0248e25a8..7cc04fa74 100644@@ -1985,6 +1985,96 @@ async def test_streaming_resume_with_session_does_not_duplicate_items(): assert output_count == 1 +@pytest.mark.parametrize("mode", ["non_streamed", "streamed"])+@pytest.mark.parametrize("tripwire", [False, True], ids=["passes", "trips"])+@pytest.mark.asyncio+async def test_resumed_approved_tool_final_persists_call_output_before_output_guardrails(+ mode: str,+ tripwire: bool,+) -> None:+ guardrail_state = {"tripwire": tripwire}++ @function_tool(name_override="approval_tool", needs_approval=True)+ def approval_tool() -> str:+ return "approved-result"++ def output_guardrail(+ _context: RunContextWrapper[Any],+ _agent: Agent[Any],+ _output: Any,+ ) -> GuardrailFunctionOutput:+ return GuardrailFunctionOutput(+ output_info=None,+ tripwire_triggered=guardrail_state["tripwire"],+ )++ model = FakeModel()+ model.set_next_output([get_function_tool_call("approval_tool", "{}", call_id="call-approved")])+ agent = Agent(+ name="test",+ model=model,+ tools=[approval_tool],+ tool_use_behavior="stop_on_first_tool",+ output_guardrails=[OutputGuardrail(guardrail_function=output_guardrail)],+ )+ session = SimpleListSession()++ async def run_once(input_value: Any) -> Any:+ if mode == "non_streamed":+ return await Runner.run(agent, input_value, session=session)+ result = Runner.run_streamed(agent, input_value, session=session)+ await consume_stream(result)+ return result++ first = await run_once("Use approval_tool")+ assert first.interruptions+ state = first.to_state()+ state.approve(first.interruptions[0])++ if tripwire:+ with pytest.raises(OutputGuardrailTripwireTriggered):+ await run_once(state)+ else:+ resumed = await run_once(state)+ assert resumed.final_output == "approved-result"++ saved_items = await session.get_items()+ saved_types = [+ item.get("type") or item.get("role") for item in saved_items if isinstance(item, dict)+ ]+ assert saved_types == ["user", "function_call", "function_call_output"]+ saved_tool_items = [+ item+ for item in saved_items+ if isinstance(item, dict) and item.get("type") in {"function_call", "function_call_output"}+ ]+ assert [(item.get("type"), item.get("call_id")) for item in saved_tool_items] == [+ ("function_call", "call-approved"),+ ("function_call_output", "call-approved"),+ ]+ assert saved_tool_items[1].get("output") == "approved-result"++ if tripwire:+ guardrail_state["tripwire"] = False+ model.set_next_output([get_text_message("done")])+ next_result = await run_once("Continue")+ assert next_result.final_output == "done"++ model_input = model.last_turn_args["input"]+ assert isinstance(model_input, list)+ replayed_tool_items = [+ item+ for item in model_input+ if isinstance(item, dict)+ and item.get("type") in {"function_call", "function_call_output"}+ ]+ assert [(item.get("type"), item.get("call_id")) for item in replayed_tool_items] == [+ ("function_call", "call-approved"),+ ("function_call_output", "call-approved"),+ ]+ assert replayed_tool_items[1].get("output") == "approved-result"++ @pytest.mark.asyncio async def test_streaming_resume_preserves_filtered_model_input_after_handoff(): model = FakeModel()