Backtestsopenai-agents-pythonrun-20261024-184640#4690
Resumed terminal tool output cannot recover after a Session append failure
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches stop the retry from re-entering the model, and the independent matrix passes for both, but they implement different contracts from the issue and A matches the issue's ask far more fully. Patch A implements contract 1 (recoverable terminal continuation): once output guardrails accept the resumed NextStepFinalOutput it retains the output, the guardrail results, the tool-use snapshot, and routes the final append through the existing pending-session-write ownership so a fail-before or lost-ack batch is reconciled on resume and the original `receipt-7` is returned with no model, tool, hook, or guardrail repetition. It serializes the step as `next_step_final_output` with an explicit restorability flag, validates that payload on load (schema gate, exact key set, exact-JSON check), and for non-JSON-exact outputs still reconciles the Session and then raises a clear UserError rather than guessing. Its tests cover the full 16-row sync/stream x live/JSON x before/after matrix, the concurrent-writer fail-closed case, structured outputs, and malformed checkpoints. Patch B implements a form of contract 2, but only a boolean `terminal_unrecoverable` marker: it does not preserve the pending batch, the accepted output, or the guardrail results, so in the fail-before rows the Session permanently lacks the tool call/output pair and the caller (who in the sync path never received `receipt-7`) is told only to start a new run. The issue explicitly asked that even the reject option 'preserve an actionable, serializable recovery payload', which B does not do, and B never exercises the changed-history fail-closed path because it rejects before any Session work. B is simpler and lower-risk, and its extra max-turns-handler coverage is careful, but that is adjacent scope rather than the reported terminal-finalization failure. A carries more surface area (new step type, new fields, reconciliation reuse), yet it addresses the root cause described in the issue and is backed by substantive, matrix-shaped regression tests, so it is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
27m 59s
Wall-clock time in the container
Full test suite
No new failures
3,059 passed · 1 skipped · 3,060 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
504vs 350 merged
Changed lines, ours vs merged
The issue
View on GitHubPlease read this first
- I read the Session,
RunState, approval, lifecycle, and output-guardrail docs. - I searched open and closed issues/PRs for
Session append,NextStepFinalOutput, and terminal tool-output persistence. The related work listed under Scope boundaries does not cover this terminal-finalization failure mode.
Describe the bug
After an approval-gated local tool completes, tool_use_behavior="stop_on_first_tool"
can make that tool output the terminal agent output. If the final client-managedSession.add_items() append then fails, the exception is correctly propagated,
but the resulting RunState cannot recover or explicitly reject that accepted
terminal result.
At the failure boundary:
- the live runtime step is
NextStepFinalOutput; - JSON serialization records
current_step = null; - there is no
pending_session_write; - the passing output guardrail has already evaluated the original output, but its result is not retained in the failed state.
Retrying the same live or JSON-restored state therefore enters the model again,
runs the output guardrail again, and returns a new final output. The completed
tool effect is not repeated in this reproducer, but terminal output identity,
guardrail evidence, and durable Session history can diverge.
Debug information
- Agents SDK: latest release
v0.22.0, and exact currentmain@a40ae9803e6b7a79faa246293f56adb100d5868b - Exact
srctree:56eeb964e8924a6d0034b3e2289699c719f5450b - Python: 3.13.15
- Operating system: macOS 26.5, arm64
- Model/provider: deterministic public
ScriptedModel; no provider, API key, network request, or timing race - Latest release: yes
- Reproducibility: consistent; focused repro and all 16 frozen matrix rows
The injected exception is:
RuntimeError: injected Session.add_items failure before commit
Repro steps
This focused fail-before-commit case is self-contained and also reproduces
against the installed openai-agents==0.22.0 package:
<details>
<summary>Reproducer</summary>
import asyncio
import copy
import json
from typing import Any
from openai.types.responses import (
ResponseFunctionToolCall,
ResponseOutputMessage,
ResponseOutputText,
)
from agents import (
Agent,
GuardrailFunctionOutput,
RunContextWrapper,
Runner,
function_tool,
output_guardrail,
)
from agents.run import RunConfig
from agents.testing import ScriptedModel
class FailingSession:
def __init__(self):
self.session_id = "terminal-write-repro"
self.items = []
self.fail_next = False
async def get_items(self, limit=None):
items = copy.deepcopy(self.items)
return items[-limit:] if limit is not None else items
async def add_items(self, items):
if self.fail_next:
self.fail_next = False
raise RuntimeError("injected Session.add_items failure before commit")
self.items.extend(copy.deepcopy(items))
async def pop_item(self):
return self.items.pop() if self.items else None
async def clear_session(self):
self.items.clear()
async def main():
effects = []
guardrail_outputs = []
@function_tool(needs_approval=True)
async def charge(amount: int) -> str:
effects.append(amount)
return f"receipt-{amount}"
@output_guardrail
async def record_output(
context: RunContextWrapper[Any], agent: Agent[Any], output: Any
) -> GuardrailFunctionOutput:
guardrail_outputs.append(str(output))
return GuardrailFunctionOutput(output_info=output, tripwire_triggered=False)
model = ScriptedModel(
steps=[
[
ResponseFunctionToolCall(
id="fc-1",
call_id="charge-1",
type="function_call",
name="charge",
arguments=json.dumps({"amount": 7}),
)
],
[
ResponseOutputMessage(
id="msg-2",
type="message",
role="assistant",
status="completed",
content=[
ResponseOutputText(
type="output_text",
text="retry-final",
annotations=[],
logprobs=[],
)
],
)
],
]
)
agent = Agent(
name="terminal-write-repro",
model=model,
tools=[charge],
tool_use_behavior="stop_on_first_tool",
output_guardrails=[record_output],
)
session = FailingSession()
config = RunConfig(tracing_disabled=True)
paused = await Runner.run(agent, "charge 7", session=session, run_config=config)
state = paused.to_state()
state.approve(paused.interruptions[0])
session.fail_next = True
try:
await Runner.run(agent, state, session=session, run_config=config)
except RuntimeError as error:
print(type(error).__name__, str(error))
serialized = state.to_json()
print(
{
"runtime_step": type(state._current_step).__name__,
"serialized_step": serialized.get("current_step"),
"pending_session_write": serialized.get("pending_session_write"),
"retained_output_guardrails": len(state._output_guardrail_results),
"model_calls": len(model.calls),
"effects": effects,
"guardrail_outputs": guardrail_outputs,
}
)
retried = await Runner.run(agent, state, session=session, run_config=config)
print(
{
"retry_output": retried.final_output,
"model_calls": len(model.calls),
"effects": effects,
"guardrail_outputs": guardrail_outputs,
}
)
asyncio.run(main())
</details>
Observed output:
RuntimeError injected Session.add_items failure before commit
{'runtime_step': 'NextStepFinalOutput', 'serialized_step': None,
'pending_session_write': None, 'retained_output_guardrails': 0,
'model_calls': 1, 'effects': [7], 'guardrail_outputs': ['receipt-7']}
{'retry_output': 'retry-final', 'model_calls': 2, 'effects': [7],
'guardrail_outputs': ['receipt-7', 'retry-final']}
Frozen behavior matrix
I also froze a 16-row Cartesian product before inspecting results:
| Dimension | Values |
|---|---|
| failing resume runner | sync, stream |
| retry runner | sync, stream |
| retry state | same live object, JSON round trip |
| append failure | fail before commit, commit then raise (lost acknowledgement) |
The tool is charge(amount=7), its exact output is receipt-7, and a second
scripted response retry-final exists only to expose a new model call.
| Observation | Rows |
|---|---|
| injected append failure propagated | 16 / 16 |
| retry completed | 16 / 16 |
| tool effect and tool start/end hooks remained exactly once | 16 / 16 |
first output guardrail evaluated receipt-7 before failure | 16 / 16 |
| pending terminal Session checkpoint retained after failure | 0 / 16 |
| first passing output-guardrail result retained after failure | 0 / 16 |
original receipt-7 recovered without another model call | 0 / 16 |
retry made a second model call and returned retry-final | 16 / 16 |
| durable Session contained one exact call/output pair after retry | 8 / 16 |
| public replay and retry model input contained one exact pair | 16 / 16 |
The eight durable-Session successes are only the append-then-raise rows: the
pair had already committed before the simulated lost acknowledgement. Those
rows still make a second model call because no terminal checkpoint recognizes
the committed batch as the already accepted result. In all eight
fail-before-commit rows, the Session lacks the tool call/output even after
retry, while replay/model input contain the pair and the Session contains the
later assistant message.
Relevant control flow
At the pinned commit: