Backtestsopenai-agents-pythonrun-20261024-184640#5181

Issue #5181 Merged fix PR #5187 Run 24 Oct 2026 · run-20261024-184640 openai/openai-agents-python

Responses streaming: the error event is not treated as terminal, so a teardown failure masks the provider's error

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches apply the identical root-cause fix: adding "error" to `_ResponseStreamWithRequestId._TERMINAL_EVENT_TYPES`, which is exactly what the issue asks for, and both pass the independent tests (base FAIL-EXC with the masking RuntimeError, A and B PASS). Correctness on the reported scenario is therefore equal.

The difference is in scope and test quality. Patch A adds a second production change: a `break` out of the `async for` in `stream_response` once `terminal_failure_error` is set. With the terminal-set fix in place this is redundant (the next `__anext__` would hit `_cleanup_after_exhaustion`, which now debug-logs the teardown failure), so it is unrequested hardening that changes the post-terminal control flow for `response.failed`/`response.incomplete` as well as `error`: the stream is now closed via the explicit close path instead of being exhausted. Nothing in the issue motivates this, and it obscures which change actually fixes the bug. Patch B keeps the production change to the single line the issue identifies.

On tests, B adds a parametrised `_ResponseStreamWithRequestId` unit test over `error` and `response.failed` (fails on base for `error` with the RuntimeError, passes after) and an end-to-end `stream_response` test asserting the `ModelBehaviorError` carries the provider's code and message. These are focused, obviously runnable, and mirror the issue's own repro table. A's tests are broader (a parametrised rewrite of an existing test, a unit test, and a real httpx MockTransport SSE test with failing `aclose`), and the SSE-level test is attractive because it exercises real transport teardown. But it depends on an undefined-in-diff helper (`_response_with_terminal_status`) and on exact SDK SSE parsing of `event: error`, which cannot be confirmed here, and its `None` case locks in behaviour (teardown error surfacing when no terminal event) that the issue does not ask for. Since patch-authored test outcomes are not in evidence, A's extra coverage is unverified while B's is straightforward.

Net: equal correctness, adequate and precise tests on both sides, but A carries an unrequested behavioural change in the hot streaming loop and heavier, less certain tests. B is the cleaner fix for the issue as stated.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

4m 55s

Wall-clock time in the container

Full test suite

No new failures

609 passed · 0 skipped · 609 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

76vs 145 merged

Changed lines, ours vs merged

Please read this first

  • Have you read the docs? Yes.
  • Have you searched for related issues? Yes. Ten open pull requests, none touching _TERMINAL_EVENT_TYPES; no open or closed issue naming the error stream event as non-terminal.

Describe the bug

_ResponseStreamWithRequestId._TERMINAL_EVENT_TYPES at src/agents/models/openai_responses.py:261 lists "response.error" but not "error". The SDK's error event is ResponseErrorEvent, whose type is Literal["error"]:

# openai/types/responses/response_error_event.py  (openai 3.0.0)
    type: Literal["error"]
    """The type of the event. Always `error`."""

So an error-terminated stream never sets _yielded_terminal_event. _cleanup_after_exhaustion then re-raises a transport teardown failure rather than debug-logging it. That teardown error escapes, and it masks the ModelBehaviorError built from the provider's own code and message.

The same file already holds the correct set. stream_response carries its own at :761-767, and that one lists both:

if chunk_type in {
    "response.completed",
    "response.failed",
    "response.incomplete",
    "error",
    "response.error",
}:

One line brings :261 into line with :761.

This is not a test-only path. _ResponseStreamWithRequestId is built at :916 with cleanup=lambda: api_response_cm.__aexit__(...), so the cleanup is a real httpx teardown.

Debug information

  • Agents SDK version: repository main at 588826c
  • Related library versions: openai 3.0.0 (the pin is openai>=3.0.0,<4)
  • Python version: 3.12
  • Operating system: Ubuntu 24.04
  • Model and model provider: OpenAI Responses API, any model. The trigger is the provider emitting an error event, not a particular model.
  • Does the issue reproduce with the latest release? Yes, on main at 588826c.
  • Does it occur consistently or intermittently? Consistently, whenever an error event arrives and the transport teardown then fails.

Repro steps

Driving _ResponseStreamWithRequestId with a cleanup that raises, one working event type and one that does not:

event type           terminal_seen   what __anext__ raised
------------------------------------------------------------------------------
response.failed      True            StopAsyncIteration (cleanup error suppressed)
error                False           RuntimeError: transport close failed

End to end on the real stream_response path, with the provider emitting an error event and the teardown failing:

on main:      caller sees -> RuntimeError: transport close failed
with the fix: caller sees -> ModelBehaviorError: Responses stream ended with terminal
              event `error`. code=server_error; message=the provider gave up.

The first hides the diagnosis the provider sent behind a transport artefact.

Expected behavior

An error event ends the stream, so the teardown failure is debug-logged and the ModelBehaviorError carrying the provider's code and message reaches the caller. That is what response.failed already does.

One related place that is not a second instance

A third terminal set at :1437-1442, in the websocket _stream_response, also omits "error". It is unreachable. event_type == "error" raises ResponsesWebSocketError at :1426, before that set is read. I left it alone.

Offer

CONTRIBUTING.md limits pull requests to collaborators and asks non-collaborators to open an issue, so this is an issue. The fix is written and verified locally all the same. One line in openai_responses.py, plus one parametrised test beside the existing _ResponseStreamWithRequestId cleanup tests.

uv run ruff format --check   → 994 files already formatted
uv run ruff check            → All checks passed!
uv run mypy src              → Success: no issues found in 317 source files
pyright                      → 0 errors, 0 warnings, 0 informations
pytest -n 8 -m "not serial"  → 11209 passed, 29 skipped
run_serial_tests.py          → 77 passed, 4 skipped, 77 deselected
.agents/skills/code-change-verification/scripts/run.sh → all commands passed

Baseline on unmodified main is 11207 passed, 29 skipped, so the delta is exactly the two new cases. Falsification, with the production file reverted and the test kept: the [error] case fails with RuntimeError: transport close failed, and the [response.failed] control still passes.

Say the word and I will open the pull request, or take the one line as it stands.

The two patches

Our patch+75 −1
src/agents/models/openai_responses.pyindex 47d4fa7a..e9208233 100644@@ -262,6 +262,7 @@ class _ResponseStreamWithRequestId:         "response.completed",         "response.failed",         "response.incomplete",+        "error",         "response.error",     } tests/models/test_openai_responses.pyindex 66f69573..e8cb8e09 100644@@ -11,7 +11,12 @@ from typing import Any, cast import httpx2 import pytest from openai import NOT_GIVEN, APIConnectionError, AsyncOpenAI, RateLimitError, omit-from openai.types.responses import Response, ResponseCompletedEvent, ResponseErrorEvent+from openai.types.responses import (+    Response,+    ResponseCompletedEvent,+    ResponseErrorEvent,+    ResponseFailedEvent,+) from openai.types.responses.response import IncompleteDetails from openai.types.responses.response_create_params import ContextManagement, PromptCacheOptions from openai.types.responses.response_usage import ResponseUsage@@ -5483,6 +5488,74 @@ async def test_response_stream_closes_distinct_source_after_normal_exhaustion()     assert cleanup_calls == 1  +def _terminal_stream_event(event_type: str) -> Any:+    if event_type == "error":+        return ResponseErrorEvent(+            type="error",+            code="server_error",+            message="the provider gave up.",+            param=None,+            sequence_number=0,+        )+    response = get_response_obj([])+    response.status = "failed"+    return ResponseFailedEvent(type="response.failed", response=response, sequence_number=0)+++@pytest.mark.asyncio+@pytest.mark.parametrize("event_type", ["error", "response.failed"])+async def test_response_stream_ignores_cleanup_failure_after_terminal_event(+    event_type: str,+) -> None:+    async def source() -> Any:+        yield _terminal_stream_event(event_type)++    async def cleanup() -> None:+        raise RuntimeError("transport close failed")++    stream = _ResponseStreamWithRequestId(source(), request_id=None, cleanup=cleanup)+    event = await stream.__anext__()+    assert event.type == event_type++    with pytest.raises(StopAsyncIteration):+        await stream.__anext__()+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_stream_response_error_event_surfaces_provider_error_despite_cleanup_failure(+    monkeypatch: pytest.MonkeyPatch,+) -> None:+    model = OpenAIResponsesModel(model="gpt-4", openai_client=object())  # type: ignore[arg-type]++    async def source() -> Any:+        yield _terminal_stream_event("error")++    async def cleanup() -> None:+        raise RuntimeError("transport close failed")++    async def fake_fetch_response(*args: Any, **kwargs: Any) -> _ResponseStreamWithRequestId:+        return _ResponseStreamWithRequestId(source(), request_id=None, cleanup=cleanup)++    monkeypatch.setattr(model, "_fetch_response", fake_fetch_response)++    events = []+    with pytest.raises(ModelBehaviorError, match="server_error") as exc_info:+        async for event in model.stream_response(+            system_instructions=None,+            input="hi",+            model_settings=ModelSettings(),+            tools=[],+            output_schema=None,+            handoffs=[],+            tracing=ModelTracing.DISABLED,+        ):+            events.append(event)++    assert "the provider gave up." in str(exc_info.value)+    assert [event.type for event in events] == ["error"]++ @pytest.mark.allow_call_model_methods @pytest.mark.asyncio async def test_websocket_request_is_counted_when_responses_provider_omits_usage( 
Merged fix · PR #5187+131 −14
src/agents/models/openai_responses.pyindex 47d4fa7a9..87e4b5bec 100644@@ -262,6 +262,7 @@ class _ResponseStreamWithRequestId:         "response.completed",         "response.failed",         "response.incomplete",+        "error",         "response.error",     } @@ -778,6 +779,10 @@ class OpenAIResponsesModel(Model):                                     trace_include_sensitive_data=tracing.include_data(),                                 )                         yield chunk+                        if terminal_failure_error is not None:+                            # Close explicitly rather than advancing a failed stream: transport+                            # teardown during iteration must not replace the provider's error.+                            break                 except asyncio.CancelledError:                     close_stream_in_background = True                     self._schedule_async_iterator_close(stream)tests/models/test_openai_responses.pyindex 66f69573c..8f17583d5 100644@@ -4,6 +4,7 @@ import asyncio import copy import json import logging+from contextlib import nullcontext from dataclasses import asdict, fields, replace from types import SimpleNamespace from typing import Any, cast@@ -908,7 +909,10 @@ async def test_fetch_response_stream_without_request_id_still_returns_events():  @pytest.mark.allow_call_model_methods @pytest.mark.asyncio-async def test_stream_response_ignores_streaming_context_exit_failure_after_terminal_event():+@pytest.mark.parametrize("event_type", ["response.completed", "error"])+async def test_stream_response_ignores_streaming_context_exit_failure_after_terminal_event(+    event_type,+):     class DummyHTTPStream:         def __init__(self):             self._yielded = False@@ -920,6 +924,14 @@ async def test_stream_response_ignores_streaming_context_exit_failure_after_term             if self._yielded:                 raise StopAsyncIteration             self._yielded = True+            if event_type == "error":+                return ResponseErrorEvent(+                    type="error",+                    code="server_error",+                    message="synthetic provider failure",+                    param=None,+                    sequence_number=0,+                )             return ResponseCompletedEvent(                 type="response.completed",                 response=get_response_obj([], response_id="resp-stream-request-id"),@@ -958,23 +970,123 @@ async def test_stream_response_ignores_streaming_context_exit_failure_after_term      model = OpenAIResponsesModel(model="gpt-4", openai_client=DummyResponsesClient())  # type: ignore[arg-type] -    events: list[ResponseCompletedEvent] = []-    async for event in model.stream_response(-        system_instructions=None,-        input="hi",-        model_settings=ModelSettings(),-        tools=[],-        output_schema=None,-        handoffs=[],-        tracing=ModelTracing.DISABLED,-    ):-        assert isinstance(event, ResponseCompletedEvent)-        events.append(event)+    events = []+    expected_error = (+        pytest.raises(ModelBehaviorError, match="synthetic provider failure")+        if event_type == "error"+        else nullcontext()+    )+    with expected_error:+        async for event in model.stream_response(+            system_instructions=None,+            input="hi",+            model_settings=ModelSettings(),+            tools=[],+            output_schema=None,+            handoffs=[],+            tracing=ModelTracing.DISABLED,+        ):+            events.append(event) -    assert len(events) == 1+    assert [event.type for event in events] == [event_type]     assert aexit_calls == [(None, None, None)]  +@pytest.mark.asyncio+async def test_response_stream_preserves_error_event_when_cleanup_fails() -> None:+    event = ResponseErrorEvent(+        type="error",+        code="server_error",+        message="synthetic provider failure",+        param=None,+        sequence_number=0,+    )+    cleanup_calls = 0++    async def events():+        yield event++    async def cleanup():+        nonlocal cleanup_calls+        cleanup_calls += 1+        raise RuntimeError("transport close failed")++    stream = _ResponseStreamWithRequestId(events(), request_id=None, cleanup=cleanup)+    assert [item async for item in stream] == [event]+    assert cleanup_calls == 1+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+@pytest.mark.parametrize("event_type", ["error", "response.failed", "response.incomplete", None])+async def test_stream_response_preserves_terminal_failure_when_http_close_fails(+    event_type: str | None,+) -> None:+    # Exercise real SSE parsing and transport teardown, which a model fake would bypass.+    close_calls = 0+    if event_type == "error":+        payload = ResponseErrorEvent(+            type="error",+            code="server_error",+            message="synthetic provider failure",+            param=None,+            sequence_number=0,+        ).model_dump()+        expected_message = "code=server_error; message=synthetic provider failure"+    elif event_type is not None:+        status = event_type.removeprefix("response.")+        payload = {+            "type": event_type,+            "sequence_number": 0,+            "response": _response_with_terminal_status(status).model_dump(),+        }+        expected_message = f"status={status}"+    else:+        payload = None+        expected_message = "transport close failed"++    class FailingCloseStream(httpx2.AsyncByteStream):+        async def __aiter__(self):+            if payload is not None:+                yield f"event: {event_type}\ndata: {json.dumps(payload)}\n\n".encode()+            yield b"data: [DONE]\n\n"++        async def aclose(self):+            nonlocal close_calls+            close_calls += 1+            raise RuntimeError("transport close failed")++    async def handler(request: httpx2.Request) -> httpx2.Response:+        return httpx2.Response(+            200,+            headers={"content-type": "text/event-stream"},+            stream=FailingCloseStream(),+            request=request,+        )++    events = []+    async with httpx2.AsyncClient(transport=httpx2.MockTransport(handler)) as http_client:+        model = OpenAIResponsesModel(+            model="gpt-4",+            openai_client=AsyncOpenAI(api_key="test-key", http_client=http_client),+        )+        expected_exception = ModelBehaviorError if event_type is not None else RuntimeError+        with pytest.raises(expected_exception, match=expected_message):+            async for event in model.stream_response(+                system_instructions=None,+                input="hi",+                model_settings=ModelSettings(),+                tools=[],+                output_schema=None,+                handoffs=[],+                tracing=ModelTracing.DISABLED,+            ):+                events.append(event)++    assert [event.type for event in events] == ([event_type] if event_type is not None else [])+    assert close_calls == 1++ @pytest.mark.allow_call_model_methods @pytest.mark.asyncio async def test_stream_response_close_closes_inner_http_stream_with_async_close(monkeypatch):