Backtestsopenai-agents-pythonrun-20261024-184640#3990

Issue #3990 Merged fix PR #3991 Run 24 Oct 2026 · run-20261024-184640 openai/openai-agents-python

Responses WebSocket server_error frames bypass model retry policies

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches touch the same condition in OpenAIResponsesWSModel.get_retry_advice and both pass every independent test, so the difference lies in how faithfully each matches the issue's specification and in test quality. The issue states that a pre-response frame with event_type="error" and error_type="server_error" should yield suggested=True, analogous to an HTTP 500, with no qualification on the code field. Patch B implements exactly that (`error_type == "server_error"`). Patch A adds an unrequested `code is None` restriction, so a server_error frame carrying any non-null code would still fall through to the common classifier and terminate the run, recreating the reported failure mode for a slightly different frame. Patch A even codifies this narrowing in a test asserting that `server_error` with code `invalid_request_error` is not retried, but that combination is not what the issue's 'invalid-request errors remain unaffected' boundary refers to; invalid-request frames have error_type="invalid_request_error", which neither patch matches. The existing `server_is_overloaded` branch, by contrast, keys on a code because that frame's type is service_unavailable_error, so B's type-based match is the consistent generalization. On tests, A only adds unit checks on get_retry_advice (plus a parametrized rewrite of the partial-overload test). B adds the same unit coverage for both stateless and previous_response_id cases, a partial-response unsafe case, and three end-to-end tests through stream_response_with_retry with provider_suggested() that verify the real retry path: stateless retry succeeds on a second connection, stateful request with previous_response_id is not replayed, and a server_error after response.created is not retried. Those exercise the exact runtime scenarios the issue enumerates and would fail on base. Scope and craft are otherwise comparable; B's extra test helpers are contained and reuse existing DummyWSConnection fixtures. B is the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

4m 27s

Wall-clock time in the container

Full test suite

No new failures

380 passed · 0 skipped · 380 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

185vs 79 merged

Changed lines, ours vs merged

Please read this first

  • I read the Agents SDK model retry and Responses WebSocket documentation.
  • I searched open and closed issues for server_error, ResponsesWebSocketError, and related retry behavior.
  • I reviewed #3969 and its fix in #3978.

Describe the bug

When a Responses WebSocket returns a server_error frame before any response.* event for the current request, Runner.run_streamed() terminates with ResponsesWebSocketError instead of applying the configured model retry policy.

We captured the following raw server frame in four production occurrences within approximately three hours:

{
  "type": "error",
  "error": {
    "type": "server_error",
    "code": null,
    "message": "Sorry, something went wrong.",
    "param": null
  },
  "sequence_number": 2
}

The run had ModelRetrySettings(max_retries=2) configured with policies covering:

  • retry_policies.provider_suggested()
  • network errors
  • transient HTTP statuses

The upstream failure is intermittent, but the missing retry classification is deterministic.

Version 0.19.0 added retry advice for pre-response WebSocket errors whose code is exactly server_is_overloaded. This related error instead has:

  • event_type="error"
  • error_type="server_error"
  • code=None
  • no HTTP status
  • no retry headers

Consequently, OpenAIResponsesWSModel.get_retry_advice() delegates to the common OpenAI classifier, which returns no retry advice. Network and HTTP-status policies cannot match because this WebSocket error frame carries neither a network-error classification nor the equivalent 5xx status.

Debug information

  • Agents SDK version when observed: 0.18.2
  • Behavior also appears unhandled by the 0.19.0 retry classifier
  • openai package version: 2.45.0
  • Python version: 3.14.6
  • Transport: OpenAI Responses WebSocket
  • API interface: streaming

Reproduction

The provider error itself is intermittent, but the classifier behavior can be reproduced without making an API request:

from openai import AsyncOpenAI

from agents.models.openai_responses import (
    OpenAIResponsesWSModel,
    ResponsesWebSocketError,
)
from agents.retry import ModelRetryAdviceRequest


error = ResponsesWebSocketError(
    {
        "type": "error",
        "error": {
            "type": "server_error",
            "code": None,
            "message": "Sorry, something went wrong.",
            "param": None,
        },
        "sequence_number": 2,
    }
)

model = OpenAIResponsesWSModel(
    model="gpt-5.4",
    openai_client=AsyncOpenAI(api_key="test"),
)

for previous_response_id in (None, "resp_existing"):
    advice = model.get_retry_advice(
        ModelRetryAdviceRequest(
            error=error,
            attempt=1,
            stream=True,
            previous_response_id=previous_response_id,
        )
    )
    print(advice)

Actual output:

None
None

At runtime, retry_policies.provider_suggested() therefore receives no positive provider advice. The error terminates a stateless streamed run without applying the configured retry and backoff policy.

Expected behavior

A Responses WebSocket error with event_type="error" and error_type="server_error" received before any response.* event should produce positive provider retry advice, analogous to a transient HTTP 500 error.

Semantically, the advice should be equivalent to:

ModelRetryAdvice(
    suggested=True,
    replay_safety=None,
)

This should allow stateless requests using retry_policies.provider_suggested() to retry without universally declaring stateful requests replay-safe.

Existing retry-safety boundaries should remain unchanged:

  • Stateless pre-response requests may retry through provider_suggested().
  • Errors received after response processing begins must remain non-retryable.
  • Requests using previous_response_id, conversation_id, or other stateful continuation remain subject to the existing replay-safety gate.
  • Invalid-request and other non-transient WebSocket errors remain unaffected.

Additional context

This is related to, but distinct from, #3969 and #3978.

That fix matches only:

request.error.event_type == "error"
and request.error.code == "server_is_overloaded"

The production frame reported here instead uses error.type="server_error" with code=null, so it still falls through the retry classifier.

The two patches

Our patch+183 −2
src/agents/models/openai_responses.pyindex 55808eda..f2c75502 100644@@ -1034,7 +1034,10 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel):         if (             isinstance(request.error, ResponsesWebSocketError)             and request.error.event_type == "error"-            and request.error.code == "server_is_overloaded"+            and (+                request.error.code == "server_is_overloaded"+                or request.error.error_type == "server_error"+            )         ):             return ModelRetryAdvice(                 suggested=True,tests/models/test_openai_responses.pyindex bd555fdc..6ba729e6 100644@@ -38,7 +38,13 @@ from agents.models.openai_responses import (     ResponsesWebSocketError,     _should_retry_pre_event_websocket_disconnect, )-from agents.retry import ModelRetryAdviceRequest+from agents.retry import (+    ModelRetryAdviceRequest,+    ModelRetryBackoffSettings,+    ModelRetrySettings,+    retry_policies,+)+from agents.run_internal.model_retry import stream_response_with_retry from agents.usage import Usage from tests.fake_model import get_response_obj from tests.testing_processor import fetch_ordered_spans@@ -4094,6 +4100,178 @@ def test_websocket_get_retry_advice_ignores_other_pre_response_error_codes() ->     assert advice is None  +def _server_error_frame(sequence_number: int) -> str:+    return json.dumps(+        {+            "type": "error",+            "error": {+                "type": "server_error",+                "code": None,+                "message": "Sorry, something went wrong.",+                "param": None,+            },+            "sequence_number": sequence_number,+        }+    )+++@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize("previous_response_id", [None, "resp_prev"])+def test_websocket_get_retry_advice_marks_pre_response_server_error_retryable(+    previous_response_id: str | None,+) -> None:+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    error = ResponsesWebSocketError(json.loads(_server_error_frame(2)))++    advice = model.get_retry_advice(+        ModelRetryAdviceRequest(+            error=error,+            attempt=1,+            stream=True,+            previous_response_id=previous_response_id,+        )+    )++    assert advice is not None+    assert advice.suggested is True+    assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_keeps_partial_server_error_unsafe() -> None:+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    error = ResponsesWebSocketError(json.loads(_server_error_frame(2)))+    setattr(error, "_openai_agents_ws_replay_safety", "unsafe")  # noqa: B010+    setattr(error, "_openai_agents_ws_response_started", True)  # noqa: B010++    advice = model.get_retry_advice(+        ModelRetryAdviceRequest(+            error=error,+            attempt=1,+            stream=True,+        )+    )++    assert advice is not None+    assert advice.suggested is False+    assert advice.replay_safety == "unsafe"+++def _patch_websocket_connections(+    monkeypatch: pytest.MonkeyPatch,+    model: OpenAIResponsesWSModel,+    connections: list[DummyWSConnection],+) -> list[DummyWSConnection]:+    opened: list[DummyWSConnection] = []++    async def fake_open(+        ws_url: str, headers: dict[str, str], *, connect_timeout: float | None = None+    ) -> DummyWSConnection:+        connection = connections.pop(0)+        opened.append(connection)+        return connection++    monkeypatch.setattr(model, "_open_websocket_connection", fake_open)+    return opened+++def _stream_with_provider_suggested_retry(+    model: OpenAIResponsesWSModel,+    *,+    previous_response_id: str | None = None,+):+    async def rewind() -> None:+        return None++    return stream_response_with_retry(+        get_stream=lambda: model.stream_response(+            system_instructions=None,+            input="hi",+            model_settings=ModelSettings(),+            tools=[],+            output_schema=None,+            handoffs=[],+            tracing=ModelTracing.DISABLED,+            previous_response_id=previous_response_id,+        ),+        rewind=rewind,+        retry_settings=ModelRetrySettings(+            max_retries=2,+            backoff=ModelRetryBackoffSettings(initial_delay=0, jitter=False),+            policy=retry_policies.provider_suggested(),+        ),+        get_retry_advice=model.get_retry_advice,+        previous_response_id=previous_response_id,+        conversation_id=None,+    )+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_retries_stateless_pre_response_server_error(monkeypatch):+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    opened = _patch_websocket_connections(+        monkeypatch,+        model,+        [+            DummyWSConnection([_server_error_frame(2)]),+            DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+        ],+    )++    events = [event async for event in _stream_with_provider_suggested_retry(model)]++    assert len(opened) == 2+    assert [event.type for event in events] == ["response.completed"]+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_does_not_retry_stateful_pre_response_server_error(monkeypatch):+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    opened = _patch_websocket_connections(+        monkeypatch,+        model,+        [+            DummyWSConnection([_server_error_frame(2)]),+            DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+        ],+    )++    with pytest.raises(ResponsesWebSocketError, match="server_error"):+        async for _event in _stream_with_provider_suggested_retry(+            model, previous_response_id="resp_prev"+        ):+            pass++    assert len(opened) == 1+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_does_not_retry_server_error_after_response_started(monkeypatch):+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    opened = _patch_websocket_connections(+        monkeypatch,+        model,+        [+            DummyWSConnection(+                [+                    _response_event_frame("response.created", "resp_partial", 1),+                    _server_error_frame(2),+                ]+            ),+            DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+        ],+    )++    with pytest.raises(ResponsesWebSocketError, match="server_error"):+        async for _event in _stream_with_provider_suggested_retry(model):+            pass++    assert len(opened) == 1++ def test_get_client_disables_provider_managed_retries_when_requested() -> None:     class DummyClient:         def __init__(self): 
Merged fix · PR #3991+74 −5
src/agents/models/openai_responses.pyindex 55808eda1..006aa203a 100644@@ -1034,7 +1034,10 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel):         if (             isinstance(request.error, ResponsesWebSocketError)             and request.error.event_type == "error"-            and request.error.code == "server_is_overloaded"+            and (+                request.error.code == "server_is_overloaded"+                or (request.error.error_type == "server_error" and request.error.code is None)+            )         ):             return ModelRetryAdvice(                 suggested=True,tests/models/test_openai_responses.pyindex bd555fdcd..bc1d33c64 100644@@ -4041,15 +4041,81 @@ def test_websocket_get_retry_advice_marks_pre_response_overload_retryable(   @pytest.mark.allow_call_model_methods-def test_websocket_get_retry_advice_keeps_partial_overload_unsafe() -> None:+@pytest.mark.parametrize("previous_response_id", [None, "resp_prev"])+def test_websocket_get_retry_advice_marks_pre_response_server_error_retryable(+    previous_response_id: str | None,+) -> None:     model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))     error = ResponsesWebSocketError(         {             "type": "error",             "error": {-                "type": "service_unavailable_error",-                "code": "server_is_overloaded",-                "message": "Our servers are currently overloaded. Please try again later.",+                "type": "server_error",+                "code": None,+                "message": "Sorry, something went wrong.",+            },+        }+    )++    advice = model.get_retry_advice(+        ModelRetryAdviceRequest(+            error=error,+            attempt=1,+            stream=True,+            previous_response_id=previous_response_id,+        )+    )++    assert advice is not None+    assert advice.suggested is True+    assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_does_not_override_non_transient_error_code() -> None:+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    error = ResponsesWebSocketError(+        {+            "type": "error",+            "error": {+                "type": "server_error",+                "code": "invalid_request_error",+                "message": "Invalid request.",+            },+        }+    )++    advice = model.get_retry_advice(+        ModelRetryAdviceRequest(+            error=error,+            attempt=1,+            stream=True,+        )+    )++    assert advice is None+++@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize(+    ("error_type", "code"),+    [+        ("service_unavailable_error", "server_is_overloaded"),+        ("server_error", None),+    ],+)+def test_websocket_get_retry_advice_keeps_partial_transient_error_unsafe(+    error_type: str,+    code: str | None,+) -> None:+    model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+    error = ResponsesWebSocketError(+        {+            "type": "error",+            "error": {+                "type": error_type,+                "code": code,+                "message": "Transient provider error.",             },         }     )