Backtestsopenai-agents-pythonrun-20261024-184640#3990
Responses WebSocket server_error frames bypass model retry policies
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches touch the same condition in OpenAIResponsesWSModel.get_retry_advice and both pass every independent test, so the difference lies in how faithfully each matches the issue's specification and in test quality. The issue states that a pre-response frame with event_type="error" and error_type="server_error" should yield suggested=True, analogous to an HTTP 500, with no qualification on the code field. Patch B implements exactly that (`error_type == "server_error"`). Patch A adds an unrequested `code is None` restriction, so a server_error frame carrying any non-null code would still fall through to the common classifier and terminate the run, recreating the reported failure mode for a slightly different frame. Patch A even codifies this narrowing in a test asserting that `server_error` with code `invalid_request_error` is not retried, but that combination is not what the issue's 'invalid-request errors remain unaffected' boundary refers to; invalid-request frames have error_type="invalid_request_error", which neither patch matches. The existing `server_is_overloaded` branch, by contrast, keys on a code because that frame's type is service_unavailable_error, so B's type-based match is the consistent generalization. On tests, A only adds unit checks on get_retry_advice (plus a parametrized rewrite of the partial-overload test). B adds the same unit coverage for both stateless and previous_response_id cases, a partial-response unsafe case, and three end-to-end tests through stream_response_with_retry with provider_suggested() that verify the real retry path: stateless retry succeeds on a second connection, stateful request with previous_response_id is not replayed, and a server_error after response.created is not retried. Those exercise the exact runtime scenarios the issue enumerates and would fail on base. Scope and craft are otherwise comparable; B's extra test helpers are contained and reuse existing DummyWSConnection fixtures. B is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
4m 27s
Wall-clock time in the container
Full test suite
No new failures
380 passed · 0 skipped · 380 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
185vs 79 merged
Changed lines, ours vs merged
The issue
View on GitHubPlease read this first
- I read the Agents SDK model retry and Responses WebSocket documentation.
- I searched open and closed issues for
server_error,ResponsesWebSocketError, and related retry behavior. - I reviewed #3969 and its fix in #3978.
Describe the bug
When a Responses WebSocket returns a server_error frame before any response.* event for the current request, Runner.run_streamed() terminates with ResponsesWebSocketError instead of applying the configured model retry policy.
We captured the following raw server frame in four production occurrences within approximately three hours:
{
"type": "error",
"error": {
"type": "server_error",
"code": null,
"message": "Sorry, something went wrong.",
"param": null
},
"sequence_number": 2
}
The run had ModelRetrySettings(max_retries=2) configured with policies covering:
retry_policies.provider_suggested()- network errors
- transient HTTP statuses
The upstream failure is intermittent, but the missing retry classification is deterministic.
Version 0.19.0 added retry advice for pre-response WebSocket errors whose code is exactly server_is_overloaded. This related error instead has:
event_type="error"error_type="server_error"code=None- no HTTP status
- no retry headers
Consequently, OpenAIResponsesWSModel.get_retry_advice() delegates to the common OpenAI classifier, which returns no retry advice. Network and HTTP-status policies cannot match because this WebSocket error frame carries neither a network-error classification nor the equivalent 5xx status.
Debug information
- Agents SDK version when observed:
0.18.2 - Behavior also appears unhandled by the
0.19.0retry classifier openaipackage version:2.45.0- Python version:
3.14.6 - Transport: OpenAI Responses WebSocket
- API interface: streaming
Reproduction
The provider error itself is intermittent, but the classifier behavior can be reproduced without making an API request:
from openai import AsyncOpenAI
from agents.models.openai_responses import (
OpenAIResponsesWSModel,
ResponsesWebSocketError,
)
from agents.retry import ModelRetryAdviceRequest
error = ResponsesWebSocketError(
{
"type": "error",
"error": {
"type": "server_error",
"code": None,
"message": "Sorry, something went wrong.",
"param": None,
},
"sequence_number": 2,
}
)
model = OpenAIResponsesWSModel(
model="gpt-5.4",
openai_client=AsyncOpenAI(api_key="test"),
)
for previous_response_id in (None, "resp_existing"):
advice = model.get_retry_advice(
ModelRetryAdviceRequest(
error=error,
attempt=1,
stream=True,
previous_response_id=previous_response_id,
)
)
print(advice)
Actual output:
None
None
At runtime, retry_policies.provider_suggested() therefore receives no positive provider advice. The error terminates a stateless streamed run without applying the configured retry and backoff policy.
Expected behavior
A Responses WebSocket error with event_type="error" and error_type="server_error" received before any response.* event should produce positive provider retry advice, analogous to a transient HTTP 500 error.
Semantically, the advice should be equivalent to:
ModelRetryAdvice(
suggested=True,
replay_safety=None,
)
This should allow stateless requests using retry_policies.provider_suggested() to retry without universally declaring stateful requests replay-safe.
Existing retry-safety boundaries should remain unchanged:
- Stateless pre-response requests may retry through
provider_suggested(). - Errors received after response processing begins must remain non-retryable.
- Requests using
previous_response_id,conversation_id, or other stateful continuation remain subject to the existing replay-safety gate. - Invalid-request and other non-transient WebSocket errors remain unaffected.
Additional context
This is related to, but distinct from, #3969 and #3978.
That fix matches only:
request.error.event_type == "error"
and request.error.code == "server_is_overloaded"
The production frame reported here instead uses error.type="server_error" with code=null, so it still falls through the retry classifier.
The two patches
src/agents/models/openai_responses.pyindex 55808eda..f2c75502 100644@@ -1034,7 +1034,10 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel): if ( isinstance(request.error, ResponsesWebSocketError) and request.error.event_type == "error"- and request.error.code == "server_is_overloaded"+ and (+ request.error.code == "server_is_overloaded"+ or request.error.error_type == "server_error"+ ) ): return ModelRetryAdvice( suggested=True,tests/models/test_openai_responses.pyindex bd555fdc..6ba729e6 100644@@ -38,7 +38,13 @@ from agents.models.openai_responses import ( ResponsesWebSocketError, _should_retry_pre_event_websocket_disconnect, )-from agents.retry import ModelRetryAdviceRequest+from agents.retry import (+ ModelRetryAdviceRequest,+ ModelRetryBackoffSettings,+ ModelRetrySettings,+ retry_policies,+)+from agents.run_internal.model_retry import stream_response_with_retry from agents.usage import Usage from tests.fake_model import get_response_obj from tests.testing_processor import fetch_ordered_spans@@ -4094,6 +4100,178 @@ def test_websocket_get_retry_advice_ignores_other_pre_response_error_codes() -> assert advice is None +def _server_error_frame(sequence_number: int) -> str:+ return json.dumps(+ {+ "type": "error",+ "error": {+ "type": "server_error",+ "code": None,+ "message": "Sorry, something went wrong.",+ "param": None,+ },+ "sequence_number": sequence_number,+ }+ )+++@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize("previous_response_id", [None, "resp_prev"])+def test_websocket_get_retry_advice_marks_pre_response_server_error_retryable(+ previous_response_id: str | None,+) -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(json.loads(_server_error_frame(2)))++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=True,+ previous_response_id=previous_response_id,+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_keeps_partial_server_error_unsafe() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(json.loads(_server_error_frame(2)))+ setattr(error, "_openai_agents_ws_replay_safety", "unsafe") # noqa: B010+ setattr(error, "_openai_agents_ws_response_started", True) # noqa: B010++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=True,+ )+ )++ assert advice is not None+ assert advice.suggested is False+ assert advice.replay_safety == "unsafe"+++def _patch_websocket_connections(+ monkeypatch: pytest.MonkeyPatch,+ model: OpenAIResponsesWSModel,+ connections: list[DummyWSConnection],+) -> list[DummyWSConnection]:+ opened: list[DummyWSConnection] = []++ async def fake_open(+ ws_url: str, headers: dict[str, str], *, connect_timeout: float | None = None+ ) -> DummyWSConnection:+ connection = connections.pop(0)+ opened.append(connection)+ return connection++ monkeypatch.setattr(model, "_open_websocket_connection", fake_open)+ return opened+++def _stream_with_provider_suggested_retry(+ model: OpenAIResponsesWSModel,+ *,+ previous_response_id: str | None = None,+):+ async def rewind() -> None:+ return None++ return stream_response_with_retry(+ get_stream=lambda: model.stream_response(+ system_instructions=None,+ input="hi",+ model_settings=ModelSettings(),+ tools=[],+ output_schema=None,+ handoffs=[],+ tracing=ModelTracing.DISABLED,+ previous_response_id=previous_response_id,+ ),+ rewind=rewind,+ retry_settings=ModelRetrySettings(+ max_retries=2,+ backoff=ModelRetryBackoffSettings(initial_delay=0, jitter=False),+ policy=retry_policies.provider_suggested(),+ ),+ get_retry_advice=model.get_retry_advice,+ previous_response_id=previous_response_id,+ conversation_id=None,+ )+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_retries_stateless_pre_response_server_error(monkeypatch):+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ opened = _patch_websocket_connections(+ monkeypatch,+ model,+ [+ DummyWSConnection([_server_error_frame(2)]),+ DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+ ],+ )++ events = [event async for event in _stream_with_provider_suggested_retry(model)]++ assert len(opened) == 2+ assert [event.type for event in events] == ["response.completed"]+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_does_not_retry_stateful_pre_response_server_error(monkeypatch):+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ opened = _patch_websocket_connections(+ monkeypatch,+ model,+ [+ DummyWSConnection([_server_error_frame(2)]),+ DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+ ],+ )++ with pytest.raises(ResponsesWebSocketError, match="server_error"):+ async for _event in _stream_with_provider_suggested_retry(+ model, previous_response_id="resp_prev"+ ):+ pass++ assert len(opened) == 1+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_stream_does_not_retry_server_error_after_response_started(monkeypatch):+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ opened = _patch_websocket_connections(+ monkeypatch,+ model,+ [+ DummyWSConnection(+ [+ _response_event_frame("response.created", "resp_partial", 1),+ _server_error_frame(2),+ ]+ ),+ DummyWSConnection([_response_completed_frame("resp_ok", 1)]),+ ],+ )++ with pytest.raises(ResponsesWebSocketError, match="server_error"):+ async for _event in _stream_with_provider_suggested_retry(model):+ pass++ assert len(opened) == 1++ def test_get_client_disables_provider_managed_retries_when_requested() -> None: class DummyClient: def __init__(self):
src/agents/models/openai_responses.pyindex 55808eda1..006aa203a 100644@@ -1034,7 +1034,10 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel): if ( isinstance(request.error, ResponsesWebSocketError) and request.error.event_type == "error"- and request.error.code == "server_is_overloaded"+ and (+ request.error.code == "server_is_overloaded"+ or (request.error.error_type == "server_error" and request.error.code is None)+ ) ): return ModelRetryAdvice( suggested=True,tests/models/test_openai_responses.pyindex bd555fdcd..bc1d33c64 100644@@ -4041,15 +4041,81 @@ def test_websocket_get_retry_advice_marks_pre_response_overload_retryable( @pytest.mark.allow_call_model_methods-def test_websocket_get_retry_advice_keeps_partial_overload_unsafe() -> None:+@pytest.mark.parametrize("previous_response_id", [None, "resp_prev"])+def test_websocket_get_retry_advice_marks_pre_response_server_error_retryable(+ previous_response_id: str | None,+) -> None: model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient())) error = ResponsesWebSocketError( { "type": "error", "error": {- "type": "service_unavailable_error",- "code": "server_is_overloaded",- "message": "Our servers are currently overloaded. Please try again later.",+ "type": "server_error",+ "code": None,+ "message": "Sorry, something went wrong.",+ },+ }+ )++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=True,+ previous_response_id=previous_response_id,+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_does_not_override_non_transient_error_code() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(+ {+ "type": "error",+ "error": {+ "type": "server_error",+ "code": "invalid_request_error",+ "message": "Invalid request.",+ },+ }+ )++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=True,+ )+ )++ assert advice is None+++@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize(+ ("error_type", "code"),+ [+ ("service_unavailable_error", "server_is_overloaded"),+ ("server_error", None),+ ],+)+def test_websocket_get_retry_advice_keeps_partial_transient_error_unsafe(+ error_type: str,+ code: str | None,+) -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(+ {+ "type": "error",+ "error": {+ "type": error_type,+ "code": code,+ "message": "Transient provider error.", }, } )