Backtestsopenai-agents-pythonrun-20261024-184640#3969
Responses WebSocket server_is_overloaded errors bypass model retry policies
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the pre-response `server_is_overloaded` WebSocket frame return positive provider advice, which fixes the issue's literal repro. But Patch A stops there: it returns `suggested=True` with no `replay_safety`, so stateful requests (a `conversation_id` or a store-backed `previous_response_id`) still are not replayed. The independent tests confirm this: both stateful runner scenarios still raise `ResponsesWebSocketError` under Patch A, while Patch B passes all eight. The issue explicitly calls out that stateful replays after an explicit pre-response rejection should be permitted when state is persisted, and withheld only for connection-local `store=false` continuations. Patch B implements exactly that distinction: it marks the replay `safe` for stateless and persisted-state requests, records the request's `store` value on the raised error so advice can detect the connection-local case, and withholds replay approval there. It also adds `server_is_overloaded` error-code classification to the shared OpenAI classifier, which is the root-cause gap the issue describes (the classifier normalizes `error_code` but never uses it), so the statusless overload signal is handled consistently beyond the WS subclass. Patch B's tests are stronger too: parametrized advice tests over stateless/previous_response_id/conversation_id, the store=false withholding case, the post-response-started veto, the statusless `APIError` path through the common classifier, and end-to-end retry via `get_response_with_retry` with both success and the store=false non-replay. Patch A's tests only cover the advice function. Patch A's extra `event_type == "error"` guard is fine but B's pre-existing response-started check already covers `response.error` frames, as its passing veto test shows. The `setattr`-on-exception approach in B mirrors the existing `_openai_agents_ws_replay_safety` convention. Scope is slightly broader in B but directly serves the issue's stated requirements rather than adding risk.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
6m 40s
Wall-clock time in the container
Full test suite
No new failures
452 passed · 0 skipped · 452 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
263vs 94 merged
Changed lines, ours vs merged
The issue
View on GitHubPlease read this first
- I read the Agents SDK model retry and Responses WebSocket documentation.
- I searched open and closed issues/PRs for
server_is_overloaded,service_unavailable_error, andResponsesWebSocketError.
Describe the bug
When an established Responses WebSocket returns this error frame before any response.* event for the current request:
{
"type": "error",
"error": {
"type": "service_unavailable_error",
"code": "server_is_overloaded",
"message": "Our servers are currently overloaded. Please try again later.",
"param": null
},
"sequence_number": 2
}
Runner.run() terminates with ResponsesWebSocketError even when the run has ModelRetrySettings(max_retries=2) configured with retry_policies.provider_suggested() and transient HTTP status policies. No runner-managed retry or configured backoff occurs.
The equivalent transient failure over HTTP carries a 5xx status and is retryable. Over WebSocket transport, the overload is delivered as an error event without an HTTP status, so the signal currently falls through the retry classifier.
This is particularly costly for long-running agent workflows. Retrying the entire run at the application or worker level can repeat already-completed tool and sub-agent work, especially when running evals on the flex processing tier.
Debug information
- Agents SDK version:
0.18.2 - Python version:
3.14.3 - OS: macOS
- Transport: OpenAI Responses WebSocket
openaipackage version:2.45.0- The relevant classifier is unchanged in
0.18.3and onmainatf663a06aea23c859be8e8555c005ac912bd58337
Repro steps
The service overload itself is intermittent, but the missing retry classification is deterministic:
from openai import AsyncOpenAI
from agents.models.openai_responses import (
OpenAIResponsesWSModel,
ResponsesWebSocketError,
)
from agents.retry import ModelRetryAdviceRequest
error = ResponsesWebSocketError(
{
"type": "error",
"error": {
"type": "service_unavailable_error",
"code": "server_is_overloaded",
"message": "Our servers are currently overloaded. Please try again later.",
"param": None,
},
"sequence_number": 2,
}
)
model = OpenAIResponsesWSModel(
model="gpt-5.4",
openai_client=AsyncOpenAI(api_key="test"),
)
for previous_response_id in (None, "resp_existing"):
advice = model.get_retry_advice(
ModelRetryAdviceRequest(
error=error,
attempt=1,
stream=False,
previous_response_id=previous_response_id,
)
)
print(advice)
Actual output:
None
None
At runtime, this means retry_policies.provider_suggested() receives no positive provider advice. HTTP status policies cannot match either because the WebSocket error has no status code, so the transient overload terminates the run.
Expected behavior
A pre-response-event WebSocket rejection with code="server_is_overloaded" should be classified as a transient, provider-suggested retry.
Existing replay safeguards should remain unchanged once any response event has begun.
For stateful requests, replay safety may need to depend on the state owner and storage mode rather than solely on the absence of output:
- A
conversation_idor store-backedprevious_response_idmay be replayable after an explicit pre-response overload rejection. - A connection-local
previous_response_idwithstore=false/ZDR should not receive unconditional replay approval, because a failed WebSocket continuation evicts that ID from the connection-local cache and there is no persisted fallback.
Additional context
ResponsesWebSocketError already retains code="server_is_overloaded" and error_type="service_unavailable_error". However:
OpenAIResponsesWSModel.get_retry_advice()eventually delegates this error to the common OpenAI classifier.- The common classifier recognizes retry headers, network/timeouts, and HTTP statuses such as 408/409/429/5xx.
- It normalizes
error_codebut does not currently use it to classify this transient overload. - WebSocket error frames do not necessarily provide the equivalent HTTP 503 status, so this case returns no advice.
Would it make sense to classify this explicit pre-response server_is_overloaded rejection as provider-suggested retryable while preserving the current hard veto after response processing has begun?
The two patches
src/agents/models/_openai_retry.pyindex f7a941be..9b597112 100644@@ -12,6 +12,17 @@ from ._retry_runtime import ( iter_error_chain as _iter_error_chain, ) +_SERVER_OVERLOADED_ERROR_CODES = frozenset({"server_is_overloaded"})+++def is_server_overloaded_error_code(error_code: str | None) -> bool:+ """Return whether an error code reports a transient provider overload.++ Transports such as Responses WebSocket error frames and streamed error events can+ deliver this signal without the HTTP 5xx status that would otherwise classify it.+ """+ return error_code in _SERVER_OVERLOADED_ERROR_CODES+ def _is_stateful_request(request: ModelRetryAdviceRequest) -> bool: return bool(request.previous_response_id or request.conversation_id)@@ -89,8 +100,10 @@ def get_openai_retry_advice(request: ModelRetryAdviceRequest) -> ModelRetryAdvic normalized=normalized, ) - if normalized.status_code in {408, 409, 429} or (- isinstance(normalized.status_code, int) and normalized.status_code >= 500+ if (+ normalized.status_code in {408, 409, 429}+ or (isinstance(normalized.status_code, int) and normalized.status_code >= 500)+ or is_server_overloaded_error_code(normalized.error_code) ): advice = ModelRetryAdvice( suggested=True,src/agents/models/openai_responses.pyindex ff17cea4..181a79c7 100644@@ -77,7 +77,7 @@ from ..tracing import SpanError, response_span from ..usage import Usage, _response_usage_to_usage, model_usage_to_span_usage from ..util._json import _to_dump_compatible from ..version import __version__-from ._openai_retry import get_openai_retry_advice+from ._openai_retry import get_openai_retry_advice, is_server_overloaded_error_code from ._response_terminal import response_error_event_failure_error, response_terminal_failure_error from ._retry_runtime import ( should_disable_provider_managed_retries,@@ -402,6 +402,24 @@ def _get_websocket_timeout_phase(error: Exception) -> str | None: return None +def _is_websocket_server_overloaded_error(error: Exception) -> bool:+ return isinstance(error, ResponsesWebSocketError) and is_server_overloaded_error_code(+ error.code+ )+++def _is_connection_local_websocket_continuation(+ error: Exception, request: ModelRetryAdviceRequest+) -> bool:+ # Without `store`, a `previous_response_id` only lives in the websocket connection cache,+ # and a failed continuation evicts it, so there is no persisted state to replay against.+ return (+ bool(request.previous_response_id)+ and not request.conversation_id+ and getattr(error, "_openai_agents_ws_request_store", None) is False+ )++ def _should_retry_pre_event_websocket_disconnect() -> bool: return not should_disable_websocket_pre_event_retries() @@ -1031,6 +1049,16 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel): replay_safety="safe", reason=str(request.error), )+ if _is_websocket_server_overloaded_error(request.error):+ # The server explicitly rejected the request before any response event, so a+ # replay is safe unless it depends on connection-local state that is now gone.+ advice = ModelRetryAdvice(+ suggested=True,+ reason=str(request.error),+ )+ if not _is_connection_local_websocket_continuation(request.error, request):+ advice.replay_safety = "safe"+ return advice return super().get_retry_advice(request) def _get_ws_request_lock(self) -> asyncio.Lock:@@ -1192,7 +1220,15 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel): event_type = payload.get("type") if event_type == "error":- raise ResponsesWebSocketError(payload)+ ws_error = ResponsesWebSocketError(payload)+ # Record the request storage mode so retry advice can tell+ # persisted state apart from connection-local continuations.+ setattr( # noqa: B010+ ws_error,+ "_openai_agents_ws_request_store",+ request_frame.get("store"),+ )+ raise ws_error if event_type == "response.error": received_any_event = True raise ResponsesWebSocketError(payload)tests/models/test_openai_responses.pyindex 00c6a4cf..4e9c9ad5 100644@@ -7,7 +7,7 @@ from typing import Any, cast import httpx import pytest-from openai import NOT_GIVEN, APIConnectionError, AsyncOpenAI, RateLimitError, omit+from openai import NOT_GIVEN, APIConnectionError, APIError, AsyncOpenAI, RateLimitError, omit from openai.types.responses import ResponseCompletedEvent, ResponseErrorEvent from openai.types.responses.response_create_params import ContextManagement, PromptCacheOptions from openai.types.shared.reasoning import Reasoning@@ -38,7 +38,8 @@ from agents.models.openai_responses import ( ResponsesWebSocketError, _should_retry_pre_event_websocket_disconnect, )-from agents.retry import ModelRetryAdviceRequest+from agents.retry import ModelRetryAdviceRequest, ModelRetrySettings, retry_policies+from agents.run_internal import model_retry as model_retry_module from agents.usage import Usage from tests.fake_model import get_response_obj from tests.testing_processor import fetch_ordered_spans@@ -4034,3 +4035,204 @@ def test_websocket_pre_event_disconnect_retry_respects_websocket_retry_disable() with websocket_pre_event_retries_disabled(True): assert _should_retry_pre_event_websocket_disconnect() is False+++def _server_overloaded_error_payload() -> dict[str, Any]:+ return {+ "type": "error",+ "error": {+ "type": "service_unavailable_error",+ "code": "server_is_overloaded",+ "message": "Our servers are currently overloaded. Please try again later.",+ "param": None,+ },+ "sequence_number": 2,+ }+++@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize(+ ("previous_response_id", "conversation_id"),+ [(None, None), ("resp_existing", None), (None, "conv_123")],+)+def test_websocket_get_retry_advice_retries_pre_event_server_overloaded(+ previous_response_id: str | None, conversation_id: str | None+) -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(_server_overloaded_error_payload())++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ previous_response_id=previous_response_id,+ conversation_id=conversation_id,+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety == "safe"+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_withholds_replay_approval_for_store_false_overload() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(_server_overloaded_error_payload())+ setattr(error, "_openai_agents_ws_request_store", False) # noqa: B010++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ previous_response_id="resp_connection_local",+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_keeps_overload_after_response_started_unsafe() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(_server_overloaded_error_payload())+ setattr(error, "_openai_agents_ws_replay_safety", "unsafe") # noqa: B010+ setattr(error, "_openai_agents_ws_response_started", True) # noqa: B010++ for previous_response_id in (None, "resp_existing"):+ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ previous_response_id=previous_response_id,+ )+ )++ assert advice is not None+ assert advice.suggested is False+ assert advice.replay_safety == "unsafe"+++@pytest.mark.allow_call_model_methods+def test_get_retry_advice_retries_statusless_server_overloaded_error() -> None:+ model = OpenAIResponsesModel(model="gpt-4", openai_client=cast(Any, object()))+ error = APIError(+ "Our servers are currently overloaded. Please try again later.",+ request=httpx.Request("POST", "https://api.openai.com/v1/responses"),+ body={"type": "service_unavailable_error", "code": "server_is_overloaded"},+ )++ for previous_response_id, replay_safety in ((None, None), ("resp_prev", "safe")):+ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=True,+ previous_response_id=previous_response_id,+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety == replay_safety+ assert advice.normalized is not None+ assert advice.normalized.status_code is None+ assert advice.normalized.error_code == "server_is_overloaded"+++async def _run_websocket_overload_with_retry(+ monkeypatch: pytest.MonkeyPatch,+ *,+ model_settings: ModelSettings,+ previous_response_id: str | None,+) -> tuple[Any, list[DummyWSConnection]]:+ client = DummyWSClient()+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=client) # type: ignore[arg-type]+ queue = [+ DummyWSConnection([json.dumps(_server_overloaded_error_payload())]),+ DummyWSConnection([_response_completed_frame("resp-2", 1)]),+ ]+ opened: list[DummyWSConnection] = []++ async def fake_open(+ ws_url: str, headers: dict[str, str], *, connect_timeout: float | None = None+ ) -> DummyWSConnection:+ connection = queue.pop(0)+ opened.append(connection)+ return connection++ monkeypatch.setattr(model, "_open_websocket_connection", fake_open)++ async def get_response() -> Any:+ return await model.get_response(+ system_instructions=None,+ input="hi",+ model_settings=model_settings,+ tools=[],+ output_schema=None,+ handoffs=[],+ tracing=ModelTracing.DISABLED,+ previous_response_id=previous_response_id,+ )++ async def rewind() -> None:+ return None++ async def no_sleep(delay: float) -> None:+ return None++ monkeypatch.setattr(model_retry_module, "_sleep_for_retry", no_sleep)++ retry_settings = ModelRetrySettings(+ max_retries=2,+ policy=retry_policies.any(+ retry_policies.provider_suggested(),+ retry_policies.http_status([429, 500, 502, 503, 504]),+ ),+ )+ response = await model_retry_module.get_response_with_retry(+ get_response=get_response,+ rewind=rewind,+ retry_settings=retry_settings,+ get_retry_advice=model.get_retry_advice,+ previous_response_id=previous_response_id,+ conversation_id=None,+ )+ return response, opened+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+@pytest.mark.parametrize("previous_response_id", [None, "resp_stored"])+async def test_websocket_model_retries_pre_event_server_overloaded_error(+ monkeypatch: pytest.MonkeyPatch, previous_response_id: str | None+) -> None:+ response, opened = await _run_websocket_overload_with_retry(+ monkeypatch,+ model_settings=ModelSettings(),+ previous_response_id=previous_response_id,+ )++ assert response.response_id == "resp-2"+ assert len(opened) == 2+ assert all(len(connection.sent_messages) == 1 for connection in opened)+++@pytest.mark.allow_call_model_methods+@pytest.mark.asyncio+async def test_websocket_model_does_not_replay_store_false_continuation_after_overload(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ with pytest.raises(ResponsesWebSocketError) as exc_info:+ await _run_websocket_overload_with_retry(+ monkeypatch,+ model_settings=ModelSettings(store=False),+ previous_response_id="resp_connection_local",+ )++ assert exc_info.value.code == "server_is_overloaded"
src/agents/models/openai_responses.pyindex ff17cea4a..55808eda1 100644@@ -1031,6 +1031,15 @@ class OpenAIResponsesWSModel(OpenAIResponsesModel): replay_safety="safe", reason=str(request.error), )+ if (+ isinstance(request.error, ResponsesWebSocketError)+ and request.error.event_type == "error"+ and request.error.code == "server_is_overloaded"+ ):+ return ModelRetryAdvice(+ suggested=True,+ reason=str(request.error),+ ) return super().get_retry_advice(request) def _get_ws_request_lock(self) -> asyncio.Lock:tests/models/test_openai_responses.pyindex 00c6a4cf5..bd555fdcd 100644@@ -4009,6 +4009,91 @@ def test_websocket_get_retry_advice_allows_stateless_receive_timeout_retry() -> assert advice.replay_safety is None +@pytest.mark.allow_call_model_methods+@pytest.mark.parametrize("previous_response_id", [None, "resp_prev"])+def test_websocket_get_retry_advice_marks_pre_response_overload_retryable(+ previous_response_id: str | None,+) -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(+ {+ "type": "error",+ "error": {+ "type": "service_unavailable_error",+ "code": "server_is_overloaded",+ "message": "Our servers are currently overloaded. Please try again later.",+ },+ }+ )++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ previous_response_id=previous_response_id,+ )+ )++ assert advice is not None+ assert advice.suggested is True+ assert advice.replay_safety is None+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_keeps_partial_overload_unsafe() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(+ {+ "type": "error",+ "error": {+ "type": "service_unavailable_error",+ "code": "server_is_overloaded",+ "message": "Our servers are currently overloaded. Please try again later.",+ },+ }+ )+ setattr(error, "_openai_agents_ws_replay_safety", "unsafe") # noqa: B010+ setattr(error, "_openai_agents_ws_response_started", True) # noqa: B010++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ )+ )++ assert advice is not None+ assert advice.suggested is False+ assert advice.replay_safety == "unsafe"+++@pytest.mark.allow_call_model_methods+def test_websocket_get_retry_advice_ignores_other_pre_response_error_codes() -> None:+ model = OpenAIResponsesWSModel(model="gpt-4", openai_client=cast(Any, DummyWSClient()))+ error = ResponsesWebSocketError(+ {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "code": "invalid_request",+ "message": "Invalid request.",+ },+ }+ )++ advice = model.get_retry_advice(+ ModelRetryAdviceRequest(+ error=error,+ attempt=1,+ stream=False,+ )+ )++ assert advice is None++ def test_get_client_disables_provider_managed_retries_when_requested() -> None: class DummyClient: def __init__(self):