Backtestsopenai-agents-pythonrun-20261024-184640#5023
BackendSpanExporter drops the whole batch on HTTP 429 instead of retrying with Retry-After
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause the same way: 408/409/429 join the retry path, Retry-After / retry-after-ms are parsed with the existing agents.models._retry_runtime helpers, and the wait is bounded by max_delay. The independent tests confirm both deliver the batch on retry, stop at max_retries, honour both headers, cap large values, and leave other 4xx non-retried. The difference is scope and craft. Patch B is a tight change: a module constant, a small helper that reuses the existing header_lookup utility (case-insensitive, None-safe), and `max(backoff, min(retry_after, max_delay))` which matches the issue's wording exactly while leaving the existing backoff/jitter untouched. Its tests cover retry, give-up at max_retries, header precedence (ms over seconds), the max_delay bound, and the deadline interaction (a 30s Retry-After against a 5s drain deadline is cut at the deadline), all of which fail on base. Parsing Retry-After on 5xx too is a small, low-risk extension. Patch A does the core fix correctly but bundles unrelated work: it introduces new `x-should-retry: false` semantics that change existing 5xx behaviour (a 503 with that header is now dropped instead of retried), rewrites `_sleep_before_retry` in a way that is behaviourally equivalent to the old code (pure churn in the deadline path), adds a timed-shutdown drain test that would already pass on base since the deadline branch never consulted the shutdown event, and edits an existing test's patch target for no functional reason. The lazy import with a 'must not become an import-time dependency' comment is unsupported given B's top-level import works in the harness. A's `min(max(sleep, retry_after), max_delay)` also silently clips jitter above max_delay only when a header is present, an inconsistent side effect. A's tests are fine (including unparsable/negative cases), but the extra behaviour change on 5xx and the unrelated refactor add risk and obscure the fix, so B is the better patch.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
6m 38s
Wall-clock time in the container
Full test suite
No new failures
78 passed · 0 skipped · 78 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
123vs 227 merged
Changed lines, ours vs merged
The issue
View on GitHubPlease read this first
- Have you read the docs? Yes: Tracing and the
BackendSpanExporterreference. - Have you searched for related issues? Yes. #4683 / #4711 cover retries being disabled after shutdown, #4983 covers a non-serializable value dropping a batch, #5008 covers the endpoint. None covers how the exporter classifies HTTP status codes.
Describe the bug
BackendSpanExporter._export_with_deadline() treats every 4xx response as final:
# If the response is a client error (4xx), we won't retry
if 400 <= response.status_code < 500:
logger.error("[non-fatal] Tracing client error %s: %s", ...)
break
That includes 429 Too Many Requests. A single rate-limited response from /v1/traces/ingest discards the whole batch, up to max_batch_size (128) traces and spans, with no retry and without looking at Retry-After. The run itself succeeds, so the only symptom is a gap in the dashboard and one [non-fatal] log line. The same applies to 408 and 409, which the OpenAI Python client (_base_client._should_retry) retries together with 429 and 5xx; the exporter only mirrors the 5xx half of that policy.
The export path already has everything needed to retry: max_retries, base_delay, max_delay, the backoff loop, and agents.models._retry_runtime.parse_retry_after_value / parse_retry_after_ms for the header.
Debug information
- Agents SDK version:
mainatfbf59a40(also 0.22.0) - Python version: 3.10.12
- Operating system: Linux
- Model and model provider: none needed;
httpx2.Clientis patched - Does the issue reproduce with the latest Agents SDK release? Yes
- Does the issue occur consistently or intermittently? Consistently for any 429
ERROR:openai.agents:[non-fatal] Tracing client error 429. Response data is redacted.
post calls: 1
Repro steps
import logging
from unittest.mock import MagicMock, patch
from agents.tracing.processors import BackendSpanExporter
from agents.tracing.spans import SpanImpl
from agents.tracing.span_data import CustomSpanData
logging.basicConfig(level=logging.WARNING)
with patch("httpx2.Client") as client:
limited = MagicMock(status_code=429, headers={"retry-after": "1"})
limited.text = '{"error": {"message": "Rate limit reached"}}'
ok = MagicMock(status_code=200)
client.return_value.post.side_effect = [limited, ok]
exporter = BackendSpanExporter(api_key="sk-test", max_retries=3, base_delay=0.1)
span = SpanImpl(
trace_id="trace_1",
span_id="span_1",
parent_id=None,
processor=MagicMock(),
span_data=CustomSpanData(name="probe", data={}),
tracing_api_key=None,
)
exporter.export([span])
print("post calls:", client.return_value.post.call_count) # 1: the batch is gone
Expected behavior
408, 409 and 429 are retried with the existing backoff up to max_retries, waiting at least the advertised Retry-After / retry-after-ms (bounded by max_delay), the way the OpenAI client and the SDK's own model retry helpers already treat these statuses. Other 4xx responses stay non-retried. In the script above the second call succeeds and post calls is 2.
A fix with regression tests is ready and I will attach it right away.
The two patches
src/agents/tracing/processors.pyindex 95998353..236e7818 100644@@ -19,6 +19,11 @@ from ..logger import ( log_model_and_tool_action_warning, logger, )+from ..models._retry_runtime import (+ header_lookup,+ parse_retry_after_ms,+ parse_retry_after_value,+) from .processor_interface import TracingExporter, TracingProcessor from .spans import Span from .traces import Trace@@ -41,6 +46,11 @@ class ConsoleSpanExporter(TracingExporter): print(f"[Exporter] Export span: {item.export()}") +# Client error statuses that are transient and worth retrying, matching the OpenAI client's+# retry policy (request timeout, conflict, rate limit).+_RETRYABLE_CLIENT_ERROR_STATUS_CODES = frozenset({408, 409, 429})++ class BackendSpanExporter(TracingExporter): _OPENAI_TRACING_INGEST_ENDPOINT = "https://api.openai.com/v1/traces/ingest" _OPENAI_TRACING_MAX_FIELD_BYTES = 100_000@@ -168,6 +178,7 @@ class BackendSpanExporter(TracingExporter): attempt = 0 delay = self.base_delay while True:+ retry_after: float | None = None request_timeout = self._timeout_for_deadline(deadline) if deadline is not None and request_timeout is None: logger.warning(@@ -191,8 +202,15 @@ class BackendSpanExporter(TracingExporter): logger.debug("Exported %s items", len(grouped)) break - # If the response is a client error (4xx), we won't retry- if 400 <= response.status_code < 500:+ # Transient client errors (408, 409, 429) are retried like the OpenAI+ # client does; other client errors (4xx) are final.+ if response.status_code in _RETRYABLE_CLIENT_ERROR_STATUS_CODES:+ retry_after = self._retry_after_from_response(response)+ logger.warning(+ "[non-fatal] Tracing: client error %s, retrying.",+ response.status_code,+ )+ elif 400 <= response.status_code < 500: if _debug.DONT_LOG_MODEL_DATA or _debug.DONT_LOG_TOOL_DATA: logger.error( "[non-fatal] Tracing client error %s. Response data is redacted.",@@ -205,11 +223,13 @@ class BackendSpanExporter(TracingExporter): response.text, ) break-- # For 5xx or other unexpected codes, treat it as transient and retry- logger.warning(- "[non-fatal] Tracing: server error %s, retrying.", response.status_code- )+ else:+ # For 5xx or other unexpected codes, treat it as transient and retry+ retry_after = self._retry_after_from_response(response)+ logger.warning(+ "[non-fatal] Tracing: server error %s, retrying.",+ response.status_code,+ ) except httpx2.RequestError as exc: # Network or other I/O error, we'll retry log_model_and_tool_action_warning(@@ -225,10 +245,20 @@ class BackendSpanExporter(TracingExporter): # Exponential backoff + jitter sleep_time = delay + random.uniform(0, 0.1 * delay) # 10% jitter+ if retry_after is not None:+ # Wait at least as long as the server asked, bounded by max_delay.+ sleep_time = max(sleep_time, min(retry_after, self.max_delay)) if not self._sleep_before_retry(sleep_time, deadline): break delay = min(delay * 2, self.max_delay) + def _retry_after_from_response(self, response: Any) -> float | None:+ headers = getattr(response, "headers", None)+ retry_after = parse_retry_after_ms(header_lookup(headers, "retry-after-ms"))+ if retry_after is not None:+ return retry_after+ return parse_retry_after_value(header_lookup(headers, "retry-after"))+ def _timeout_for_deadline(self, deadline: float | None) -> httpx2.Timeout | None: if deadline is None: return Nonetests/test_trace_processor.pyindex 6182f9d2..29dd827e 100644@@ -564,6 +564,85 @@ def test_backend_span_exporter_4xx_client_error(mock_client, monkeypatch, caplog exporter.close() +@patch("httpx2.Client")+@pytest.mark.parametrize("status_code", [408, 409, 429])+def test_backend_span_exporter_retries_transient_client_errors(mock_client, status_code: int):+ limited = MagicMock(status_code=status_code, headers={})+ ok = MagicMock(status_code=200)+ mock_client.return_value.post.side_effect = [limited, ok]++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1)+ with patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry:+ exporter.export([get_span(mock_processor())])++ # The batch is delivered on the second attempt instead of being dropped.+ assert mock_client.return_value.post.call_count == 2+ assert wait_for_retry.call_count == 1+ exporter.close()+++@patch("httpx2.Client")+def test_backend_span_exporter_429_gives_up_after_max_retries(mock_client):+ mock_client.return_value.post.return_value = MagicMock(status_code=429, headers={})++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1)+ with patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry:+ exporter.export([get_span(mock_processor())])++ assert mock_client.return_value.post.call_count == 3+ assert wait_for_retry.call_count == 2+ exporter.close()+++@patch("httpx2.Client")+@pytest.mark.parametrize(+ "headers, expected_wait",+ [+ ({"retry-after": "2"}, 2.0),+ ({"retry-after-ms": "1500"}, 1.5),+ ({"retry-after-ms": "1500", "retry-after": "5"}, 1.5),+ # Retry-After is bounded by max_delay.+ ({"retry-after": "120"}, 10.0),+ ],+)+def test_backend_span_exporter_429_honors_retry_after(+ mock_client, headers: dict[str, str], expected_wait: float+):+ limited = MagicMock(status_code=429, headers=headers)+ ok = MagicMock(status_code=200)+ mock_client.return_value.post.side_effect = [limited, ok]++ exporter = BackendSpanExporter(+ api_key="test_key", max_retries=3, base_delay=0.1, max_delay=10.0+ )+ with patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry:+ exporter.export([get_span(mock_processor())])++ assert mock_client.return_value.post.call_count == 2+ wait_for_retry.assert_called_once()+ assert wait_for_retry.call_args.args[0] == pytest.approx(expected_wait)+ exporter.close()+++@patch("httpx2.Client")+def test_backend_span_exporter_429_retry_after_respects_deadline(mock_client):+ limited = MagicMock(status_code=429, headers={"retry-after": "30"})+ mock_client.return_value.post.return_value = limited++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1)+ with patch("time.sleep") as sleep_for_retry:+ exporter._export_with_deadline(+ [get_span(mock_processor())], deadline=time.monotonic() + 5.0+ )++ # Retry-After (30s) exceeds the remaining deadline, so the wait is cut at the deadline+ # instead of retrying after the 0.1s backoff.+ assert mock_client.return_value.post.call_count == 1+ sleep_for_retry.assert_called_once()+ assert 4.0 < sleep_for_retry.call_args.args[0] <= 5.0+ exporter.close()++ @patch("httpx2.Client") def test_backend_span_exporter_5xx_retry(mock_client): mock_response = MagicMock()
src/agents/tracing/processors.pyindex 959983537..451ac78df 100644@@ -52,6 +52,9 @@ class BackendSpanExporter(TracingExporter): } ) _OPENAI_TRACING_USAGE_SPAN_TYPES = frozenset({"generation"})+ # 4xx statuses that are transient, mirroring the OpenAI client's own retry policy:+ # request timeout, conflict, and rate limiting.+ _RETRYABLE_CLIENT_STATUS_CODES = frozenset({408, 409, 429}) _UNSERIALIZABLE = object() def __init__(@@ -167,7 +170,9 @@ class BackendSpanExporter(TracingExporter): # Exponential backoff loop attempt = 0 delay = self.base_delay+ retry_after: float | None = None while True:+ retry_after = None request_timeout = self._timeout_for_deadline(deadline) if deadline is not None and request_timeout is None: logger.warning(@@ -191,8 +196,18 @@ class BackendSpanExporter(TracingExporter): logger.debug("Exported %s items", len(grouped)) break - # If the response is a client error (4xx), we won't retry- if 400 <= response.status_code < 500:+ allows_retry = self._server_allows_retry(response)+ # A rate limit, request timeout, or conflict is transient: retry it+ # like a server error, waiting at least the advertised Retry-After,+ # unless the server says outright not to retry.+ if response.status_code in self._RETRYABLE_CLIENT_STATUS_CODES and allows_retry:+ retry_after = self._retry_after_seconds(response)+ logger.warning(+ "[non-fatal] Tracing: client error %s, retrying.",+ response.status_code,+ )+ # Any other client error (4xx) won't be retried+ elif 400 <= response.status_code < 500: if _debug.DONT_LOG_MODEL_DATA or _debug.DONT_LOG_TOOL_DATA: logger.error( "[non-fatal] Tracing client error %s. Response data is redacted.",@@ -205,11 +220,18 @@ class BackendSpanExporter(TracingExporter): response.text, ) break-- # For 5xx or other unexpected codes, treat it as transient and retry- logger.warning(- "[non-fatal] Tracing: server error %s, retrying.", response.status_code- )+ elif not allows_retry:+ logger.error(+ "[non-fatal] Tracing: server forbade retry for %s.",+ response.status_code,+ )+ break+ else:+ # For 5xx or other unexpected codes, treat it as transient and retry+ logger.warning(+ "[non-fatal] Tracing: server error %s, retrying.",+ response.status_code,+ ) except httpx2.RequestError as exc: # Network or other I/O error, we'll retry log_model_and_tool_action_warning(@@ -225,10 +247,40 @@ class BackendSpanExporter(TracingExporter): # Exponential backoff + jitter sleep_time = delay + random.uniform(0, 0.1 * delay) # 10% jitter+ if retry_after is not None:+ # Honour the server's wait, bounded by max_delay so a large+ # Retry-After cannot stall the export thread.+ sleep_time = min(max(sleep_time, retry_after), self.max_delay) if not self._sleep_before_retry(sleep_time, deadline): break delay = min(delay * 2, self.max_delay) + def _server_allows_retry(self, response: httpx2.Response) -> bool:+ # Mirror the OpenAI client: an explicit `x-should-retry: false` wins over the+ # status code classification.+ headers = getattr(response, "headers", None)+ if headers is None:+ return True+ should_retry = headers.get("x-should-retry")+ if not isinstance(should_retry, str):+ return True+ return should_retry.strip().lower() != "false"++ def _retry_after_seconds(self, response: httpx2.Response) -> float | None:+ # Imported lazily: the models package pulls in the OpenAI client, which must not+ # become an import-time dependency of tracing.+ from ..models._retry_runtime import parse_retry_after_ms, parse_retry_after_value++ headers = getattr(response, "headers", None)+ if headers is None:+ return None+ retry_after = parse_retry_after_ms(headers.get("retry-after-ms"))+ if retry_after is None:+ retry_after = parse_retry_after_value(headers.get("retry-after"))+ if retry_after is None or not math.isfinite(retry_after):+ return None+ return retry_after+ def _timeout_for_deadline(self, deadline: float | None) -> httpx2.Timeout | None: if deadline is None: return None@@ -241,27 +293,25 @@ class BackendSpanExporter(TracingExporter): return httpx2.Timeout(remaining, connect=connect_timeout) def _sleep_before_retry(self, sleep_time: float, deadline: float | None) -> bool:- if deadline is None:- if self._shutdown_event.wait(sleep_time):- logger.warning(- "[non-fatal] Tracing: shutdown requested during retry backoff, giving up."- )- return False- return not self._shutdown_event.is_set()-- remaining = deadline - time.monotonic()- if remaining <= 0:+ remaining = None if deadline is None else deadline - time.monotonic()+ if remaining is not None and remaining <= 0: logger.warning("[non-fatal] Tracing: export deadline reached before retry, giving up.") return False - if sleep_time >= remaining:- time.sleep(remaining)+ wait_for = sleep_time if remaining is None else min(sleep_time, remaining)+ if deadline is None and self._shutdown_event.wait(wait_for): logger.warning(- "[non-fatal] Tracing: export deadline reached during retry backoff, giving up."+ "[non-fatal] Tracing: shutdown requested during retry backoff, giving up." ) return False-- time.sleep(sleep_time)+ if deadline is not None:+ # The final drain runs after shutdown is requested; its deadline bounds retries.+ time.sleep(wait_for)+ if remaining is not None and (sleep_time >= remaining or time.monotonic() >= deadline):+ logger.warning(+ "[non-fatal] Tracing: export deadline reached during retry backoff, giving up."+ )+ return False return True def _should_sanitize_for_openai_tracing_api(self) -> bool:tests/test_trace_processor.pyindex 6182f9d2b..aed7536f5 100644@@ -284,6 +284,22 @@ def test_batch_trace_processor_shutdown_passes_deadline_to_exporter() -> None: assert seen_deadlines[0] is not None +@patch("httpx2.Client")+def test_batch_trace_processor_timed_shutdown_retries_final_drain(mock_client) -> None:+ transient = MagicMock(status_code=503, headers={})+ success = MagicMock(status_code=200, headers={})+ mock_client.return_value.post.side_effect = [transient, success]++ exporter = BackendSpanExporter(api_key="test_key", max_retries=2, base_delay=0.001)+ processor = BatchTraceProcessor(exporter=exporter)+ processor._queue.put_nowait(get_span(processor))++ processor.shutdown(timeout=1.0)++ assert mock_client.return_value.post.call_count == 2+ exporter.close()++ def test_batch_trace_processor_survives_exporter_exception(): """A failing exporter must not kill the background worker thread. @@ -564,6 +580,101 @@ def test_backend_span_exporter_4xx_client_error(mock_client, monkeypatch, caplog exporter.close() +@patch("httpx2.Client")+@pytest.mark.parametrize("status_code", [408, 409, 429])+def test_backend_span_exporter_retries_transient_client_errors(mock_client, status_code: int):+ """A rate limit, timeout, or conflict is transient, as the OpenAI client treats it;+ dropping the batch on the first response loses every trace in it."""+ mock_response = MagicMock()+ mock_response.status_code = status_code+ mock_response.headers = {}+ mock_client.return_value.post.return_value = mock_response++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2)+ with patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry:+ exporter.export([get_span(mock_processor())])++ assert mock_client.return_value.post.call_count == 3+ assert wait_for_retry.call_count == 2+ exporter.close()+++@patch("httpx2.Client")+def test_backend_span_exporter_rate_limit_obeys_x_should_retry_false(mock_client, caplog):+ """An explicit `x-should-retry: false` wins over the status classification, as in the+ OpenAI client."""+ mock_response = MagicMock()+ mock_response.status_code = 429+ mock_response.headers = {"x-should-retry": "false", "retry-after": "1"}+ mock_response.text = "rate limited"+ mock_client.return_value.post.return_value = mock_response++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2)+ with (+ patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry,+ caplog.at_level(logging.ERROR, logger="openai.agents"),+ ):+ exporter.export([get_span(mock_processor())])++ mock_client.return_value.post.assert_called_once()+ wait_for_retry.assert_not_called()+ assert "Tracing client error 429" in caplog.text+ exporter.close()+++@patch("httpx2.Client")+def test_backend_span_exporter_rate_limit_recovers_after_retry(mock_client):+ ok_response = MagicMock()+ ok_response.status_code = 200+ limited_response = MagicMock()+ limited_response.status_code = 429+ limited_response.headers = {}+ mock_client.return_value.post.side_effect = [limited_response, ok_response]++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2)+ with patch.object(exporter._shutdown_event, "wait", return_value=False):+ exporter.export([get_span(mock_processor())])++ assert mock_client.return_value.post.call_count == 2+ exporter.close()+++@patch("httpx2.Client")+@pytest.mark.parametrize(+ ("headers", "expected_wait"),+ [+ ({"retry-after": "3"}, 3.0),+ ({"retry-after-ms": "1500"}, 1.5),+ # Larger than max_delay: bounded so the export thread cannot stall.+ ({"retry-after": "600"}, 5.0),+ # Unparsable or negative values fall back to exponential backoff.+ ({"retry-after": "soon"}, None),+ ({"retry-after": "-1"}, None),+ ],+)+def test_backend_span_exporter_rate_limit_honours_retry_after(+ mock_client, headers: dict[str, str], expected_wait: float | None+):+ limited_response = MagicMock()+ limited_response.status_code = 429+ limited_response.headers = headers+ ok_response = MagicMock()+ ok_response.status_code = 200+ mock_client.return_value.post.side_effect = [limited_response, ok_response]++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=5.0)+ with patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry:+ exporter.export([get_span(mock_processor())])++ assert mock_client.return_value.post.call_count == 2+ waited = wait_for_retry.call_args.args[0]+ if expected_wait is None:+ assert 0.1 <= waited <= 0.11+ else:+ assert waited == pytest.approx(expected_wait)+ exporter.close()++ @patch("httpx2.Client") def test_backend_span_exporter_5xx_retry(mock_client): mock_response = MagicMock()@@ -583,6 +694,26 @@ def test_backend_span_exporter_5xx_retry(mock_client): exporter.close() +@patch("httpx2.Client")+def test_backend_span_exporter_5xx_obeys_x_should_retry_false(mock_client, caplog):+ mock_response = MagicMock()+ mock_response.status_code = 503+ mock_response.headers = {"x-should-retry": "false"}+ mock_client.return_value.post.return_value = mock_response++ exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2)+ with (+ patch.object(exporter._shutdown_event, "wait", return_value=False) as wait_for_retry,+ caplog.at_level(logging.ERROR, logger="openai.agents"),+ ):+ exporter.export([get_span(mock_processor())])++ mock_client.return_value.post.assert_called_once()+ wait_for_retry.assert_not_called()+ assert "server forbade retry for 503" in caplog.text+ exporter.close()++ @patch("httpx2.Client") def test_backend_span_exporter_deadline_stops_during_5xx_retry_backoff(mock_client): mock_response = MagicMock()@@ -590,7 +721,7 @@ def test_backend_span_exporter_deadline_stops_during_5xx_retry_backoff(mock_clie mock_client.return_value.post.return_value = mock_response exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=1.0)- with patch("time.sleep") as sleep_for_retry:+ with patch("agents.tracing.processors.time.sleep") as sleep_for_retry: exporter._export_with_deadline( [get_span(mock_processor())], deadline=time.monotonic() + 0.01 )