Backtestslitellmrun-20261019-033420#38571
[Bug]: Bedrock Realtime hides provider throttles as normal WebSocket closes and lacks disconnect cleanup
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Patch A addresses both halves of the issue. For the throttle signal it adds classify_realtime_failure (status 429 or ThrottlingException/ServiceQuotaExceededException anywhere in the cause chain), emits a Realtime error event with type rate_limit_error and a stable code rate_limit_exceeded, and closes with 1013 instead of 1011/1000; it also remaps ServiceQuotaExceededException from 400 to 429 so the router cools down and fails over, and chains the committed-stream failure so a post-readiness throttle still classifies correctly. For slot cleanup it adds a shielded _release_realtime_session_resources in a finally plus a BaseException guard around pre-call, and makes the limiter idempotent at the root by claiming the stashed acquisition before any await (claim_parallel_slot), so a logging callback and disconnect cleanup racing concurrently cannot double-decrement. Its tests cover every exit path the issue lists (normal, disconnect, pre-readiness throttle, post-readiness failure, task cancellation, cancellation during pre-call), the concurrent double-release race, and router fallback before session.updated. Patch B only handles slot release: it releases in finally but gates it on success/failure stamps, deferring the release to the logging worker rather than making cleanup idempotent, and it does nothing for the client-facing throttle signal (error event stays server_error, close stays 1011, ServiceQuotaExceededException stays a non-retryable 400). The independent tests confirm this: B fails both throttle-classification tests that A passes, while both pass the slot tests. B's asyncio.wait FIRST_COMPLETED change is a reasonable adjacent fix but outside the ask. A is the more complete and more correct fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
1h 22m
Wall-clock time in the container
Full test suite
No new failures
21,320 passed · 26 skipped · 21,358 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
798vs 225 merged
Changed lines, ours vs merged
The issue
View on GitHub[Bug]: Bedrock Realtime hides provider throttles as normal WebSocket closes and lacks disconnect cleanup
Check for existing issues
- I searched the existing issues and did not find one covering Bedrock Nova Sonic throttling and request-slot cleanup on
/v1/realtime.
What happened?
When Bedrock's regional concurrent-session quota is exhausted, LiteLLM acknowledges the throttled Nova Sonic session as ready, emits no Realtime error event, and closes the client WebSocket with code 1000 and an empty reason.
This makes a capacity rejection indistinguishable from a successful, intentional session close. The application cannot identify a retryable throttle, apply backoff, fail over before sending audio, or report an actionable error.
There is also no WebSocket-lifecycle cleanup around the proxy's /v1/realtime route. Pre-call processing acquires proxy rate-limit state, including max_parallel_requests slots, but unlike the SSE streaming path the Realtime endpoint has no shielded finally that calls _arelease_max_parallel_requests_on_disconnect. When a WebSocket is cancelled or a provider failure is suppressed before a normal LiteLLM success/failure callback, the slot can remain until the limiter's one-hour TTL prunes it.
The Bedrock handler currently suppresses forwarding failures and then performs a normal close:
# litellm/llms/bedrock/realtime/handler.py
await asyncio.gather(
client_to_bedrock_task,
bedrock_to_client_task,
return_exceptions=True,
)
...
except Exception as e:
verbose_proxy_logger.debug(
"Bedrock to client forwarding ended: %s", e, exc_info=True
)
finally:
await client_ws.close()
Because client_ws.close() has no error code or reason, Starlette sends a normal WebSocket close. Because the forwarding exception does not escape, the router cannot classify the attempt as throttled, update deployment health/cooldown state, or attempt an ordered deployment fallback before the session is committed.
Expected behavior:
- A Bedrock throttling or service-quota exception before provider readiness is propagated to the router as one logical failure.
- If another deployment is configured, LiteLLM attempts that deployment before acknowledging the session as ready.
- If no fallback is available, the client receives a Realtime
errorevent containing a stable, retryable throttle classification, followed by an error close such as1013(try again later) or1011, not1000. session.updatedis emitted only after the provider has accepted the session configuration and is ready to receive audio.- Every WebSocket exit path releases exactly the proxy concurrency slot acquired for that session: normal completion, client disconnect, provider failure, cancellation, and pre-readiness fallback.
- Cleanup is idempotent so a logging callback and disconnect cleanup cannot double-decrement the limiter.
User flow
- Twenty clients establish Nova Sonic sessions in an AWS account/region whose concurrent-request quota is 20.
- A twenty-first client connects to LiteLLM and sends
session.update. - LiteLLM sends
session.createdandsession.updated, so the client reasonably begins sending audio. - Bedrock rejects that session because capacity is exhausted.
- LiteLLM emits no
errorevent and closes with1000, reason"". - The client cannot distinguish this from a clean end-of-session and has no reliable throttle signal on which to base retry or failover behavior.
After a fix, step 3 does not occur until upstream readiness is established. A rejected attempt either falls back before audio is accepted or produces an explicit retryable error and non-normal close, and its concurrency slot is released immediately.
Proof the bug occurs
The reproduction used the deployed LiteLLM proxy, a team-scoped virtual key, and real Bedrock Nova 2 Sonic sessions. No mock provider was used.
Environment:
LiteLLM: v1.98.0
Commit: d8f71d7bdbd7c9873d98293f83d64c6db72847e6
Endpoint: /v1/realtime
Provider/model: Bedrock amazon.nova-2-sonic-v1:0
Region: us-east-1
Proxy replicas: 4
AWS Bedrock quota code: L-995EBD81
AWS concurrent-session quota: 20
The test opened 21 concurrent sessions and continued reading each socket for 10 seconds after session.updated.
Observed client result:
21/21 sessions received session.updated
20/21 sessions remained connected for the hold period
1/21 sessions closed after readiness
close code: 1000
close reason: ""
Realtime error event: none
CloudWatch for the exact test minute showed:
AWS/Bedrock Invocations = 20
AWS/Bedrock InvocationThrottles = 1
The alternate us-west-2 deployment received no invocation, despite being registered under the same model group with lower routing priority.
The observed normal close follows directly from the handler source quoted above. The cleanup concern follows from the proxy lifecycle:
common_processing_pre_call_logic()runs before routing and invokes the proxy limiter hooks.- The v3 parallel limiter stores each acquired slot and gives it a 3,600-second TTL.
- LiteLLM already provides
ProxyLogging._arelease_max_parallel_requests_on_disconnect()and uses it from shielded SSE cleanup when no logging callback owns the release. - The
/v1/realtimeroute has onlytry/exceptaround routing; it has no correspondingfinallycleanup. - The Bedrock forwarding coroutines suppress exceptions, so the outer route can return without a failure callback that would otherwise release/account for the request.
Suggested regression coverage:
- Make the Bedrock duplex output raise a throttling exception while applying the first
session.update. - Assert
session.updatedhas not been sent. - With a second deployment configured, assert the exception reaches the router and the retained configuration is attempted there.
- Without a fallback, assert a Realtime
errorevent and non-normal close are sent with a retryable throttle classification. - Configure a virtual key with
max_parallel_requests=1, connect and disconnect normally, and assert another session can immediately acquire the slot. - Repeat step 5 for client cancellation, pre-readiness throttle, provider failure after readiness, and proxy task cancellation.
- Assert every acquired slot is released exactly once and no path waits for the 3,600-second TTL.
- Assert CloudWatch/provider throttles become LiteLLM deployment failures rather than successful sessions.
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on?
v1.98.0
The two patches
litellm/litellm_core_utils/realtime_errors.pyindex 3c064728a6..49f7d9101d 100644@@ -9,15 +9,70 @@ frame itself fail, which is how a loud failure turns back into a silent one. """ import json+from dataclasses import dataclass from typing import Final from litellm.types.realtime import RealtimeErrorDetail, RealtimeErrorEvent WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123 +# RFC 6455 1013 "Try Again Later": the server is overloaded, so the client should retry with backoff.+WEBSOCKET_CLOSE_TRY_AGAIN_LATER: Final = 1013+WEBSOCKET_CLOSE_INTERNAL_ERROR: Final = 1011 -def realtime_error_event(message: str, error_type: str) -> str:- detail: Final[RealtimeErrorDetail] = {"type": error_type, "message": message}+_THROTTLE_STATUS_CODE: Final = 429+_THROTTLE_EXCEPTION_NAMES: Final = frozenset({"ThrottlingException", "ServiceQuotaExceededException"})+_MAX_CAUSE_DEPTH: Final = 8+++@dataclass(frozen=True, slots=True)+class RealtimeFailureClass:+ """How a failed realtime session is reported to the client: error event fields plus close frame."""++ error_type: str+ code: str | None+ close_code: int+ fallback_reason: str+++REALTIME_THROTTLE_FAILURE: Final = RealtimeFailureClass(+ error_type="rate_limit_error",+ code="rate_limit_exceeded",+ close_code=WEBSOCKET_CLOSE_TRY_AGAIN_LATER,+ fallback_reason="Provider throttled the session; try again later",+)+REALTIME_SERVER_FAILURE: Final = RealtimeFailureClass(+ error_type="server_error",+ code=None,+ close_code=WEBSOCKET_CLOSE_INTERNAL_ERROR,+ fallback_reason="Internal server error",+)+++def is_throttle_error(error: BaseException) -> bool:+ """True when the error, or any error it was raised from, is a provider throttle / capacity rejection."""+ current: BaseException | None = error+ for _ in range(_MAX_CAUSE_DEPTH): # bounded: a raise-from chain can be made cyclic+ if current is None:+ return False+ if getattr(current, "status_code", None) == _THROTTLE_STATUS_CODE:+ return True+ if type(current).__name__ in _THROTTLE_EXCEPTION_NAMES:+ return True+ current = current.__cause__+ return False+++def classify_realtime_failure(error: BaseException) -> RealtimeFailureClass:+ return REALTIME_THROTTLE_FAILURE if is_throttle_error(error) else REALTIME_SERVER_FAILURE+++def realtime_error_event(message: str, error_type: str, code: str | None = None) -> str:+ detail: Final[RealtimeErrorDetail] = (+ {"type": error_type, "message": message}+ if code is None+ else {"type": error_type, "code": code, "message": message}+ ) event: Final[RealtimeErrorEvent] = {"type": "error", "error": detail} return json.dumps(event) litellm/llms/bedrock/realtime/handler.pyindex 7ab5a14dfc..e90bbc65c7 100644@@ -47,7 +47,9 @@ _BEDROCK_STREAM_ERROR_STATUS: Final[Mapping[str, int]] = MappingProxyType( "ModelStreamErrorException": 424, "ModelTimeoutException": 408, "ResourceNotFoundException": 404,- "ServiceQuotaExceededException": 400,+ # Nova Sonic reports an exhausted concurrent-session quota this way: a retryable capacity rejection,+ # so it must reach the router as a 429 to cool the deployment down and fail over, not as a 400.+ "ServiceQuotaExceededException": 429, "ServiceUnavailableException": 503, "ThrottlingException": 429, "ValidationException": 400,@@ -142,13 +144,18 @@ def _pending_session_update(scope: Mapping[str, object]) -> str | None: """A fallback attempt on the same websocket replays the session.update the failed attempt never acked.""" if scope.get(BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY) is True: committed_failure: Final = scope.get(BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY)+ # Chained so the committed stream's own failure (e.g. a throttle) still classifies the client's error close. raise BedrockError( status_code=400, message=( "Bedrock realtime session already committed to a provider stream; it cannot be replayed"- + (f". The committed stream failed with: {committed_failure}" if committed_failure else "")+ + (+ f". The committed stream failed with: {_redact_string(str(committed_failure))}"+ if committed_failure+ else ""+ ) ),- )+ ) from (committed_failure if isinstance(committed_failure, BaseException) else None) pending: Final = scope.get(BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY) return pending if isinstance(pending, str) else None @@ -157,7 +164,7 @@ def _raise_provider_failure(scope: MutableMapping[str, object], failure: BaseExc error: Final = _as_bedrock_error(failure) verbose_proxy_logger.error("Bedrock Realtime: provider stream failed: %s", _redact_string(str(error))) if scope.get(BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY) is True:- scope[BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY] = _redact_string(str(error))+ scope[BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY] = error raise error from failure litellm/llms/custom_httpx/llm_http_handler.pyindex 2fe4130a31..411fa0c247 100644@@ -30,7 +30,11 @@ from litellm.litellm_core_utils.audio_utils.subtitle_utils import ( ) from litellm.litellm_core_utils.get_litellm_params import AWS_CREDENTIAL_KWARGS_KEYS from litellm.litellm_core_utils.llm_request_utils import serialize_multipart_form_fields-from litellm.litellm_core_utils.realtime_errors import realtime_error_event, websocket_close_reason+from litellm.litellm_core_utils.realtime_errors import (+ classify_realtime_failure,+ realtime_error_event,+ websocket_close_reason,+) from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming from litellm.litellm_core_utils.url_utils import encode_url_path_segment from litellm.llms.base_llm.anthropic_messages.transformation import (@@ -6320,16 +6324,19 @@ class BaseLLMHTTPHandler: except Exception as e: verbose_logger.exception("Error connecting to backend: %s", e) redacted_error: Final = _redact_string(str(e))+ failure: Final = classify_realtime_failure(e) try:- await websocket.send_text(realtime_error_event(redacted_error, error_type="server_error"))+ await websocket.send_text(+ realtime_error_event(redacted_error, error_type=failure.error_type, code=failure.code)+ ) except Exception: # noqa: BLE001 # best-effort notice: a dead client socket must not skip the close below verbose_logger.debug("Could not send realtime error event to client; closing anyway") try: await websocket.close(- code=1011,+ code=failure.close_code, reason=websocket_close_reason(- _redact_string(f"Internal server error: {e}"),- fallback="Internal server error",+ _redact_string(f"{failure.fallback_reason}: {e}"),+ fallback=failure.fallback_reason, ), ) except RuntimeError as close_error:litellm/proxy/hooks/parallel_request_limiter_v3.pyindex c398abff09..81d24369aa 100644@@ -564,6 +564,19 @@ def claim_request_stash_for_data(data: dict) -> RequestRateLimiterStash: return stash +def claim_parallel_slot(stash: RequestRateLimiterStash | None) -> ParallelSlotAcquisition | None:+ """+ Take ownership of the stashed max_parallel_requests acquisition, clearing it+ before any await so exactly one of the success / failure / disconnect+ callbacks releases it even when they run concurrently on the event loop.+ """+ if stash is None:+ return None+ acquisition: Final = stash.parallel_slot+ stash.parallel_slot = None+ return acquisition++ def get_request_stash_for_call(litellm_call_id: str | None) -> RequestRateLimiterStash | None: stash: Final = _request_stash.get() if stash is None:@@ -4450,13 +4463,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): verbose_proxy_logger.debug("INSIDE parallel request limiter ASYNC SUCCESS LOGGING") stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:+ acquisition: Final = claim_parallel_slot(stash)+ if acquisition is not None: await self._release_parallel_request_slots( acquisition=acquisition, parent_otel_span=litellm_parent_otel_span, )- stash.parallel_slot = None pipeline_operations: Final = self._build_success_event_pipeline_operations( kwargs=kwargs,@@ -4576,13 +4588,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): pipeline_operations: Final[list[RedisPipelineIncrementOperation]] = [] stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:+ acquisition: Final = claim_parallel_slot(stash)+ if acquisition is not None: await self._release_parallel_request_slots( acquisition=acquisition, parent_otel_span=litellm_parent_otel_span, )- stash.parallel_slot = None # Skip the reservation refund if async_post_call_failure_hook # already released it (proxy-level rejection that also bubbles up@@ -4690,15 +4701,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): object's current max_parallel_requests configuration, which can change mid-request) decides whether there is anything to release. """- stash: Final = get_request_stash()- if stash is None or stash.parallel_slot is None:+ acquisition: Final = claim_parallel_slot(get_request_stash())+ if acquisition is None: return await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,+ acquisition=acquisition, parent_otel_span=None, )- stash.parallel_slot = None async def async_post_call_success_hook(self, data: dict, user_api_key_dict: UserAPIKeyAuth, response): """@@ -4774,12 +4784,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash() if stash is None: return- if stash.parallel_slot is not None:+ acquisition: Final = claim_parallel_slot(stash)+ if acquisition is not None: await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,+ acquisition=acquisition, parent_otel_span=user_api_key_dict.parent_otel_span, )- stash.parallel_slot = None if stash.batch_enqueued_reservation is not None: await self.batch_enqueued_token_store.refund(litellm/proxy/proxy_server.pyindex dd7a39eb94..c69b6d0966 100644@@ -294,6 +294,7 @@ from litellm.litellm_core_utils.core_helpers import ( from litellm.litellm_core_utils.credential_accessor import CredentialAccessor from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj from litellm.litellm_core_utils.realtime_errors import (+ classify_realtime_failure, realtime_error_event, websocket_close_reason, )@@ -11899,6 +11900,27 @@ async def _release_realtime_budget_reservation(user_api_key_dict: UserAPIKeyAuth ) +async def _release_realtime_session_resources(user_api_key_dict: UserAPIKeyAuth, *, release_budget: bool) -> None:+ """+ Release what the pre-call phase acquired for this websocket session: the+ max_parallel_requests slot and, unless a success log already settled it,+ the budget reservation.++ Shielded so a cancelled session (client disconnect, proxy shutdown) still+ frees its slot immediately instead of holding it until the limiter's TTL+ prunes it. Idempotent: the limiter claims the stashed acquisition before+ releasing it, so a success/failure callback that already released the slot+ (or races this call) turns the slot release into a no-op.+ """+ with anyio.CancelScope(shield=True):+ try:+ await proxy_logging_obj._arelease_max_parallel_requests_on_disconnect(user_api_key_dict)+ except Exception: # noqa: BLE001 # cleanup must not mask the session's outcome; the slot TTL is the backstop+ verbose_proxy_logger.exception("Could not release realtime max_parallel_requests slot")+ if release_budget:+ await _release_realtime_budget_reservation(user_api_key_dict)++ async def _reject_realtime_session( websocket: WebSocket, user_api_key_dict: UserAPIKeyAuth,@@ -11917,7 +11939,7 @@ async def _reject_realtime_session( verbose_proxy_logger.debug("Could not send realtime pre-call error event to client; closing anyway") await websocket.close(code=code, reason=reason) finally:- await _release_realtime_budget_reservation(user_api_key_dict)+ await _release_realtime_session_resources(user_api_key_dict, release_budget=True) @app.websocket("/openai/v1/realtime")@@ -12021,6 +12043,10 @@ async def realtime_websocket_endpoint( websocket, user_api_key_dict, code=1011, reason="Pre-call error", error_message=str(e) ) return+ except BaseException:+ # Cancelled mid pre-call (client gone, proxy shutdown) after the limiter may already hold a slot.+ await _release_realtime_session_resources(user_api_key_dict, release_budget=True)+ raise # Phase 2: route to upstream LLM. try:@@ -12038,14 +12064,17 @@ async def realtime_websocket_endpoint( except Exception as e: verbose_proxy_logger.exception("Internal server error") redacted_error: Final = _redact_string(str(e))+ failure: Final = classify_realtime_failure(e) try:- await websocket.send_text(realtime_error_event(redacted_error, error_type="server_error"))+ await websocket.send_text(+ realtime_error_event(redacted_error, error_type=failure.error_type, code=failure.code)+ ) except Exception: # noqa: BLE001 # best-effort notice: a dead client socket must not skip the close below verbose_proxy_logger.debug("Could not send realtime error event to client; closing anyway") try: await websocket.close(- code=1011,- reason=websocket_close_reason(redacted_error, fallback="Internal server error"),+ code=failure.close_code,+ reason=websocket_close_reason(redacted_error, fallback=failure.fallback_reason), ) except Exception: # noqa: BLE001 # the lower layer may have closed the socket already; closing twice is not an error verbose_proxy_logger.debug("Could not close realtime client websocket; it is already gone")@@ -12054,8 +12083,10 @@ async def realtime_websocket_endpoint( REALTIME_SESSION_SUCCESS_LOGGED_KEY, ) - if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY):- await _release_realtime_budget_reservation(user_api_key_dict)+ await _release_realtime_session_resources(+ user_api_key_dict,+ release_budget=not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY),+ ) ######################################################################litellm/proxy/response_api_endpoints/endpoints.pyindex 5907ffc64e..247f7877c4 100644@@ -7,6 +7,7 @@ from types import MappingProxyType from typing import TYPE_CHECKING, Any, Final, NamedTuple, Protocol, cast, get_args from uuid import uuid4 +import anyio import fastapi from fastapi import APIRouter, Depends, HTTPException, Request, Response from fastapi.responses import JSONResponse@@ -39,6 +40,7 @@ from litellm.types.responses.main import DeleteResponseResult from litellm.types.utils import TokenCountResponse if TYPE_CHECKING:+ from litellm.proxy.utils import ProxyLogging from litellm.router import Router router: Final = APIRouter()@@ -1412,6 +1414,20 @@ async def _enforce_responses_ws_first_frame_model_auth( ) +async def _release_ws_parallel_request_slot(+ proxy_logging_obj: "ProxyLogging", user_api_key_dict: UserAPIKeyAuth+) -> None:+ """+ Release this websocket session's max_parallel_requests slot; a no-op if a logging callback already did.+ Shielded so a cancelled session still frees its slot instead of holding it until the limiter's TTL.+ """+ with anyio.CancelScope(shield=True):+ try:+ await proxy_logging_obj._arelease_max_parallel_requests_on_disconnect(user_api_key_dict)+ except Exception: # noqa: BLE001 # cleanup must not mask the session's outcome; the slot TTL is the backstop+ verbose_proxy_logger.exception("Could not release responses websocket max_parallel_requests slot")++ @router.websocket("/v1/responses") @router.websocket("/responses") async def responses_websocket_endpoint(@@ -1531,8 +1547,15 @@ async def responses_websocket_endpoint( ) except Exception: pass- await websocket.close(code=1008, reason="Pre-call error")+ try:+ await websocket.close(code=1008, reason="Pre-call error")+ finally:+ await _release_ws_parallel_request_slot(proxy_logging_obj, user_api_key_dict) return+ except BaseException:+ # Cancelled mid pre-call (client gone, proxy shutdown) after the limiter may already hold a slot.+ await _release_ws_parallel_request_slot(proxy_logging_obj, user_api_key_dict)+ raise # Phase 2: route to upstream provider try:@@ -1547,3 +1570,5 @@ async def responses_websocket_endpoint( except Exception: verbose_proxy_logger.exception("Responses WebSocket error") await websocket.close(code=1011, reason="Internal server error")+ finally:+ await _release_ws_parallel_request_slot(proxy_logging_obj, user_api_key_dict)litellm/types/realtime.pyindex 30db794c96..fda8bb71b0 100644@@ -1,7 +1,7 @@ from typing import Any, Literal from pydantic import BaseModel-from typing_extensions import ReadOnly, TypedDict+from typing_extensions import NotRequired, ReadOnly, TypedDict from .llms.openai import ( OpenAIRealtimeEvents,@@ -156,6 +156,7 @@ class RealtimeTranscriptionSessionResponse(BaseModel): class RealtimeErrorDetail(TypedDict): type: ReadOnly[str]+ code: NotRequired[ReadOnly[str]] message: ReadOnly[str] tests/test_litellm/litellm_core_utils/test_realtime_errors.pyindex 1d2cf905f4..2f682eeb9f 100644@@ -52,3 +52,56 @@ def test_websocket_close_reason_truncates_multibyte_message_by_bytes(): ) def test_client_close_code_only_forwards_codes_a_server_may_send(upstream_code, expected): assert client_close_code(upstream_code) == expected+++class ThrottlingException(Exception):+ pass+++class ServiceQuotaExceededException(Exception):+ pass+++@pytest.mark.parametrize(+ "error",+ [+ ThrottlingException("Too many requests"),+ ServiceQuotaExceededException("concurrent sessions exhausted"),+ type("ProviderError", (Exception,), {"status_code": 429})("rate limited"),+ ],+)+def test_classify_realtime_failure_marks_provider_throttles_retryable(error):+ from litellm.litellm_core_utils.realtime_errors import classify_realtime_failure++ failure = classify_realtime_failure(error)++ assert (failure.error_type, failure.code, failure.close_code) == ("rate_limit_error", "rate_limit_exceeded", 1013)+++def test_classify_realtime_failure_follows_raise_from_chain():+ from litellm.litellm_core_utils.realtime_errors import classify_realtime_failure++ try:+ try:+ raise ThrottlingException("Too many requests")+ except ThrottlingException as cause:+ raise RuntimeError("bedrock stream failed") from cause+ except RuntimeError as wrapped:+ assert classify_realtime_failure(wrapped).close_code == 1013+++def test_classify_realtime_failure_keeps_other_failures_as_server_errors():+ from litellm.litellm_core_utils.realtime_errors import classify_realtime_failure++ failure = classify_realtime_failure(RuntimeError("token refresh failed"))++ assert (failure.error_type, failure.code, failure.close_code) == ("server_error", None, 1011)+++def test_realtime_error_event_carries_stable_code_when_given():+ event = json.loads(realtime_error_event("throttled", error_type="rate_limit_error", code="rate_limit_exceeded"))++ assert event == {+ "type": "error",+ "error": {"type": "rate_limit_error", "code": "rate_limit_exceeded", "message": "throttled"},+ }tests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.pyindex 2eadaee9e1..aab7c5c7e6 100644@@ -782,3 +782,94 @@ class TestBedrockRealtimeAwsAuth: if __name__ == "__main__": pytest.main([__file__, "-v"])+++class ThrottlingException(Exception):+ """Named like the modeled AWS SDK error Nova Sonic raises when the account's request rate is throttled"""+++class ServiceQuotaExceededException(Exception):+ """Named like the modeled AWS SDK error Nova Sonic raises when the concurrent-session quota is exhausted"""+++class CapacityRejectedBedrockStream:+ """Duplex stream that Bedrock rejects while applying the first session.update, before readiness"""++ def __init__(self, error):+ self.input_stream = FakeInputStream()+ self._error = error++ async def await_output(self):+ raise self._error+++class TestBedrockRealtimeThrottlePropagation:+ """A capacity rejection must reach the router as a retryable 429 before session.updated is sent"""++ SESSION_UPDATE = json.dumps({"type": "session.update", "session": {"instructions": "hi", "modalities": ["audio"]}})+ AWS_PARAMS = {"aws_region_name": "us-east-1", "aws_access_key_id": "k", "aws_secret_access_key": "s"}++ @pytest.mark.asyncio+ @pytest.mark.parametrize(+ "error",+ [+ ThrottlingException("Too many requests, please wait before trying again."),+ ServiceQuotaExceededException("concurrent session quota L-995EBD81 exhausted"),+ ],+ )+ async def test_pre_readiness_capacity_rejection_escapes_as_retryable_throttle(self, stub_aws_sdk_client, error):+ websocket = ConnectedClientWS([self.SESSION_UPDATE])+ stub_aws_sdk_client["streams"] = [CapacityRejectedBedrockStream(error)]++ with pytest.raises(BedrockError) as failure:+ await BedrockRealtime().async_realtime(+ model="amazon.nova-2-sonic-v1:0", websocket=websocket, logging_obj=FakeLogging(), **self.AWS_PARAMS+ )++ assert failure.value.status_code == 429+ assert litellm._should_retry(failure.value.status_code), "the router must fail over / cool down on a throttle"+ assert type(error).__name__ in failure.value.message+ sent_types = [json.loads(m)["type"] for m in websocket.sent_to_client]+ assert "session.updated" not in sent_types, "the client must not be told the session is ready"+ assert not websocket.closed, "the proxy route owns the client-facing throttle error and close"+++class ThrottledAfterReadinessReceiver(ScriptedBedrockReceiver):+ """Bedrock accepts the session, then throttles it: what the issue's 21st concurrent Nova Sonic session saw"""++ async def receive(self):+ await asyncio.sleep(0)+ raise ThrottlingException("Too many requests, please wait before trying again.")+++class TestBedrockRealtimeCommittedThrottle:+ """A throttle after session.updated cannot fail over, but the client must still learn it was a throttle"""++ SESSION_UPDATE = json.dumps({"type": "session.update", "session": {"instructions": "hi", "modalities": ["audio"]}})+ AWS_PARAMS = {"aws_region_name": "us-east-1", "aws_access_key_id": "k", "aws_secret_access_key": "s"}++ @pytest.mark.asyncio+ async def test_router_retry_after_committed_throttle_still_classifies_as_throttle(self, stub_aws_sdk_client):+ from litellm.litellm_core_utils.realtime_errors import REALTIME_THROTTLE_FAILURE, classify_realtime_failure++ websocket = ConnectedClientWS([self.SESSION_UPDATE])+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream([], receiver_type=ThrottledAfterReadinessReceiver)]+ handler = BedrockRealtime()++ with pytest.raises(BedrockError) as failure:+ await handler.async_realtime(+ model="amazon.nova-2-sonic-v1:0", websocket=websocket, logging_obj=FakeLogging(), **self.AWS_PARAMS+ )+ assert failure.value.status_code == 429+ assert "session.updated" in [json.loads(m)["type"] for m in websocket.sent_to_client]++ # The router retries the 429 on the same websocket; the committed session refuses the replay, and that+ # refusal is the error the proxy route reports to the client.+ with pytest.raises(BedrockError) as replay:+ await handler.async_realtime(+ model="amazon.nova-2-sonic-v1:0", websocket=websocket, logging_obj=FakeLogging(), **self.AWS_PARAMS+ )++ assert replay.value.status_code == 400+ assert classify_realtime_failure(replay.value) == REALTIME_THROTTLE_FAILURE+ assert "ThrottlingException" in replay.value.messagetests/test_litellm/proxy/realtime_endpoints/test_realtime_websocket_session_cleanup.pynew file mode 100644index 0000000000..67ca1dfe35@@ -0,0 +1,454 @@+"""+/v1/realtime session lifecycle regressions for Bedrock Nova Sonic throttling:++- a provider throttle surfaces as a retryable Realtime ``error`` event plus a+ 1013 close, never as a normal 1000 close;+- every websocket exit path releases exactly the max_parallel_requests slot the+ pre-call hook acquired, so the next session can connect immediately instead of+ waiting for the limiter's slot TTL;+- a pre-readiness throttle on one deployment fails over to the next deployment+ in the model group before the session is acknowledged.+"""++import asyncio+import json+from contextlib import contextmanager+from unittest.mock import AsyncMock, MagicMock, patch++import pytest+from fastapi import HTTPException++from litellm.caching.caching import DualCache+from litellm.llms.bedrock.common_utils import BedrockError+from litellm.proxy._types import UserAPIKeyAuth+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3,+)+from litellm.proxy.utils import InternalUsageCache, hash_token++MODEL = "nova-sonic"+++class RecordingWebSocket:+ def __init__(self):+ self.headers = {}+ self.scope = {"headers": []}+ self.url = "ws://testserver/v1/realtime?model=nova-sonic"+ self.events = []++ async def accept(self, **kwargs):+ self.events.append(("accept", kwargs))++ async def send_text(self, payload):+ self.events.append(("send_text", payload))++ async def close(self, code=1000, reason=None):+ self.events.append(("close", {"code": code, "reason": reason}))++ def closes(self):+ return [payload for name, payload in self.events if name == "close"]++ def error_events(self):+ return [+ json.loads(payload)+ for name, payload in self.events+ if name == "send_text" and json.loads(payload).get("type") == "error"+ ]+++class SlotHarness:+ """A real v3 limiter with max_parallel_requests=1 wired in as the proxy's parallel_request_limiter."""++ def __init__(self):+ self.api_key = hash_token("sk-realtime-session")+ self.cache = DualCache()+ self.limiter = _PROXY_MaxParallelRequestsHandler_v3(internal_usage_cache=InternalUsageCache(self.cache))+ self.user_api_key_dict = UserAPIKeyAuth(api_key=self.api_key, max_parallel_requests=1, models=["*"])+ self.counter_key = f"{{api_key:{self.api_key}}}:max_parallel_requests"+ self.release_calls = 0+ original_release = self.limiter._release_parallel_request_slots++ async def counting_release(*args, **kwargs):+ self.release_calls += 1+ await asyncio.sleep(0) # a real release is a Redis round-trip: let a concurrent releaser interleave+ return await original_release(*args, **kwargs)++ self.limiter._release_parallel_request_slots = counting_release++ async def in_flight(self):+ return self.limiter._gauge_in_flight_from_cache_value(await self.cache.async_get_cache(key=self.counter_key))++ async def pre_call(self, *args, **kwargs):+ data = {"model": MODEL, "litellm_call_id": "realtime-call"}+ await self.limiter.async_pre_call_hook(+ user_api_key_dict=self.user_api_key_dict, cache=self.cache, data=data, call_type="_arealtime"+ )+ return data, MagicMock(model_call_details={})+++@contextmanager+def realtime_endpoint_wiring(harness, llm_call):+ """+ The /v1/realtime route has no injection seam for its pre-call processor,+ router or limiter, so they are swapped at the module boundary. The limiter+ itself is the real v3 implementation: slot accounting is what is asserted.+ """+ from litellm.proxy import proxy_server++ processor = MagicMock()+ processor.common_processing_pre_call_logic = AsyncMock(side_effect=harness.pre_call)++ async def route_request(**kwargs):+ return llm_call()++ with (+ patch( # test-quality-ok: the route has no seam; key-model access is out of scope here+ "litellm.proxy.proxy_server.can_key_call_resolved_model", new=AsyncMock(return_value=None)+ ),+ patch( # test-quality-ok: the route has no seam; pre-call runs the real limiter via harness.pre_call+ "litellm.proxy.proxy_server.ProxyBaseLLMRequestProcessing", return_value=processor+ ),+ patch( # test-quality-ok: the route has no seam; stands in for the provider session lifetime+ "litellm.proxy.proxy_server.route_request", new=route_request+ ),+ patch.object( # test-quality-ok: installs the real v3 limiter as the proxy's parallel_request_limiter+ proxy_server.proxy_logging_obj, "get_proxy_hook", return_value=harness.limiter+ ),+ ):+ yield+++def start_session(harness, websocket):+ """Run one realtime session in its own task, as Starlette does per websocket connection."""+ from litellm.proxy import proxy_server++ return asyncio.create_task(+ proxy_server.realtime_websocket_endpoint(+ websocket=websocket,+ model=MODEL,+ intent=None,+ guardrails=None,+ user_api_key_dict=harness.user_api_key_dict,+ )+ )+++async def run_session(harness, websocket, llm_call):+ with realtime_endpoint_wiring(harness, llm_call):+ await start_session(harness, websocket)+++async def assert_next_session_admitted(harness):+ websocket = RecordingWebSocket()+ reached_provider = []++ async def healthy_session():+ reached_provider.append(True)+ assert await harness.in_flight() == 1, "the new session must hold the only slot while it runs"++ await run_session(harness, websocket, healthy_session)+ assert reached_provider, f"second session was rejected: {websocket.closes()}"+ assert await harness.in_flight() == 0+++@pytest.mark.asyncio+async def test_bedrock_throttle_sends_retryable_error_event_and_try_again_later_close():+ harness = SlotHarness()+ websocket = RecordingWebSocket()++ async def throttled_session():+ raise BedrockError(status_code=429, message="ServiceQuotaExceededException: concurrent sessions exhausted")++ await run_session(harness, websocket, throttled_session)++ errors = websocket.error_events()+ assert len(errors) == 1+ assert errors[0]["error"]["type"] == "rate_limit_error"+ assert errors[0]["error"]["code"] == "rate_limit_exceeded"+ assert "ServiceQuotaExceededException" in errors[0]["error"]["message"]+ assert [close["code"] for close in websocket.closes()] == [1013], "a throttle must not look like a normal close"+ assert websocket.closes()[0]["reason"]+ assert await harness.in_flight() == 0+ assert harness.release_calls == 1+++@pytest.mark.asyncio+async def test_non_throttle_provider_failure_keeps_server_error_close():+ harness = SlotHarness()+ websocket = RecordingWebSocket()++ async def broken_session():+ raise BedrockError(status_code=424, message="ModelStreamErrorException: Nova Sonic stream broke")++ await run_session(harness, websocket, broken_session)++ assert websocket.error_events()[0]["error"]["type"] == "server_error"+ assert "code" not in websocket.error_events()[0]["error"]+ assert [close["code"] for close in websocket.closes()] == [1011]+++async def _normal_completion():+ return None+++async def _client_disconnect():+ from starlette.websockets import WebSocketDisconnect++ raise WebSocketDisconnect(code=1001)+++async def _pre_readiness_throttle():+ raise BedrockError(status_code=429, message="ThrottlingException: Too many requests")+++async def _provider_failure_after_readiness():+ raise BedrockError(status_code=424, message="ModelStreamErrorException: stream broke after session.updated")+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "session",+ [_normal_completion, _client_disconnect, _pre_readiness_throttle, _provider_failure_after_readiness],+ ids=["normal", "client_disconnect", "pre_readiness_throttle", "provider_failure_after_readiness"],+)+async def test_every_session_exit_releases_its_parallel_slot_exactly_once(session):+ harness = SlotHarness()++ await run_session(harness, RecordingWebSocket(), session)++ assert await harness.in_flight() == 0, "the slot must be freed now, not after the limiter TTL"+ assert harness.release_calls == 1+ await assert_next_session_admitted(harness)+++@pytest.mark.asyncio+async def test_cancelled_session_task_releases_its_parallel_slot():+ harness = SlotHarness()+ session_started = asyncio.Event()++ async def long_lived_session():+ session_started.set()+ await asyncio.Event().wait()++ with realtime_endpoint_wiring(harness, long_lived_session):+ task = start_session(harness, RecordingWebSocket())+ await session_started.wait()+ assert await harness.in_flight() == 1+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ assert await harness.in_flight() == 0+ assert harness.release_calls == 1+ await assert_next_session_admitted(harness)+++@pytest.mark.asyncio+async def test_session_cancelled_during_pre_call_releases_its_parallel_slot():+ """Cancelled after the limiter took the slot but before routing (e.g. a slow guardrail, proxy shutdown)."""+ harness = SlotHarness()+ slot_acquired = asyncio.Event()+ acquire_slot = harness.pre_call++ async def pre_call_stuck_in_guardrail(*args, **kwargs):+ await acquire_slot()+ slot_acquired.set()+ await asyncio.Event().wait()++ harness.pre_call = pre_call_stuck_in_guardrail+ with realtime_endpoint_wiring(harness, _normal_completion):+ task = start_session(harness, RecordingWebSocket())+ await slot_acquired.wait()+ assert await harness.in_flight() == 1+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ assert await harness.in_flight() == 0, "the slot must be freed now, not after the limiter TTL"+ assert harness.release_calls == 1+ harness.pre_call = acquire_slot+ await assert_next_session_admitted(harness)+++@pytest.mark.asyncio+async def test_slot_released_by_logging_callback_is_not_released_again_on_disconnect():+ harness = SlotHarness()++ async def session_logged_by_success_callback():+ await harness.limiter.async_log_success_event(+ kwargs={"litellm_call_id": "realtime-call", "standard_logging_object": None},+ response_obj=None,+ start_time=None,+ end_time=None,+ )++ await run_session(harness, RecordingWebSocket(), session_logged_by_success_callback)++ assert harness.release_calls == 1, "callback and disconnect cleanup must not double-decrement the gauge"+ assert await harness.in_flight() == 0+ await assert_next_session_admitted(harness)+++@pytest.mark.asyncio+async def test_concurrent_callback_and_disconnect_release_claim_the_slot_once():+ harness = SlotHarness()++ async def scenario():+ await harness.pre_call()+ assert await harness.in_flight() == 1+ await asyncio.gather(+ harness.limiter.async_log_failure_event(+ kwargs={"litellm_call_id": "realtime-call"}, response_obj=None, start_time=None, end_time=None+ ),+ harness.limiter.async_release_max_parallel_requests_on_disconnect(harness.user_api_key_dict),+ )++ await asyncio.create_task(scenario())++ assert harness.release_calls == 1+ assert await harness.in_flight() == 0+++@pytest.mark.asyncio+async def test_slot_held_by_live_session_rejects_second_session():+ """Sanity check for the harness: the slot really is exclusive while a session runs."""+ harness = SlotHarness()++ async def scenario():+ await harness.pre_call()+ with pytest.raises(HTTPException) as rejected:+ await asyncio.create_task(harness.pre_call())+ assert rejected.value.status_code == 429++ await asyncio.create_task(scenario())+++@pytest.mark.asyncio+async def test_pre_readiness_throttle_fails_over_to_next_deployment_before_ack():+ import litellm.realtime_api.main as realtime_main+ from litellm import Router++ attempts = []++ async def fake_async_realtime(*, model, websocket, aws_region_name=None, **kwargs):+ attempts.append(aws_region_name)+ if aws_region_name == "us-east-1":+ raise BedrockError(status_code=429, message="ServiceQuotaExceededException: concurrent sessions exhausted")+ await websocket.send_text(json.dumps({"type": "session.updated"}))++ router = Router(+ model_list=[+ {+ "model_name": MODEL,+ "litellm_params": {+ "model": "bedrock/amazon.nova-2-sonic-v1:0",+ "aws_region_name": "us-east-1",+ "aws_access_key_id": "k",+ "aws_secret_access_key": "s",+ "order": 1,+ },+ },+ {+ "model_name": MODEL,+ "litellm_params": {+ "model": "bedrock/amazon.nova-2-sonic-v1:0",+ "aws_region_name": "us-west-2",+ "aws_access_key_id": "k",+ "aws_secret_access_key": "s",+ "order": 2,+ },+ },+ ],+ num_retries=1,+ retry_after=0,+ )+ websocket = RecordingWebSocket()++ with patch.object( # test-quality-ok: stands in for Nova Sonic at the provider-handler boundary+ realtime_main.bedrock_realtime, "async_realtime", new=fake_async_realtime+ ):+ await router._arealtime(model=MODEL, websocket=websocket)++ assert attempts == ["us-east-1", "us-west-2"]+ assert [json.loads(p)["type"] for name, p in websocket.events if name == "send_text"] == ["session.updated"]+ assert websocket.closes() == []+++@pytest.mark.asyncio+@pytest.mark.parametrize("fails", [False, True], ids=["normal", "provider_failure"])+async def test_responses_websocket_session_releases_its_parallel_slot(fails):+ """The /v1/responses websocket acquires the same pre-call slot and must free it on exit too."""+ from litellm.proxy import proxy_server+ from litellm.proxy.response_api_endpoints import endpoints++ harness = SlotHarness()++ async def session():+ assert await harness.in_flight() == 1+ if fails:+ raise RuntimeError("upstream responses websocket failed")++ processor = MagicMock()+ processor.common_processing_pre_call_logic = AsyncMock(side_effect=harness.pre_call)++ async def route_request(**kwargs):+ return session()++ with (+ patch.object( # test-quality-ok: the route has no seam; pre-call runs the real limiter via harness.pre_call+ endpoints, "ProxyBaseLLMRequestProcessing", return_value=processor+ ),+ patch( # test-quality-ok: the route has no seam; stands in for the provider session lifetime+ "litellm.proxy.route_llm_request.route_request", new=route_request+ ),+ patch.object( # test-quality-ok: installs the real v3 limiter as the proxy's parallel_request_limiter+ proxy_server.proxy_logging_obj, "get_proxy_hook", return_value=harness.limiter+ ),+ ):+ await asyncio.create_task(+ endpoints.responses_websocket_endpoint(+ websocket=RecordingWebSocket(), model="gpt-4o", user_api_key_dict=harness.user_api_key_dict+ )+ )++ assert await harness.in_flight() == 0+ assert harness.release_calls == 1+++@pytest.mark.asyncio+async def test_responses_websocket_cancelled_during_pre_call_releases_its_parallel_slot():+ from litellm.proxy import proxy_server+ from litellm.proxy.response_api_endpoints import endpoints++ harness = SlotHarness()+ slot_acquired = asyncio.Event()++ async def pre_call_stuck_in_guardrail(*args, **kwargs):+ await harness.pre_call()+ slot_acquired.set()+ await asyncio.Event().wait()++ processor = MagicMock()+ processor.common_processing_pre_call_logic = AsyncMock(side_effect=pre_call_stuck_in_guardrail)++ with (+ patch.object( # test-quality-ok: the route has no seam; pre-call runs the real limiter via harness.pre_call+ endpoints, "ProxyBaseLLMRequestProcessing", return_value=processor+ ),+ patch.object( # test-quality-ok: installs the real v3 limiter as the proxy's parallel_request_limiter+ proxy_server.proxy_logging_obj, "get_proxy_hook", return_value=harness.limiter+ ),+ ):+ task = asyncio.create_task(+ endpoints.responses_websocket_endpoint(+ websocket=RecordingWebSocket(), model="gpt-4o", user_api_key_dict=harness.user_api_key_dict+ )+ )+ await slot_acquired.wait()+ assert await harness.in_flight() == 1+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ assert await harness.in_flight() == 0+ assert harness.release_calls == 1
litellm/constants.pyindex e8aafda1797..c106be688e4 100644@@ -317,6 +317,8 @@ WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123 BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY: Final = "litellm.bedrock_realtime.pending_session_update" BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY: Final = "litellm.bedrock_realtime.session_committed" BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY: Final = "litellm.bedrock_realtime.committed_failure"+REALTIME_SESSION_SUCCESS_LOGGED_KEY: Final = "realtime_session_success_logged"+REALTIME_SESSION_FAILURE_LOGGED_KEY: Final = "realtime_session_failure_logged" # SSL/TLS cipher configuration for faster handshakes # Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported oneslitellm/litellm_core_utils/realtime_streaming.pyindex e3f8786a39a..fa567bdf4c9 100644@@ -10,6 +10,7 @@ from typing_extensions import ReadOnly import litellm from litellm._logging import redact_internal_details_from_client_message, verbose_logger+from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig from litellm.types.llms.openai import (@@ -35,9 +36,6 @@ else: CLIENT_CONNECTION_CLASS = Any -REALTIME_SESSION_SUCCESS_LOGGED_KEY: Final = "realtime_session_success_logged"-- @dataclass(frozen=True, slots=True) class BackendClose: code: int@@ -1153,6 +1151,7 @@ class RealTimeStreaming: self._logging_worker.ensure_initialized_and_enqueue( self.logging_obj.dispatch_failure_handlers(error, traceback.format_exc(), prefer_async_handlers=True) )+ self.logging_obj.model_call_details[REALTIME_SESSION_FAILURE_LOGGED_KEY] = True @staticmethod def _detect_beta_header(websocket: ScopedWebSocket) -> bool:litellm/llms/bedrock/realtime/handler.pyindex 7ab5a14dfc2..2c1ce6068b2 100644@@ -20,6 +20,7 @@ from litellm.constants import ( BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY, BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY, BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY,+ REALTIME_SESSION_SUCCESS_LOGGED_KEY, ) from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging@@ -346,6 +347,7 @@ class BedrockRealtime(BaseAWSLLM): prefer_async_handlers=True, ) )+ logging_obj.model_call_details[REALTIME_SESSION_SUCCESS_LOGGED_KEY] = True if outcome.provider_failure is None: return@@ -382,7 +384,7 @@ class BedrockRealtime(BaseAWSLLM): ) bedrock_task: Final = asyncio.create_task(collect_logged_events()) - await asyncio.wait((client_task, bedrock_task), return_when=asyncio.FIRST_EXCEPTION)+ await asyncio.wait((client_task, bedrock_task), return_when=asyncio.FIRST_COMPLETED) client_disconnected: Final = ( client_task.done() and not client_task.cancelled() and client_task.exception() is None )litellm/proxy/proxy_server.pyindex dd7a39eb94c..1c931863a2f 100644@@ -274,6 +274,8 @@ from litellm.constants import ( PROXY_BUDGET_RESCHEDULER_MAX_TIME, PROXY_BUDGET_RESCHEDULER_MIN_TIME, PROXY_CONFIG_RELOAD_INTERVAL_SECONDS,+ REALTIME_SESSION_FAILURE_LOGGED_KEY,+ REALTIME_SESSION_SUCCESS_LOGGED_KEY, ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG, USER_SPEND_ALERTS_JOB_ID, WEEKLY_SPEND_REPORT_JOB_ID,@@ -11899,6 +11901,13 @@ async def _release_realtime_budget_reservation(user_api_key_dict: UserAPIKeyAuth ) +async def _release_realtime_max_parallel_slot(user_api_key_dict: UserAPIKeyAuth) -> None:+ release_like_http_disconnect: Final = (+ proxy_logging_obj._arelease_max_parallel_requests_on_disconnect # pyright: ignore[reportPrivateUsage] # shared+ )+ await release_like_http_disconnect(user_api_key_dict)++ async def _reject_realtime_session( websocket: WebSocket, user_api_key_dict: UserAPIKeyAuth,@@ -11918,6 +11927,7 @@ async def _reject_realtime_session( await websocket.close(code=code, reason=reason) finally: await _release_realtime_budget_reservation(user_api_key_dict)+ await _release_realtime_max_parallel_slot(user_api_key_dict) @app.websocket("/openai/v1/realtime")@@ -12021,6 +12031,9 @@ async def realtime_websocket_endpoint( websocket, user_api_key_dict, code=1011, reason="Pre-call error", error_message=str(e) ) return+ except BaseException:+ await _release_realtime_max_parallel_slot(user_api_key_dict)+ raise # Phase 2: route to upstream LLM. try:@@ -12050,12 +12063,10 @@ async def realtime_websocket_endpoint( except Exception: # noqa: BLE001 # the lower layer may have closed the socket already; closing twice is not an error verbose_proxy_logger.debug("Could not close realtime client websocket; it is already gone") finally:- from litellm.litellm_core_utils.realtime_streaming import (- REALTIME_SESSION_SUCCESS_LOGGED_KEY,- )- if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY): await _release_realtime_budget_reservation(user_api_key_dict)+ if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_FAILURE_LOGGED_KEY):+ await _release_realtime_max_parallel_slot(user_api_key_dict) ######################################################################tests/test_litellm/litellm_core_utils/test_realtime_streaming.pyindex 2a33d84ec78..7e6d4d24905 100644@@ -10,9 +10,9 @@ from websockets.exceptions import ConnectionClosed from websockets.frames import Close import litellm+from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.integrations.custom_guardrail import CustomGuardrail from litellm.litellm_core_utils.realtime_streaming import (- REALTIME_SESSION_SUCCESS_LOGGED_KEY, RealTimeStreaming, client_sent_openai_beta_realtime_header, )@@ -3399,6 +3399,26 @@ async def test_refused_session_does_not_stamp_the_reservation_ownership_marker() assert REALTIME_SESSION_SUCCESS_LOGGED_KEY not in session.logging.model_call_details +@pytest.mark.asyncio+async def test_refused_session_stamps_the_failure_ownership_marker():+ """LIT-6463: the enqueued failure callback releases the key's max_parallel_requests+ slot from the logging worker, so a refusal stamps REALTIME_SESSION_FAILURE_LOGGED_KEY.+ The proxy endpoint reads it to leave the slot to that callback instead of racing it.+ A session that relayed frames logs a success and must not carry the failure stamp."""+ upstream_close: Final = ConnectionClosed(Close(1008, _UPSTREAM_REFUSAL), None)+ refused: Final = _relay_session(_client_ws_that_never_sends(), _backend_ws_closing_with(upstream_close))+ session_created: Final = json.dumps({"type": "session.created", "session": {"id": "sess_1"}}).encode()+ relayed: Final = _relay_session(+ _client_ws_that_never_sends(), _backend_ws_closing_with(session_created, upstream_close)+ )++ await refused.run()+ await relayed.run()++ assert refused.logging.model_call_details.get(REALTIME_SESSION_FAILURE_LOGGED_KEY) is True+ assert REALTIME_SESSION_FAILURE_LOGGED_KEY not in relayed.logging.model_call_details++ @pytest.mark.asyncio async def test_transformed_transcription_completion_never_sends_response_create(): from typing import Finaltests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.pyindex 2eadaee9e1a..ac3a43b742f 100644@@ -8,6 +8,7 @@ from unittest.mock import MagicMock import pytest import litellm+from litellm.constants import REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.llms.bedrock.common_utils import BedrockError from litellm.llms.bedrock.realtime.handler import BedrockRealtime from litellm.llms.bedrock.realtime.transformation import BedrockRealtimeConfig@@ -77,6 +78,7 @@ class UnavailableBedrockStream: class FakeLogging: def __init__(self, trace_id="trace-nova-sonic"): self.litellm_trace_id = trace_id+ self.model_call_details = {} class DisconnectingClientWS:@@ -673,6 +675,31 @@ class TestBedrockRealtimeProviderFailurePropagation: await spend_dispatch["coro"] assert [event["type"] for event in spend_dispatch["events"]] == ["response.done"] + @pytest.mark.asyncio+ async def test_success_dispatch_stamps_the_ownership_marker_only_when_spend_was_logged(+ self, stub_aws_sdk_client, spend_dispatch+ ):+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream(self.TEXT_TURN)]+ await BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=ConnectedClientWS([self.SESSION_UPDATE]),+ logging_obj=spend_dispatch["logging_obj"],+ **self.AWS_PARAMS,+ )+ await spend_dispatch["coro"]+ assert [event["type"] for event in spend_dispatch["events"]] == ["response.done"]+ assert spend_dispatch["logging_obj"].model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY) is True++ idle_logging = FakeLogging()+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream([])]+ await BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=ConnectedClientWS([self.SESSION_UPDATE]),+ logging_obj=idle_logging,+ **self.AWS_PARAMS,+ )+ assert REALTIME_SESSION_SUCCESS_LOGGED_KEY not in idle_logging.model_call_details+ @pytest.mark.asyncio async def test_stream_failure_after_client_disconnect_is_not_a_provider_failure(self, stub_aws_sdk_client): stream = ScriptedBedrockStream([], receiver_type=BreakingBedrockReceiver)@@ -684,6 +711,25 @@ class TestBedrockRealtimeProviderFailurePropagation: assert stream.input_stream.closed + @pytest.mark.asyncio+ async def test_client_disconnect_ends_the_session_while_bedrock_output_stays_open(self, stub_aws_sdk_client):+ receiver = DrainedThenOpenBedrockReceiver([])+ stream = ScriptedBedrockStream([], receiver_type=lambda _payloads: receiver)+ stub_aws_sdk_client["streams"] = [stream]++ await asyncio.wait_for(+ BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=RealtimeClientWS(),+ logging_obj=FakeLogging(),+ **self.AWS_PARAMS,+ ),+ timeout=1,+ )++ assert receiver.drained.is_set(), "the handler must have been waiting on the open provider stream"+ assert stream.input_stream.closed+ @pytest.mark.asyncio async def test_session_updated_is_not_sent_before_bedrock_is_ready(self, stub_aws_models): handler = BedrockRealtime()tests/test_litellm/proxy/test_proxy_server.pyindex a5e6cf285ad..e09dddfec5b 100644@@ -33,6 +33,7 @@ from litellm.litellm_core_utils.get_model_cost_map import ModelCostMapReloaded from litellm.caching.dual_cache import DualCache from litellm.proxy._types import LitellmUserRoles, TokenCountRequest, UserAPIKeyAuth from litellm.proxy.auth.user_api_key_auth import user_api_key_auth+from litellm.proxy.hooks.parallel_request_limiter_v3 import RequestRateLimiterStash from litellm.proxy.proxy_server import app, initialize from litellm.utils import _invalidate_model_cost_lowercase_map @@ -9999,6 +10000,7 @@ async def _lit6973_drive_realtime_session( reservation: dict, *, backend_logged_success: bool,+ backend_logged_failure: bool = False, phase_one_exit: str | None = None, websocket: MagicMock | None = None, ) -> MagicMock:@@ -10006,8 +10008,9 @@ async def _lit6973_drive_realtime_session( phase_one_exit picks a rejection before the relay: "model_access" makes the key/model check raise ProxyException, "pre_call" makes pre-call processing- (rate limits, guardrails) raise. Neither reaches route_request, so no success- log can own the reservation and the endpoint has to release it on that exit.+ (rate limits, guardrails) raise, "pre_call_cancelled" cancels the task inside+ pre-call processing. None reaches route_request, so no success log can own the+ reservation and the endpoint has to release it on that exit. route_request resolves normally in both cases: the relay owns the session once route_request returns. A successful session enqueues its success cost@@ -10017,7 +10020,7 @@ async def _lit6973_drive_realtime_session( logging object carries a real model_call_details dict so the stamp is observable, and the reservation has empty entries so the real release touches no counter store."""- from litellm.litellm_core_utils.realtime_streaming import REALTIME_SESSION_SUCCESS_LOGGED_KEY+ from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.proxy import proxy_server as ps user_api_key_dict: Final = UserAPIKeyAuth(api_key="sk-test", token="hashed-token")@@ -10029,6 +10032,8 @@ async def _lit6973_drive_realtime_session( async def fake_llm_call() -> None: if backend_logged_success: logging_obj.model_call_details[REALTIME_SESSION_SUCCESS_LOGGED_KEY] = True+ if backend_logged_failure:+ logging_obj.model_call_details[REALTIME_SESSION_FAILURE_LOGGED_KEY] = True from litellm.proxy._types import ProxyException @@ -10037,7 +10042,13 @@ async def _lit6973_drive_realtime_session( if phase_one_exit == "model_access" else None )- pre_call_error: Final = Exception("Rate limit exceeded") if phase_one_exit == "pre_call" else None+ pre_call_error: Final = (+ asyncio.CancelledError()+ if phase_one_exit == "pre_call_cancelled"+ else Exception("Rate limit exceeded")+ if phase_one_exit == "pre_call"+ else None+ ) pre_call: Final = AsyncMock( side_effect=pre_call_error, return_value=({"model": "vertex_ai/gemini-live-2.5-flash"}, logging_obj) )@@ -10155,6 +10166,114 @@ async def test_successful_realtime_session_leaves_the_reservation_for_the_cost_c assert reservation["finalized"] is False +_LIT6463_COUNTER_KEY: Final = "{api_key:hashed-token}:max_parallel_requests"+++async def _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ *,+ backend_logged_success: bool,+ backend_logged_failure: bool = False,+ phase_one_exit: str | None = None,+) -> tuple[DualCache, RequestRateLimiterStash]:+ """Run the realtime endpoint with a real v3 limiter registered and the request's+ stash already holding slot-1 of a two-slot counter, the state pre-call leaves+ behind. Returns the limiter's cache and the stash so the test can read what the+ endpoint did to the slot."""+ from litellm.proxy import proxy_server as ps+ from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3,+ _request_stash,+ )+ from litellm.proxy.utils import InternalUsageCache++ dual_cache: Final = DualCache()+ await dual_cache.async_set_cache(+ key=_LIT6463_COUNTER_KEY, value={"slot-1": 1.0, "slot-2": 2.0}, local_only=True+ )+ limiter: Final = _PROXY_MaxParallelRequestsHandler_v3(internal_usage_cache=InternalUsageCache(dual_cache))+ stash: Final = RequestRateLimiterStash(+ parallel_slot={"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}+ )+ reservation: Final = {"reserved_cost": 0.55, "input_cost": 0.0, "finalized": False, "entries": []}++ stash_token: Final = _request_stash.set(stash)+ try:+ hooks: Final = patch.dict( # test-quality-ok: registers the real limiter the route's release reads+ ps.proxy_logging_obj.proxy_hook_mapping, {"parallel_request_limiter": limiter}+ )+ expected_exit: Final = (+ pytest.raises(asyncio.CancelledError)+ if phase_one_exit == "pre_call_cancelled"+ else contextlib.nullcontext()+ )+ with hooks, expected_exit:+ await _lit6973_drive_realtime_session(+ reservation,+ backend_logged_success=backend_logged_success,+ backend_logged_failure=backend_logged_failure,+ phase_one_exit=phase_one_exit,+ )+ finally:+ _request_stash.reset(stash_token)+ return dual_cache, stash+++@pytest.mark.asyncio+@pytest.mark.parametrize("phase_one_exit", [None, "pre_call", "pre_call_cancelled"])+async def test_realtime_session_ending_without_llm_callbacks_releases_the_max_parallel_slot(+ phase_one_exit: str | None,+):+ """The rate limiter acquires the key's max_parallel_requests slot in pre-call and+ only frees it from the LLM success/failure callbacks. A realtime session that ends+ without either callback (Bedrock closes without usage events, a later pre-call hook+ rejects the session, or the task is cancelled while still in pre-call) has to be+ released by the route itself, or the slot stays occupied until its TTL and the key's+ next session is refused with a 429."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=False, phase_one_exit=phase_one_exit+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {"slot-2": 2.0}+ assert stash.parallel_slot is None+++@pytest.mark.asyncio+async def test_successful_realtime_session_leaves_the_max_parallel_slot_for_the_limiter_callback():+ """A session that enqueued its success callback hands the slot to the limiter's+ own success handler, which runs on the logging worker. If the route also released+ it, the two releases would race on the same stashed acquisition and, under the+ limiter's integer in-memory fallback, double-decrement the counter so the key+ admits more sessions than max_parallel_requests allows. With the success stamp+ present the route leaves the slot and the stash alone."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=True+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {+ "slot-1": 1.0,+ "slot-2": 2.0,+ }+ assert stash.parallel_slot == {"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}+++@pytest.mark.asyncio+async def test_refused_realtime_session_leaves_the_max_parallel_slot_for_the_limiter_failure_callback():+ """An upstream refusal before any frame enqueues the failure callback instead, and+ the limiter's failure handler releases the slot from the logging worker just like+ the success handler does. The route sees no success stamp, so it still settles the+ budget reservation, but it must leave the slot to that callback or the two releases+ race on the same acquisition."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=False, backend_logged_failure=True+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {+ "slot-1": 1.0,+ "slot-2": 2.0,+ }+ assert stash.parallel_slot == {"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}++ @pytest.mark.asyncio async def test_release_or_invalidate_falls_back_to_invalidating_the_counters(): """If releasing the reservation itself fails (e.g. the counter store is down),