Backtestslitellmrun-20261018-182052#38571
[Bug]: Bedrock Realtime hides provider throttles as normal WebSocket closes and lacks disconnect cleanup
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Patch B addresses both halves of the issue; Patch A addresses only the slot-cleanup half. B adds a throttle classifier (`realtime_failure_classification`) that maps 429 / RouterRateLimitError (following `__cause__` through the committed-session replay refusal) to a `rate_limit_error` Realtime error event and a 1013 close, remaps Bedrock `ServiceQuotaExceededException` from 400 to 429 so the router cools down and falls back, and stores the committed error in scope so a post-readiness throttle stays retryable. A does none of this: the client still receives `server_error`/1011 for a throttle, and the two independent 'throttle without fallback' tests fail on A exactly as on base. On cleanup, B makes idempotency a property of the limiter itself (claim `stash.parallel_slot = None` before awaiting the release in all four release paths) and wraps the route's release in a shielded, exception-swallowing helper applied to the pre-call rejection, the normal finally, and the responses websocket. A instead relies on success/failure stamps set by callers to decide whether the route may release, which is fragile (any path that dispatches a callback without stamping double-releases) and leaves the underlying race unaddressed; its shielding is also absent, so a cancelled session can still skip release. A's switch from FIRST_EXCEPTION to FIRST_COMPLETED is a reasonable fix for client-disconnect hang but is incidental. Tests: B adds handler tests proving no `session.updated` before readiness, 429 propagation, fallback replay, and a limiter race test plus end-to-end route tests using the real v3 limiter with max_parallel_requests=1 for every exit path the issue lists. A's tests cover stamp ownership and slot release but not the throttle signaling. The independent suite confirms: B passes all 10, A fails the 2 throttle-classification tests. B's extra touch on /v1/responses is adjacent but low risk and does not obscure the fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
1h 20m
Wall-clock time in the container
Full test suite
No new failures
21,416 passed · 26 skipped · 21,471 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
607vs 225 merged
Changed lines, ours vs merged
The issue
View on GitHub[Bug]: Bedrock Realtime hides provider throttles as normal WebSocket closes and lacks disconnect cleanup
Check for existing issues
- I searched the existing issues and did not find one covering Bedrock Nova Sonic throttling and request-slot cleanup on
/v1/realtime.
What happened?
When Bedrock's regional concurrent-session quota is exhausted, LiteLLM acknowledges the throttled Nova Sonic session as ready, emits no Realtime error event, and closes the client WebSocket with code 1000 and an empty reason.
This makes a capacity rejection indistinguishable from a successful, intentional session close. The application cannot identify a retryable throttle, apply backoff, fail over before sending audio, or report an actionable error.
There is also no WebSocket-lifecycle cleanup around the proxy's /v1/realtime route. Pre-call processing acquires proxy rate-limit state, including max_parallel_requests slots, but unlike the SSE streaming path the Realtime endpoint has no shielded finally that calls _arelease_max_parallel_requests_on_disconnect. When a WebSocket is cancelled or a provider failure is suppressed before a normal LiteLLM success/failure callback, the slot can remain until the limiter's one-hour TTL prunes it.
The Bedrock handler currently suppresses forwarding failures and then performs a normal close:
# litellm/llms/bedrock/realtime/handler.py
await asyncio.gather(
client_to_bedrock_task,
bedrock_to_client_task,
return_exceptions=True,
)
...
except Exception as e:
verbose_proxy_logger.debug(
"Bedrock to client forwarding ended: %s", e, exc_info=True
)
finally:
await client_ws.close()
Because client_ws.close() has no error code or reason, Starlette sends a normal WebSocket close. Because the forwarding exception does not escape, the router cannot classify the attempt as throttled, update deployment health/cooldown state, or attempt an ordered deployment fallback before the session is committed.
Expected behavior:
- A Bedrock throttling or service-quota exception before provider readiness is propagated to the router as one logical failure.
- If another deployment is configured, LiteLLM attempts that deployment before acknowledging the session as ready.
- If no fallback is available, the client receives a Realtime
errorevent containing a stable, retryable throttle classification, followed by an error close such as1013(try again later) or1011, not1000. session.updatedis emitted only after the provider has accepted the session configuration and is ready to receive audio.- Every WebSocket exit path releases exactly the proxy concurrency slot acquired for that session: normal completion, client disconnect, provider failure, cancellation, and pre-readiness fallback.
- Cleanup is idempotent so a logging callback and disconnect cleanup cannot double-decrement the limiter.
User flow
- Twenty clients establish Nova Sonic sessions in an AWS account/region whose concurrent-request quota is 20.
- A twenty-first client connects to LiteLLM and sends
session.update. - LiteLLM sends
session.createdandsession.updated, so the client reasonably begins sending audio. - Bedrock rejects that session because capacity is exhausted.
- LiteLLM emits no
errorevent and closes with1000, reason"". - The client cannot distinguish this from a clean end-of-session and has no reliable throttle signal on which to base retry or failover behavior.
After a fix, step 3 does not occur until upstream readiness is established. A rejected attempt either falls back before audio is accepted or produces an explicit retryable error and non-normal close, and its concurrency slot is released immediately.
Proof the bug occurs
The reproduction used the deployed LiteLLM proxy, a team-scoped virtual key, and real Bedrock Nova 2 Sonic sessions. No mock provider was used.
Environment:
LiteLLM: v1.98.0
Commit: d8f71d7bdbd7c9873d98293f83d64c6db72847e6
Endpoint: /v1/realtime
Provider/model: Bedrock amazon.nova-2-sonic-v1:0
Region: us-east-1
Proxy replicas: 4
AWS Bedrock quota code: L-995EBD81
AWS concurrent-session quota: 20
The test opened 21 concurrent sessions and continued reading each socket for 10 seconds after session.updated.
Observed client result:
21/21 sessions received session.updated
20/21 sessions remained connected for the hold period
1/21 sessions closed after readiness
close code: 1000
close reason: ""
Realtime error event: none
CloudWatch for the exact test minute showed:
AWS/Bedrock Invocations = 20
AWS/Bedrock InvocationThrottles = 1
The alternate us-west-2 deployment received no invocation, despite being registered under the same model group with lower routing priority.
The observed normal close follows directly from the handler source quoted above. The cleanup concern follows from the proxy lifecycle:
common_processing_pre_call_logic()runs before routing and invokes the proxy limiter hooks.- The v3 parallel limiter stores each acquired slot and gives it a 3,600-second TTL.
- LiteLLM already provides
ProxyLogging._arelease_max_parallel_requests_on_disconnect()and uses it from shielded SSE cleanup when no logging callback owns the release. - The
/v1/realtimeroute has onlytry/exceptaround routing; it has no correspondingfinallycleanup. - The Bedrock forwarding coroutines suppress exceptions, so the outer route can return without a failure callback that would otherwise release/account for the request.
Suggested regression coverage:
- Make the Bedrock duplex output raise a throttling exception while applying the first
session.update. - Assert
session.updatedhas not been sent. - With a second deployment configured, assert the exception reaches the router and the retained configuration is attempted there.
- Without a fallback, assert a Realtime
errorevent and non-normal close are sent with a retryable throttle classification. - Configure a virtual key with
max_parallel_requests=1, connect and disconnect normally, and assert another session can immediately acquire the slot. - Repeat step 5 for client cancellation, pre-readiness throttle, provider failure after readiness, and proxy task cancellation.
- Assert every acquired slot is released exactly once and no path waits for the 3,600-second TTL.
- Assert CloudWatch/provider throttles become LiteLLM deployment failures rather than successful sessions.
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on?
v1.98.0
The two patches
litellm/constants.pyindex e8aafda179..ffd07cd19f 100644@@ -317,6 +317,7 @@ WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123 BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY: Final = "litellm.bedrock_realtime.pending_session_update" BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY: Final = "litellm.bedrock_realtime.session_committed" BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY: Final = "litellm.bedrock_realtime.committed_failure"+BEDROCK_REALTIME_COMMITTED_ERROR_SCOPE_KEY: Final = "litellm.bedrock_realtime.committed_error" # SSL/TLS cipher configuration for faster handshakes # Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported oneslitellm/litellm_core_utils/realtime_errors.pyindex 3c064728a6..dc046d4fea 100644@@ -15,6 +15,37 @@ from litellm.types.realtime import RealtimeErrorDetail, RealtimeErrorEvent WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123 +# A provider throttle (HTTP 429, e.g. Bedrock ThrottlingException or an+# exhausted concurrent-session quota) is retryable, so it gets its own stable+# error type and the RFC 6455 "Try Again Later" close instead of a generic+# server error, letting clients back off or fail over instead of treating it+# as a clean or fatal end of session.+REALTIME_RATE_LIMIT_ERROR_TYPE: Final = "rate_limit_error"+REALTIME_SERVER_ERROR_TYPE: Final = "server_error"+WEBSOCKET_CLOSE_TRY_AGAIN_LATER: Final = 1013+WEBSOCKET_CLOSE_INTERNAL_ERROR: Final = 1011+++def realtime_failure_classification(error: BaseException) -> tuple[str, int]:+ """(error event type, close code) for a realtime session that failed with ``error``.++ Every deployment being cooled down after throttles surfaces from the router+ as ``RouterRateLimitError`` rather than the provider's 429, and is just as+ retryable. The explicit ``__cause__`` chain is followed because a retry's+ refusal to replay a committed session is what the router surfaces, chained+ to the throttle that actually ended the session.+ """+ from litellm.types.router import RouterRateLimitError++ current: BaseException | None = error+ seen: set[int] = set()+ while current is not None and id(current) not in seen:+ seen.add(id(current))+ if getattr(current, "status_code", None) == 429 or isinstance(current, RouterRateLimitError):+ return REALTIME_RATE_LIMIT_ERROR_TYPE, WEBSOCKET_CLOSE_TRY_AGAIN_LATER+ current = current.__cause__+ return REALTIME_SERVER_ERROR_TYPE, WEBSOCKET_CLOSE_INTERNAL_ERROR+ def realtime_error_event(message: str, error_type: str) -> str: detail: Final[RealtimeErrorDetail] = {"type": error_type, "message": message}litellm/llms/bedrock/realtime/handler.pyindex 7ab5a14dfc..c776139438 100644@@ -17,6 +17,7 @@ from pydantic import JsonValue, TypeAdapter import litellm from litellm._logging import _redact_string, verbose_proxy_logger from litellm.constants import (+ BEDROCK_REALTIME_COMMITTED_ERROR_SCOPE_KEY, BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY, BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY, BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY,@@ -47,7 +48,9 @@ _BEDROCK_STREAM_ERROR_STATUS: Final[Mapping[str, int]] = MappingProxyType( "ModelStreamErrorException": 424, "ModelTimeoutException": 408, "ResourceNotFoundException": 404,- "ServiceQuotaExceededException": 400,+ # An exhausted concurrent-session quota is a capacity throttle: 429 lets the router cool the+ # deployment down and fall back, and the proxy tell the client to retry later.+ "ServiceQuotaExceededException": 429, "ServiceUnavailableException": 503, "ThrottlingException": 429, "ValidationException": 400,@@ -142,13 +145,16 @@ def _pending_session_update(scope: Mapping[str, object]) -> str | None: """A fallback attempt on the same websocket replays the session.update the failed attempt never acked.""" if scope.get(BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY) is True: committed_failure: Final = scope.get(BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY)+ committed_error: Final = scope.get(BEDROCK_REALTIME_COMMITTED_ERROR_SCOPE_KEY)+ # Chained so the proxy still classifies a committed throttle as retryable even though+ # the router surfaces this non-retryable refusal as the last attempt's error. raise BedrockError( status_code=400, message=( "Bedrock realtime session already committed to a provider stream; it cannot be replayed" + (f". The committed stream failed with: {committed_failure}" if committed_failure else "") ),- )+ ) from (committed_error if isinstance(committed_error, BaseException) else None) pending: Final = scope.get(BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY) return pending if isinstance(pending, str) else None @@ -158,6 +164,7 @@ def _raise_provider_failure(scope: MutableMapping[str, object], failure: BaseExc verbose_proxy_logger.error("Bedrock Realtime: provider stream failed: %s", _redact_string(str(error))) if scope.get(BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY) is True: scope[BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY] = _redact_string(str(error))+ scope[BEDROCK_REALTIME_COMMITTED_ERROR_SCOPE_KEY] = error raise error from failure litellm/proxy/hooks/parallel_request_limiter_v3.pyindex c398abff09..3a259ad522 100644@@ -4452,11 +4452,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs)) acquisition: Final = stash.parallel_slot if stash is not None else None if stash is not None and acquisition is not None:+ # Claim before awaiting so a concurrent release (websocket+ # disconnect cleanup, the sibling logging callback) sees the+ # slot gone and cannot decrement it a second time.+ stash.parallel_slot = None await self._release_parallel_request_slots( acquisition=acquisition, parent_otel_span=litellm_parent_otel_span, )- stash.parallel_slot = None pipeline_operations: Final = self._build_success_event_pipeline_operations( kwargs=kwargs,@@ -4578,11 +4581,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs)) acquisition: Final = stash.parallel_slot if stash is not None else None if stash is not None and acquisition is not None:+ # Claim before awaiting so a concurrent release (websocket+ # disconnect cleanup, the sibling logging callback) sees the+ # slot gone and cannot decrement it a second time.+ stash.parallel_slot = None await self._release_parallel_request_slots( acquisition=acquisition, parent_otel_span=litellm_parent_otel_span, )- stash.parallel_slot = None # Skip the reservation refund if async_post_call_failure_hook # already released it (proxy-level rejection that also bubbles up@@ -4694,11 +4700,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): if stash is None or stash.parallel_slot is None: return + # Claim the acquisition before awaiting: the success/failure callback+ # may run concurrently on the logging worker, and whichever claims the+ # slot first is the only one that releases it.+ acquisition: Final = stash.parallel_slot+ stash.parallel_slot = None await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,+ acquisition=acquisition, parent_otel_span=None, )- stash.parallel_slot = None async def async_post_call_success_hook(self, data: dict, user_api_key_dict: UserAPIKeyAuth, response): """@@ -4774,12 +4784,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash() if stash is None: return- if stash.parallel_slot is not None:+ claimed_slot: Final = stash.parallel_slot+ if claimed_slot is not None:+ stash.parallel_slot = None await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,+ acquisition=claimed_slot, parent_otel_span=user_api_key_dict.parent_otel_span, )- stash.parallel_slot = None if stash.batch_enqueued_reservation is not None: await self.batch_enqueued_token_store.refund(litellm/proxy/proxy_server.pyindex dd7a39eb94..20f75b2428 100644@@ -295,6 +295,7 @@ from litellm.litellm_core_utils.credential_accessor import CredentialAccessor from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj from litellm.litellm_core_utils.realtime_errors import ( realtime_error_event,+ realtime_failure_classification, websocket_close_reason, ) from litellm.litellm_core_utils.sensitive_data_masker import (@@ -11917,6 +11918,7 @@ async def _reject_realtime_session( verbose_proxy_logger.debug("Could not send realtime pre-call error event to client; closing anyway") await websocket.close(code=code, reason=reason) finally:+ await proxy_logging_obj.arelease_websocket_session_parallel_slot(user_api_key_dict) await _release_realtime_budget_reservation(user_api_key_dict) @@ -12038,13 +12040,14 @@ async def realtime_websocket_endpoint( except Exception as e: verbose_proxy_logger.exception("Internal server error") redacted_error: Final = _redact_string(str(e))+ error_type, close_code = realtime_failure_classification(e) try:- await websocket.send_text(realtime_error_event(redacted_error, error_type="server_error"))+ await websocket.send_text(realtime_error_event(redacted_error, error_type=error_type)) except Exception: # noqa: BLE001 # best-effort notice: a dead client socket must not skip the close below verbose_proxy_logger.debug("Could not send realtime error event to client; closing anyway") try: await websocket.close(- code=1011,+ code=close_code, reason=websocket_close_reason(redacted_error, fallback="Internal server error"), ) except Exception: # noqa: BLE001 # the lower layer may have closed the socket already; closing twice is not an error@@ -12054,6 +12057,7 @@ async def realtime_websocket_endpoint( REALTIME_SESSION_SUCCESS_LOGGED_KEY, ) + await proxy_logging_obj.arelease_websocket_session_parallel_slot(user_api_key_dict) if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY): await _release_realtime_budget_reservation(user_api_key_dict) litellm/proxy/response_api_endpoints/endpoints.pyindex 5907ffc64e..d772290702 100644@@ -1531,7 +1531,11 @@ async def responses_websocket_endpoint( ) except Exception: pass- await websocket.close(code=1008, reason="Pre-call error")+ try:+ await websocket.close(code=1008, reason="Pre-call error")+ finally:+ # A hook failing after the limiter admitted the request leaves its slot held.+ await proxy_logging_obj.arelease_websocket_session_parallel_slot(user_api_key_dict) return # Phase 2: route to upstream provider@@ -1547,3 +1551,5 @@ async def responses_websocket_endpoint( except Exception: verbose_proxy_logger.exception("Responses WebSocket error") await websocket.close(code=1011, reason="Internal server error")+ finally:+ await proxy_logging_obj.arelease_websocket_session_parallel_slot(user_api_key_dict)litellm/proxy/utils.pyindex c095586b6c..77bd290469 100644@@ -19,6 +19,7 @@ from email.mime.text import MIMEText from types import MappingProxyType from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, Optional, Protocol, TypeVar, Union, cast, overload +import anyio from typing_extensions import ReadOnly, TypedDict from litellm import _custom_logger_compatible_callbacks_literal@@ -3752,6 +3753,29 @@ class ProxyLogging: return await limiter.async_release_max_parallel_requests_on_disconnect(user_api_key_dict) + async def arelease_websocket_session_parallel_slot(+ self,+ user_api_key_dict: UserAPIKeyAuth,+ ) -> None:+ """+ Release the max_parallel_requests slot pre-call acquired for a+ websocket session (/v1/realtime, /v1/responses), on every exit path.++ A websocket session can end without either logging callback owning+ the release: a cancelled session task, a client disconnect, or a+ provider failure the relay reports to the client itself. Shielded so a+ cancelled task still frees its slot instead of holding it until the+ limiter's TTL prunes it. The limiter claims the stashed acquisition+ before releasing it, so this is a no-op when a logging callback+ already released the slot (or none was acquired) and can never+ double-decrement.+ """+ with anyio.CancelScope(shield=True):+ try:+ await self._arelease_max_parallel_requests_on_disconnect(user_api_key_dict)+ except Exception: # noqa: BLE001 - cleanup must not mask the session's outcome; the slot TTL is the backstop+ verbose_proxy_logger.exception("Could not release the websocket session's max_parallel_requests slot")+ def _init_response_taking_too_long_task(self, data: dict | None = None): """ Initialize the response taking too long task if user is using slack alertingtests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.pyindex 2eadaee9e1..e98396811d 100644@@ -8,6 +8,7 @@ from unittest.mock import MagicMock import pytest import litellm+from litellm.litellm_core_utils.realtime_errors import realtime_failure_classification from litellm.llms.bedrock.common_utils import BedrockError from litellm.llms.bedrock.realtime.handler import BedrockRealtime from litellm.llms.bedrock.realtime.transformation import BedrockRealtimeConfig@@ -649,6 +650,42 @@ class TestBedrockRealtimeProviderFailurePropagation: assert not litellm._should_retry(replay.value.status_code), "the router must not retry the replay refusal" assert "Nova Sonic stream broke" in replay.value.message, "the router surfaces the last attempt's error" + @pytest.mark.asyncio+ async def test_throttle_after_readiness_stays_retryable_through_the_replay_refusal(+ self, stub_aws_sdk_client, spend_dispatch+ ):+ class ThrottlingException(Exception):+ pass++ class ThrottledAfterReadyReceiver(ScriptedBedrockReceiver):+ async def receive(self):+ if not self._payloads:+ await asyncio.sleep(0)+ raise ThrottlingException("Too many concurrent requests")+ return await super().receive()++ handler = BedrockRealtime()+ websocket = ConnectedClientWS([self.SESSION_UPDATE])+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream(self.TEXT_TURN, receiver_type=ThrottledAfterReadyReceiver)]++ with pytest.raises(BedrockError) as throttled:+ await handler.async_realtime(+ model="amazon.nova-sonic-v1:0", websocket=websocket, logging_obj=spend_dispatch["logging_obj"], **self.AWS_PARAMS+ )+ assert throttled.value.status_code == 429+ await spend_dispatch.pop("coro")++ # The router retries the 429; the replay refusal is what it finally surfaces to the proxy.+ with pytest.raises(BedrockError) as replay:+ await handler.async_realtime(+ model="amazon.nova-sonic-v1:0", websocket=websocket, logging_obj=spend_dispatch["logging_obj"], **self.AWS_PARAMS+ )++ assert replay.value.status_code == 400+ assert realtime_failure_classification(replay.value) == ("rate_limit_error", 1013), (+ "the client must still get a retryable throttle, not a generic server error"+ )+ @pytest.mark.asyncio async def test_input_side_failure_keeps_spend_for_responses_already_delivered( self, stub_aws_sdk_client, spend_dispatch@@ -782,3 +819,58 @@ class TestBedrockRealtimeAwsAuth: if __name__ == "__main__": pytest.main([__file__, "-v"])+++class ThrottlingException(Exception):+ """Named like the modeled AWS SDK error Bedrock raises when its request rate is exceeded"""+++class ServiceQuotaExceededException(Exception):+ """Named like the modeled AWS SDK error Bedrock raises when the concurrent-session quota is exhausted"""+++class ThrottledBedrockStream:+ """Duplex stream Bedrock rejects for capacity once the first session.update is applied"""++ def __init__(self, error_type):+ self.input_stream = FakeInputStream()+ self._error_type = error_type++ async def await_output(self):+ raise self._error_type("Too many concurrent requests for model amazon.nova-2-sonic-v1:0")+++class TestBedrockRealtimeCapacityThrottle:+ """A capacity rejection must not look like a ready session: no session.updated, a retryable 429 for+ the router (cooldown + next deployment), and the retained session.update replayed on that deployment"""++ SESSION_UPDATE = json.dumps({"type": "session.update", "session": {"instructions": "hi", "modalities": ["text"]}})+ AWS_PARAMS = {"aws_region_name": "us-east-1", "aws_access_key_id": "k", "aws_secret_access_key": "s"}++ @pytest.mark.asyncio+ @pytest.mark.parametrize("error_type", [ThrottlingException, ServiceQuotaExceededException])+ async def test_throttle_before_readiness_is_a_retryable_429_and_falls_back_before_ready(+ self, stub_aws_sdk_client, error_type+ ):+ websocket = ConnectedClientWS([self.SESSION_UPDATE])+ fallback_stream = ScriptedBedrockStream([])+ stub_aws_sdk_client["streams"] = [ThrottledBedrockStream(error_type), fallback_stream]++ with pytest.raises(BedrockError) as throttled:+ await BedrockRealtime().async_realtime(+ model="amazon.nova-2-sonic-v1:0", websocket=websocket, logging_obj=FakeLogging(), **self.AWS_PARAMS+ )++ assert throttled.value.status_code == 429+ assert error_type.__name__ in throttled.value.message+ assert litellm._should_retry(throttled.value.status_code), "the router must try the next deployment"+ assert [json.loads(m)["type"] for m in websocket.sent_to_client] == ["session.created"]+ assert not websocket.closed, "the proxy route owns the client-facing error event and error close"++ await BedrockRealtime().async_realtime(+ model="amazon.nova-2-sonic-v1:0", websocket=websocket, logging_obj=FakeLogging(), **self.AWS_PARAMS+ )++ assert [json.loads(m)["type"] for m in websocket.sent_to_client] == ["session.created", "session.updated"]+ replayed = [json.loads(chunk.value.bytes_.decode("utf-8")) for chunk in fallback_stream.input_stream.sent]+ assert [next(iter(event["event"])) for event in replayed][:2] == ["sessionStart", "promptStart"]tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.pyindex 10c0bb88a8..36e5d56903 100644@@ -6284,3 +6284,53 @@ async def test_an_open_circuit_breaker_reads_the_sliding_window_locally_without_ assert isinstance(values, list) assert [record.getMessage() for record in caplog.records if record.levelno >= logging.WARNING] == [] assert any("circuit breaker is open" in record.getMessage() for record in caplog.records)+++@pytest.mark.asyncio+async def test_concurrent_disconnect_and_failure_callback_release_the_slot_once_v3():+ """+ A websocket session's shielded disconnect cleanup and the limiter's own+ logging callback can run at the same time (the callback runs on the logging+ worker). Under the in-memory integer mirror, which carries no per-slot ids,+ both releasing the same acquisition would decrement the gauge twice and+ free a slot held by another in-flight request. Whichever claims the stashed+ acquisition first must be the only one that releases it.+ """+ _api_key = hash_token("sk-websocket-session")+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache)+ )+ counter_key = f"{{api_key:{_api_key}}}:max_parallel_requests"+ # Two requests in flight: this session and another one that must stay counted.+ await local_cache.async_set_cache(key=counter_key, value=2, local_only=True)+ get_or_create_request_stash().parallel_slot = ParallelSlotAcquisition(+ slot_id="session-slot",+ counter_keys=[counter_key],+ )+ # A real cache store suspends on reads; the in-memory one would otherwise+ # finish each release without ever yielding, hiding the interleaving.+ original_get_cache = handler.internal_usage_cache.async_get_cache++ async def suspending_get_cache(*args, **kwargs):+ await asyncio.sleep(0)+ return await original_get_cache(*args, **kwargs)++ handler.internal_usage_cache.async_get_cache = suspending_get_cache++ await asyncio.gather(+ handler.async_release_max_parallel_requests_on_disconnect(+ UserAPIKeyAuth(api_key=_api_key, max_parallel_requests=2)+ ),+ handler.async_log_failure_event(+ kwargs={"standard_logging_object": {"metadata": {"user_api_key_hash": _api_key}}},+ response_obj=None,+ start_time=datetime.now(),+ end_time=datetime.now(),+ ),+ )++ assert get_request_stash().parallel_slot is None+ assert handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=counter_key, local_only=True)+ ) == 1tests/test_litellm/proxy/realtime_endpoints/test_realtime_websocket_session_cleanup.pynew file mode 100644index 0000000000..e069f87048@@ -0,0 +1,246 @@+"""+/v1/realtime session exits: provider throttles reach the client as a retryable+error (not a normal 1000 close), and every exit path frees the key's+max_parallel_requests slot right away instead of leaving it to the limiter's+one-hour TTL.++The sessions run through the real realtime_websocket_endpoint and a real v3+parallel request limiter with max_parallel_requests=1; only pre-call wiring and+the routed provider call are faked. "The slot was released" is checked the way+a client would see it: a second session's admission succeeds.+"""++import asyncio+import json+from collections.abc import Awaitable, Callable+from typing import Final+from unittest.mock import AsyncMock, MagicMock, patch++import pytest+from fastapi import HTTPException++from litellm.caching.caching import DualCache+from litellm.llms.bedrock.common_utils import BedrockError+from litellm.proxy import proxy_server as ps+from litellm.proxy._types import UserAPIKeyAuth+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3,+)+from litellm.proxy.utils import InternalUsageCache, hash_token+from litellm.types.router import RouterRateLimitError++MODEL: Final = "bedrock/amazon.nova-2-sonic-v1:0"+++def _fake_websocket() -> MagicMock:+ ws = MagicMock()+ ws.headers = {}+ ws.scope = {"headers": [], "type": "websocket"}+ ws.url = "ws://testserver/v1/realtime"+ ws.accept = AsyncMock()+ ws.send_text = AsyncMock()+ ws.close = AsyncMock()+ return ws+++class _Session:+ def __init__(self) -> None:+ self.cache: Final = DualCache()+ self.limiter: Final = _PROXY_MaxParallelRequestsHandler_v3(internal_usage_cache=InternalUsageCache(self.cache))+ self.user_api_key_dict: Final = UserAPIKeyAuth(+ api_key=hash_token("sk-realtime-session"), token="hashed-token", max_parallel_requests=1+ )++ async def admit(self, data: dict) -> None:+ await self.limiter.async_pre_call_hook(+ user_api_key_dict=self.user_api_key_dict, cache=self.cache, data=data, call_type="_arealtime"+ )++ async def run(+ self,+ llm_call: Callable[[], Awaitable[None]],+ *,+ ws: MagicMock | None = None,+ fail_after_admission: Exception | None = None,+ ) -> MagicMock:+ """Run one realtime session in its own task, as the ASGI server does per connection."""+ websocket: Final = ws if ws is not None else _fake_websocket()+ logging_obj: Final = MagicMock()+ logging_obj.model_call_details = {}++ async def pre_call(*_args: object, **kwargs: object):+ data: Final = {"model": MODEL}+ await self.admit(data)+ if fail_after_admission is not None:+ raise fail_after_admission+ return data, logging_obj++ async def route_request(**_kwargs: object):+ return llm_call()++ with (+ patch.object(+ ps, "can_key_call_resolved_model", new=AsyncMock()+ ), # test-quality-ok: no HTTP boundary; key/model access is not under test+ patch.object(+ ps.ProxyBaseLLMRequestProcessing, "common_processing_pre_call_logic", new=pre_call+ ), # test-quality-ok: pre-call runs the real limiter admission+ patch.object(+ ps, "route_request", new=route_request+ ), # test-quality-ok: fakes the provider relay whose exit is under test+ patch.object(+ ps.proxy_logging_obj,+ "get_proxy_hook",+ new=lambda name: self.limiter if name == "parallel_request_limiter" else None,+ ), # test-quality-ok: wires the real limiter in place of the proxy's configured hooks+ ):+ await asyncio.create_task(+ ps.realtime_websocket_endpoint(+ websocket=websocket,+ model=MODEL,+ intent=None,+ guardrails=None,+ user_api_key_dict=self.user_api_key_dict,+ )+ )+ return websocket++ async def assert_slot_is_free(self) -> None:+ async def admit_next() -> None:+ await self.admit({"model": MODEL})++ # A fresh task has a fresh request stash, like the next client connection.+ await asyncio.create_task(admit_next())+++async def _completes() -> None:+ return None+++def _raises(error: BaseException) -> Callable[[], Awaitable[None]]:+ async def call() -> None:+ raise error++ return call+++@pytest.mark.asyncio+async def test_slot_is_held_while_a_session_is_open():+ """Guards the other assertions: an open session really does hold the only slot."""+ session: Final = _Session()+ opened: Final = asyncio.Event()++ async def stays_open() -> None:+ opened.set()+ await asyncio.Event().wait()++ task: Final = asyncio.create_task(session.run(stays_open))+ await opened.wait()+ with pytest.raises(HTTPException) as rejected:+ await session.assert_slot_is_free()+ assert rejected.value.status_code == 429+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task+ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_bedrock_throttle_reaches_the_client_as_a_retryable_error_close():+ session: Final = _Session()+ throttle: Final = BedrockError(status_code=429, message="ThrottlingException: Too many concurrent requests")++ ws: Final = await session.run(_raises(throttle))++ event: Final = json.loads(ws.send_text.await_args.args[0])+ assert event["type"] == "error"+ assert event["error"]["type"] == "rate_limit_error"+ assert "ThrottlingException" in event["error"]["message"]+ assert ws.close.await_args.kwargs["code"] == 1013+ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_all_deployments_cooled_down_after_throttles_is_a_retryable_error_close():+ session: Final = _Session()+ cooled_down: Final = RouterRateLimitError(+ model=MODEL, cooldown_time=5, enable_pre_call_checks=False, cooldown_list=["a", "b"], model_ids=["a", "b"]+ )++ ws: Final = await session.run(_raises(cooled_down))++ assert json.loads(ws.send_text.await_args.args[0])["error"]["type"] == "rate_limit_error"+ assert ws.close.await_args.kwargs["code"] == 1013+++@pytest.mark.asyncio+async def test_provider_failure_after_readiness_is_a_server_error_close_and_frees_the_slot():+ session: Final = _Session()++ ws: Final = await session.run(_raises(BedrockError(status_code=424, message="ModelStreamErrorException: broke")))++ assert json.loads(ws.send_text.await_args.args[0])["error"]["type"] == "server_error"+ assert ws.close.await_args.kwargs["code"] == 1011+ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_normal_session_end_frees_the_slot():+ session: Final = _Session()++ ws: Final = await session.run(_completes)++ ws.send_text.assert_not_awaited()+ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_cancelled_session_task_frees_the_slot():+ """A client cancel or proxy shutdown cancels the session task mid-relay; no+ logging callback fires, so the endpoint's shielded cleanup must release."""+ session: Final = _Session()+ relaying: Final = asyncio.Event()++ async def relay_until_cancelled() -> None:+ relaying.set()+ await asyncio.Event().wait()++ task: Final = asyncio.create_task(session.run(relay_until_cancelled))+ await relaying.wait()+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_pre_call_failure_after_admission_frees_the_slot():+ """A guardrail or later hook failing after the limiter admitted the session+ rejects it before routing; that slot must not wait for the TTL either."""+ session: Final = _Session()++ ws: Final = await session.run(_completes, fail_after_admission=Exception("blocked by guardrail"))++ ws.close.assert_awaited_once_with(code=1011, reason="Pre-call error")+ await session.assert_slot_is_free()+++@pytest.mark.asyncio+async def test_slot_released_by_the_logging_callback_is_not_released_again():+ """The limiter's own logging callback already freed the slot (e.g. the+ session's failure was logged); the endpoint cleanup must then be a no-op+ rather than decrement another session's slot."""+ session: Final = _Session()+ release: Final = AsyncMock(wraps=session.limiter._release_parallel_request_slots)++ async def logged_then_ends() -> None:+ await session.limiter.async_log_failure_event(kwargs={}, response_obj=None, start_time=None, end_time=None)++ with patch.object(+ session.limiter, "_release_parallel_request_slots", new=release+ ): # test-quality-ok: counts releases on the real limiter+ await session.run(logged_then_ends)++ assert release.await_count == 1+ await session.assert_slot_is_free()tests/test_litellm/proxy/response_api_endpoints/test_endpoints.pyindex d7010de640..51ea0878da 100644@@ -2,6 +2,8 @@ Test for response_api_endpoints/endpoints.py """ +import asyncio+import contextlib import unittest from typing import Any from unittest.mock import AsyncMock, MagicMock, patch@@ -1959,3 +1961,112 @@ class TestResponsesInputTokens: assert response.status_code == 429, response.text assert response.json()["error"]["message"] == "rate limited"+++class TestResponsesWSParallelSlotCleanup:+ """The /v1/responses websocket takes a max_parallel_requests slot in pre-call+ like /v1/realtime; every session exit must free it instead of leaving it to+ the limiter's one-hour TTL. Checked as the next client sees it: its+ admission with max_parallel_requests=1 succeeds."""++ @staticmethod+ async def _run_session(llm_call, *, fail_after_admission=None):+ from litellm.caching.caching import DualCache+ from litellm.proxy import proxy_server as ps+ from litellm.proxy._types import UserAPIKeyAuth+ from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3,+ )+ from litellm.proxy.response_api_endpoints.endpoints import (+ responses_websocket_endpoint,+ )+ from litellm.proxy.utils import InternalUsageCache, hash_token++ cache = DualCache()+ limiter = _PROXY_MaxParallelRequestsHandler_v3(internal_usage_cache=InternalUsageCache(cache))+ user_api_key_dict = UserAPIKeyAuth(api_key=hash_token("sk-responses-ws"), max_parallel_requests=1)++ async def admit():+ await limiter.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=cache,+ data={"model": "gpt-4o-mini"},+ call_type="_aresponses_websocket",+ )++ async def pre_call(**_kwargs):+ await admit()+ if fail_after_admission is not None:+ raise fail_after_admission+ return {"model": "gpt-4o-mini"}, MagicMock()++ processor = MagicMock()+ processor.common_processing_pre_call_logic = pre_call++ async def route_request(**_kwargs):+ return llm_call()++ ws = MagicMock()+ ws.headers = {}+ ws.scope = {"headers": []}+ ws.url = "ws://testserver/v1/responses"+ ws.accept = AsyncMock()+ ws.send_text = AsyncMock()+ ws.close = AsyncMock()++ with (+ patch(+ "litellm.proxy.response_api_endpoints.endpoints.ProxyBaseLLMRequestProcessing",+ return_value=processor,+ ),+ patch("litellm.proxy.route_llm_request.route_request", new=route_request),+ patch.object(+ ps.proxy_logging_obj,+ "get_proxy_hook",+ new=lambda name: limiter if name == "parallel_request_limiter" else None,+ ),+ ):+ session = asyncio.create_task(+ responses_websocket_endpoint(websocket=ws, model="gpt-4o-mini", user_api_key_dict=user_api_key_dict)+ )+ await asyncio.sleep(0.05)+ if not session.done():+ session.cancel()+ with contextlib.suppress(asyncio.CancelledError):+ await session++ # The next connection runs in a fresh task, so it has its own request stash.+ await asyncio.create_task(admit())+ return ws++ @pytest.mark.asyncio+ async def test_normal_session_end_frees_the_slot(self):+ async def completes():+ return None++ await self._run_session(completes)++ @pytest.mark.asyncio+ async def test_provider_failure_frees_the_slot(self):+ async def fails():+ raise RuntimeError("upstream websocket reset")++ ws = await self._run_session(fails)++ ws.close.assert_awaited_once_with(code=1011, reason="Internal server error")++ @pytest.mark.asyncio+ async def test_cancelled_session_frees_the_slot(self):+ async def relays_forever():+ await asyncio.Event().wait()++ await self._run_session(relays_forever)++ @pytest.mark.asyncio+ async def test_pre_call_failure_after_admission_frees_the_slot(self):+ async def never_routed():+ raise AssertionError("a rejected session must not be routed")++ ws = await self._run_session(never_routed, fail_after_admission=Exception("blocked by guardrail"))++ ws.close.assert_awaited_once_with(code=1008, reason="Pre-call error")
litellm/constants.pyindex e8aafda1797..c106be688e4 100644@@ -317,6 +317,8 @@ WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123 BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY: Final = "litellm.bedrock_realtime.pending_session_update" BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY: Final = "litellm.bedrock_realtime.session_committed" BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY: Final = "litellm.bedrock_realtime.committed_failure"+REALTIME_SESSION_SUCCESS_LOGGED_KEY: Final = "realtime_session_success_logged"+REALTIME_SESSION_FAILURE_LOGGED_KEY: Final = "realtime_session_failure_logged" # SSL/TLS cipher configuration for faster handshakes # Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported oneslitellm/litellm_core_utils/realtime_streaming.pyindex e3f8786a39a..fa567bdf4c9 100644@@ -10,6 +10,7 @@ from typing_extensions import ReadOnly import litellm from litellm._logging import redact_internal_details_from_client_message, verbose_logger+from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig from litellm.types.llms.openai import (@@ -35,9 +36,6 @@ else: CLIENT_CONNECTION_CLASS = Any -REALTIME_SESSION_SUCCESS_LOGGED_KEY: Final = "realtime_session_success_logged"-- @dataclass(frozen=True, slots=True) class BackendClose: code: int@@ -1153,6 +1151,7 @@ class RealTimeStreaming: self._logging_worker.ensure_initialized_and_enqueue( self.logging_obj.dispatch_failure_handlers(error, traceback.format_exc(), prefer_async_handlers=True) )+ self.logging_obj.model_call_details[REALTIME_SESSION_FAILURE_LOGGED_KEY] = True @staticmethod def _detect_beta_header(websocket: ScopedWebSocket) -> bool:litellm/llms/bedrock/realtime/handler.pyindex 7ab5a14dfc2..2c1ce6068b2 100644@@ -20,6 +20,7 @@ from litellm.constants import ( BEDROCK_REALTIME_COMMITTED_FAILURE_SCOPE_KEY, BEDROCK_REALTIME_PENDING_SESSION_UPDATE_SCOPE_KEY, BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY,+ REALTIME_SESSION_SUCCESS_LOGGED_KEY, ) from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging@@ -346,6 +347,7 @@ class BedrockRealtime(BaseAWSLLM): prefer_async_handlers=True, ) )+ logging_obj.model_call_details[REALTIME_SESSION_SUCCESS_LOGGED_KEY] = True if outcome.provider_failure is None: return@@ -382,7 +384,7 @@ class BedrockRealtime(BaseAWSLLM): ) bedrock_task: Final = asyncio.create_task(collect_logged_events()) - await asyncio.wait((client_task, bedrock_task), return_when=asyncio.FIRST_EXCEPTION)+ await asyncio.wait((client_task, bedrock_task), return_when=asyncio.FIRST_COMPLETED) client_disconnected: Final = ( client_task.done() and not client_task.cancelled() and client_task.exception() is None )litellm/proxy/proxy_server.pyindex dd7a39eb94c..1c931863a2f 100644@@ -274,6 +274,8 @@ from litellm.constants import ( PROXY_BUDGET_RESCHEDULER_MAX_TIME, PROXY_BUDGET_RESCHEDULER_MIN_TIME, PROXY_CONFIG_RELOAD_INTERVAL_SECONDS,+ REALTIME_SESSION_FAILURE_LOGGED_KEY,+ REALTIME_SESSION_SUCCESS_LOGGED_KEY, ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG, USER_SPEND_ALERTS_JOB_ID, WEEKLY_SPEND_REPORT_JOB_ID,@@ -11899,6 +11901,13 @@ async def _release_realtime_budget_reservation(user_api_key_dict: UserAPIKeyAuth ) +async def _release_realtime_max_parallel_slot(user_api_key_dict: UserAPIKeyAuth) -> None:+ release_like_http_disconnect: Final = (+ proxy_logging_obj._arelease_max_parallel_requests_on_disconnect # pyright: ignore[reportPrivateUsage] # shared+ )+ await release_like_http_disconnect(user_api_key_dict)++ async def _reject_realtime_session( websocket: WebSocket, user_api_key_dict: UserAPIKeyAuth,@@ -11918,6 +11927,7 @@ async def _reject_realtime_session( await websocket.close(code=code, reason=reason) finally: await _release_realtime_budget_reservation(user_api_key_dict)+ await _release_realtime_max_parallel_slot(user_api_key_dict) @app.websocket("/openai/v1/realtime")@@ -12021,6 +12031,9 @@ async def realtime_websocket_endpoint( websocket, user_api_key_dict, code=1011, reason="Pre-call error", error_message=str(e) ) return+ except BaseException:+ await _release_realtime_max_parallel_slot(user_api_key_dict)+ raise # Phase 2: route to upstream LLM. try:@@ -12050,12 +12063,10 @@ async def realtime_websocket_endpoint( except Exception: # noqa: BLE001 # the lower layer may have closed the socket already; closing twice is not an error verbose_proxy_logger.debug("Could not close realtime client websocket; it is already gone") finally:- from litellm.litellm_core_utils.realtime_streaming import (- REALTIME_SESSION_SUCCESS_LOGGED_KEY,- )- if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY): await _release_realtime_budget_reservation(user_api_key_dict)+ if not litellm_logging_obj.model_call_details.get(REALTIME_SESSION_FAILURE_LOGGED_KEY):+ await _release_realtime_max_parallel_slot(user_api_key_dict) ######################################################################tests/test_litellm/litellm_core_utils/test_realtime_streaming.pyindex 2a33d84ec78..7e6d4d24905 100644@@ -10,9 +10,9 @@ from websockets.exceptions import ConnectionClosed from websockets.frames import Close import litellm+from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.integrations.custom_guardrail import CustomGuardrail from litellm.litellm_core_utils.realtime_streaming import (- REALTIME_SESSION_SUCCESS_LOGGED_KEY, RealTimeStreaming, client_sent_openai_beta_realtime_header, )@@ -3399,6 +3399,26 @@ async def test_refused_session_does_not_stamp_the_reservation_ownership_marker() assert REALTIME_SESSION_SUCCESS_LOGGED_KEY not in session.logging.model_call_details +@pytest.mark.asyncio+async def test_refused_session_stamps_the_failure_ownership_marker():+ """LIT-6463: the enqueued failure callback releases the key's max_parallel_requests+ slot from the logging worker, so a refusal stamps REALTIME_SESSION_FAILURE_LOGGED_KEY.+ The proxy endpoint reads it to leave the slot to that callback instead of racing it.+ A session that relayed frames logs a success and must not carry the failure stamp."""+ upstream_close: Final = ConnectionClosed(Close(1008, _UPSTREAM_REFUSAL), None)+ refused: Final = _relay_session(_client_ws_that_never_sends(), _backend_ws_closing_with(upstream_close))+ session_created: Final = json.dumps({"type": "session.created", "session": {"id": "sess_1"}}).encode()+ relayed: Final = _relay_session(+ _client_ws_that_never_sends(), _backend_ws_closing_with(session_created, upstream_close)+ )++ await refused.run()+ await relayed.run()++ assert refused.logging.model_call_details.get(REALTIME_SESSION_FAILURE_LOGGED_KEY) is True+ assert REALTIME_SESSION_FAILURE_LOGGED_KEY not in relayed.logging.model_call_details++ @pytest.mark.asyncio async def test_transformed_transcription_completion_never_sends_response_create(): from typing import Finaltests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.pyindex 2eadaee9e1a..ac3a43b742f 100644@@ -8,6 +8,7 @@ from unittest.mock import MagicMock import pytest import litellm+from litellm.constants import REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.llms.bedrock.common_utils import BedrockError from litellm.llms.bedrock.realtime.handler import BedrockRealtime from litellm.llms.bedrock.realtime.transformation import BedrockRealtimeConfig@@ -77,6 +78,7 @@ class UnavailableBedrockStream: class FakeLogging: def __init__(self, trace_id="trace-nova-sonic"): self.litellm_trace_id = trace_id+ self.model_call_details = {} class DisconnectingClientWS:@@ -673,6 +675,31 @@ class TestBedrockRealtimeProviderFailurePropagation: await spend_dispatch["coro"] assert [event["type"] for event in spend_dispatch["events"]] == ["response.done"] + @pytest.mark.asyncio+ async def test_success_dispatch_stamps_the_ownership_marker_only_when_spend_was_logged(+ self, stub_aws_sdk_client, spend_dispatch+ ):+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream(self.TEXT_TURN)]+ await BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=ConnectedClientWS([self.SESSION_UPDATE]),+ logging_obj=spend_dispatch["logging_obj"],+ **self.AWS_PARAMS,+ )+ await spend_dispatch["coro"]+ assert [event["type"] for event in spend_dispatch["events"]] == ["response.done"]+ assert spend_dispatch["logging_obj"].model_call_details.get(REALTIME_SESSION_SUCCESS_LOGGED_KEY) is True++ idle_logging = FakeLogging()+ stub_aws_sdk_client["streams"] = [ScriptedBedrockStream([])]+ await BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=ConnectedClientWS([self.SESSION_UPDATE]),+ logging_obj=idle_logging,+ **self.AWS_PARAMS,+ )+ assert REALTIME_SESSION_SUCCESS_LOGGED_KEY not in idle_logging.model_call_details+ @pytest.mark.asyncio async def test_stream_failure_after_client_disconnect_is_not_a_provider_failure(self, stub_aws_sdk_client): stream = ScriptedBedrockStream([], receiver_type=BreakingBedrockReceiver)@@ -684,6 +711,25 @@ class TestBedrockRealtimeProviderFailurePropagation: assert stream.input_stream.closed + @pytest.mark.asyncio+ async def test_client_disconnect_ends_the_session_while_bedrock_output_stays_open(self, stub_aws_sdk_client):+ receiver = DrainedThenOpenBedrockReceiver([])+ stream = ScriptedBedrockStream([], receiver_type=lambda _payloads: receiver)+ stub_aws_sdk_client["streams"] = [stream]++ await asyncio.wait_for(+ BedrockRealtime().async_realtime(+ model="amazon.nova-sonic-v1:0",+ websocket=RealtimeClientWS(),+ logging_obj=FakeLogging(),+ **self.AWS_PARAMS,+ ),+ timeout=1,+ )++ assert receiver.drained.is_set(), "the handler must have been waiting on the open provider stream"+ assert stream.input_stream.closed+ @pytest.mark.asyncio async def test_session_updated_is_not_sent_before_bedrock_is_ready(self, stub_aws_models): handler = BedrockRealtime()tests/test_litellm/proxy/test_proxy_server.pyindex a5e6cf285ad..e09dddfec5b 100644@@ -33,6 +33,7 @@ from litellm.litellm_core_utils.get_model_cost_map import ModelCostMapReloaded from litellm.caching.dual_cache import DualCache from litellm.proxy._types import LitellmUserRoles, TokenCountRequest, UserAPIKeyAuth from litellm.proxy.auth.user_api_key_auth import user_api_key_auth+from litellm.proxy.hooks.parallel_request_limiter_v3 import RequestRateLimiterStash from litellm.proxy.proxy_server import app, initialize from litellm.utils import _invalidate_model_cost_lowercase_map @@ -9999,6 +10000,7 @@ async def _lit6973_drive_realtime_session( reservation: dict, *, backend_logged_success: bool,+ backend_logged_failure: bool = False, phase_one_exit: str | None = None, websocket: MagicMock | None = None, ) -> MagicMock:@@ -10006,8 +10008,9 @@ async def _lit6973_drive_realtime_session( phase_one_exit picks a rejection before the relay: "model_access" makes the key/model check raise ProxyException, "pre_call" makes pre-call processing- (rate limits, guardrails) raise. Neither reaches route_request, so no success- log can own the reservation and the endpoint has to release it on that exit.+ (rate limits, guardrails) raise, "pre_call_cancelled" cancels the task inside+ pre-call processing. None reaches route_request, so no success log can own the+ reservation and the endpoint has to release it on that exit. route_request resolves normally in both cases: the relay owns the session once route_request returns. A successful session enqueues its success cost@@ -10017,7 +10020,7 @@ async def _lit6973_drive_realtime_session( logging object carries a real model_call_details dict so the stamp is observable, and the reservation has empty entries so the real release touches no counter store."""- from litellm.litellm_core_utils.realtime_streaming import REALTIME_SESSION_SUCCESS_LOGGED_KEY+ from litellm.constants import REALTIME_SESSION_FAILURE_LOGGED_KEY, REALTIME_SESSION_SUCCESS_LOGGED_KEY from litellm.proxy import proxy_server as ps user_api_key_dict: Final = UserAPIKeyAuth(api_key="sk-test", token="hashed-token")@@ -10029,6 +10032,8 @@ async def _lit6973_drive_realtime_session( async def fake_llm_call() -> None: if backend_logged_success: logging_obj.model_call_details[REALTIME_SESSION_SUCCESS_LOGGED_KEY] = True+ if backend_logged_failure:+ logging_obj.model_call_details[REALTIME_SESSION_FAILURE_LOGGED_KEY] = True from litellm.proxy._types import ProxyException @@ -10037,7 +10042,13 @@ async def _lit6973_drive_realtime_session( if phase_one_exit == "model_access" else None )- pre_call_error: Final = Exception("Rate limit exceeded") if phase_one_exit == "pre_call" else None+ pre_call_error: Final = (+ asyncio.CancelledError()+ if phase_one_exit == "pre_call_cancelled"+ else Exception("Rate limit exceeded")+ if phase_one_exit == "pre_call"+ else None+ ) pre_call: Final = AsyncMock( side_effect=pre_call_error, return_value=({"model": "vertex_ai/gemini-live-2.5-flash"}, logging_obj) )@@ -10155,6 +10166,114 @@ async def test_successful_realtime_session_leaves_the_reservation_for_the_cost_c assert reservation["finalized"] is False +_LIT6463_COUNTER_KEY: Final = "{api_key:hashed-token}:max_parallel_requests"+++async def _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ *,+ backend_logged_success: bool,+ backend_logged_failure: bool = False,+ phase_one_exit: str | None = None,+) -> tuple[DualCache, RequestRateLimiterStash]:+ """Run the realtime endpoint with a real v3 limiter registered and the request's+ stash already holding slot-1 of a two-slot counter, the state pre-call leaves+ behind. Returns the limiter's cache and the stash so the test can read what the+ endpoint did to the slot."""+ from litellm.proxy import proxy_server as ps+ from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3,+ _request_stash,+ )+ from litellm.proxy.utils import InternalUsageCache++ dual_cache: Final = DualCache()+ await dual_cache.async_set_cache(+ key=_LIT6463_COUNTER_KEY, value={"slot-1": 1.0, "slot-2": 2.0}, local_only=True+ )+ limiter: Final = _PROXY_MaxParallelRequestsHandler_v3(internal_usage_cache=InternalUsageCache(dual_cache))+ stash: Final = RequestRateLimiterStash(+ parallel_slot={"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}+ )+ reservation: Final = {"reserved_cost": 0.55, "input_cost": 0.0, "finalized": False, "entries": []}++ stash_token: Final = _request_stash.set(stash)+ try:+ hooks: Final = patch.dict( # test-quality-ok: registers the real limiter the route's release reads+ ps.proxy_logging_obj.proxy_hook_mapping, {"parallel_request_limiter": limiter}+ )+ expected_exit: Final = (+ pytest.raises(asyncio.CancelledError)+ if phase_one_exit == "pre_call_cancelled"+ else contextlib.nullcontext()+ )+ with hooks, expected_exit:+ await _lit6973_drive_realtime_session(+ reservation,+ backend_logged_success=backend_logged_success,+ backend_logged_failure=backend_logged_failure,+ phase_one_exit=phase_one_exit,+ )+ finally:+ _request_stash.reset(stash_token)+ return dual_cache, stash+++@pytest.mark.asyncio+@pytest.mark.parametrize("phase_one_exit", [None, "pre_call", "pre_call_cancelled"])+async def test_realtime_session_ending_without_llm_callbacks_releases_the_max_parallel_slot(+ phase_one_exit: str | None,+):+ """The rate limiter acquires the key's max_parallel_requests slot in pre-call and+ only frees it from the LLM success/failure callbacks. A realtime session that ends+ without either callback (Bedrock closes without usage events, a later pre-call hook+ rejects the session, or the task is cancelled while still in pre-call) has to be+ released by the route itself, or the slot stays occupied until its TTL and the key's+ next session is refused with a 429."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=False, phase_one_exit=phase_one_exit+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {"slot-2": 2.0}+ assert stash.parallel_slot is None+++@pytest.mark.asyncio+async def test_successful_realtime_session_leaves_the_max_parallel_slot_for_the_limiter_callback():+ """A session that enqueued its success callback hands the slot to the limiter's+ own success handler, which runs on the logging worker. If the route also released+ it, the two releases would race on the same stashed acquisition and, under the+ limiter's integer in-memory fallback, double-decrement the counter so the key+ admits more sessions than max_parallel_requests allows. With the success stamp+ present the route leaves the slot and the stash alone."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=True+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {+ "slot-1": 1.0,+ "slot-2": 2.0,+ }+ assert stash.parallel_slot == {"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}+++@pytest.mark.asyncio+async def test_refused_realtime_session_leaves_the_max_parallel_slot_for_the_limiter_failure_callback():+ """An upstream refusal before any frame enqueues the failure callback instead, and+ the limiter's failure handler releases the slot from the logging worker just like+ the success handler does. The route sees no success stamp, so it still settles the+ budget reservation, but it must leave the slot to that callback or the two releases+ race on the same acquisition."""+ dual_cache, stash = await _lit6463_drive_realtime_session_holding_a_max_parallel_slot(+ backend_logged_success=False, backend_logged_failure=True+ )++ assert await dual_cache.async_get_cache(key=_LIT6463_COUNTER_KEY, local_only=True) == {+ "slot-1": 1.0,+ "slot-2": 2.0,+ }+ assert stash.parallel_slot == {"slot_id": "slot-1", "counter_keys": [_LIT6463_COUNTER_KEY]}++ @pytest.mark.asyncio async def test_release_or_invalidate_falls_back_to_invalidating_the_counters(): """If releasing the reservation itself fails (e.g. the counter store is down),