Backtestslitellmrun-20261018-182052#24677
[Bug]: Incorrect TPM limiting for virtual keys
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches identify the same root cause: the RPM `:requests` and TPM `:tokens` counters share one `{key:value}:window` key, and whichever counter observes expiry resets only itself, so the sibling carries prior-window usage forward indefinitely. Both fix this in BATCH_RATE_LIMITER_SCRIPT, CHECK_AND_INCREMENT_BY_N_SCRIPT, the in-memory sliding window, and the in-memory atomic path, and both pass all five independent tests that fail on base.
Patch B goes further on the issue's second symptom (cooldowns of 2-5 minutes). In a Redis-backed multi-pod deployment, the local pre-check in is_cache_list_over_limit rejects on the locally mirrored counter before consulting Redis; after a sibling reset in Redis, that local mirror of the sibling is never refreshed (a local rejection short-circuits the Redis call), so stale rejection persists until the local TTL lapses. Patch B treats a counter whose window start is expired as 0 in the local pre-check and invalidates local sibling mirrors when Redis reports a freshly reset window. Patch A leaves the local mirror stale, so under the multi-pod scenario the issue describes it only partially shortens the cooldown.
Patch B also handles a consequence the fix itself introduces: once counters reset at rollover, settling a reservation made in the old window (`actual - reserved`) into the new window corrupts it. It records reservation window identities and routes those settlements through the existing window-guarded reservation path, with parametrized success/failure tests. Patch A ignores this.
Patch A's added `_check_and_increment_lock` around in_memory_cache_sliding_window is a reasonable concurrency guard but introduces a non-reentrant asyncio.Lock on a path whose callers I cannot see; if any caller already holds that lock this deadlocks. Patch B's Lua helper also guards on the ':window' suffix before deriving the prefix, where Patch A derives blindly.
On scope, Patch B is considerably larger and the reservation-settlement rework adds risk, but it is coherent with the fix rather than unrelated churn, and the new test file exercises in-memory, fake-Redis multi-pod, atomic, and post-rollover settlement scenarios. Patch A's tests are focused and valid but narrower. B is the more complete fix for both reported symptoms.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
43m 33s
Wall-clock time in the container
Full test suite
No new failures
4,367 passed · 1 skipped · 4,369 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
543vs 195 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
This was initially reported on #18953 for v1.80.0. It was marked as solved, but we are still seeing this behaviour in v1.82.3
Actual rate limit enforcement is usually ~30% below the configured limit, and the cooldown period is longer than a minute. Usually 2, but can reach 5 minutes in some instances.
This is with the default rate limit type and configuration.
Steps to Reproduce
- Configure an API key with TPM limit (e.g. 500k)
- Use the API key
- Rate limit enforcement hits at approximately 300-350k.
Relevant log output
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.82.3
Twitter / LinkedIn details
_No response_
The two patches
litellm/caching/in_memory_cache.pyindex 56c9147e06..b93fc8fb40 100644@@ -265,7 +265,7 @@ class InMemoryCache(BaseCache): async def disconnect(self): pass - def delete_cache(self, key):+ def delete_cache(self, key: str) -> None: self._remove_key(key) async def async_get_ttl(self, key: str) -> int | None:litellm/proxy/hooks/parallel_request_limiter_v3.pyindex 89826694bb..29f70e6c4c 100644@@ -92,6 +92,19 @@ _REQUEST_RATE_LIMIT_DATA: Final = TypeAdapter(Mapping[str, object]) BATCH_RATE_LIMITER_SCRIPT: Final = """+-- RPM (`:requests`) and TPM (`:tokens`) counters of one descriptor share a+-- single `{key:value}:window` key. Whichever counter observes the window+-- expiry resets the window, so it must also clear its sibling counter --+-- otherwise the sibling carries the previous window's usage into the new+-- window and is never reset by window expiry again. Siblings share the+-- window key's hash tag, so they always live on the same cluster slot.+local function reset_window_counters(window_key)+ if string.sub(window_key, -7) == ':window' then+ local prefix = string.sub(window_key, 1, -8)+ redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')+ end+end+ local results = {} local now = tonumber(ARGV[1]) local window_size = tonumber(ARGV[2])@@ -105,7 +118,8 @@ for i = 1, #KEYS, 2 do -- Check if window exists and is valid local window_start = redis.call('GET', window_key) if not window_start or (now - tonumber(window_start)) >= window_size then- -- Reset window and counter+ -- Reset window and every counter sharing it+ reset_window_counters(window_key) redis.call('SET', window_key, tostring(now)) redis.call('SET', counter_key, increment_value) redis.call('EXPIRE', window_key, window_size)@@ -148,6 +162,19 @@ CHECK_AND_INCREMENT_BY_N_SCRIPT: Final = """ -- Return on success: -- { 0, new_counter_1, window_start_1, new_counter_2, window_start_2, ... } -- Return on over-limit: { 1, descriptor_index, current_counter, limit }+-- RPM (`:requests`) and TPM (`:tokens`) counters of one descriptor share a+-- single `{key:value}:window` key. Whichever counter observes the window+-- expiry resets the window, so it must also clear its sibling counter --+-- otherwise the sibling carries the previous window's usage into the new+-- window and is never reset by window expiry again. Siblings share the+-- window key's hash tag, so they always live on the same cluster slot.+local function reset_window_counters(window_key)+ if string.sub(window_key, -7) == ':window' then+ local prefix = string.sub(window_key, 1, -8)+ redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')+ end+end+ local time_reply = redis.call('TIME') local now = tonumber(time_reply[1]) local descriptor_count = #KEYS / 2@@ -186,7 +213,17 @@ for i = 1, descriptor_count do descriptor_state[i] = { window_expired, current_counter, window_start } end +-- Pass 2: all checks passed. Clear every counter of each expired window+-- once (before any counter of that window is set), then apply increments.+local reset_windows = {}+for i = 1, descriptor_count do+ local window_key = KEYS[(i - 1) * 2 + 1]+ if descriptor_state[i][1] and not reset_windows[window_key] then+ reset_windows[window_key] = true+ reset_window_counters(window_key)+ end+end+ local results = { 0 } for i = 1, descriptor_count do local window_key = KEYS[(i - 1) * 2 + 1]@@ -340,6 +377,10 @@ end return results """ +# RPM and TPM counters of one descriptor share the `{key:value}:window` key.+WINDOW_KEY_SUFFIX: Final = ":window"+WINDOW_SHARED_COUNTER_TYPES: Final = ("requests", "tokens")+ # Redis cluster slot count REDIS_CLUSTER_SLOTS: Final = 16384 REDIS_NODE_HASHTAG_NAME: Final = "all_keys"@@ -535,6 +576,9 @@ class RequestRateLimiterStash: reserved_tokens: int = 0 reserved_model: str | None = None reserved_scopes: frozenset[tuple[str, str]] = field(default_factory=frozenset)+ reserved_window_identities: frozenset[tuple[str, str, Literal["redis", "local"]]] = field(+ default_factory=frozenset+ ) itpm_reserved_tokens: int = 0 itpm_reserved_scopes: frozenset[tuple[str, str]] = field(default_factory=frozenset) itpm_reserved_window_identities: frozenset[tuple[str, str, Literal["redis", "local"]]] = field(@@ -1008,6 +1052,36 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): self.internal_usage_cache.dual_cache.redis_cache, RedisClusterCache ) + @staticmethod+ def _window_counter_keys(window_key: str) -> tuple[str, ...]:+ """Every windowed counter key sharing ``window_key`` (RPM and TPM)."""+ if not window_key.endswith(WINDOW_KEY_SUFFIX):+ return ()+ prefix: Final = window_key[: -len(WINDOW_KEY_SUFFIX)]+ return tuple(f"{prefix}:{counter_type}" for counter_type in WINDOW_SHARED_COUNTER_TYPES)++ def _reset_local_window_counters(self, window_key: str) -> None:+ """Local mirror of the Lua ``reset_window_counters``: a window reset+ clears every counter sharing the window, not just the one that+ observed the expiry, so no counter carries a stale window's usage."""+ for counter_key in self._window_counter_keys(window_key):+ self.internal_usage_cache.dual_cache.in_memory_cache.delete_cache(counter_key)++ @staticmethod+ def _parse_window_start(window_start: CacheCounterValue | None) -> int | None:+ if window_start is None:+ return None+ try:+ return int(float(window_start))+ except (TypeError, ValueError, OverflowError):+ return None++ @classmethod+ def _is_window_expired(cls, window_start: CacheCounterValue | None, now_int: int, window_size: int) -> bool:+ """A missing or unparseable window start means the window is gone."""+ parsed_start: Final = cls._parse_window_start(window_start)+ return parsed_start is None or now_int - parsed_start >= window_size+ async def in_memory_cache_sliding_window( self, keys: list[str],@@ -1035,7 +1109,8 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): # Check if window exists and is valid if window_start is None or (now_int - int(window_start)) >= window_size:- # Reset window and counter+ # Reset window and every counter sharing it+ self._reset_local_window_counters(window_key) await self.internal_usage_cache.async_set_cache( key=window_key, value=str(now_int),@@ -1090,9 +1165,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): keys_to_fetch: list[str], cache_values: CacheCounterValues, key_metadata: dict[str, WindowKeyMetadata],+ now_int: int | None = None, ) -> RateLimitResponse: """ Check if the cache values are over the limit.++ When ``now_int`` is given, a counter whose window has already expired+ (or whose window key is gone) counts as 0: it belongs to a past window+ and must not keep rejecting requests after that window has rolled. """ statuses: Final[list[RateLimitStatus]] = [] overall_code = "OK"@@ -1102,6 +1182,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): window_key = keys_to_fetch[i] counter_key = keys_to_fetch[i + 1] counter_value = cache_values[i + 1]+ if now_int is not None and self._is_window_expired(+ cache_values[i], now_int, key_metadata[window_key]["window_size"]+ ):+ counter_value = None requests_limit = key_metadata[window_key]["requests_limit"] tokens_limit = key_metadata[window_key]["tokens_limit"] @@ -1309,7 +1393,9 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) if cache_values is not None:- rate_limit_response: Final = self.is_cache_list_over_limit(keys_to_fetch, cache_values, key_metadata)+ rate_limit_response: Final = self.is_cache_list_over_limit(+ keys_to_fetch, cache_values, key_metadata, now_int=now_int+ ) if rate_limit_response["overall_code"] == "OVER_LIMIT": return rate_limit_response @@ -1335,6 +1421,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): now_int=now_int, ) + # A window Redis just reset invalidates every locally mirrored+ # counter of that window, including siblings not refreshed by+ # this call. Clear them once, before mirroring the new values.+ for window_key in dict.fromkeys(+ keys_to_fetch[i]+ for i in range(0, len(cache_values), 2)+ if self._parse_window_start(cache_values[i]) == now_int+ ):+ self._reset_local_window_counters(window_key) # update in-memory cache with new values for i in range(0, len(cache_values), 2): window_key = keys_to_fetch[i]@@ -2047,7 +2142,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): } ) - # Pass 2: apply increments.+ # Pass 2: clear every counter of each expired window once, before any+ # counter of that window is set, then apply increments.+ for window_key in dict.fromkeys(+ meta["window_key"] for meta, state in zip(per_counter_meta, descriptor_state) if state["window_expired"]+ ):+ self._reset_local_window_counters(window_key) statuses: Final[list[RateLimitStatus]] = [] for meta, state in zip(per_counter_meta, descriptor_state): new_counter = meta["increment"] if state["window_expired"] else state["current"] + meta["increment"]@@ -3731,6 +3831,9 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) is not None )+ # Window each reservation landed in, so reconciliation+ # can skip a delta whose window has since rolled.+ stash.reserved_window_identities = tpm_response.get("reservation_windows", frozenset()) tpm_reservation_scopes = tuple( # rebind-ok: record successful reservation scopes stash.reserved_scopes )@@ -4353,6 +4456,44 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) return ops + def _build_window_guarded_tpm_ops(+ self,+ targets: Sequence[tuple[str, str]],+ reserved_scopes: Set[tuple[str, str]],+ actual_tokens: int,+ reserved_tokens: int,+ reservation_window_identities: frozenset[tuple[str, str, Literal["redis", "local"]]],+ ) -> tuple[list[RedisPipelineIncrementOperation], tuple[ReservationAwareIncrementOperation, ...]]: # mutable-ok: increment pipeline APIs take a list+ """+ Split TPM reconciliation into plain ops and window-guarded ops.++ A reserved scope whose reservation window is known settles through the+ window guard: once that window rolls, its counters were reset, so the+ ``actual - reserved`` delta must not land in the new window. Every+ other scope keeps the plain ``_build_reservation_aware_tpm_ops`` path.+ """+ guarded_counter_keys: Final = frozenset(counter_key for counter_key, _, _ in reservation_window_identities)+ guarded_targets: Final = tuple(+ scope+ for scope in targets+ if scope in reserved_scopes+ and self.create_rate_limit_keys(scope[0], scope[1], "tokens") in guarded_counter_keys+ )+ plain_ops: Final = self._build_reservation_aware_tpm_ops(+ targets=tuple(scope for scope in targets if scope not in guarded_targets),+ reserved_scopes=reserved_scopes,+ actual_tokens=actual_tokens,+ reserved_tokens=reserved_tokens,+ )+ guarded_ops: Final = self._build_project_reservation_ops(+ targets=guarded_targets,+ reserved_scopes=reserved_scopes,+ actual_tokens=actual_tokens,+ reserved_tokens=reserved_tokens,+ reservation_window_identities=reservation_window_identities,+ )+ return plain_ops, guarded_ops+ def _build_project_reservation_op( self, scope: tuple[str, str],@@ -4419,6 +4560,18 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): response_obj: object, rate_limit_type: Literal["output", "input", "total"], ) -> list[RedisPipelineIncrementOperation]:+ return self._build_success_event_tpm_operations(+ kwargs=kwargs,+ response_obj=response_obj,+ rate_limit_type=rate_limit_type,+ )[0]++ def _build_success_event_tpm_operations(+ self,+ kwargs: dict[str, Any], # mutable-ok: logging callback kwargs are a plain dict+ response_obj: object,+ rate_limit_type: Literal["output", "input", "total"],+ ) -> tuple[list[RedisPipelineIncrementOperation], tuple[ReservationAwareIncrementOperation, ...]]: # mutable-ok: increment pipeline APIs take a list """Build Redis pipeline increment ops for TPM / parallel-request counters.""" from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs from litellm.proxy.common_utils.callback_utils import (@@ -4432,7 +4585,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): if request_metadata.get(INTERNAL_CALL_ORIGIN_METADATA_KEY): # Internal sub-calls bill spend to the caller but are not the caller's # traffic; charging them here would let background evals eat TPM headroom.- return []+ return [], () # mutable-ok: increment pipeline APIs take a list standard_logging_metadata: Final = standard_logging_object.get("metadata") or {} model_group: Final = get_model_group_from_litellm_kwargs(kwargs)@@ -4465,14 +4618,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): reserved_tokens: Final = stash.reserved_tokens if stash is not None else 0 reserved_model: Final = stash.reserved_model if stash is not None else None reserved_scopes: Final[frozenset[tuple[str, str]]] = stash.reserved_scopes if stash is not None else frozenset()+ reserved_window_identities: Final = stash.reserved_window_identities if stash is not None else frozenset() # Reconciliation must target the same model-scoped counter that the # pre-call reservation incremented. If a reservation was made, # ``reserved_model`` is authoritative; otherwise fall back to the # router's ``model_group`` (covers the no-reservation charge path). reconcile_model: Final = reserved_model or model_group - pipeline_operations: Final[list[RedisPipelineIncrementOperation]] = []- # ---------------------------------------------------------------- # TPM reconciliation # Per-scope behavior:@@ -4501,17 +4653,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): total_tokens, reserved_tokens - total_tokens, )- pipeline_operations.extend(- self._build_reservation_aware_tpm_ops(- targets=charged_targets,- reserved_scopes=reserved_scopes,- actual_tokens=total_tokens,- reserved_tokens=reserved_tokens,- )+ return self._build_window_guarded_tpm_ops(+ targets=charged_targets,+ reserved_scopes=reserved_scopes,+ actual_tokens=total_tokens,+ reserved_tokens=reserved_tokens,+ reservation_window_identities=reserved_window_identities, ) - return pipeline_operations- async def async_log_success_event(self, kwargs, response_obj, start_time, end_time): """ Update TPM usage on successful API calls by incrementing counters using pipeline@@ -4529,7 +4678,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs)) await self._release_stashed_parallel_slot(stash, litellm_parent_otel_span) - pipeline_operations: Final = self._build_success_event_pipeline_operations(+ pipeline_operations, guarded_operations = self._build_success_event_tpm_operations( kwargs=kwargs, response_obj=response_obj, rate_limit_type=rate_limit_type,@@ -4539,6 +4688,11 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): pipeline_operations=pipeline_operations, parent_otel_span=litellm_parent_otel_span, )+ if guarded_operations:+ await self.async_increment_reservation_aware_tokens(+ pipeline_operations=guarded_operations,+ parent_otel_span=litellm_parent_otel_span,+ ) io_token_operations: Final = self._build_io_token_reservation_ops( kwargs=kwargs, response_obj=response_obj,@@ -4660,21 +4814,25 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) tpm_actual, itpm_actual, otpm_actual = self._recovered_partial_usage_tokens(kwargs) - if stash is not None and reserved_tokens > 0:+ # Settle only against the scopes the reservation actually+ # charged: unreserved scopes were never incremented, so a+ # refund there would drive their counter negative.+ tpm_plain_operations, tpm_guarded_operations = (+ self._build_window_guarded_tpm_ops(+ targets=tuple(stash.reserved_scopes),+ reserved_scopes=stash.reserved_scopes,+ actual_tokens=tpm_actual,+ reserved_tokens=reserved_tokens,+ reservation_window_identities=stash.reserved_window_identities,+ )+ if stash is not None and reserved_tokens > 0+ else ([], ()) # mutable-ok: increment pipeline APIs take a list+ )+ if tpm_plain_operations or tpm_guarded_operations: verbose_proxy_logger.debug( "Settling reserved TPM tokens on failure: reserved=%s actual=%s", reserved_tokens, tpm_actual )- # Settle only against the scopes the reservation actually- # charged: unreserved scopes were never incremented, so a- # refund there would drive their counter negative.- pipeline_operations.extend(- self._build_reservation_aware_tpm_ops(- targets=list(stash.reserved_scopes),- reserved_scopes=stash.reserved_scopes,- actual_tokens=tpm_actual,- reserved_tokens=reserved_tokens,- )- )+ pipeline_operations.extend(tpm_plain_operations) # Settle project ITPM/OTPM reservations the same way: at the # recovered partial usage, or a full refund when there is none.@@ -4721,6 +4879,11 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): increment_list=pipeline_operations, litellm_parent_otel_span=litellm_parent_otel_span, )+ if tpm_guarded_operations:+ await self.async_increment_reservation_aware_tokens(+ pipeline_operations=tpm_guarded_operations,+ parent_otel_span=litellm_parent_otel_span,+ ) for project_operations in (itpm_operations, otpm_operations): if isinstance(project_operations, list): await self.internal_usage_cache.dual_cache.async_increment_cache_pipeline(@@ -4864,15 +5027,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): return tpm_actual, itpm_actual, otpm_actual = self._recovered_partial_usage_tokens(request_data) - combined_ops: Final = (- self._build_reservation_aware_tpm_ops(+ combined_ops, combined_guarded_ops = (+ self._build_window_guarded_tpm_ops( targets=tuple(stash.reserved_scopes), reserved_scopes=stash.reserved_scopes, actual_tokens=tpm_actual, reserved_tokens=reserved_tokens,+ reservation_window_identities=stash.reserved_window_identities, ) if reserved_tokens > 0- else ()+ else ([], ()) # mutable-ok: increment pipeline APIs take a list ) itpm_ops: Final = ( self._build_project_reservation_ops(@@ -4910,7 +5074,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): if otpm_reserved > 0 else () )- if combined_ops or itpm_ops or otpm_ops:+ if combined_ops or combined_guarded_ops or itpm_ops or otpm_ops: verbose_proxy_logger.debug( "Releasing reserved tokens on proxy-level rejection: tpm=%s, itpm=%s, otpm=%s", reserved_tokens,@@ -4922,6 +5086,11 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): increment_list=combined_ops, litellm_parent_otel_span=user_api_key_dict.parent_otel_span, )+ if combined_guarded_ops:+ await self.async_increment_reservation_aware_tokens(+ pipeline_operations=combined_guarded_ops,+ parent_otel_span=user_api_key_dict.parent_otel_span,+ ) for project_ops in (itpm_ops, otpm_ops): if isinstance(project_ops, list): await self.internal_usage_cache.dual_cache.async_increment_cache_pipeline(tests/test_litellm/proxy/hooks/test_rate_limit_window_rollover.pynew file mode 100644index 0000000000..775721108f@@ -0,0 +1,305 @@+"""+Regression tests: RPM/TPM limits must reset when the rate-limit window rolls.++The RPM (`:requests`) and TPM (`:tokens`) counters of one descriptor share a+single `{key:value}:window` key. Previously a window reset only cleared the+counter that observed the expiry, so the sibling counter carried the previous+window's usage into the new one (TPM tripping well below the configured+limit), and the local pre-check kept rejecting on a stale counter after the+window had rolled (cooldowns longer than one window).+"""++import time+from datetime import datetime, timedelta++import pytest+from fastapi import HTTPException++from litellm.caching.caching import DualCache+from litellm.proxy._types import UserAPIKeyAuth+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ BATCH_RATE_LIMITER_SCRIPT,+ CHECK_AND_INCREMENT_BY_N_SCRIPT,+ _request_stash,+ get_request_stash,+)+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+ _PROXY_MaxParallelRequestsHandler_v3 as RateLimitHandler,+)+from litellm.proxy.utils import InternalUsageCache, hash_token+from litellm.types.utils import ModelResponse, Usage+++class TimeController:+ def __init__(self):+ self._current = datetime.now()++ def now(self) -> datetime:+ return self._current++ def advance(self, seconds: float) -> None:+ self._current += timedelta(seconds=seconds)+++@pytest.fixture+def time_controller(monkeypatch):+ controller = TimeController()+ # The in-memory cache expires keys against time.time().+ monkeypatch.setattr(time, "time", lambda: controller.now().timestamp())+ return controller+++@pytest.fixture(autouse=True)+def _isolated_request_stash():+ token = _request_stash.set(None)+ yield+ _request_stash.reset(token)+++def _handler(time_controller: TimeController) -> RateLimitHandler:+ return RateLimitHandler(+ internal_usage_cache=InternalUsageCache(DualCache()),+ time_provider=time_controller.now,+ )+++async def _pre_call(handler: RateLimitHandler, user: UserAPIKeyAuth, max_tokens: int) -> None:+ token = _request_stash.set(None)+ try:+ await handler.async_pre_call_hook(+ user_api_key_dict=user,+ cache=handler.internal_usage_cache.dual_cache,+ data={+ "model": "gpt-4o",+ "messages": [{"role": "user", "content": "hi"}],+ "max_tokens": max_tokens,+ },+ call_type="completion",+ )+ finally:+ _request_stash.reset(token)+++def _success_kwargs(api_key: str) -> dict:+ return {+ "standard_logging_object": {"metadata": {"user_api_key_hash": api_key}},+ "model": "gpt-4o",+ }+++def _usage_response(total_tokens: int) -> ModelResponse:+ return ModelResponse(+ id="r",+ model="gpt-4o",+ usage=Usage(+ prompt_tokens=total_tokens // 2,+ completion_tokens=total_tokens - total_tokens // 2,+ total_tokens=total_tokens,+ ),+ choices=[],+ )+++@pytest.mark.asyncio+async def test_tpm_resets_when_rpm_check_rolls_the_shared_window(time_controller):+ """A key with both RPM and TPM limits gets its full TPM budget back in the+ next window, even though the RPM pass is the one that resets the window."""+ handler = _handler(time_controller)+ api_key = hash_token("sk-window-rollover")+ user = UserAPIKeyAuth(api_key=api_key, rpm_limit=100, tpm_limit=1000)++ # The window opens with a request that is charged to RPM only (a+ # non-generation provider call), so the TPM counter is created later.+ async with handler.request_capacity(user, "gpt-4o"):+ pass++ # Mid-window: a completion reserves tokens and reports 900 of usage.+ time_controller.advance(30)+ await handler.async_pre_call_hook(+ user_api_key_dict=user,+ cache=handler.internal_usage_cache.dual_cache,+ data={"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 100},+ call_type="completion",+ )+ stash = get_request_stash()+ assert stash is not None and stash.reserved_tokens > 0+ await handler.async_log_success_event(+ kwargs=_success_kwargs(api_key),+ response_obj=_usage_response(900),+ start_time=datetime.now(),+ end_time=datetime.now(),+ )++ # The window is nearly exhausted: a 500-token request is rejected.+ with pytest.raises(HTTPException) as exc_info:+ await _pre_call(handler, user, max_tokens=500)+ assert exc_info.value.status_code == 429++ # Next window: the full 1000-token TPM budget is available again.+ time_controller.advance(handler.window_size - 30 + 1)+ await _pre_call(handler, user, max_tokens=500)+ await _pre_call(handler, user, max_tokens=400)+++def _fake_redis_batch_script(store: dict):+ """In-process stand-in for BATCH_RATE_LIMITER_SCRIPT on a Redis shared by+ several proxy pods (window reset clears every counter of the window)."""++ async def script(keys, args):+ now, window_size = int(args[0]), int(args[1])+ results = []+ for i in range(0, len(keys), 2):+ window_key, counter_key = keys[i], keys[i + 1]+ window_start = store.get(window_key)+ if window_start is None or now - int(window_start) >= window_size:+ prefix = window_key[: -len(":window")]+ store.pop(f"{prefix}:requests", None)+ store.pop(f"{prefix}:tokens", None)+ store[window_key] = str(now)+ store[counter_key] = 1+ else:+ store[counter_key] = store.get(counter_key, 0) + 1+ results.extend([store[window_key], store[counter_key]])+ return results++ return script+++@pytest.mark.asyncio+async def test_rpm_rejection_lifts_once_the_shared_window_rolls(time_controller):+ """A pod that first sees a Redis window late (another pod opened it) must+ admit requests as soon as that window rolls, not keep rejecting on its+ local mirror of the previous window's counter."""+ handler = _handler(time_controller)+ api_key = hash_token("sk-rpm-cooldown")+ user = UserAPIKeyAuth(api_key=api_key, rpm_limit=2)+ scope = f"{{api_key:{api_key}}}"+ now_int = int(time_controller.now().timestamp())+ # Another pod opened the window and already used the whole RPM budget.+ redis_store = {f"{scope}:window": str(now_int), f"{scope}:requests": 2}+ handler.batch_rate_limiter_script = _fake_redis_batch_script(redis_store)++ time_controller.advance(50)+ with pytest.raises(HTTPException):+ await _pre_call(handler, user, max_tokens=10)++ # The window opened 61s ago, so it has rolled in Redis.+ time_controller.advance(handler.window_size - 50 + 1)+ await _pre_call(handler, user, max_tokens=10)+ assert redis_store[f"{scope}:requests"] == 1+++@pytest.mark.asyncio+async def test_post_call_tpm_rejection_lifts_once_the_shared_window_rolls(time_controller, monkeypatch):+ """With TPM reservation disabled (post-call TPM enforcement), an exhausted+ TPM budget is restored once the window rolls instead of a stale local+ tokens counter extending the cooldown past the window."""+ monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", "false")+ handler = _handler(time_controller)+ api_key = hash_token("sk-tpm-cooldown")+ user = UserAPIKeyAuth(api_key=api_key, rpm_limit=100, tpm_limit=1000)+ scope = f"{{api_key:{api_key}}}"+ now_int = int(time_controller.now().timestamp())+ # Another pod opened the window and the key already spent its TPM budget.+ redis_store = {f"{scope}:window": str(now_int), f"{scope}:requests": 5, f"{scope}:tokens": 1200}+ handler.batch_rate_limiter_script = _fake_redis_batch_script(redis_store)++ time_controller.advance(50)+ with pytest.raises(HTTPException) as exc_info:+ await _pre_call(handler, user, max_tokens=10)+ assert "tokens" in str(exc_info.value.detail)++ time_controller.advance(handler.window_size - 50 + 1)+ await _pre_call(handler, user, max_tokens=10)+ assert redis_store[f"{scope}:tokens"] == 1+++@pytest.mark.asyncio+async def test_atomic_in_memory_window_reset_clears_sibling_counters(time_controller):+ """The atomic check-and-increment path (TPM reservation, no Redis) resets+ every counter of an expired window, so the RPM counter does not inherit+ the previous window's request count when TPM observes the expiry first."""+ handler = _handler(time_controller)+ cache = handler.internal_usage_cache.dual_cache+ scope = "{api_key:k}"+ await cache.async_set_cache(key=f"{scope}:window", value=str(int(time_controller.now().timestamp())), ttl=600)+ await cache.async_set_cache(key=f"{scope}:requests", value=99, ttl=600)+ await cache.async_set_cache(key=f"{scope}:tokens", value=999, ttl=600)++ time_controller.advance(handler.window_size + 1)+ response = await handler.reserve_tpm_tokens(+ descriptors=[{"key": "api_key", "value": "k", "rate_limit": {"tokens_per_unit": 1000}}],+ estimated_tokens=500,+ )++ assert response["overall_code"] == "OK"+ assert await cache.async_get_cache(key=f"{scope}:tokens") == 500+ assert await cache.async_get_cache(key=f"{scope}:requests") is None+++def test_redis_scripts_reset_every_counter_sharing_the_window():+ """Both Redis window-reset paths clear the sibling counters of the reset+ window before writing their own counter."""+ for script in (BATCH_RATE_LIMITER_SCRIPT, CHECK_AND_INCREMENT_BY_N_SCRIPT):+ assert "redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')" in script+ reset_call = script.index("reset_window_counters(window_key)\n")+ assert reset_call < script.index("redis.call('SET', window_key, tostring(now))")+++async def _pre_call_capturing_stash(handler: RateLimitHandler, user: UserAPIKeyAuth, max_tokens: int):+ token = _request_stash.set(None)+ try:+ await handler.async_pre_call_hook(+ user_api_key_dict=user,+ cache=handler.internal_usage_cache.dual_cache,+ data={"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}], "max_tokens": max_tokens},+ call_type="completion",+ )+ stash = get_request_stash()+ finally:+ _request_stash.reset(token)+ assert stash is not None and stash.reserved_tokens > 0+ return stash+++@pytest.mark.asyncio+@pytest.mark.parametrize("outcome", ["success", "failure"])+async def test_settling_a_reservation_from_a_rolled_window_leaves_the_new_window_alone(time_controller, outcome):+ """A request that reserved tokens in one window and finishes after the+ window rolled must not apply its ``actual - reserved`` delta to the new+ window's counter: the reset already discarded that reservation, so the+ delta would wipe out (or inflate) reservations made in the new window."""+ handler = _handler(time_controller)+ api_key = hash_token("sk-settle-after-rollover")+ user = UserAPIKeyAuth(api_key=api_key, rpm_limit=100, tpm_limit=10_000)+ tokens_key = f"{{api_key:{api_key}}}:tokens"+ cache = handler.internal_usage_cache.dual_cache++ old_window_stash = await _pre_call_capturing_stash(handler, user, max_tokens=3000)++ # The window rolls; a new request reserves against the fresh window.+ time_controller.advance(handler.window_size + 1)+ new_window_stash = await _pre_call_capturing_stash(handler, user, max_tokens=500)+ assert await cache.async_get_cache(key=tokens_key) == new_window_stash.reserved_tokens++ # The old-window request now finishes with almost no usage.+ token = _request_stash.set(old_window_stash)+ try:+ if outcome == "success":+ await handler.async_log_success_event(+ kwargs=_success_kwargs(api_key),+ response_obj=_usage_response(10),+ start_time=datetime.now(),+ end_time=datetime.now(),+ )+ else:+ await handler.async_log_failure_event(+ kwargs=_success_kwargs(api_key),+ response_obj=None,+ start_time=datetime.now(),+ end_time=datetime.now(),+ )+ finally:+ _request_stash.reset(token)++ assert await cache.async_get_cache(key=tokens_key) == new_window_stash.reserved_tokens
litellm/proxy/hooks/parallel_request_limiter_v3.pyindex 89826694bb6..cdec5922fff 100644@@ -91,6 +91,11 @@ else: _REQUEST_RATE_LIMIT_DATA: Final = TypeAdapter(Mapping[str, object]) +def _sibling_counter_keys(window_key: str) -> tuple[str, str]:+ prefix: Final = window_key.removesuffix(":window")+ return f"{prefix}:requests", f"{prefix}:tokens"++ BATCH_RATE_LIMITER_SCRIPT: Final = """ local results = {} local now = tonumber(ARGV[1])@@ -106,6 +111,8 @@ for i = 1, #KEYS, 2 do local window_start = redis.call('GET', window_key) if not window_start or (now - tonumber(window_start)) >= window_size then -- Reset window and counter+ local prefix = string.sub(window_key, 1, -(#':window') - 1)+ redis.call('DEL', prefix .. ':requests', prefix .. ':tokens') redis.call('SET', window_key, tostring(now)) redis.call('SET', counter_key, increment_value) redis.call('EXPIRE', window_key, window_size)@@ -151,6 +158,7 @@ CHECK_AND_INCREMENT_BY_N_SCRIPT: Final = """ local time_reply = redis.call('TIME') local now = tonumber(time_reply[1]) local descriptor_count = #KEYS / 2+local reset_windows = {} -- Pass 1: read state, validate. Abort without writing if any over limit. local descriptor_state = {}@@ -201,6 +209,11 @@ for i = 1, descriptor_count do if window_expired then active_window_start = now+ if not reset_windows[window_key] then+ local prefix = string.sub(window_key, 1, -(#':window') - 1)+ redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')+ reset_windows[window_key] = true+ end redis.call('SET', window_key, tostring(now)) redis.call('SET', counter_key, increment) redis.call('EXPIRE', window_key, window_size)@@ -1018,6 +1031,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): Implement sliding window rate limiting logic using in-memory cache operations. This follows the same logic as the Redis Lua script but uses async cache operations. """+ async with self._check_and_increment_lock:+ return await self._in_memory_cache_sliding_window(keys=keys, now_int=now_int, window_size=window_size)++ async def _in_memory_cache_sliding_window(+ self,+ keys: list[str],+ now_int: int,+ window_size: int,+ ) -> CacheCounterValues: results: Final[list[CacheCounterValue | None]] = [] # Process each window/counter pair@@ -1036,6 +1058,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): # Check if window exists and is valid if window_start is None or (now_int - int(window_start)) >= window_size: # Reset window and counter+ for sibling_counter_key in _sibling_counter_keys(window_key):+ await self.internal_usage_cache.async_set_cache(+ key=sibling_counter_key,+ value=0,+ ttl=window_size,+ litellm_parent_otel_span=None,+ local_only=True,+ ) await self.internal_usage_cache.async_set_cache( key=window_key, value=str(now_int),@@ -2048,6 +2078,20 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) # Pass 2: apply increments.+ expired_windows: Final[Mapping[str, int]] = {+ meta["window_key"]: meta["window_size"]+ for meta, state in zip(per_counter_meta, descriptor_state)+ if state["window_expired"]+ }+ for window_key, window_size in expired_windows.items():+ for sibling_counter_key in _sibling_counter_keys(window_key):+ await self.internal_usage_cache.async_set_cache(+ key=sibling_counter_key,+ value=0,+ ttl=window_size,+ litellm_parent_otel_span=parent_otel_span,+ local_only=True,+ ) statuses: Final[list[RateLimitStatus]] = [] for meta, state in zip(per_counter_meta, descriptor_state): new_counter = meta["increment"] if state["window_expired"] else state["current"] + meta["increment"]tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.pyindex 85023b94207..5889b1b513f 100644@@ -18,11 +18,13 @@ from fastapi import HTTPException import litellm from litellm import Router from litellm.caching.caching import DualCache+from litellm.caching.in_memory_cache import InMemoryCache from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY from litellm.proxy._types import UserAPIKeyAuth from litellm.proxy.hooks.parallel_request_limiter_v3 import ( PARALLEL_REQUEST_SLOT_TTL_SECONDS, ParallelSlotAcquisition,+ RateLimitDescriptor, RequestRateLimiterStash, _request_stash, get_or_create_request_stash,@@ -5602,6 +5604,155 @@ async def _reserved_tokens_for( return int(await local_cache.async_get_cache(key=tokens_key) or 0) +@pytest.mark.asyncio+async def test_tpm_reservation_resets_sibling_tokens_with_request_window(monkeypatch):+ monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", "true")+ time_controller = TimeController()+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache),+ time_provider=time_controller.now,+ )+ user_api_key_dict = UserAPIKeyAuth(+ api_key=hash_token("sk-window-reset-siblings"),+ tpm_limit=1000,+ rpm_limit=1000,+ )++ async def request(call_id):+ data = {+ "model": "gpt-4o",+ "messages": [{"role": "user", "content": "hi"}],+ "max_tokens": 200,+ "litellm_call_id": call_id,+ "metadata": {+ "user_api_key": user_api_key_dict.api_key,+ "user_api_key_user_id": user_api_key_dict.user_id,+ },+ }+ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data=data,+ call_type="completion",+ )+ await handler.async_log_success_event(+ kwargs={+ "litellm_call_id": call_id,+ "litellm_params": {+ "metadata": {+ "user_api_key": user_api_key_dict.api_key,+ "user_api_key_user_id": user_api_key_dict.user_id,+ "model_group": "gpt-4o",+ }+ },+ "standard_logging_object": {+ "metadata": {+ "user_api_key_hash": user_api_key_dict.api_key,+ "user_api_key_user_id": user_api_key_dict.user_id,+ }+ },+ },+ response_obj=ModelResponse(+ model="gpt-4o",+ usage=Usage(prompt_tokens=100, completion_tokens=200, total_tokens=300),+ ),+ start_time=datetime.now(),+ end_time=datetime.now(),+ )++ tokens_key = handler.create_rate_limit_keys(+ key="api_key", value=user_api_key_dict.api_key, rate_limit_type="tokens"+ )+ for index in range(3):+ await request(f"call-{index}")+ assert await local_cache.async_get_cache(key=tokens_key) == (index + 1) * 300++ time_controller.advance(61)+ await request("call-after-window-reset")+ assert await local_cache.async_get_cache(key=tokens_key) == 300+++@pytest.mark.asyncio+async def test_atomic_tpm_reservation_rollover_resets_sibling_requests_counter():+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(local_cache))+ window_size = 60+ now_int = int(time.time())+ window_key = "{api_key:atomic-rollover}:window"+ requests_key = handler.create_rate_limit_keys("api_key", "atomic-rollover", "requests")+ tokens_key = handler.create_rate_limit_keys("api_key", "atomic-rollover", "tokens")+ for key, value in ((window_key, str(now_int - window_size - 1)), (requests_key, 3), (tokens_key, 900)):+ await local_cache.async_set_cache(key=key, value=value, ttl=window_size)++ tpm_pass = await handler.atomic_check_and_increment_by_n(+ descriptors=[+ RateLimitDescriptor(+ key="api_key",+ value="atomic-rollover",+ rate_limit={"tokens_per_unit": 1000, "window_size": window_size},+ )+ ],+ increments=[{"tokens": 200}],+ )+ assert tpm_pass["overall_code"] == "OK"+ assert await local_cache.async_get_cache(key=tokens_key) == 200++ rpm_pass = await handler.should_rate_limit(+ descriptors=[+ RateLimitDescriptor(+ key="api_key",+ value="atomic-rollover",+ rate_limit={"requests_per_unit": 5, "window_size": window_size},+ )+ ],+ skip_tpm_check=True,+ )+ assert rpm_pass["overall_code"] == "OK"+ assert [status["limit_remaining"] for status in rpm_pass["statuses"]] == [4]+ assert await local_cache.async_get_cache(key=requests_key) == 1+++class _YieldingInMemoryCache(InMemoryCache):+ async def async_get_cache(self, key: str, **kwargs: object) -> object:+ value = await super().async_get_cache(key, **kwargs)+ await asyncio.sleep(0)+ return value+++@pytest.mark.asyncio+async def test_window_rollover_reset_does_not_erase_concurrent_sibling_increment():+ local_cache = DualCache(in_memory_cache=_YieldingInMemoryCache())+ handler = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(local_cache))+ window_size = 60+ now_int = int(time.time())+ window_key = "{api_key:concurrent-rollover}:window"+ requests_key = handler.create_rate_limit_keys("api_key", "concurrent-rollover", "requests")+ tokens_key = handler.create_rate_limit_keys("api_key", "concurrent-rollover", "tokens")+ for key, value in ((window_key, str(now_int - window_size - 1)), (requests_key, 3), (tokens_key, 900)):+ await local_cache.async_set_cache(key=key, value=value, ttl=window_size)++ tpm_descriptor = RateLimitDescriptor(+ key="api_key",+ value="concurrent-rollover",+ rate_limit={"tokens_per_unit": 1000, "window_size": window_size},+ )+ rpm_pass, tpm_pass = await asyncio.gather(+ handler.in_memory_cache_sliding_window(+ keys=[window_key, requests_key], now_int=now_int, window_size=window_size+ ),+ handler.atomic_check_and_increment_by_n(+ descriptors=[tpm_descriptor],+ increments=[{"requests": 0, "tokens": 200}],+ ),+ )++ assert rpm_pass == [str(now_int), 1]+ assert tpm_pass["overall_code"] == "OK"+ assert await local_cache.async_get_cache(key=requests_key) == 1+ assert await local_cache.async_get_cache(key=tokens_key) == 200++ @pytest.mark.asyncio @pytest.mark.parametrize( "key_metadata, team_metadata, expected_output_estimate, tier",