Backtestslitellmrun-20261019-033420#24677

Issue #24677 Merged fix PR #41838 Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: Incorrect TPM limiting for virtual keys

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches identify the same root cause: the `:requests` and `:tokens` counters share one `:window` key, so a window reset triggered through the RPM counter left the tokens counter carrying the previous window's count, which explains both the ~30% under-enforcement and the multi-minute cooldown. Both fix this in BATCH_RATE_LIMITER_SCRIPT, CHECK_AND_INCREMENT_BY_N_SCRIPT, in_memory_cache_sliding_window, and the in-memory pass of atomic_check_and_increment_by_n, with per-run dedup so a later pair on the same window does not wipe a counter just started. Patch B goes one step further that matters for the issue: is_cache_list_over_limit now treats a mirrored counter as empty when its window start is missing or older than window_size, so the in-memory pre-check stops rejecting from a stale counter after the window ends. That is exactly the 'cooldown longer than a minute' symptom, and the independent tests confirm the gap: A still fails test_rate_limited_key_admitted_after_one_minute_without_reservation with the identical 363s stuck-limit as base, while B passes all six issue-derived scenarios. Patch A's extra asyncio lock around in_memory_cache_sliding_window is plausible hardening against a concurrent reset erasing a sibling increment, but it relies on an attribute not shown and does not close the stale pre-check path. On tests, A's are behavioral and include a concurrency case; B's include end-to-end time-advanced regressions for reservation, non-reservation, and RPM rollover plus a helper unit test, though its Lua tests are string assertions and count for little. Scope and craft are comparable; B's shared helper and comments make the invariant clearer. B resolves more of the reported behavior at the root.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

44m 5s

Wall-clock time in the container

Full test suite

No new failures

2,184 passed · 1 skipped · 2,186 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

334vs 195 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

This was initially reported on #18953 for v1.80.0. It was marked as solved, but we are still seeing this behaviour in v1.82.3

Actual rate limit enforcement is usually ~30% below the configured limit, and the cooldown period is longer than a minute. Usually 2, but can reach 5 minutes in some instances.

This is with the default rate limit type and configuration.

Steps to Reproduce

  1. Configure an API key with TPM limit (e.g. 500k)
  2. Use the API key
  3. Rate limit enforcement hits at approximately 300-350k.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.3

Twitter / LinkedIn details

_No response_

The two patches

Our patch+329 −5
litellm/proxy/hooks/parallel_request_limiter_v3.pyindex 89826694bb..5c592232dd 100644@@ -92,6 +92,27 @@ _REQUEST_RATE_LIMIT_DATA: Final = TypeAdapter(Mapping[str, object])   BATCH_RATE_LIMITER_SCRIPT: Final = """+-- The :requests and :tokens counters of a descriptor share one window key.+-- A window reset must zero every counter on that window, not just the one+-- being processed, or the sibling keeps the previous window's count while+-- the next check sees a fresh window and never resets it. Each window is+-- reset at most once per script run so a later pair on the same window does+-- not wipe a counter an earlier pair just started.+local reset_windows = {}+local function reset_sibling_counters(window_key, counter_key)+    if reset_windows[window_key] or string.sub(window_key, -7) ~= ':window' then+        return+    end+    reset_windows[window_key] = true+    local prefix = string.sub(window_key, 1, -7)+    for _, counter_type in ipairs({ 'requests', 'tokens' }) do+        local sibling_key = prefix .. counter_type+        if sibling_key ~= counter_key then+            redis.call('DEL', sibling_key)+        end+    end+end+ local results = {} local now = tonumber(ARGV[1]) local window_size = tonumber(ARGV[2])@@ -105,7 +126,8 @@ for i = 1, #KEYS, 2 do     -- Check if window exists and is valid     local window_start = redis.call('GET', window_key)     if not window_start or (now - tonumber(window_start)) >= window_size then-        -- Reset window and counter+        -- Reset window and every counter sharing it+        reset_sibling_counters(window_key, counter_key)         redis.call('SET', window_key, tostring(now))         redis.call('SET', counter_key, increment_value)         redis.call('EXPIRE', window_key, window_size)@@ -148,6 +170,27 @@ CHECK_AND_INCREMENT_BY_N_SCRIPT: Final = """ -- Return on success: --     { 0, new_counter_1, window_start_1, new_counter_2, window_start_2, ... } -- Return on over-limit: { 1, descriptor_index, current_counter, limit }+-- The :requests and :tokens counters of a descriptor share one window key.+-- A window reset must zero every counter on that window, not just the one+-- being processed, or the sibling keeps the previous window's count while+-- the next check sees a fresh window and never resets it. Each window is+-- reset at most once per script run so a later pair on the same window does+-- not wipe a counter an earlier pair just started.+local reset_windows = {}+local function reset_sibling_counters(window_key, counter_key)+    if reset_windows[window_key] or string.sub(window_key, -7) ~= ':window' then+        return+    end+    reset_windows[window_key] = true+    local prefix = string.sub(window_key, 1, -7)+    for _, counter_type in ipairs({ 'requests', 'tokens' }) do+        local sibling_key = prefix .. counter_type+        if sibling_key ~= counter_key then+            redis.call('DEL', sibling_key)+        end+    end+end+ local time_reply = redis.call('TIME') local now = tonumber(time_reply[1]) local descriptor_count = #KEYS / 2@@ -201,6 +244,7 @@ for i = 1, descriptor_count do      if window_expired then         active_window_start = now+        reset_sibling_counters(window_key, counter_key)         redis.call('SET', window_key, tostring(now))         redis.call('SET', counter_key, increment)         redis.call('EXPIRE', window_key, window_size)@@ -340,6 +384,29 @@ end return results """ +# Counter types that share a descriptor's ``{key:value}:window`` key. Must+# match the counter types reset by ``reset_sibling_counters`` in the Lua+# scripts above.+WINDOWED_COUNTER_TYPES: Final = ("requests", "tokens")+_WINDOW_KEY_SUFFIX: Final = ":window"+++def sibling_counter_keys(window_key: str, counter_key: str) -> tuple[str, ...]:+    """Counter keys other than ``counter_key`` that share ``window_key``.++    A window reset must zero these too, otherwise they carry the previous+    window's count into the new one.+    """+    if not window_key.endswith(_WINDOW_KEY_SUFFIX):+        return ()+    prefix: Final = window_key[: -len("window")]+    return tuple(+        sibling_key+        for counter_type in WINDOWED_COUNTER_TYPES+        if (sibling_key := f"{prefix}{counter_type}") != counter_key+    )++ # Redis cluster slot count REDIS_CLUSTER_SLOTS: Final = 16384 REDIS_NODE_HASHTAG_NAME: Final = "all_keys"@@ -1008,6 +1075,29 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):             self.internal_usage_cache.dual_cache.redis_cache, RedisClusterCache         ) +    async def _reset_sibling_counters_in_memory(+        self,+        window_key: str,+        counter_key: str,+        ttl: int,+        reset_windows: set[str],+        parent_otel_span: Span | None = None,+    ) -> None:+        """In-memory twin of the Lua ``reset_sibling_counters``: zero every+        other counter sharing ``window_key`` the first time that window is+        reset within one check pass."""+        if window_key in reset_windows:+            return+        reset_windows.add(window_key)+        for sibling_key in sibling_counter_keys(window_key, counter_key):+            await self.internal_usage_cache.async_set_cache(+                key=sibling_key,+                value=0,+                ttl=ttl,+                litellm_parent_otel_span=parent_otel_span,+                local_only=True,+            )+     async def in_memory_cache_sliding_window(         self,         keys: list[str],@@ -1019,6 +1109,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):         This follows the same logic as the Redis Lua script but uses async cache operations.         """         results: Final[list[CacheCounterValue | None]] = []+        reset_windows: Final[set[str]] = set()  # mutable-ok: tracks windows reset during this pass          # Process each window/counter pair         for i in range(0, len(keys), 2):@@ -1035,7 +1126,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):              # Check if window exists and is valid             if window_start is None or (now_int - int(window_start)) >= window_size:-                # Reset window and counter+                # Reset window and every counter sharing it+                await self._reset_sibling_counters_in_memory(+                    window_key=window_key,+                    counter_key=counter_key,+                    ttl=window_size,+                    reset_windows=reset_windows,+                )                 await self.internal_usage_cache.async_set_cache(                     key=window_key,                     value=str(now_int),@@ -1090,9 +1187,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):         keys_to_fetch: list[str],         cache_values: CacheCounterValues,         key_metadata: dict[str, WindowKeyMetadata],+        now_int: int | None = None,     ) -> RateLimitResponse:         """         Check if the cache values are over the limit.++        When ``now_int`` is given, a counter whose window has already ended+        (or whose window start is unknown) is treated as empty: it belongs to+        a previous window and must not keep rejecting requests after the+        window rolls over. Expiry uses ``self.window_size``, the same window+        the batch script and its in-memory twin reset these keys on.         """         statuses: Final[list[RateLimitStatus]] = []         overall_code = "OK"@@ -1101,7 +1205,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):             item_code = "OK"             window_key = keys_to_fetch[i]             counter_key = keys_to_fetch[i + 1]-            counter_value = cache_values[i + 1]+            window_value = cache_values[i]+            counter_value = (+                None+                if now_int is not None+                and (window_value is None or now_int - int(window_value) >= self.window_size)+                else cache_values[i + 1]+            )             requests_limit = key_metadata[window_key]["requests_limit"]             tokens_limit = key_metadata[window_key]["tokens_limit"] @@ -1309,7 +1419,9 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):             )              if cache_values is not None:-                rate_limit_response: Final = self.is_cache_list_over_limit(keys_to_fetch, cache_values, key_metadata)+                rate_limit_response: Final = self.is_cache_list_over_limit(+                    keys_to_fetch, cache_values, key_metadata, now_int=now_int+                )                 if rate_limit_response["overall_code"] == "OVER_LIMIT":                     return rate_limit_response @@ -1363,7 +1475,9 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):                     window_size=self.window_size,                 ) -            windowed_response = self.is_cache_list_over_limit(keys_to_fetch, cache_values, key_metadata)+            windowed_response = self.is_cache_list_over_limit(+                keys_to_fetch, cache_values, key_metadata, now_int=now_int+            )             if windowed_response["overall_code"] == "OVER_LIMIT":                 return windowed_response @@ -2049,9 +2163,17 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):          # Pass 2: apply increments.         statuses: Final[list[RateLimitStatus]] = []+        reset_windows: Final[set[str]] = set()  # mutable-ok: tracks windows reset during this pass         for meta, state in zip(per_counter_meta, descriptor_state):             new_counter = meta["increment"] if state["window_expired"] else state["current"] + meta["increment"]             if state["window_expired"]:+                await self._reset_sibling_counters_in_memory(+                    window_key=meta["window_key"],+                    counter_key=meta["counter_key"],+                    ttl=meta["ttl"],+                    reset_windows=reset_windows,+                    parent_otel_span=parent_otel_span,+                )                 await self.internal_usage_cache.async_set_cache(                     key=meta["window_key"],                     value=str(now_int),tests/test_litellm/proxy/hooks/test_rate_limit_window_rollover.pynew file mode 100644index 0000000000..c244dbe144@@ -0,0 +1,202 @@+"""+Regression tests for TPM limits being enforced below the configured value and+for longer than one window.++A descriptor's ``:requests`` and ``:tokens`` counters share a single+``{key:value}:window`` key. Resetting an expired window used to reset only the+counter being processed, so when the RPM pass rolled the window over first,+the TPM check then saw a "fresh" window and kept the previous window's token+count -- the tokens counter was effectively never reset. Separately, the+in-memory pre-check trusted a mirrored counter even after its window had+ended, so a key stayed rejected well past the window.++Only the system clock seen by the limiter (``time_provider``) is advanced, the+in-memory cache TTLs are not, which mirrors a counter key whose TTL outlives+its window (in-memory TTLs refreshed on every write, Redis TTLs refreshed by+pipeline increments).+"""++from datetime import datetime, timedelta++import pytest+from fastapi import HTTPException++from litellm.caching.caching import DualCache+from litellm.proxy._types import UserAPIKeyAuth+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+    BATCH_RATE_LIMITER_SCRIPT,+    CHECK_AND_INCREMENT_BY_N_SCRIPT,+    _request_stash,+    sibling_counter_keys,+)+from litellm.proxy.hooks.parallel_request_limiter_v3 import (+    _PROXY_MaxParallelRequestsHandler_v3 as RateLimitHandler,+)+from litellm.proxy.utils import InternalUsageCache, hash_token+from litellm.types.utils import ModelResponse, Usage++WINDOW_SIZE = 60+++class Clock:+    def __init__(self) -> None:+        self.current = datetime.now()++    def now(self) -> datetime:+        return self.current++    def advance(self, seconds: float) -> None:+        self.current += timedelta(seconds=seconds)+++@pytest.fixture(autouse=True)+def _isolated_request_stash():+    token = _request_stash.set(None)+    yield+    _request_stash.reset(token)+++@pytest.fixture+def clock() -> Clock:+    return Clock()+++def _make_handler(clock: Clock) -> tuple[RateLimitHandler, DualCache]:+    cache = DualCache()+    handler = RateLimitHandler(internal_usage_cache=InternalUsageCache(cache), time_provider=clock.now)+    assert handler.window_size == WINDOW_SIZE+    return handler, cache+++def _request(max_tokens: int) -> dict:+    return {+        "model": "gpt-3.5-turbo",+        "messages": [{"role": "user", "content": "hi"}],+        "max_tokens": max_tokens,+    }+++async def _pre_call(handler: RateLimitHandler, cache: DualCache, key: UserAPIKeyAuth, data: dict) -> None:+    _request_stash.set(None)+    await handler.async_pre_call_hook(user_api_key_dict=key, cache=cache, data=data, call_type="acompletion")+++@pytest.mark.asyncio+async def test_tpm_budget_is_restored_after_window_when_key_also_has_rpm_limit(clock):+    """Default (reservation) mode: once the window rolls over, a key with both+    rpm_limit and tpm_limit must get its full TPM budget back. The RPM pass+    resets the shared window first; the TPM reservation must not inherit the+    previous window's tokens."""+    handler, cache = _make_handler(clock)+    key = UserAPIKeyAuth(api_key=hash_token("sk-rollover-reservation"), rpm_limit=100, tpm_limit=1000)++    await _pre_call(handler, cache, key, _request(max_tokens=800))+    with pytest.raises(HTTPException) as exc_info:+        await _pre_call(handler, cache, key, _request(max_tokens=800))+    assert exc_info.value.status_code == 429++    clock.advance(WINDOW_SIZE + 1)++    await _pre_call(handler, cache, key, _request(max_tokens=800))++    tokens_counter = await cache.async_get_cache(+        key=handler.create_rate_limit_keys("api_key", key.api_key, "tokens"), local_only=True+    )+    assert 800 <= int(tokens_counter) < 1000+++@pytest.mark.asyncio+async def test_tpm_limit_recovers_after_one_window_without_reservation(clock, monkeypatch):+    """Post-call TPM enforcement (reservation disabled): a key that went over+    its TPM limit must be admitted again as soon as the window ends, and the+    new window must start from an empty tokens counter."""+    monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", "false")+    handler, cache = _make_handler(clock)+    api_key = hash_token("sk-rollover-post-call")+    key = UserAPIKeyAuth(api_key=api_key, rpm_limit=100, tpm_limit=1000)++    await _pre_call(handler, cache, key, _request(max_tokens=10))+    await handler.async_log_success_event(+        kwargs={+            "standard_logging_object": {"metadata": {"user_api_key_hash": api_key}},+            "model": "gpt-3.5-turbo",+        },+        response_obj=ModelResponse(+            model="gpt-3.5-turbo",+            usage=Usage(prompt_tokens=600, completion_tokens=600, total_tokens=1200),+        ),+        start_time=clock.now(),+        end_time=clock.now(),+    )+    with pytest.raises(HTTPException) as exc_info:+        await _pre_call(handler, cache, key, _request(max_tokens=10))+    assert exc_info.value.status_code == 429++    clock.advance(WINDOW_SIZE + 1)++    await _pre_call(handler, cache, key, _request(max_tokens=10))++    tokens_counter = await cache.async_get_cache(+        key=handler.create_rate_limit_keys("api_key", api_key, "tokens"), local_only=True+    )+    assert int(tokens_counter) == 1+++@pytest.mark.asyncio+async def test_rpm_limit_recovers_after_one_window(clock):+    """The in-memory pre-check must not keep rejecting from a counter whose+    window has already ended."""+    handler, cache = _make_handler(clock)+    key = UserAPIKeyAuth(api_key=hash_token("sk-rollover-rpm"), rpm_limit=2)++    for _ in range(2):+        await _pre_call(handler, cache, key, _request(max_tokens=10))+    with pytest.raises(HTTPException):+        await _pre_call(handler, cache, key, _request(max_tokens=10))++    clock.advance(WINDOW_SIZE + 1)++    await _pre_call(handler, cache, key, _request(max_tokens=10))+++@pytest.mark.asyncio+async def test_sliding_window_reset_zeroes_sibling_counter(clock):+    """Resetting a shared window through one counter zeroes the other counter+    on that window, whether or not it is part of the same pass."""+    handler, cache = _make_handler(clock)+    window_key = "{api_key:abc}:window"+    requests_key = "{api_key:abc}:requests"+    tokens_key = "{api_key:abc}:tokens"+    stale_start = int(clock.now().timestamp())+    for cache_key, value in ((window_key, str(stale_start)), (requests_key, 7), (tokens_key, 5000)):+        await cache.async_set_cache(key=cache_key, value=value, local_only=True)++    clock.advance(WINDOW_SIZE + 1)+    now_int = int(clock.now().timestamp())++    results = await handler.in_memory_cache_sliding_window(+        keys=[window_key, requests_key], now_int=now_int, window_size=WINDOW_SIZE+    )++    assert results == [str(now_int), 1]+    assert await cache.async_get_cache(key=tokens_key, local_only=True) == 0+++def test_sibling_counter_keys_cover_the_other_windowed_counters():+    assert sibling_counter_keys("{api_key:abc}:window", "{api_key:abc}:tokens") == ("{api_key:abc}:requests",)+    assert sibling_counter_keys("{api_key:abc}:window", "{api_key:abc}:requests") == ("{api_key:abc}:tokens",)+    assert sibling_counter_keys("not-a-window-key", "{api_key:abc}:tokens") == ()+++@pytest.mark.parametrize(+    "script",+    [BATCH_RATE_LIMITER_SCRIPT, CHECK_AND_INCREMENT_BY_N_SCRIPT],+    ids=["batch_rate_limiter", "check_and_increment_by_n"],+)+def test_redis_scripts_reset_sibling_counters_with_the_window(script):+    """The Redis Lua scripts (not executable in the unit-test environment)+    must clear sibling counters in the same branch that restarts the window."""+    window_reset = "redis.call('SET', window_key, tostring(now))"+    assert "for _, counter_type in ipairs({ 'requests', 'tokens' }) do" in script+    assert script.count(window_reset) == 1+    assert f"reset_sibling_counters(window_key, counter_key)\n        {window_reset}" in script 
Merged fix · PR #41838+195 −0
litellm/proxy/hooks/parallel_request_limiter_v3.pyindex 89826694bb6..cdec5922fff 100644@@ -91,6 +91,11 @@ else: _REQUEST_RATE_LIMIT_DATA: Final = TypeAdapter(Mapping[str, object])  +def _sibling_counter_keys(window_key: str) -> tuple[str, str]:+    prefix: Final = window_key.removesuffix(":window")+    return f"{prefix}:requests", f"{prefix}:tokens"++ BATCH_RATE_LIMITER_SCRIPT: Final = """ local results = {} local now = tonumber(ARGV[1])@@ -106,6 +111,8 @@ for i = 1, #KEYS, 2 do     local window_start = redis.call('GET', window_key)     if not window_start or (now - tonumber(window_start)) >= window_size then         -- Reset window and counter+        local prefix = string.sub(window_key, 1, -(#':window') - 1)+        redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')         redis.call('SET', window_key, tostring(now))         redis.call('SET', counter_key, increment_value)         redis.call('EXPIRE', window_key, window_size)@@ -151,6 +158,7 @@ CHECK_AND_INCREMENT_BY_N_SCRIPT: Final = """ local time_reply = redis.call('TIME') local now = tonumber(time_reply[1]) local descriptor_count = #KEYS / 2+local reset_windows = {}  -- Pass 1: read state, validate. Abort without writing if any over limit. local descriptor_state = {}@@ -201,6 +209,11 @@ for i = 1, descriptor_count do      if window_expired then         active_window_start = now+        if not reset_windows[window_key] then+            local prefix = string.sub(window_key, 1, -(#':window') - 1)+            redis.call('DEL', prefix .. ':requests', prefix .. ':tokens')+            reset_windows[window_key] = true+        end         redis.call('SET', window_key, tostring(now))         redis.call('SET', counter_key, increment)         redis.call('EXPIRE', window_key, window_size)@@ -1018,6 +1031,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):         Implement sliding window rate limiting logic using in-memory cache operations.         This follows the same logic as the Redis Lua script but uses async cache operations.         """+        async with self._check_and_increment_lock:+            return await self._in_memory_cache_sliding_window(keys=keys, now_int=now_int, window_size=window_size)++    async def _in_memory_cache_sliding_window(+        self,+        keys: list[str],+        now_int: int,+        window_size: int,+    ) -> CacheCounterValues:         results: Final[list[CacheCounterValue | None]] = []          # Process each window/counter pair@@ -1036,6 +1058,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):             # Check if window exists and is valid             if window_start is None or (now_int - int(window_start)) >= window_size:                 # Reset window and counter+                for sibling_counter_key in _sibling_counter_keys(window_key):+                    await self.internal_usage_cache.async_set_cache(+                        key=sibling_counter_key,+                        value=0,+                        ttl=window_size,+                        litellm_parent_otel_span=None,+                        local_only=True,+                    )                 await self.internal_usage_cache.async_set_cache(                     key=window_key,                     value=str(now_int),@@ -2048,6 +2078,20 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):             )          # Pass 2: apply increments.+        expired_windows: Final[Mapping[str, int]] = {+            meta["window_key"]: meta["window_size"]+            for meta, state in zip(per_counter_meta, descriptor_state)+            if state["window_expired"]+        }+        for window_key, window_size in expired_windows.items():+            for sibling_counter_key in _sibling_counter_keys(window_key):+                await self.internal_usage_cache.async_set_cache(+                    key=sibling_counter_key,+                    value=0,+                    ttl=window_size,+                    litellm_parent_otel_span=parent_otel_span,+                    local_only=True,+                )         statuses: Final[list[RateLimitStatus]] = []         for meta, state in zip(per_counter_meta, descriptor_state):             new_counter = meta["increment"] if state["window_expired"] else state["current"] + meta["increment"]tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.pyindex 85023b94207..5889b1b513f 100644@@ -18,11 +18,13 @@ from fastapi import HTTPException import litellm from litellm import Router from litellm.caching.caching import DualCache+from litellm.caching.in_memory_cache import InMemoryCache from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY from litellm.proxy._types import UserAPIKeyAuth from litellm.proxy.hooks.parallel_request_limiter_v3 import (     PARALLEL_REQUEST_SLOT_TTL_SECONDS,     ParallelSlotAcquisition,+    RateLimitDescriptor,     RequestRateLimiterStash,     _request_stash,     get_or_create_request_stash,@@ -5602,6 +5604,155 @@ async def _reserved_tokens_for(     return int(await local_cache.async_get_cache(key=tokens_key) or 0)  +@pytest.mark.asyncio+async def test_tpm_reservation_resets_sibling_tokens_with_request_window(monkeypatch):+    monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", "true")+    time_controller = TimeController()+    local_cache = DualCache()+    handler = _PROXY_MaxParallelRequestsHandler(+        internal_usage_cache=InternalUsageCache(local_cache),+        time_provider=time_controller.now,+    )+    user_api_key_dict = UserAPIKeyAuth(+        api_key=hash_token("sk-window-reset-siblings"),+        tpm_limit=1000,+        rpm_limit=1000,+    )++    async def request(call_id):+        data = {+            "model": "gpt-4o",+            "messages": [{"role": "user", "content": "hi"}],+            "max_tokens": 200,+            "litellm_call_id": call_id,+            "metadata": {+                "user_api_key": user_api_key_dict.api_key,+                "user_api_key_user_id": user_api_key_dict.user_id,+            },+        }+        await handler.async_pre_call_hook(+            user_api_key_dict=user_api_key_dict,+            cache=local_cache,+            data=data,+            call_type="completion",+        )+        await handler.async_log_success_event(+            kwargs={+                "litellm_call_id": call_id,+                "litellm_params": {+                    "metadata": {+                        "user_api_key": user_api_key_dict.api_key,+                        "user_api_key_user_id": user_api_key_dict.user_id,+                        "model_group": "gpt-4o",+                    }+                },+                "standard_logging_object": {+                    "metadata": {+                        "user_api_key_hash": user_api_key_dict.api_key,+                        "user_api_key_user_id": user_api_key_dict.user_id,+                    }+                },+            },+            response_obj=ModelResponse(+                model="gpt-4o",+                usage=Usage(prompt_tokens=100, completion_tokens=200, total_tokens=300),+            ),+            start_time=datetime.now(),+            end_time=datetime.now(),+        )++    tokens_key = handler.create_rate_limit_keys(+        key="api_key", value=user_api_key_dict.api_key, rate_limit_type="tokens"+    )+    for index in range(3):+        await request(f"call-{index}")+        assert await local_cache.async_get_cache(key=tokens_key) == (index + 1) * 300++    time_controller.advance(61)+    await request("call-after-window-reset")+    assert await local_cache.async_get_cache(key=tokens_key) == 300+++@pytest.mark.asyncio+async def test_atomic_tpm_reservation_rollover_resets_sibling_requests_counter():+    local_cache = DualCache()+    handler = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(local_cache))+    window_size = 60+    now_int = int(time.time())+    window_key = "{api_key:atomic-rollover}:window"+    requests_key = handler.create_rate_limit_keys("api_key", "atomic-rollover", "requests")+    tokens_key = handler.create_rate_limit_keys("api_key", "atomic-rollover", "tokens")+    for key, value in ((window_key, str(now_int - window_size - 1)), (requests_key, 3), (tokens_key, 900)):+        await local_cache.async_set_cache(key=key, value=value, ttl=window_size)++    tpm_pass = await handler.atomic_check_and_increment_by_n(+        descriptors=[+            RateLimitDescriptor(+                key="api_key",+                value="atomic-rollover",+                rate_limit={"tokens_per_unit": 1000, "window_size": window_size},+            )+        ],+        increments=[{"tokens": 200}],+    )+    assert tpm_pass["overall_code"] == "OK"+    assert await local_cache.async_get_cache(key=tokens_key) == 200++    rpm_pass = await handler.should_rate_limit(+        descriptors=[+            RateLimitDescriptor(+                key="api_key",+                value="atomic-rollover",+                rate_limit={"requests_per_unit": 5, "window_size": window_size},+            )+        ],+        skip_tpm_check=True,+    )+    assert rpm_pass["overall_code"] == "OK"+    assert [status["limit_remaining"] for status in rpm_pass["statuses"]] == [4]+    assert await local_cache.async_get_cache(key=requests_key) == 1+++class _YieldingInMemoryCache(InMemoryCache):+    async def async_get_cache(self, key: str, **kwargs: object) -> object:+        value = await super().async_get_cache(key, **kwargs)+        await asyncio.sleep(0)+        return value+++@pytest.mark.asyncio+async def test_window_rollover_reset_does_not_erase_concurrent_sibling_increment():+    local_cache = DualCache(in_memory_cache=_YieldingInMemoryCache())+    handler = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(local_cache))+    window_size = 60+    now_int = int(time.time())+    window_key = "{api_key:concurrent-rollover}:window"+    requests_key = handler.create_rate_limit_keys("api_key", "concurrent-rollover", "requests")+    tokens_key = handler.create_rate_limit_keys("api_key", "concurrent-rollover", "tokens")+    for key, value in ((window_key, str(now_int - window_size - 1)), (requests_key, 3), (tokens_key, 900)):+        await local_cache.async_set_cache(key=key, value=value, ttl=window_size)++    tpm_descriptor = RateLimitDescriptor(+        key="api_key",+        value="concurrent-rollover",+        rate_limit={"tokens_per_unit": 1000, "window_size": window_size},+    )+    rpm_pass, tpm_pass = await asyncio.gather(+        handler.in_memory_cache_sliding_window(+            keys=[window_key, requests_key], now_int=now_int, window_size=window_size+        ),+        handler.atomic_check_and_increment_by_n(+            descriptors=[tpm_descriptor],+            increments=[{"requests": 0, "tokens": 200}],+        ),+    )++    assert rpm_pass == [str(now_int), 1]+    assert tpm_pass["overall_code"] == "OK"+    assert await local_cache.async_get_cache(key=requests_key) == 1+    assert await local_cache.async_get_cache(key=tokens_key) == 200++ @pytest.mark.asyncio @pytest.mark.parametrize(     "key_metadata, team_metadata, expected_output_estimate, tier",