Backtestslitellmrun-20261111-220245#32614

Issue #32614 Merged fix PR #43125 Run 11 Nov 2026 · run-20261111-220245 BerriAI/litellm

[Bug]: Router budget sync can overwrite memory cache with stale Redis spend

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches fix the headline sync race: the sync now awaits the Redis pipeline before reading Redis back, and both snapshot/clear the queue and requeue on failure (independent tests 1-4 pass on both). The difference is the second path the issue names explicitly: `_PROXY_VirtualKeyModelMaxBudgetLimiter` awaits `_push_in_memory_increments_to_redis()` after logging and the issue says that await should mean Redis has been updated. Patch A keeps that method fire-and-forget (it just tracks the task in `in_flight_redis_pushes`), so the virtual-key path is unchanged; the independent test for it fails on A exactly as on base, and reading A's diff confirms the push still returns before the pipeline runs. A also leaves a residual window: a success callback can create a push task after sync's `gather` over in-flight tasks but before sync's own flush, so sync can still read Redis while that task is in flight. Patch B makes the push genuinely await the pipeline (through a shielded task so cancellation of the caller cannot lose or double-apply a batch), serializes flush/sync with a lock, requeues with per-key coalescing so the queue stays bounded while Redis is down, and folds any increments recorded during Redis I/O into the value written back to memory. That resolves every scenario in the issue, including the virtual-key limiter, and all five independent tests pass. B's tests are also stronger: they pin event ordering (pipeline before batch_get), requeue, in-flight increments surviving a flush, and cancellation edges. Costs on B's side: it is a considerably larger redesign with intricate cancellation handling, it adds a lock around the in-memory increment on the hot path, it flips the existing circuit-breaker test expectation (now requeues rather than drops), and it includes unrelated reformatting of existing tests. Those are real scope and risk concerns, but A leaves a path the issue names still broken, and correctness weighs first.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

1h 10m

Wall-clock time in the container

Full test suite

No new failures

263 passed · 0 skipped · 263 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

213vs 675 merged

Changed lines, ours vs merged

Check for existing issues

I found #20886, but it is closed as stale/not planned and cannot be reopened. This issue adds current-branch verification and additional affected code paths

What happened?

RouterBudgetLimiting._push_in_memory_increments_to_redis() still schedules redis_cache.async_increment_pipeline(...) with asyncio.create_task(...) and immediately clears the queue. _sync_in_memory_spend_with_redis() awaits that method, but the await only waits for task scheduling. It then fetches Redis and writes those values into in-memory cache before the pipeline has necessarily completed

On current litellm_internal_staging at 60729f733e, fetched on 2026-07-09, I reproduced memory being overwritten from 160 to stale Redis 100. After the background pipeline finishes, Redis is 160, but memory remains 100

Why this matters

async_filter_deployments() reads budget spend through DualCache.async_batch_get_cache(). DualCache returns in-memory values first and only checks Redis for misses, so the stale memory value is trusted by later provider, deployment, and tag budget checks

The same inherited flush method also affects _PROXY_VirtualKeyModelMaxBudgetLimiter: it awaits _push_in_memory_increments_to_redis() after logging virtual-key and end-user model spend, but that await does not currently mean Redis has been updated

A related implementation already uses a safer pattern. BaseRoutingStrategy._push_in_memory_increments_to_redis() awaits redis_cache.async_increment_pipeline(...) and _sync_in_memory_spend_with_redis() consumes that awaited result before merging values back into memory

Current code references

FileDetail
litellm/router_strategy/budget_limiter.py:522Defines RouterBudgetLimiting._push_in_memory_increments_to_redis()
litellm/router_strategy/budget_limiter.py:539Schedules the Redis pipeline with asyncio.create_task(...)
litellm/router_strategy/budget_limiter.py:569_sync_in_memory_spend_with_redis() awaits _push_in_memory_increments_to_redis() before reading Redis
litellm/router_strategy/budget_limiter.py:593Sync reads Redis with async_batch_get_cache(key_list=cache_keys) immediately after scheduling the pipeline
litellm/router_strategy/budget_limiter.py:155Budget filtering reads spend through self.dual_cache.async_batch_get_cache(...)
litellm/caching/dual_cache.py:274Defines DualCache.async_batch_get_cache(...)
litellm/caching/dual_cache.py:284Reads in-memory cache first
litellm/caching/dual_cache.py:289Falls through to Redis only when the result contains None
litellm/proxy/hooks/model_max_budget_limiter.py:21_PROXY_VirtualKeyModelMaxBudgetLimiter inherits RouterBudgetLimiting
litellm/proxy/hooks/model_max_budget_limiter.py:320Model-budget success logging awaits _push_in_memory_increments_to_redis()
litellm/router_strategy/base_routing_strategy.py:106Related base strategy flush helper
litellm/router_strategy/base_routing_strategy.py:206Related base strategy sync awaits the flush result before merging

Steps to reproduce

This is a minimal reproduction using __new__ to avoid the background periodic task interfering with the test

import asyncio
from types import SimpleNamespace

from litellm.router_strategy.budget_limiter import RouterBudgetLimiting
from litellm.types.utils import BudgetConfig


class MockRedisCache:
    def __init__(self, values):
        self.values = values
        self.events = []

    async def async_increment_pipeline(self, increment_list, **kwargs):
        self.events.append("increment_pipeline:start")
        await asyncio.sleep(0.05)
        for op in increment_list:
            key = op["key"]
            self.values[key] = float(self.values.get(key, 0.0) or 0.0) + float(op["increment_value"])
        self.events.append("increment_pipeline:done")

    async def async_batch_get_cache(self, key_list, **kwargs):
        self.events.append("batch_get")
        return {key: self.values.get(key) for key in key_list}


class MockInMemoryCache:
    def __init__(self, values):
        self.values = values

    async def async_set_cache(self, key, value, **kwargs):
        self.values[key] = float(value)


async def main():
    spend_key = "provider_spend:openai:1d"
    redis_cache = MockRedisCache({spend_key: 100.0})
    in_memory_cache = MockInMemoryCache({spend_key: 160.0})

    budget_limiter = RouterBudgetLimiting.__new__(RouterBudgetLimiting)
    budget_limiter.dual_cache = SimpleNamespace(redis_cache=redis_cache, in_memory_cache=in_memory_cache)
    budget_limiter.provider_budget_config = {"openai": BudgetConfig(time_period="1d", budget_limit=500.0)}
    budget_limiter.deployment_budget_config = None
    budget_limiter.tag_budget_config = None
    budget_limiter.redis_increment_operation_queue = [
        {"key": spend_key, "increment_value": 60.0, "ttl": 86400}
    ]

    await budget_limiter._sync_in_memory_spend_with_redis()
    memory_immediately = in_memory_cache.values[spend_key]
    redis_immediately = redis_cache.values[spend_key]
    events_immediately = list(redis_cache.events)
    await asyncio.sleep(0.075)

    print({
        "memory_immediately_after_sync": memory_immediately,
        "redis_immediately_after_sync": redis_immediately,
        "memory_after_pipeline_finishes": in_memory_cache.values[spend_key],
        "redis_after_pipeline_finishes": redis_cache.values[spend_key],
        "events": redis_cache.events,
        "events_immediately_after_sync": events_immediately,
        "queue": budget_limiter.redis_increment_operation_queue,
    })


asyncio.run(main())

Observed result

{
  'memory_immediately_after_sync': 100.0,
  'redis_immediately_after_sync': 100.0,
  'memory_after_pipeline_finishes': 100.0,
  'redis_after_pipeline_finishes': 160.0,
  'events': ['batch_get', 'increment_pipeline:start', 'increment_pipeline:done'],
  'events_immediately_after_sync': ['batch_get'],
  'queue': []
}

The event order shows batch_get happens before increment_pipeline:start, so sync reads Redis before queued increments have been applied

Expected result

_push_in_memory_increments_to_redis() should await the Redis increment pipeline before _sync_in_memory_spend_with_redis() reads Redis. It should snapshot and clear the queue before awaiting so new increments that arrive during the flush remain queued for the next sync, and it should requeue the flushed batch if the pipeline fails

There is an existing unmerged branch, origin/fix/issue-20886-budget-sync-race, with commit 620fa2dcfd fix(router-budget): await redis pipeline before sync reads, that appears to implement that queue snapshot, await, and requeue behavior

What part of LiteLLM is this about?

Proxy

LiteLLM version

Current litellm_internal_staging at 60729f733e, fetched from upstream on 2026-07-09

The two patches

Our patch+194 −19
litellm/proxy/hooks/model_max_budget_limiter.pyindex cfa54ae01a..3f4f07b92d 100644@@ -1,3 +1,4 @@+import asyncio import json import time from collections.abc import Iterable, Mapping, Sequence@@ -297,6 +298,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting):     def __init__(self, dual_cache: DualCache):         self.dual_cache = dual_cache         self.redis_increment_operation_queue = []+        self.in_flight_redis_pushes: set[asyncio.Task[bool]] = set()         self.deployment_budget_config = None      async def is_key_within_model_budget(litellm/router_strategy/budget_limiter.pyindex 3e094df7ac..422739bd25 100644@@ -28,7 +28,12 @@ from typing import Any, Final import litellm from litellm._logging import verbose_router_logger from litellm.caching.caching import DualCache-from litellm.caching.redis_cache import RedisCache, RedisPipelineIncrementOperation, log_redis_failure+from litellm.caching.redis_cache import (+    RedisCache,+    RedisCircuitBreakerOpenError,+    RedisPipelineIncrementOperation,+    log_redis_failure,+) from litellm.integrations.custom_logger import CustomLogger, Span from litellm.litellm_core_utils.core_helpers import (     get_metadata_variable_name_from_kwargs,@@ -93,13 +98,6 @@ class _LiteLLMParamsDictView:         return dict(self._params)  -async def _push_increments_to_redis(redis_cache: RedisCache, queued: list[RedisPipelineIncrementOperation]) -> None:-    try:-        await redis_cache.async_increment_pipeline(increment_list=queued)-    except Exception as e:-        log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)-- class RouterBudgetLimiting(CustomLogger):     def __init__(         self,@@ -109,6 +107,7 @@ class RouterBudgetLimiting(CustomLogger):     ):         self.dual_cache = dual_cache         self.redis_increment_operation_queue: list[RedisPipelineIncrementOperation] = []+        self.in_flight_redis_pushes: set[asyncio.Task[bool]] = set()         asyncio.create_task(self.periodic_sync_in_memory_spend_with_redis())         self.provider_budget_config: GenericBudgetConfigType | None = provider_budget_config         self.deployment_budget_config: GenericBudgetConfigType | None = None@@ -534,23 +533,53 @@ class RouterBudgetLimiting(CustomLogger):         - async_log_success_event collects all provider spend increments in `redis_increment_operation_queue`         - This function pushes all increments to Redis in a batched pipeline to optimize performance +        Runs inside request success callbacks, so the Redis round trip is handed to a task. The task is+        tracked in `in_flight_redis_pushes` so `_sync_in_memory_spend_with_redis` can wait for it before+        reading Redis back into memory.+         Only runs if Redis is initialized         """         try:-            if not self.dual_cache.redis_cache:+            redis_cache: Final = self.dual_cache.redis_cache+            if not redis_cache:                 return  # Redis is not initialized+            if not self.redis_increment_operation_queue:+                return -            verbose_router_logger.debug(-                "Pushing Redis Increment Pipeline for queue: %s",-                self.redis_increment_operation_queue,-            )-            queued: Final = self.redis_increment_operation_queue-            self.redis_increment_operation_queue = []-            if queued:-                asyncio.create_task(_push_increments_to_redis(self.dual_cache.redis_cache, queued))+            push: Final = asyncio.create_task(self._flush_increment_queue_to_redis(redis_cache))+            self.in_flight_redis_pushes.add(push)+            push.add_done_callback(self.in_flight_redis_pushes.discard)++        except Exception as e:+            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e) +    async def _flush_increment_queue_to_redis(self, redis_cache: RedisCache) -> bool:+        """+        Push the queued spend increments to Redis and wait for the pipeline to finish.++        The queue is snapshotted and cleared before awaiting, so increments logged during the flush stay+        queued for the next one. If the pipeline fails, the snapshot is put back in front of the queue so+        the next flush retries it; an open circuit breaker drops it instead, so the queue cannot grow+        without bound while Redis is marked down.++        Returns True when every queued increment is in Redis.+        """+        queued: Final = self.redis_increment_operation_queue+        self.redis_increment_operation_queue = []+        if not queued:+            return True++        verbose_router_logger.debug("Pushing Redis Increment Pipeline for queue: %s", queued)+        try:+            await redis_cache.async_increment_pipeline(increment_list=queued)+        except RedisCircuitBreakerOpenError as e:+            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)+            return False         except Exception as e:             log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)+            self.redis_increment_operation_queue = queued + self.redis_increment_operation_queue+            return False+        return True      async def _sync_in_memory_spend_with_redis(self):         """@@ -570,8 +599,12 @@ class RouterBudgetLimiting(CustomLogger):             if self.dual_cache.redis_cache is None:                 return -            # 1. Push all provider spend increments to Redis-            await self._push_in_memory_increments_to_redis()+            # 1. Push all provider spend increments to Redis, waiting for any push a success callback+            # already started, so the read below sees every increment this instance has logged+            if self.in_flight_redis_pushes:+                await asyncio.gather(*self.in_flight_redis_pushes, return_exceptions=True)+            if not await self._flush_increment_queue_to_redis(self.dual_cache.redis_cache):+                return  # Redis is missing this instance's increments, so its values would overwrite fresher memory              # 2. Fetch all current provider spend from Redis to update in-memory cache             cache_keys: Final = []tests/test_litellm/router_strategy/test_budget_limiter_hotpath.pyindex 4cc8fe7881..aea385eecc 100644@@ -396,3 +396,98 @@ async def test_push_task_failure_is_logged_once_and_not_leaked(disable_budget_sy         "Error syncing in-memory cache with Redis: Error 61 connecting to 127.0.0.1:6379"     ]     unretrieved.assert_not_called()+++SPEND_KEY = "provider_spend:openai:1d"+++def _redis_holding(values: dict[str, float], *, fail_pipeline: BaseException | None = None) -> MagicMock:+    """A Redis double whose increment pipeline yields to the loop before applying, like a real round trip."""+    redis_cache = MagicMock(spec=RedisCache)++    async def increment_pipeline(increment_list, **_: object) -> list[float]:+        await asyncio.sleep(0.01)+        if fail_pipeline is not None:+            raise fail_pipeline+        for op in increment_list:+            values[op["key"]] = values.get(op["key"], 0.0) + op["increment_value"]+        return [values[op["key"]] for op in increment_list]++    async def batch_get(key_list, **_: object) -> dict[str, float | None]:+        return {key: values.get(key) for key in key_list}++    redis_cache.async_increment_pipeline = AsyncMock(side_effect=increment_pipeline)+    redis_cache.async_batch_get_cache = AsyncMock(side_effect=batch_get)+    return redis_cache+++async def _limiter_with_local_spend(redis_cache: MagicMock, local_spend: float) -> RouterBudgetLimiting:+    limiter = RouterBudgetLimiting(+        dual_cache=DualCache(redis_cache=redis_cache),+        provider_budget_config={"openai": BudgetConfig(max_budget=500.0, budget_duration="1d")},+    )+    await asyncio.gather(*(task for task in asyncio.all_tasks() if task is not asyncio.current_task()))+    await limiter.dual_cache.in_memory_cache.async_set_cache(key=SPEND_KEY, value=local_spend)+    return limiter+++@pytest.mark.asyncio+async def test_sync_does_not_overwrite_memory_with_redis_spend_missing_queued_increments(disable_budget_sync):+    redis_values = {SPEND_KEY: 100.0}+    limiter = await _limiter_with_local_spend(_redis_holding(redis_values), local_spend=160.0)+    limiter.redis_increment_operation_queue = [{"key": SPEND_KEY, "increment_value": 60.0, "ttl": 86400}]++    await limiter._sync_in_memory_spend_with_redis()++    assert redis_values == {SPEND_KEY: 160.0}+    assert await limiter.dual_cache.in_memory_cache.async_get_cache(key=SPEND_KEY) == 160.0, (+        "sync read Redis before the queued +60 landed and wrote the stale 100 into memory"+    )+    assert limiter.redis_increment_operation_queue == []+++@pytest.mark.asyncio+async def test_sync_waits_for_a_push_a_success_callback_already_started(disable_budget_sync):+    redis_values = {SPEND_KEY: 100.0}+    limiter = await _limiter_with_local_spend(_redis_holding(redis_values), local_spend=160.0)+    limiter.redis_increment_operation_queue = [{"key": SPEND_KEY, "increment_value": 60.0, "ttl": 86400}]+    await limiter._push_in_memory_increments_to_redis()+    await asyncio.sleep(0)++    await limiter._sync_in_memory_spend_with_redis()++    assert await limiter.dual_cache.in_memory_cache.async_get_cache(key=SPEND_KEY) == 160.0, (+        "sync read Redis while the callback's +60 pipeline was still in flight"+    )+    assert redis_values == {SPEND_KEY: 160.0}+++@pytest.mark.asyncio+async def test_failed_sync_flush_requeues_increments_and_keeps_memory(disable_budget_sync):+    redis_values = {SPEND_KEY: 100.0}+    redis_cache = _redis_holding(redis_values, fail_pipeline=ConnectionError("Connection reset by peer"))+    limiter = await _limiter_with_local_spend(redis_cache, local_spend=160.0)+    queued = [{"key": SPEND_KEY, "increment_value": 60.0, "ttl": 86400}]+    limiter.redis_increment_operation_queue = list(queued)++    await limiter._sync_in_memory_spend_with_redis()++    assert limiter.redis_increment_operation_queue == queued+    assert await limiter.dual_cache.in_memory_cache.async_get_cache(key=SPEND_KEY) == 160.0+    redis_cache.async_batch_get_cache.assert_not_awaited()+++@pytest.mark.asyncio+async def test_increments_logged_during_a_flush_stay_queued_for_the_next_one(disable_budget_sync):+    redis_values = {SPEND_KEY: 100.0}+    limiter = await _limiter_with_local_spend(_redis_holding(redis_values), local_spend=160.0)+    limiter.redis_increment_operation_queue = [{"key": SPEND_KEY, "increment_value": 60.0, "ttl": 86400}]+    late_increment = {"key": SPEND_KEY, "increment_value": 5.0, "ttl": 86400}++    sync = asyncio.create_task(limiter._sync_in_memory_spend_with_redis())+    await asyncio.sleep(0)+    limiter.redis_increment_operation_queue.append(late_increment)+    await sync++    assert limiter.redis_increment_operation_queue == [late_increment]+    assert redis_values == {SPEND_KEY: 160.0}tests/unit/proxy/hooks/test_unit_test_max_model_budget_limiter.pyindex efe41e1da9..3e5de1eddd 100644@@ -1684,3 +1684,48 @@ async def test_key_entry_without_a_spend_cap_does_not_lift_the_team_cap(key_entr             key_model_max_budget=key_model_max_budget,             model="openai/gpt-4",         )+++@pytest.mark.asyncio+async def test_virtual_key_spend_survives_a_failed_redis_push():+    """A Redis error on the success-callback push must keep the spend queued so the next push delivers it."""+    pushed_increments = []+    attempts = 0++    async def increment_pipeline(increment_list, **_):+        nonlocal attempts+        attempts += 1+        await asyncio.sleep(0)+        if attempts == 1:+            raise ConnectionError("Connection reset by peer")+        pushed_increments.extend(increment_list)+        return [op["increment_value"] for op in increment_list]++    redis_cache = AsyncMock(spec=RedisCache)+    redis_cache.async_get_cache.return_value = None+    redis_cache.async_increment_pipeline.side_effect = increment_pipeline+    limiter = _PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=DualCache(redis_cache=redis_cache))+    model = "gpt-4"++    def success_kwargs(response_cost):+        return {+            "standard_logging_object": {+                "response_cost": response_cost,+                "model": model,+                "metadata": {"user_api_key_hash": "vk-hash"},+            },+            "litellm_params": {+                "metadata": {+                    "user_api_key_model_max_budget": {model: {"budget_limit": 10.0, "time_period": "1d"}},+                },+            },+        }++    await limiter.async_log_success_event(success_kwargs(0.25), response_obj=None, start_time=None, end_time=None)+    await asyncio.gather(*limiter.in_flight_redis_pushes)+    await limiter.async_log_success_event(success_kwargs(0.5), response_obj=None, start_time=None, end_time=None)+    await asyncio.gather(*limiter.in_flight_redis_pushes)++    assert attempts == 2+    assert [op["increment_value"] for op in pushed_increments] == [0.25, 0.5]+    assert limiter.redis_increment_operation_queue == [] 
Merged fix · PR #43125+587 −88
litellm/proxy/hooks/model_max_budget_limiter.pyindex cfa54ae01a2..7ce50bf5ead 100644@@ -1,3 +1,4 @@+import asyncio import json import time from collections.abc import Iterable, Mapping, Sequence@@ -297,6 +298,9 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting):     def __init__(self, dual_cache: DualCache):         self.dual_cache = dual_cache         self.redis_increment_operation_queue = []+        self._redis_increment_queue_lock = asyncio.Lock()+        self._redis_increment_flush_lock = asyncio.Lock()+        self._detached_increment_operations = None         self.deployment_budget_config = None      async def is_key_within_model_budget(litellm/router_strategy/budget_limiter.pyindex 3e094df7ac8..4e84bded9de 100644@@ -21,8 +21,10 @@ anthropic: import asyncio import builtins import logging-from collections.abc import Mapping+from collections.abc import Mapping, Sequence from datetime import datetime, timedelta, timezone+from itertools import groupby+from types import MappingProxyType from typing import Any, Final  import litellm@@ -93,11 +95,12 @@ class _LiteLLMParamsDictView:         return dict(self._params)  -async def _push_increments_to_redis(redis_cache: RedisCache, queued: list[RedisPipelineIncrementOperation]) -> None:-    try:-        await redis_cache.async_increment_pipeline(increment_list=queued)-    except Exception as e:-        log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)+def _sum_increments_by_key(operations: Sequence[RedisPipelineIncrementOperation]) -> Mapping[str, float]:+    by_key: Final = groupby(+        sorted(operations, key=lambda operation: operation["key"]),+        key=lambda operation: operation["key"],+    )+    return MappingProxyType({key: sum(operation["increment_value"] for operation in group) for key, group in by_key})   class RouterBudgetLimiting(CustomLogger):@@ -109,6 +112,9 @@ class RouterBudgetLimiting(CustomLogger):     ):         self.dual_cache = dual_cache         self.redis_increment_operation_queue: list[RedisPipelineIncrementOperation] = []+        self._redis_increment_queue_lock = asyncio.Lock()+        self._redis_increment_flush_lock = asyncio.Lock()+        self._detached_increment_operations: tuple[RedisPipelineIncrementOperation, ...] | None = None         asyncio.create_task(self.periodic_sync_in_memory_spend_with_redis())         self.provider_budget_config: GenericBudgetConfigType | None = provider_budget_config         self.deployment_budget_config: GenericBudgetConfigType | None = None@@ -392,17 +398,97 @@ class RouterBudgetLimiting(CustomLogger):         - Increments the spend in memory cache (so spend instantly updated in memory)         - Queues the increment operation to Redis Pipeline (using batched pipeline to optimize performance. Using Redis for multi instance environment of LiteLLM)         """-        await self.dual_cache.in_memory_cache.async_increment(-            key=spend_key,-            value=response_cost,-            ttl=ttl,-        )         increment_op: Final = RedisPipelineIncrementOperation(             key=spend_key,             increment_value=response_cost,             ttl=ttl,         )-        self.redis_increment_operation_queue.append(increment_op)+        async with self._get_redis_increment_queue_lock():+            await self.dual_cache.in_memory_cache.async_increment(+                key=spend_key,+                value=response_cost,+                ttl=ttl,+            )+            self.redis_increment_operation_queue.append(increment_op)++    def _get_redis_increment_queue_lock(self) -> asyncio.Lock:+        return self._redis_increment_queue_lock++    async def _detach_queued_increment_operations(self) -> tuple[RedisPipelineIncrementOperation, ...]:+        async with self._get_redis_increment_queue_lock():+            if self._detached_increment_operations is not None:+                return self._detached_increment_operations+            increment_operations_to_flush: Final = tuple(self.redis_increment_operation_queue)+            if not increment_operations_to_flush:+                return increment_operations_to_flush+            self.redis_increment_operation_queue = []  # mutable-ok: emptied queue must stay appendable+            self._detached_increment_operations = increment_operations_to_flush+            return increment_operations_to_flush++    async def _clear_detached_increment_operations(self) -> None:+        async with self._get_redis_increment_queue_lock():+            self._detached_increment_operations = None++    async def _requeue_detached_increment_operations(self) -> None:+        async with self._get_redis_increment_queue_lock():+            detached_increment_operations: Final = self._detached_increment_operations+            if detached_increment_operations is None:+                return+            operations: Final = (*detached_increment_operations, *self.redis_increment_operation_queue)+            grouped_operations: Final = (+                (key, tuple(group))+                for key, group in groupby(+                    sorted(operations, key=lambda operation: operation["key"]),+                    key=lambda operation: operation["key"],+                )+            )+            self.redis_increment_operation_queue = [+                RedisPipelineIncrementOperation(+                    key=key,+                    increment_value=sum(operation["increment_value"] for operation in group),+                    ttl=group[-1]["ttl"],+                )+                for key, group in grouped_operations+            ]+            self._detached_increment_operations = None++    async def _flush_queued_increment_operations(self, redis_cache: RedisCache) -> bool:+        flush_task: Final = asyncio.create_task(self._write_queued_increment_operations(redis_cache))+        return await self._await_flush_task(flush_task)++    async def _await_flush_task(self, flush_task: asyncio.Task[bool]) -> bool:+        try:+            return await asyncio.shield(flush_task)+        except asyncio.CancelledError:+            while not flush_task.done():+                try:+                    await asyncio.shield(flush_task)+                except asyncio.CancelledError:+                    continue+            flush_task.result()+            raise++    async def _write_queued_increment_operations(self, redis_cache: RedisCache) -> bool:+        increment_operations_to_flush: Final = await self._detach_queued_increment_operations()+        if len(increment_operations_to_flush) == 0:+            await self._clear_detached_increment_operations()+            return True++        verbose_router_logger.debug(+            "Pushing Redis Increment Pipeline for queue: %s",+            increment_operations_to_flush,+        )+        increment_list: Final = list(  # mutable-ok: Redis pipeline contract requires a list+            increment_operations_to_flush+        )+        try:+            await redis_cache.async_increment_pipeline(increment_list=increment_list)+        except Exception as error:+            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", error)+            await self._requeue_detached_increment_operations()+            return False+        await self._clear_detached_increment_operations()+        return True      async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):         """Original method now uses helper functions"""@@ -528,29 +614,25 @@ class RouterBudgetLimiting(CustomLogger):                     DEFAULT_REDIS_SYNC_INTERVAL                 )  # Still wait DEFAULT_REDIS_SYNC_INTERVAL seconds on error before retrying -    async def _push_in_memory_increments_to_redis(self):+    async def _push_in_memory_increments_to_redis(self) -> bool:         """         How this works:         - async_log_success_event collects all provider spend increments in `redis_increment_operation_queue`         - This function pushes all increments to Redis in a batched pipeline to optimize performance -        Only runs if Redis is initialized+        Only runs if Redis is initialized. Returns False when the detached batch could not be+        written, so callers must not treat Redis as up to date.         """-        try:-            if not self.dual_cache.redis_cache:-                return  # Redis is not initialized+        redis_cache: Final = self.dual_cache.redis_cache+        if redis_cache is None:+            return True -            verbose_router_logger.debug(-                "Pushing Redis Increment Pipeline for queue: %s",-                self.redis_increment_operation_queue,-            )-            queued: Final = self.redis_increment_operation_queue-            self.redis_increment_operation_queue = []-            if queued:-                asyncio.create_task(_push_increments_to_redis(self.dual_cache.redis_cache, queued))+        flush_task: Final = asyncio.create_task(self._flush_queued_increments_with_lock(redis_cache))+        return await self._await_flush_task(flush_task) -        except Exception as e:-            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)+    async def _flush_queued_increments_with_lock(self, redis_cache: RedisCache) -> bool:+        async with self._redis_increment_flush_lock:+            return await self._flush_queued_increment_operations(redis_cache)      async def _sync_in_memory_spend_with_redis(self):         """@@ -569,43 +651,50 @@ class RouterBudgetLimiting(CustomLogger):             # No need to sync if Redis cache is not initialized             if self.dual_cache.redis_cache is None:                 return+            async with self._redis_increment_flush_lock:+                await self._flush_increments_then_copy_redis_spend()+        except Exception as e:+            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e) -            # 1. Push all provider spend increments to Redis-            await self._push_in_memory_increments_to_redis()--            # 2. Fetch all current provider spend from Redis to update in-memory cache-            cache_keys: Final = []+    async def _flush_increments_then_copy_redis_spend(self) -> None:+        redis_cache: Final = self.dual_cache.redis_cache+        if redis_cache is None or not await self._flush_queued_increment_operations(redis_cache):+            return -            if self.provider_budget_config is not None:-                for provider, config in self.provider_budget_config.items():-                    if config is None:-                        continue-                    cache_keys.append(f"provider_spend:{provider}:{config.budget_duration}")+        cache_keys: Final = [] -            if self.deployment_budget_config is not None:-                for model_id, config in self.deployment_budget_config.items():-                    if config is None:-                        continue-                    cache_keys.append(f"deployment_spend:{model_id}:{config.budget_duration}")+        if self.provider_budget_config is not None:+            for provider, config in self.provider_budget_config.items():+                if config is None:+                    continue+                cache_keys.append(f"provider_spend:{provider}:{config.budget_duration}") -            if self.tag_budget_config is not None:-                for tag, config in self.tag_budget_config.items():-                    if config is None:-                        continue-                    cache_keys.append(f"tag_spend:{tag}:{config.budget_duration}")+        if self.deployment_budget_config is not None:+            for model_id, config in self.deployment_budget_config.items():+                if config is None:+                    continue+                cache_keys.append(f"deployment_spend:{model_id}:{config.budget_duration}") -            # Batch fetch current spend values from Redis-            redis_values: Final = await self.dual_cache.redis_cache.async_batch_get_cache(key_list=cache_keys)+        if self.tag_budget_config is not None:+            for tag, config in self.tag_budget_config.items():+                if config is None:+                    continue+                cache_keys.append(f"tag_spend:{tag}:{config.budget_duration}") -            # Update in-memory cache with Redis values-            if isinstance(redis_values, dict):  # Check if redis_values is a dictionary-                for key, value in redis_values.items():-                    if value is not None:-                        await self.dual_cache.in_memory_cache.async_set_cache(key=key, value=float(value))-                        verbose_router_logger.debug("Updated in-memory cache for %s: %s", key, value)+        redis_values: Final = await redis_cache.async_batch_get_cache(key_list=cache_keys) -        except Exception as e:-            log_redis_failure(verbose_router_logger, logging.ERROR, "Error syncing in-memory cache with Redis", e)+        if not isinstance(redis_values, dict):+            return+        async with self._get_redis_increment_queue_lock():+            pending_spend_by_key: Final = _sum_increments_by_key(self.redis_increment_operation_queue)+            updated_spend_by_key: Final = tuple(+                (key, float(value) + pending_spend_by_key.get(key, 0.0))+                for key, value in redis_values.items()+                if value is not None+            )+            for key, updated_spend in updated_spend_by_key:+                await self.dual_cache.in_memory_cache.async_set_cache(key=key, value=updated_spend)+                verbose_router_logger.debug("Updated in-memory cache for %s: %s", key, updated_spend)      def _get_budget_config_for_deployment(         self,tests/test_litellm/router_strategy/test_budget_limiter_hotpath.pyindex 4cc8fe78811..a2c38a898e9 100644@@ -1,6 +1,8 @@ import asyncio import gc import logging+from types import SimpleNamespace+from typing import Final from unittest.mock import AsyncMock, MagicMock  import pytest@@ -9,6 +11,7 @@ import litellm from litellm.caching.caching import DualCache from litellm.caching.redis_cache import RedisCache, RedisCircuitBreakerOpenError from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+from litellm.types.caching import RedisPipelineIncrementOperation from litellm.types.router import LiteLLM_Params from litellm.types.utils import BudgetConfig @@ -30,9 +33,7 @@ async def test_get_llm_provider_for_deployment_dict_does_not_require_litellm_par ):     class RaiseOnInit:         def __init__(self, *args, **kwargs):-            raise AssertionError(-                "LiteLLM_Params should not be instantiated in hot path"-            )+            raise AssertionError("LiteLLM_Params should not be instantiated in hot path")      monkeypatch.setattr(         "litellm.router_strategy.budget_limiter.LiteLLM_Params",@@ -99,9 +100,7 @@ async def test_get_llm_provider_for_deployment_dict_view_supports_mapping_and_at   @pytest.mark.asyncio-async def test_async_filter_deployments_resolves_provider_once_per_deployment(-    disable_budget_sync, monkeypatch-):+async def test_async_filter_deployments_resolves_provider_once_per_deployment(disable_budget_sync, monkeypatch):     provider_budget = RouterBudgetLimiting(         dual_cache=DualCache(),         provider_budget_config={@@ -207,9 +206,7 @@ def _legacy_provider_resolution(deployment):     Reference implementation used before hot-path optimization.     """     try:-        _litellm_params = LiteLLM_Params(-            **deployment.get("litellm_params", {"model": ""})-        )+        _litellm_params = LiteLLM_Params(**deployment.get("litellm_params", {"model": ""}))         _, custom_llm_provider, _, _ = litellm.get_llm_provider(             model=_litellm_params.model,             litellm_params=_litellm_params,@@ -228,9 +225,7 @@ def _legacy_provider_resolution(deployment):     ], ) @pytest.mark.asyncio-async def test_get_llm_provider_for_deployment_matches_legacy_behavior(-    disable_budget_sync, deployment-):+async def test_get_llm_provider_for_deployment_matches_legacy_behavior(disable_budget_sync, deployment):     provider_budget = RouterBudgetLimiting(         dual_cache=DualCache(),         provider_budget_config={},@@ -242,9 +237,7 @@ async def test_get_llm_provider_for_deployment_matches_legacy_behavior(     assert current_provider == legacy_provider  -def test_register_deployment_budget_for_runtime_added_deployment(-    disable_budget_sync, monkeypatch-):+def test_register_deployment_budget_for_runtime_added_deployment(disable_budget_sync, monkeypatch):     import asyncio      monkeypatch.setattr(asyncio, "create_task", lambda coro: None)@@ -274,9 +267,7 @@ def test_register_deployment_budget_for_runtime_added_deployment(     assert budget_limiter._get_budget_config_for_deployment(model_id) is None  -def test_router_add_deployment_registers_deployment_budget(-    disable_budget_sync, monkeypatch-):+def test_router_add_deployment_registers_deployment_budget(disable_budget_sync, monkeypatch):     import asyncio      from litellm import Router@@ -304,9 +295,7 @@ def test_router_add_deployment_registers_deployment_budget(      budget_limiter = router._get_router_deployment_budget_limiter()     assert budget_limiter is not None-    config = budget_limiter._get_budget_config_for_deployment(-        "runtime-budget-deployment"-    )+    config = budget_limiter._get_budget_config_for_deployment("runtime-budget-deployment")     assert config is not None     assert config.max_budget == 0.000000000001 @@ -338,7 +327,9 @@ async def test_sync_refused_by_the_open_circuit_breaker_is_quiet_and_leaks_no_ta      assert caplog.records == []     unretrieved.assert_not_called()-    assert limiter.redis_increment_operation_queue == []+    assert limiter.redis_increment_operation_queue == [+        {"key": "provider_spend:openai:1d", "increment_value": 0.5, "ttl": 60}+    ]     assert redis_cache.async_increment_pipeline.await_count == 1  @@ -353,32 +344,34 @@ async def _limiter_with_redis(redis_cache: MagicMock) -> RouterBudgetLimiting:   @pytest.mark.asyncio-async def test_push_returns_before_redis_answers(disable_budget_sync):-    """The push runs inside the request success callback, so it must hand the Redis round trip to a task instead of waiting on it."""+async def test_push_waits_for_redis_before_completing(disable_budget_sync):+    redis_started = asyncio.Event()     redis_answered = asyncio.Event()      async def wait_for_redis(**_: object) -> None:+        redis_started.set()         await redis_answered.wait()      redis_cache = MagicMock(spec=RedisCache)     redis_cache.async_increment_pipeline = AsyncMock(side_effect=wait_for_redis)     limiter = await _limiter_with_redis(redis_cache) -    await asyncio.wait_for(limiter._push_in_memory_increments_to_redis(), timeout=1)-    await asyncio.sleep(0)--    assert not redis_answered.is_set()+    push_task = asyncio.create_task(limiter._push_in_memory_increments_to_redis())+    await asyncio.wait_for(redis_started.wait(), timeout=1)+    assert not push_task.done()+    redis_answered.set()+    assert await asyncio.wait_for(push_task, timeout=1) is True     assert redis_cache.async_increment_pipeline.await_count == 1     assert limiter.redis_increment_operation_queue == []-    redis_answered.set()-    await asyncio.gather(*(task for task in asyncio.all_tasks() if task is not asyncio.current_task()))   @pytest.mark.asyncio async def test_push_task_failure_is_logged_once_and_not_leaked(disable_budget_sync, caplog):     """A real Redis failure on the background push must surface as one error line, never as an unretrieved task exception."""     redis_cache = MagicMock(spec=RedisCache)-    redis_cache.async_increment_pipeline = AsyncMock(side_effect=ConnectionError("Error 61 connecting to 127.0.0.1:6379"))+    redis_cache.async_increment_pipeline = AsyncMock(+        side_effect=ConnectionError("Error 61 connecting to 127.0.0.1:6379")+    )     limiter = await _limiter_with_redis(redis_cache)     loop = asyncio.get_running_loop()     unretrieved = MagicMock()@@ -396,3 +389,398 @@ async def test_push_task_failure_is_logged_once_and_not_leaked(disable_budget_sy         "Error syncing in-memory cache with Redis: Error 61 connecting to 127.0.0.1:6379"     ]     unretrieved.assert_not_called()+++_SPEND_KEY = "provider_spend:openai:1d"+++def _increment(increment_value: float) -> RedisPipelineIncrementOperation:+    return RedisPipelineIncrementOperation(key=_SPEND_KEY, increment_value=increment_value, ttl=86400)+++class _ObservedLock(asyncio.Lock):+    def __init__(self) -> None:+        super().__init__()+        self.waiter_started = asyncio.Event()++    async def acquire(self) -> bool:+        if self.locked():+            self.waiter_started.set()+        return await super().acquire()+++class _MockRedisCache:+    def __init__(+        self,+        initial_values: dict[str, float],+        pipeline_started: asyncio.Event | None = None,+        allow_pipeline_to_complete: asyncio.Event | None = None,+        should_fail_pipeline: bool = False,+        pipeline_completed: asyncio.Event | None = None,+        read_started: asyncio.Event | None = None,+        allow_read_to_complete: asyncio.Event | None = None,+    ) -> None:+        self.values = initial_values+        self.events: list[str] = []+        self.pipeline_started = pipeline_started+        self.allow_pipeline_to_complete = allow_pipeline_to_complete+        self.should_fail_pipeline = should_fail_pipeline+        self.pipeline_completed = pipeline_completed+        self.read_started = read_started+        self.allow_read_to_complete = allow_read_to_complete++    async def async_increment_pipeline(+        self, increment_list: list[RedisPipelineIncrementOperation], **kwargs: object+    ) -> None:+        self.events.append("increment_pipeline:start")+        if self.pipeline_started is not None:+            self.pipeline_started.set()+        if self.allow_pipeline_to_complete is not None:+            await self.allow_pipeline_to_complete.wait()+        if self.should_fail_pipeline:+            raise RuntimeError("redis down")+        for op in increment_list:+            key = op["key"]+            current = float(self.values.get(key, 0.0) or 0.0)+            self.values[key] = current + float(op["increment_value"])+        self.events.append("increment_pipeline:done")+        if self.pipeline_completed is not None:+            self.pipeline_completed.set()++    async def async_batch_get_cache(self, key_list: list[str], **kwargs: object) -> dict[str, float | None]:+        self.events.append("batch_get")+        snapshot = {key: self.values.get(key) for key in key_list}+        if self.read_started is not None:+            self.read_started.set()+        if self.allow_read_to_complete is not None:+            await self.allow_read_to_complete.wait()+        return snapshot+++class _MockInMemoryCache:+    def __init__(self, initial_values: dict[str, float]) -> None:+        self.values = initial_values++    async def async_increment(self, key: str, value: float, ttl: int, **kwargs: object) -> float:+        current = float(self.values.get(key, 0.0) or 0.0)+        self.values[key] = current + float(value)+        return self.values[key]++    async def async_set_cache(self, key: str, value: float, **kwargs: object) -> None:+        self.values[key] = float(value)+++def _new_router_budget_limiter(+    *,+    redis_cache: object,+    queue_lock: asyncio.Lock | None = None,+    in_memory_cache: object | None = None,+    redis_increment_operation_queue: list[RedisPipelineIncrementOperation] | None = None,+    provider_budget_config: dict[str, BudgetConfig] | None = None,+) -> RouterBudgetLimiting:+    budget_limiter = RouterBudgetLimiting.__new__(RouterBudgetLimiting)+    budget_limiter.dual_cache = SimpleNamespace(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache if in_memory_cache is not None else SimpleNamespace(),+    )+    budget_limiter.provider_budget_config = provider_budget_config+    budget_limiter.deployment_budget_config = None+    budget_limiter.tag_budget_config = None+    budget_limiter.redis_increment_operation_queue = (+        list(redis_increment_operation_queue) if redis_increment_operation_queue is not None else []+    )+    budget_limiter._redis_increment_queue_lock = queue_lock if queue_lock is not None else asyncio.Lock()+    budget_limiter._redis_increment_flush_lock = asyncio.Lock()+    budget_limiter._detached_increment_operations = None+    return budget_limiter+++@pytest.mark.asyncio+async def test_should_await_redis_pipeline_before_sync_reads() -> None:+    pipeline_started = asyncio.Event()+    allow_pipeline_to_complete = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={_SPEND_KEY: 100.0},+        pipeline_started=pipeline_started,+        allow_pipeline_to_complete=allow_pipeline_to_complete,+    )+    in_memory_cache = _MockInMemoryCache(initial_values={_SPEND_KEY: 160.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache,+        redis_increment_operation_queue=[_increment(60.0)],+        provider_budget_config={"openai": BudgetConfig(time_period="1d", budget_limit=500.0)},+    )++    sync_task = asyncio.create_task(budget_limiter._sync_in_memory_spend_with_redis())+    await asyncio.wait_for(pipeline_started.wait(), timeout=1)+    assert "batch_get" not in redis_cache.events+    allow_pipeline_to_complete.set()+    await sync_task++    assert redis_cache.values[_SPEND_KEY] == 160.0+    assert in_memory_cache.values[_SPEND_KEY] == 160.0+    assert budget_limiter.redis_increment_operation_queue == []+    assert redis_cache.events == [+        "increment_pipeline:start",+        "increment_pipeline:done",+        "batch_get",+    ]+++@pytest.mark.asyncio+async def test_should_requeue_increments_when_redis_pipeline_fails() -> None:+    redis_cache = _MockRedisCache(initial_values={}, should_fail_pipeline=True)+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        redis_increment_operation_queue=[_increment(10.0)],+    )++    flush_succeeded = await budget_limiter._push_in_memory_increments_to_redis()++    assert flush_succeeded is False+    assert budget_limiter.redis_increment_operation_queue == [_increment(10.0)]+++@pytest.mark.asyncio+async def test_should_keep_new_increments_when_pipeline_flush_fails() -> None:+    pipeline_started = asyncio.Event()+    allow_pipeline_to_complete = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={},+        pipeline_started=pipeline_started,+        allow_pipeline_to_complete=allow_pipeline_to_complete,+        should_fail_pipeline=True,+    )+    in_memory_cache = _MockInMemoryCache(initial_values={_SPEND_KEY: 0.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache,+        redis_increment_operation_queue=[_increment(10.0)],+    )++    push_task = asyncio.create_task(budget_limiter._push_in_memory_increments_to_redis())+    await asyncio.wait_for(pipeline_started.wait(), timeout=1)+    await budget_limiter._increment_spend_in_current_window(spend_key=_SPEND_KEY, response_cost=20.0, ttl=86400)+    allow_pipeline_to_complete.set()+    await push_task++    assert budget_limiter.redis_increment_operation_queue == [_increment(30.0)]+++@pytest.mark.asyncio+async def test_failed_redis_flushes_coalesce_spend_by_key() -> None:+    other_spend_key: Final = "provider_spend:other:1d"+    redis_cache: Final = _MockRedisCache(+        initial_values={_SPEND_KEY: 0.0, other_spend_key: 0.0}, should_fail_pipeline=True+    )+    in_memory_cache: Final = _MockInMemoryCache(initial_values={_SPEND_KEY: 0.0, other_spend_key: 0.0})+    budget_limiter: Final = _new_router_budget_limiter(redis_cache=redis_cache, in_memory_cache=in_memory_cache)++    for spend_key, response_cost, ttl in (+        (_SPEND_KEY, 10.0, 90),+        (other_spend_key, 4.0, 50),+        (_SPEND_KEY, 20.0, 80),+        (_SPEND_KEY, 30.0, 70),+    ):+        await budget_limiter._increment_spend_in_current_window(spend_key, response_cost, ttl)+        assert await budget_limiter._push_in_memory_increments_to_redis() is False++    queued: Final = {operation["key"]: operation for operation in budget_limiter.redis_increment_operation_queue}+    assert len(budget_limiter.redis_increment_operation_queue) == 2+    assert queued[_SPEND_KEY] == RedisPipelineIncrementOperation(key=_SPEND_KEY, increment_value=60.0, ttl=70)+    assert queued[other_spend_key] == RedisPipelineIncrementOperation(key=other_spend_key, increment_value=4.0, ttl=50)++    redis_cache.should_fail_pipeline = False+    assert await budget_limiter._push_in_memory_increments_to_redis() is True+    assert redis_cache.values == {_SPEND_KEY: 60.0, other_spend_key: 4.0}+    assert budget_limiter.redis_increment_operation_queue == []+++@pytest.mark.asyncio+async def test_should_keep_in_memory_spend_when_redis_pipeline_fails() -> None:+    redis_cache = _MockRedisCache(initial_values={_SPEND_KEY: 100.0}, should_fail_pipeline=True)+    in_memory_cache = _MockInMemoryCache(initial_values={_SPEND_KEY: 160.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache,+        redis_increment_operation_queue=[_increment(60.0)],+        provider_budget_config={"openai": BudgetConfig(time_period="1d", budget_limit=500.0)},+    )++    await budget_limiter._sync_in_memory_spend_with_redis()++    assert in_memory_cache.values[_SPEND_KEY] == 160.0+    assert redis_cache.values[_SPEND_KEY] == 100.0+    assert budget_limiter.redis_increment_operation_queue == [_increment(60.0)]+    assert "batch_get" not in redis_cache.events+++@pytest.mark.asyncio+async def test_should_keep_increments_when_flush_is_cancelled_after_success() -> None:+    pipeline_started = asyncio.Event()+    allow_pipeline_to_complete = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={_SPEND_KEY: 0.0},+        pipeline_started=pipeline_started,+        allow_pipeline_to_complete=allow_pipeline_to_complete,+    )+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        redis_increment_operation_queue=[_increment(10.0)],+    )++    push_task = asyncio.create_task(budget_limiter._push_in_memory_increments_to_redis())+    await asyncio.wait_for(pipeline_started.wait(), timeout=1)+    push_task.cancel()+    allow_pipeline_to_complete.set()+    with pytest.raises(asyncio.CancelledError):+        await push_task++    assert redis_cache.values[_SPEND_KEY] == 10.0+    assert budget_limiter.redis_increment_operation_queue == []+++@pytest.mark.asyncio+async def test_cancelled_push_waiting_for_flush_lock_still_writes_spend() -> None:+    redis_cache = _MockRedisCache(initial_values={_SPEND_KEY: 0.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        redis_increment_operation_queue=[_increment(10.0)],+    )+    flush_lock = _ObservedLock()+    budget_limiter._redis_increment_flush_lock = flush_lock++    async with flush_lock:+        push_task = asyncio.create_task(budget_limiter._push_in_memory_increments_to_redis())+        await asyncio.wait_for(flush_lock.waiter_started.wait(), timeout=1)+        push_task.cancel()+        await asyncio.sleep(0)+        assert not push_task.done()++    with pytest.raises(asyncio.CancelledError):+        await asyncio.wait_for(push_task, timeout=1)++    assert redis_cache.values[_SPEND_KEY] == 10.0+    assert budget_limiter.redis_increment_operation_queue == []+++@pytest.mark.asyncio+async def test_empty_flush_does_not_block_later_increment_sync() -> None:+    redis_cache = _MockRedisCache(initial_values={_SPEND_KEY: 100.0})+    in_memory_cache = _MockInMemoryCache(initial_values={_SPEND_KEY: 100.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache,+        provider_budget_config={"openai": BudgetConfig(time_period="1d", budget_limit=500.0)},+    )++    empty_flush_succeeded = await budget_limiter._push_in_memory_increments_to_redis()+    await budget_limiter._increment_spend_in_current_window(spend_key=_SPEND_KEY, response_cost=20.0, ttl=86400)+    await budget_limiter._sync_in_memory_spend_with_redis()++    assert empty_flush_succeeded is True+    assert budget_limiter.redis_increment_operation_queue == []+    assert redis_cache.values[_SPEND_KEY] == 120.0+    assert in_memory_cache.values[_SPEND_KEY] == 120.0+    assert redis_cache.events == [+        "increment_pipeline:start",+        "increment_pipeline:done",+        "batch_get",+    ]+++@pytest.mark.asyncio+async def test_should_requeue_increments_when_flush_is_cancelled_and_redis_fails() -> None:+    pipeline_started = asyncio.Event()+    allow_pipeline_to_complete = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={_SPEND_KEY: 0.0},+        pipeline_started=pipeline_started,+        allow_pipeline_to_complete=allow_pipeline_to_complete,+        should_fail_pipeline=True,+    )+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        redis_increment_operation_queue=[_increment(10.0)],+    )++    push_task = asyncio.create_task(budget_limiter._push_in_memory_increments_to_redis())+    await asyncio.wait_for(pipeline_started.wait(), timeout=1)+    push_task.cancel()+    allow_pipeline_to_complete.set()+    with pytest.raises(asyncio.CancelledError):+        await push_task++    assert redis_cache.values[_SPEND_KEY] == 0.0+    assert budget_limiter.redis_increment_operation_queue == [_increment(10.0)]+++@pytest.mark.asyncio+@pytest.mark.parametrize("pause_during", ["write", "read"])+async def test_sync_preserves_spend_recorded_during_redis_io(pause_during: str) -> None:+    io_started = asyncio.Event()+    allow_io_to_complete = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={_SPEND_KEY: 100.0},+        pipeline_started=io_started if pause_during == "write" else None,+        allow_pipeline_to_complete=allow_io_to_complete if pause_during == "write" else None,+        read_started=io_started if pause_during == "read" else None,+        allow_read_to_complete=allow_io_to_complete if pause_during == "read" else None,+    )+    in_memory_cache = _MockInMemoryCache(initial_values={_SPEND_KEY: 160.0})+    budget_limiter = _new_router_budget_limiter(+        redis_cache=redis_cache,+        in_memory_cache=in_memory_cache,+        redis_increment_operation_queue=[_increment(60.0)],+        provider_budget_config={"openai": BudgetConfig(time_period="1d", budget_limit=175.0)},+    )++    sync_task = asyncio.create_task(budget_limiter._sync_in_memory_spend_with_redis())+    await asyncio.wait_for(io_started.wait(), timeout=1)+    await budget_limiter._increment_spend_in_current_window(_SPEND_KEY, 20.0, 86400)+    allow_io_to_complete.set()+    await sync_task++    assert in_memory_cache.values[_SPEND_KEY] == 180.0+    assert redis_cache.values[_SPEND_KEY] == 160.0+    assert budget_limiter.redis_increment_operation_queue == [_increment(20.0)]++    await budget_limiter._sync_in_memory_spend_with_redis()++    assert in_memory_cache.values[_SPEND_KEY] == 180.0+    assert redis_cache.values[_SPEND_KEY] == 180.0+    assert budget_limiter.redis_increment_operation_queue == []+++@pytest.mark.asyncio+@pytest.mark.parametrize("cancellations", [1, 2])+async def test_cancelled_flush_does_not_requeue_an_applied_batch(cancellations: int) -> None:+    pipeline_started = asyncio.Event()+    pipeline_completed = asyncio.Event()+    allow_pipeline = asyncio.Event()+    redis_cache = _MockRedisCache(+        initial_values={_SPEND_KEY: 0.0},+        pipeline_started=pipeline_started,+        pipeline_completed=pipeline_completed,+        allow_pipeline_to_complete=allow_pipeline,+    )+    queue_lock = _ObservedLock()+    limiter = _new_router_budget_limiter(+        redis_cache=redis_cache, queue_lock=queue_lock, redis_increment_operation_queue=[_increment(10.0)]+    )+    push_task = asyncio.create_task(limiter._push_in_memory_increments_to_redis())+    await asyncio.wait_for(pipeline_started.wait(), timeout=1)+    async with limiter._redis_increment_queue_lock:+        allow_pipeline.set()+        await asyncio.wait_for(pipeline_completed.wait(), timeout=1)+        await asyncio.wait_for(queue_lock.waiter_started.wait(), timeout=1)+        for _ in range(cancellations):+            push_task.cancel()+            await asyncio.sleep(0)+        assert not push_task.done()+    with pytest.raises(asyncio.CancelledError):+        await push_task+    await limiter._push_in_memory_increments_to_redis()+    assert redis_cache.values[_SPEND_KEY] == 10.0+    assert limiter.redis_increment_operation_queue == []tests/unit/proxy/hooks/test_unit_test_max_model_budget_limiter.pyindex efe41e1da9a..f194e43c74a 100644@@ -19,6 +19,7 @@ from litellm.proxy.hooks.model_max_budget_limiter import (     resolve_model_budget, ) from litellm.proxy._types import UserAPIKeyAuth+from litellm.types.caching import RedisPipelineIncrementOperation from litellm.types.utils import BudgetConfig as GenericBudgetInfo  @@ -487,6 +488,23 @@ async def test_async_log_success_event_pushes_redis_increments_when_redis_config             mock_push.assert_awaited_once()  +@pytest.mark.asyncio+async def test_model_budget_limiter_initializes_redis_increment_queue_lock():+    dual_cache = DualCache()+    limiter = _PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+    spend_key = "virtual_key_spend:test-key:gpt-4:1d"++    await limiter._increment_spend_in_current_window(+        spend_key=spend_key, response_cost=0.01, ttl=86400+    )++    assert limiter.redis_increment_operation_queue == [+        RedisPipelineIncrementOperation(+            key=spend_key, increment_value=0.01, ttl=86400+        )+    ]++ @pytest.mark.asyncio async def test_get_fallback_model_within_budget_returns_none_without_fallbacks(     budget_limiter,