Backtestslitellmrun-20261019-033420#37877

Issue #37877 Merged fix PR #38172 Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: provider budget never counts embedding or rerank spend — RouterBudgetLimiting raises "custom_llm_provider is required"

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Patch A fixes the root cause the issue identifies: embedding(), transcription(), speech() and rerank() now put the already-resolved custom_llm_provider into litellm_params, exactly as completion() does and as the issue suggests. It also adds a small generic fallback in Logging.update_environment_variables that copies the provider kwarg into litellm_params when absent, which covers other hand-built surfaces (moderation, etc.) without touching each one, and it preserves an explicit litellm_params value. Its tests exercise the real pipeline: an actual aembedding call with a mock response captured through the callback and then fed into RouterBudgetLimiting, plus handler-patched rerank/transcription/speech runs asserting the provider lands in both the logged and handler litellm_params. Patch B instead changes the consumer to read custom_llm_provider from the standard logging payload and silently skips accounting when it is None. That makes the four independent tests pass, but it leaves litellm_params incomplete for any other callback that reads it, and it converts the loud 'custom_llm_provider is required' error into exactly the silent under-counting the issue warns about. B's tests are synthetic kwargs dicts that never run through embedding/rerank, so they verify the limiter's new branch rather than the reported end-to-end behaviour. A is more correct at the root, better tested, and matches the surrounding conventions; B is a narrower symptom-level change with a masking risk.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

26m 46s

Wall-clock time in the container

Full test suite

No new failures

5,022 passed · 1 skipped · 5,025 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

192vs 147 merged

Changed lines, ours vs merged

What happened?

With router_settings.provider_budget_config enabled, every /v1/embeddings and /v1/rerank call raises inside the budget callback:

LiteLLM:ERROR: litellm_logging.py:2672 - LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging
  File ".../litellm/litellm_core_utils/litellm_logging.py", line 2618, in async_success_handler
    await callback.async_log_success_event(...)
  File ".../litellm/router_strategy/budget_limiter.py", line 412, in async_log_success_event
    raise ValueError("custom_llm_provider is required")
ValueError: custom_llm_provider is required

The error is labelled non-blocking and the request succeeds, so it looks cosmetic. It is not: RouterBudgetLimiting.async_log_success_event raises before _increment_spend_for_key, so embedding and rerank spend is never added to the provider budget. The ceiling silently under-counts, and a deployment that is mostly embeddings would never reach it.

/v1/chat/completions is unaffected — its spend does accumulate.

Root cause

budget_limiter.py:410 reads the provider from litellm_params:

custom_llm_provider: str = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)
if custom_llm_provider is None:
    raise ValueError("custom_llm_provider is required")

completion() puts it there (main.py:5289):

litellm_params = get_litellm_params(
    ...
    custom_llm_provider=custom_llm_provider,
    ...
)

embedding() does not (main.py:6028):

litellm_params_dict = get_litellm_params(**kwargs)

custom_llm_provider is a named parameter of embedding(), so it is never in **kwargs; and it has already been resolved a few lines above by get_llm_provider() — it is passed to logging.update_environment_variables(custom_llm_provider=...) on the very next statement, just not into litellm_params.

Same shape at main.py:7548 (transcription) and main.py:7825 (speech). rerank_api/main.py:168 builds rerank_litellm_params without it as well.

Because the parameter is consumed by the function signature, no configuration can work around this — passing custom_llm_provider in the request body or in the deployment's litellm_params still ends up as the named argument, never in **kwargs.

Relevant log output

Reproduced 3/3 on /v1/embeddings and /v1/rerank; 0/4 on /v1/chat/completions (streaming and non-streaming, cache hit and miss).

GET /provider/budgets on the same proxy shows spend accumulating from chat only:

{"providers":{"bedrock":{"budget_limit":1000.0,"time_period":"1mo","spend":159.18631146,...}}}

Suggested fix

Pass the already-resolved value, as completion() does:

litellm_params_dict = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)

Config

model_list:
  - model_name: titan-embed-v2
    litellm_params:
      model: bedrock/amazon.titan-embed-text-v2:0
      aws_region_name: us-east-1
  - model_name: cohere-rerank-v3-5
    litellm_params:
      model: bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
      aws_region_name: us-east-1

router_settings:
  provider_budget_config:
    bedrock:
      budget_limit: 1000
      time_period: 1mo
  redis_host: os.environ/REDIS_HOST

Twitter / LinkedIn details

_No response_

The two patches

Our patch+189 −3
litellm/litellm_core_utils/litellm_logging.pyindex 83ab2bc11a..78d4b46f58 100644@@ -911,6 +911,11 @@ class Logging(LiteLLMLoggingBaseClass):             **self.litellm_params,             **scrub_sensitive_keys_in_metadata(litellm_params),         }+        # Endpoints that hand-build litellm_params pass the resolved provider only as a kwarg;+        # success callbacks (e.g. provider budgets) read it from litellm_params.+        resolved_custom_llm_provider: Final = additional_params.get("custom_llm_provider")+        if resolved_custom_llm_provider is not None and self.litellm_params.get("custom_llm_provider") is None:+            self.litellm_params["custom_llm_provider"] = resolved_custom_llm_provider         self.litellm_request_debug = litellm_params.get("litellm_request_debug", False)         self.logger_fn = litellm_params.get("logger_fn", None)         if _is_debugging_on() or self.litellm_request_debug:litellm/main.pyindex 12854db15d..ec51c6106d 100644@@ -6391,7 +6391,7 @@ def embedding(             model_info=kwargs.get("model_info"),         ) -    litellm_params_dict: Final = get_litellm_params(**kwargs)+    litellm_params_dict: Final = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)      logging: Final[LiteLLMLoggingObj] = litellm_logging_obj     logging.update_environment_variables(@@ -7957,7 +7957,7 @@ def transcription(         **non_default_params,     ) -    litellm_params_dict: Final = get_litellm_params(**kwargs)+    litellm_params_dict: Final = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)      litellm_logging_obj.update_environment_variables(         model=model,@@ -8239,7 +8239,12 @@ def speech(      if max_retries is None:         max_retries = litellm.num_retries or openai.DEFAULT_MAX_RETRIES-    litellm_params_dict: Final = get_litellm_params(metadata=metadata, api_key=api_key or dynamic_api_key, **kwargs)+    litellm_params_dict: Final = get_litellm_params(+        metadata=metadata,+        api_key=api_key or dynamic_api_key,+        custom_llm_provider=custom_llm_provider,+        **kwargs,+    )      # Get provider-specific text-to-speech config and map parameters     text_to_speech_provider_config = ProviderConfigManager.get_provider_text_to_speech_config(litellm/rerank_api/main.pyindex 18ecc250b6..c56b55999d 100644@@ -189,6 +189,7 @@ def rerank(             "preset_cache_key": None,             "stream_response": {},             **optional_params.model_dump(exclude_unset=True),+            "custom_llm_provider": _custom_llm_provider,         }          litellm_logging_obj.update_from_kwargs(tests/test_litellm/router_strategy/test_budget_limiter_non_chat_provider.pynew file mode 100644index 0000000000..0e55e86800@@ -0,0 +1,175 @@+"""+Regression tests: provider budgets must count spend from non-chat endpoints.++embedding / rerank / transcription / speech used to log `litellm_params` without+`custom_llm_provider`, so RouterBudgetLimiting.async_log_success_event raised+"custom_llm_provider is required" before incrementing provider spend.+"""++import asyncio+from typing import Final+from unittest.mock import patch++import pytest++import litellm+from litellm.caching.caching import DualCache+from litellm.integrations.custom_logger import CustomLogger+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+from litellm.types.utils import BudgetConfig+++class _CaptureSuccessKwargs(CustomLogger):+    def __init__(self) -> None:+        super().__init__()+        self.kwargs: dict | None = None++    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):+        self.kwargs = kwargs+++class _HandlerReached(Exception):+    pass+++@pytest.fixture+def disable_budget_sync(monkeypatch):+    async def noop(*args, **kwargs):+        return None++    monkeypatch.setattr(+        "litellm.router_strategy.budget_limiter.RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redis",+        noop,+    )+++@pytest.mark.asyncio+async def test_embedding_spend_is_added_to_provider_budget(disable_budget_sync, monkeypatch):+    capture: Final = _CaptureSuccessKwargs()+    monkeypatch.setattr(litellm, "callbacks", [capture])++    await litellm.aembedding(+        model="openai/text-embedding-3-small",+        input=["hello world"],+        mock_response=[0.1, 0.2, 0.3],+    )+    for _ in range(50):+        if capture.kwargs is not None:+            break+        await asyncio.sleep(0.05)+    assert capture.kwargs is not None+    assert capture.kwargs["litellm_params"]["custom_llm_provider"] == "openai"++    provider_budget: Final = RouterBudgetLimiting(+        dual_cache=DualCache(),+        provider_budget_config={"openai": BudgetConfig(budget_limit=100, time_period="1d")},+    )+    capture.kwargs["standard_logging_object"]["response_cost"] = 0.25+    await provider_budget.async_log_success_event(capture.kwargs, None, None, None)++    spend: Final = await provider_budget.dual_cache.in_memory_cache.async_get_cache("provider_spend:openai:1d")+    assert spend == pytest.approx(0.25)+++def test_rerank_logs_custom_llm_provider_in_litellm_params():+    captured: dict = {}++    def _fake_rerank(*args, **kwargs):+        captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+        captured["handler"] = kwargs["litellm_params"]+        raise _HandlerReached()++    with patch("litellm.rerank_api.main.base_llm_http_handler.rerank", side_effect=_fake_rerank):+        with pytest.raises(Exception):+            litellm.rerank(+                model="cohere/rerank-english-v3.0",+                query="hello",+                documents=["hello world"],+                api_key="fake-key",+            )++    assert captured["logged"]["custom_llm_provider"] == "cohere"+    assert captured["handler"]["custom_llm_provider"] == "cohere"+++def test_transcription_logs_custom_llm_provider_in_litellm_params():+    captured: dict = {}++    def _fake_transcription(*args, **kwargs):+        captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+        captured["handler"] = kwargs["litellm_params"]+        raise _HandlerReached()++    with patch("litellm.main.openai_audio_transcriptions.audio_transcriptions", side_effect=_fake_transcription):+        with pytest.raises(Exception):+            litellm.transcription(+                model="openai/whisper-1",+                file=("audio.wav", b"RIFF0000WAVEfmt ", "audio/wav"),+                api_key="fake-key",+            )++    assert captured["logged"]["custom_llm_provider"] == "openai"+    assert captured["handler"]["custom_llm_provider"] == "openai"+++def test_speech_logs_custom_llm_provider_in_litellm_params():+    captured: dict = {}++    def _fake_speech(*args, **kwargs):+        captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+        raise _HandlerReached()++    with patch("litellm.main.openai_chat_completions.audio_speech", side_effect=_fake_speech):+        with pytest.raises(Exception):+            litellm.speech(+                model="openai/tts-1",+                input="hello",+                voice="alloy",+                api_key="fake-key",+            )++    assert captured["logged"]["custom_llm_provider"] == "openai"+++def test_moderation_logs_custom_llm_provider_in_litellm_params():+    """amoderation hand-builds litellm_params and passes the provider only as a kwarg."""+    logging_obj: Final = LiteLLMLoggingObj(+        model="omni-moderation-latest",+        messages=[],+        stream=False,+        call_type="moderation",+        start_time=None,+        litellm_call_id="test-call-id",+        function_id="test-function-id",+    )+    logging_obj.update_environment_variables(+        model="omni-moderation-latest",+        user=None,+        optional_params={},+        litellm_params={},+        custom_llm_provider="openai",+    )++    assert logging_obj.model_call_details["litellm_params"]["custom_llm_provider"] == "openai"+++def test_explicit_litellm_params_provider_is_not_overridden():+    logging_obj: Final = LiteLLMLoggingObj(+        model="gpt-4o",+        messages=[],+        stream=False,+        call_type="completion",+        start_time=None,+        litellm_call_id="test-call-id",+        function_id="test-function-id",+    )+    logging_obj.update_environment_variables(+        model="gpt-4o",+        user=None,+        optional_params={},+        litellm_params={"custom_llm_provider": "azure"},+        custom_llm_provider="openai",+    )++    assert logging_obj.model_call_details["litellm_params"]["custom_llm_provider"] == "azure" 
Merged fix · PR #38172+142 −5
litellm/router_strategy/budget_limiter.pyindex 4e84bded9de..64252cbbfb3 100644@@ -502,12 +502,12 @@ class RouterBudgetLimiting(CustomLogger):          response_cost: Final[float] = standard_logging_payload.get("response_cost", 0)         model_id: Final[str] = str(standard_logging_payload.get("model_id", ""))-        custom_llm_provider: Final[str] = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)-        if custom_llm_provider is None:-            raise ValueError("custom_llm_provider is required")+        custom_llm_provider: Final[str | None] = standard_logging_payload.get("custom_llm_provider") -        budget_config: Final = self._get_budget_config_for_provider(custom_llm_provider)-        if budget_config:+        budget_config: Final = (+            self._get_budget_config_for_provider(custom_llm_provider) if custom_llm_provider is not None else None+        )+        if custom_llm_provider is not None and budget_config is not None:             # increment spend for provider             spend_key: Final = f"provider_spend:{custom_llm_provider}:{budget_config.budget_duration}"             start_time_key: Final = f"provider_budget_start_time:{custom_llm_provider}"tests/test_litellm/router_strategy/test_budget_limiter.pynew file mode 100644index 00000000000..62de1586fdd@@ -0,0 +1,137 @@+"""+Spend tracking in RouterBudgetLimiting.async_log_success_event.++Only chat completions puts custom_llm_provider into litellm_params. The responses,+anthropic_messages, embedding and rerank surfaces leave it unset, which used to make+the callback raise before any spend was recorded, so those budgets never moved.+"""++from typing import Final++import pytest++from litellm.caching.caching import DualCache+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+++@pytest.fixture+def disable_budget_sync(monkeypatch):+    async def noop(*args, **kwargs):+        return None++    monkeypatch.setattr(+        "litellm.router_strategy.budget_limiter.RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redis",+        noop,+    )+++def _success_kwargs(+    *,+    provider_in_litellm_params: str | None,+    provider_in_payload: str | None,+    call_type: str = "aresponses",+    response_cost: float = 0.25,+    model_id: str = "deployment-1",+) -> dict[str, object]:+    provider_params: Final[dict[str, str]] = (+        {} if provider_in_litellm_params is None else {"custom_llm_provider": provider_in_litellm_params}+    )+    litellm_params: Final[dict[str, str]] = {"model": "openai/gpt-4o", **provider_params}++    return {+        "call_type": call_type,+        "litellm_params": litellm_params,+        "standard_logging_object": {+            "response_cost": response_cost,+            "model_id": model_id,+            "custom_llm_provider": provider_in_payload,+        },+    }+++async def _log_success(limiter: RouterBudgetLimiting, kwargs: dict[str, object]) -> None:+    await limiter.async_log_success_event(kwargs=kwargs, response_obj=None, start_time=None, end_time=None)+++@pytest.mark.asyncio+@pytest.mark.parametrize("call_type", ["aresponses", "anthropic_messages", "aembedding", "arerank"])+async def test_provider_spend_tracked_when_litellm_params_omits_provider(disable_budget_sync, call_type):+    """Non-chat surfaces carry the provider only on the standard logging payload."""+    limiter = RouterBudgetLimiting(+        dual_cache=DualCache(),+        provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+    )++    await _log_success(+        limiter,+        _success_kwargs(+            provider_in_litellm_params=None,+            provider_in_payload="openai",+            call_type=call_type,+        ),+    )++    assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_chat_completions_spend_still_tracked(disable_budget_sync):+    """Chat completions fills in both sources and must keep accumulating."""+    limiter = RouterBudgetLimiting(+        dual_cache=DualCache(),+        provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+    )++    await _log_success(+        limiter,+        _success_kwargs(+            provider_in_litellm_params="openai",+            provider_in_payload="openai",+            call_type="acompletion",+        ),+    )++    assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_budget_of_other_provider_is_untouched(disable_budget_sync):+    """A provider without its own budget must not bleed into a configured one."""+    limiter = RouterBudgetLimiting(+        dual_cache=DualCache(),+        provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+    )++    await _log_success(+        limiter,+        _success_kwargs(provider_in_litellm_params=None, provider_in_payload="anthropic"),+    )++    assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") in (None, 0.0)+++@pytest.mark.asyncio+async def test_deployment_budget_tracked_when_provider_is_unresolvable(disable_budget_sync):+    """An unresolvable provider must not abort the deployment and tag budgets that follow it."""+    limiter = RouterBudgetLimiting(+        dual_cache=DualCache(),+        provider_budget_config=None,+        model_list=[+            {+                "model_name": "some-model",+                "litellm_params": {+                    "model": "openai/gpt-4o",+                    "max_budget": 10.0,+                    "budget_duration": "1d",+                },+                "model_info": {"id": "deployment-1"},+            }+        ],+    )++    await _log_success(+        limiter,+        _success_kwargs(provider_in_litellm_params=None, provider_in_payload=None),+    )++    assert await limiter.dual_cache.async_get_cache("deployment_spend:deployment-1:1d") == 0.25